Sep 18 19:12:04.874588 ip-10-0-138-252 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 18 19:12:04.874607 ip-10-0-138-252 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 18 19:12:04.874618 ip-10-0-138-252 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 18 19:12:04.874984 ip-10-0-138-252 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 18 19:12:16.007391 ip-10-0-138-252 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 18 19:12:16.007409 ip-10-0-138-252 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot ca6d0f0e94854b778d0133d532465aa7 --
Sep 18 19:13:52.988306 ip-10-0-138-252 systemd[1]: Starting Kubernetes Kubelet...
Sep 18 19:13:53.394009 ip-10-0-138-252 kubenswrapper[2532]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 19:13:53.394009 ip-10-0-138-252 kubenswrapper[2532]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 18 19:13:53.394009 ip-10-0-138-252 kubenswrapper[2532]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 19:13:53.394009 ip-10-0-138-252 kubenswrapper[2532]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 18 19:13:53.394009 ip-10-0-138-252 kubenswrapper[2532]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 19:13:53.396151 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.396071    2532 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 18 19:13:53.400201 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400186    2532 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:13:53.400234 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400203    2532 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:13:53.400234 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400207    2532 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:13:53.400234 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400210    2532 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:13:53.400234 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400214    2532 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:13:53.400234 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400217    2532 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:13:53.400234 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400227    2532 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:13:53.400234 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400231    2532 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:13:53.400234 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400234    2532 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:13:53.400234 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400237    2532 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:13:53.400457 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400240    2532 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:13:53.400457 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400243    2532 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:13:53.400457 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400246    2532 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:13:53.400457 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400249    2532 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:13:53.400457 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400252    2532 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:13:53.400457 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400255    2532 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:13:53.400457 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400258    2532 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:13:53.400457 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400261    2532 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:13:53.400457 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400263    2532 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:13:53.400457 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400266    2532 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:13:53.400457 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400268    2532 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:13:53.400457 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400271    2532 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:13:53.400457 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400273    2532 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:13:53.400457 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400276    2532 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:13:53.400457 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400279    2532 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:13:53.400457 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400281    2532 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:13:53.400457 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400285    2532 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:13:53.400457 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400287    2532 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:13:53.400457 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400290    2532 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:13:53.400457 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400293    2532 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:13:53.401015 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400295    2532 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:13:53.401015 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400298    2532 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:13:53.401015 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400301    2532 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:13:53.401015 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400304    2532 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:13:53.401015 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400306    2532 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:13:53.401015 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400309    2532 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:13:53.401015 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400311    2532 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:13:53.401015 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400314    2532 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:13:53.401015 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400316    2532 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:13:53.401015 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400319    2532 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:13:53.401015 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400321    2532 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:13:53.401015 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400324    2532 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:13:53.401015 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400327    2532 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:13:53.401015 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400331    2532 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:13:53.401015 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400334    2532 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:13:53.401015 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400337    2532 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:13:53.401015 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400339    2532 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:13:53.401015 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400342    2532 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:13:53.401015 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400345    2532 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:13:53.401015 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400348    2532 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:13:53.401521 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400350    2532 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:13:53.401521 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400353    2532 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:13:53.401521 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400355    2532 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:13:53.401521 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400358    2532 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:13:53.401521 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400361    2532 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:13:53.401521 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400363    2532 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:13:53.401521 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400366    2532 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:13:53.401521 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400368    2532 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:13:53.401521 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400371    2532 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:13:53.401521 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400373    2532 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:13:53.401521 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400376    2532 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:13:53.401521 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400379    2532 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:13:53.401521 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400382    2532 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:13:53.401521 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400384    2532 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:13:53.401521 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400387    2532 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:13:53.401521 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400391    2532 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:13:53.401521 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400396    2532 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:13:53.401521 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400400    2532 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:13:53.401521 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400403    2532 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:13:53.402219 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400406    2532 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:13:53.402219 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400408    2532 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:13:53.402219 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400411    2532 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:13:53.402219 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400414    2532 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:13:53.402219 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400418    2532 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:13:53.402219 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400421    2532 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:13:53.402219 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400425    2532 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:13:53.402219 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400428    2532 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:13:53.402219 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400431    2532 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:13:53.402219 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400434    2532 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:13:53.402219 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400437    2532 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:13:53.402219 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400440    2532 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:13:53.402219 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400442    2532 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:13:53.402219 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400445    2532 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:13:53.402219 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400447    2532 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:13:53.402219 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400450    2532 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:13:53.402219 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400452    2532 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:13:53.402219 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400455    2532 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:13:53.402219 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400457    2532 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:13:53.402983 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400460    2532 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:13:53.402983 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400462    2532 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:13:53.402983 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400465    2532 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:13:53.402983 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400468    2532 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:13:53.402983 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400470    2532 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:13:53.402983 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400473    2532 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:13:53.402983 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400475    2532 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:13:53.402983 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400478    2532 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:13:53.402983 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400480    2532 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:13:53.402983 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400483    2532 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:13:53.402983 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.400489    2532 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:13:53.402983 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401150    2532 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:13:53.402983 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401156    2532 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:13:53.402983 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401159    2532 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:13:53.402983 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401162    2532 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:13:53.402983 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401164    2532 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:13:53.402983 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401167    2532 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:13:53.402983 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401170    2532 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:13:53.402983 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401173    2532 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:13:53.402983 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401176    2532 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:13:53.403496 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401179    2532 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:13:53.403496 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401182    2532 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:13:53.403496 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401185    2532 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:13:53.403496 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401187    2532 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:13:53.403496 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401190    2532 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:13:53.403496 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401192    2532 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:13:53.403496 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401195    2532 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:13:53.403496 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401198    2532 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:13:53.403496 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401200    2532 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:13:53.403496 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401203    2532 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:13:53.403496 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401205    2532 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:13:53.403496 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401208    2532 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:13:53.403496 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401211    2532 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:13:53.403496 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401214    2532 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:13:53.403496 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401218    2532 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:13:53.403496 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401222    2532 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:13:53.403496 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401225    2532 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:13:53.403496 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401228    2532 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:13:53.404044 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401230    2532 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:13:53.404044 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401233    2532 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:13:53.404044 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401236    2532 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:13:53.404044 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401239    2532 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:13:53.404044 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401241    2532 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:13:53.404044 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401244    2532 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:13:53.404044 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401247    2532 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:13:53.404044 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401250    2532 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:13:53.404044 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401252    2532 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:13:53.404044 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401255    2532 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:13:53.404044 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401257    2532 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:13:53.404044 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401260    2532 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:13:53.404044 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401263    2532 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:13:53.404044 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401266    2532 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:13:53.404044 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401269    2532 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:13:53.404044 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401272    2532 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:13:53.404044 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401274    2532 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:13:53.404044 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401277    2532 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:13:53.404044 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401280    2532 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:13:53.404044 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401284    2532 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:13:53.404750 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401286    2532 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:13:53.404750 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401289    2532 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:13:53.404750 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401292    2532 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:13:53.404750 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401295    2532 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:13:53.404750 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401298    2532 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:13:53.404750 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401300    2532 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:13:53.404750 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401303    2532 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:13:53.404750 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401306    2532 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:13:53.404750 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401308    2532 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:13:53.404750 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401311    2532 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:13:53.404750 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401314    2532 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:13:53.404750 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401316    2532 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:13:53.404750 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401319    2532 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:13:53.404750 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401321    2532 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:13:53.404750 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401324    2532 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:13:53.404750 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401326    2532 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:13:53.404750 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401329    2532 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:13:53.404750 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401332    2532 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:13:53.404750 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401335    2532 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:13:53.404750 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401337    2532 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:13:53.405298 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401340    2532 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:13:53.405298 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401342    2532 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:13:53.405298 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401345    2532 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:13:53.405298 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401347    2532 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:13:53.405298 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401350    2532 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:13:53.405298 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401353    2532 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:13:53.405298 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401356    2532 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:13:53.405298 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401362    2532 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:13:53.405298 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401365    2532 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:13:53.405298 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401368    2532 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:13:53.405298 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401370    2532 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:13:53.405298 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401373    2532 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:13:53.405298 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401375    2532 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:13:53.405298 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401379    2532 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:13:53.405298 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401382    2532 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:13:53.405298 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401386    2532 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:13:53.405298 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401389    2532 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:13:53.405298 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401392    2532 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:13:53.405298 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401395    2532 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:13:53.405298 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401397    2532 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:13:53.405789 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401400    2532 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:13:53.405789 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401403    2532 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:13:53.405789 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401406    2532 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:13:53.405789 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401408    2532 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:13:53.405789 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401411    2532 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:13:53.405789 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401413    2532 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:13:53.405789 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401416    2532 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:13:53.405789 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401419    2532 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:13:53.405789 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401422    2532 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:13:53.405789 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401424    2532 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:13:53.405789 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401427    2532 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:13:53.405789 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.401430    2532 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:13:53.405789 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404388    2532 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:13:53.405789 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404398    2532 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:13:53.405789 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404402    2532 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:13:53.405789 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404405    2532 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:13:53.405789 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404409    2532 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:13:53.405789 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404412    2532 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:13:53.405789 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404415    2532 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:13:53.405789 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404418    2532 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:13:53.406297 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404421    2532 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:13:53.406297 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404424    2532 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:13:53.406297 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404427    2532 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:13:53.406297 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404430    2532 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:13:53.406297 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404433    2532 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:13:53.406297 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404437    2532 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:13:53.406297 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404440    2532 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:13:53.406297 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404442    2532 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:13:53.406297 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404445    2532 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:13:53.406297 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404447    2532 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:13:53.406297 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404452    2532 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:13:53.406297 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404457    2532 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:13:53.406297 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404460    2532 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:13:53.406297 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404463    2532 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:13:53.406297 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404466    2532 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:13:53.406297 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404469    2532 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:13:53.406297 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404472    2532 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:13:53.406297 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404476    2532 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:13:53.406297 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404479    2532 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:13:53.406799 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404482    2532 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:13:53.406799 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404485    2532 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:13:53.406799 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404488    2532 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:13:53.406799 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404491    2532 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:13:53.406799 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404504    2532 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:13:53.406799 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404508    2532 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:13:53.406799 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404512    2532 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:13:53.406799 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404514    2532 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:13:53.406799 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404517    2532 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:13:53.406799 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404520    2532 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:13:53.406799 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404523    2532 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:13:53.406799 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404525    2532 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:13:53.406799 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404528    2532 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:13:53.406799 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404531    2532 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:13:53.406799 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404533    2532 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:13:53.406799 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404536    2532 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:13:53.406799 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404538    2532 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:13:53.406799 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404541    2532 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:13:53.406799 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404544    2532 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:13:53.406799 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404546    2532 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:13:53.407299 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404549    2532 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:13:53.407299 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404552    2532 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:13:53.407299 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404554    2532 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:13:53.407299 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404558    2532 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:13:53.407299 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404562    2532 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:13:53.407299 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404566    2532 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:13:53.407299 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404569    2532 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:13:53.407299 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404571    2532 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:13:53.407299 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404574    2532 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:13:53.407299 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404576    2532 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:13:53.407299 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404579    2532 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:13:53.407299 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404582    2532 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:13:53.407299 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404585    2532 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:13:53.407299 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404588    2532 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:13:53.407299 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404590    2532 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:13:53.407299 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404593    2532 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:13:53.407299 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404595    2532 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:13:53.407299 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404598    2532 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:13:53.407299 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404602    2532 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:13:53.407773 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404604    2532 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:13:53.407773 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404607    2532 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:13:53.407773 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404610    2532 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:13:53.407773 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404612    2532 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:13:53.407773 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404615    2532 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:13:53.407773 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404617    2532 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:13:53.407773 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404620    2532 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:13:53.407773 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404622    2532 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:13:53.407773 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404625    2532 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:13:53.407773 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404627    2532 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:13:53.407773 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404630    2532 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:13:53.407773 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404633    2532 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:13:53.407773 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404635    2532 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:13:53.407773 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404637    2532 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:13:53.407773 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404640    2532 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:13:53.407773 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404642    2532 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:13:53.407773 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404645    2532 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:13:53.407773 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404648    2532 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:13:53.407773 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404651    2532 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:13:53.407773 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404653    2532 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:13:53.408275 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404657    2532 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:13:53.408275 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404659    2532 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:13:53.408275 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404663    2532 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:13:53.408275 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404665    2532 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:13:53.408275 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404668    2532 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:13:53.408275 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404670    2532 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:13:53.408275 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404673    2532 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:13:53.408275 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404675    2532 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:13:53.408275 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404678    2532 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:13:53.408275 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404680    2532 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:13:53.408275 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404683    2532 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:13:53.408275 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404686    2532 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:13:53.408275 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.404688    2532 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:13:53.408275 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.404760    2532 flags.go:64] FLAG: --address="0.0.0.0"
Sep 18 19:13:53.408275 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.404793    2532 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 18 19:13:53.408275 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.404799    2532 flags.go:64] FLAG: --anonymous-auth="true"
Sep 18 19:13:53.408275 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.404804    2532 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 18 19:13:53.408275 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.404808    2532 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 18 19:13:53.408275 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.404812    2532 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 18 19:13:53.408275 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.404816    2532 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 18 19:13:53.408275 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.404820    2532 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 18 19:13:53.408790 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.404824    2532 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 18 19:13:53.408790 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.404827    2532 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 18 19:13:53.408790 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.404830    2532 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 18 19:13:53.408790 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.404834    2532 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 18 19:13:53.408790 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.404837    2532 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 18 19:13:53.408790 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.404840    2532 flags.go:64] FLAG: --cgroup-root=""
Sep 18 19:13:53.408790 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.404843    2532 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 18 19:13:53.408790 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.404846    2532 flags.go:64] FLAG: --client-ca-file=""
Sep 18 19:13:53.408790 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.404849    2532 flags.go:64] FLAG: --cloud-provider="external"
Sep 18 19:13:53.408790 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.404852    2532 flags.go:64] FLAG: --cluster-dns="[]"
Sep 18 19:13:53.408790 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.404857    2532 flags.go:64] FLAG: --cluster-domain=""
Sep 18 19:13:53.408790 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.404861    2532 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 18 19:13:53.408790 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.404864    2532 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 18 19:13:53.408790 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.404867    2532 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 18 19:13:53.408790 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.404870    2532 flags.go:64] FLAG: --container-log-max-files="5"
Sep 18 19:13:53.408790 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.404874    2532 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 18 19:13:53.408790 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.404877    2532 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 18 19:13:53.408790 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.404880    2532 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 18 19:13:53.408790 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.404884    2532 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 18 19:13:53.408790 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.404887    2532 flags.go:64] FLAG: --contention-profiling="false"
Sep 18 19:13:53.408790 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.404890    2532 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 18 19:13:53.408790 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.404893    2532 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 18 19:13:53.408790 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.404896    2532 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 18 19:13:53.408790 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.404899    2532 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 18 19:13:53.409431 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.404903    2532 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 18 19:13:53.409431 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.404906    2532 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 18 19:13:53.409431 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.404909    2532 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 18 19:13:53.409431 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.404912    2532 flags.go:64] FLAG: --enable-load-reader="false"
Sep 18 19:13:53.409431 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.404915    2532 flags.go:64] FLAG: --enable-server="true"
Sep 18 19:13:53.409431 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.404918    2532 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 18 19:13:53.409431 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.404923    2532 flags.go:64] FLAG: --event-burst="100"
Sep 18 19:13:53.409431 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.404926    2532 flags.go:64] FLAG: --event-qps="50"
Sep 18 19:13:53.409431 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.404929    2532 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 18 19:13:53.409431 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.404932    2532 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 18 19:13:53.409431 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.404935    2532 flags.go:64] FLAG: --eviction-hard=""
Sep 18 19:13:53.409431 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.404939    2532 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 18 19:13:53.409431 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.404942    2532 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 18 19:13:53.409431 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.404945    2532 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 18 19:13:53.409431 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.404948    2532 flags.go:64] FLAG: --eviction-soft=""
Sep 18 19:13:53.409431 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.404950    2532 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 18 19:13:53.409431 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.404953    2532 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 18 19:13:53.409431 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.404956    2532 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 18 19:13:53.409431 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.404960    2532 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 18 19:13:53.409431 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.404963    2532 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 18 19:13:53.409431 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.404966    2532 flags.go:64] FLAG: --fail-swap-on="true"
Sep 18 19:13:53.409431 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.404969    2532 flags.go:64] FLAG: --feature-gates=""
Sep 18 19:13:53.409431 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.404972    2532 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 18 19:13:53.409431 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.404975    2532 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 18 19:13:53.409431 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.404978    2532 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 18 19:13:53.410066 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.404981    2532 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 18 19:13:53.410066 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.404984    2532 flags.go:64] FLAG: --healthz-port="10248"
Sep 18 19:13:53.410066 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.404987    2532 flags.go:64] FLAG: --help="false"
Sep 18 19:13:53.410066 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.404990    2532 flags.go:64] FLAG: --hostname-override="ip-10-0-138-252.ec2.internal"
Sep 18 19:13:53.410066 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.404993    2532 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 18 19:13:53.410066 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.404997    2532 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 18 19:13:53.410066 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405000    2532 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 18 19:13:53.410066 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405003    2532 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 18 19:13:53.410066 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405006    2532 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 18 19:13:53.410066 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405009    2532 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 18 19:13:53.410066 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405012    2532 flags.go:64] FLAG: --image-service-endpoint=""
Sep 18 19:13:53.410066 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405015    2532 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 18 19:13:53.410066 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405018    2532 flags.go:64] FLAG: --kube-api-burst="100"
Sep 18 19:13:53.410066 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405021    2532 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 18 19:13:53.410066 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405024    2532 flags.go:64] FLAG: --kube-api-qps="50"
Sep 18 19:13:53.410066 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405028    2532 flags.go:64] FLAG: --kube-reserved=""
Sep 18 19:13:53.410066 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405031    2532 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 18 19:13:53.410066 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405034    2532 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 18 19:13:53.410066 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405037    2532 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 18 19:13:53.410066 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405039    2532 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 18 19:13:53.410066 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405042    2532 flags.go:64] FLAG: --lock-file=""
Sep 18 19:13:53.410066 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405046    2532 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 18 19:13:53.410066 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405049    2532 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 18 19:13:53.410066 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405052    2532 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 18 19:13:53.410660 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405072    2532 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 18 19:13:53.410660 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405075    2532 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 18 19:13:53.410660 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405079    2532 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 18 19:13:53.410660 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405084    2532 flags.go:64] FLAG: --logging-format="text"
Sep 18 19:13:53.410660 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405087    2532 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 18 19:13:53.410660 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405091    2532 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 18 19:13:53.410660 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405094    2532 flags.go:64] FLAG: --manifest-url=""
Sep 18 19:13:53.410660 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405096    2532 flags.go:64] FLAG: --manifest-url-header=""
Sep 18 19:13:53.410660 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405100    2532 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 18 19:13:53.410660 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405104    2532 flags.go:64] FLAG: --max-open-files="1000000"
Sep 18 19:13:53.410660 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405107    2532 flags.go:64] FLAG: --max-pods="110"
Sep 18 19:13:53.410660 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405110    2532 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 18 19:13:53.410660 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405115    2532 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 18 19:13:53.410660 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405118    2532 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 18 19:13:53.410660 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405121    2532 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 18 19:13:53.410660 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405124    2532 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 18 19:13:53.410660 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405127    2532 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 18 19:13:53.410660 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405130    2532 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 18 19:13:53.410660 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405138    2532 flags.go:64] FLAG: --node-status-max-images="50"
Sep 18 19:13:53.410660 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405140    2532 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 18 19:13:53.410660 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405144    2532 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 18 19:13:53.410660 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405147    2532 flags.go:64] FLAG: --pod-cidr=""
Sep 18 19:13:53.410660 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405150    2532 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 18 19:13:53.411241 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405155    2532 flags.go:64] FLAG: --pod-manifest-path=""
Sep 18 19:13:53.411241 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405158    2532 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 18 19:13:53.411241 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405161    2532 flags.go:64] FLAG: --pods-per-core="0"
Sep 18 19:13:53.411241 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405164    2532 flags.go:64] FLAG: --port="10250"
Sep 18 19:13:53.411241 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405167    2532 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 18 19:13:53.411241 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405170    2532 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-093c71ef291bac430"
Sep 18 19:13:53.411241 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405173    2532 flags.go:64] FLAG: --qos-reserved=""
Sep 18 19:13:53.411241 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405176    2532 flags.go:64] FLAG: --read-only-port="10255"
Sep 18 19:13:53.411241 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405179    2532 flags.go:64] FLAG: --register-node="true"
Sep 18 19:13:53.411241 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405183    2532 flags.go:64] FLAG: --register-with-taints=""
Sep 18 19:13:53.411241 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405186    2532 flags.go:64] FLAG: --registry-burst="10"
Sep 18 19:13:53.411241 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405189    2532 flags.go:64] FLAG: --registry-qps="5"
Sep 18 19:13:53.411241 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405194    2532 flags.go:64] FLAG: --reserved-cpus=""
Sep 18 19:13:53.411241 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405197    2532 flags.go:64] FLAG: --reserved-memory=""
Sep 18 19:13:53.411241 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405200    2532 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 18 19:13:53.411241 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405204    2532 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 18 19:13:53.411241 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405207    2532 flags.go:64] FLAG: --rotate-certificates="false"
Sep 18 19:13:53.411241 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405209    2532 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 18 19:13:53.411241 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405212    2532 flags.go:64] FLAG: --runonce="false"
Sep 18 19:13:53.411241 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405215    2532 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 18 19:13:53.411241 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405218    2532 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 18 19:13:53.411241 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405221    2532 flags.go:64] FLAG: --seccomp-default="false"
Sep 18 19:13:53.411241 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405227    2532 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 18 19:13:53.411241 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405230    2532 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 18 19:13:53.411241 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405233    2532 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 18 19:13:53.411241 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405236    2532 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 18 19:13:53.411915 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405239    2532 flags.go:64] FLAG: --storage-driver-password="root"
Sep 18 19:13:53.411915 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405242    2532 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 18 19:13:53.411915 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405245    2532 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 18 19:13:53.411915 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405248    2532 flags.go:64] FLAG: --storage-driver-user="root"
Sep 18 19:13:53.411915 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405251    2532 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 18 19:13:53.411915 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405254    2532 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 18 19:13:53.411915 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405257    2532 flags.go:64] FLAG: --system-cgroups=""
Sep 18 19:13:53.411915 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405259    2532 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 18 19:13:53.411915 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405265    2532 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 18 19:13:53.411915 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405268    2532 flags.go:64] FLAG: --tls-cert-file=""
Sep 18 19:13:53.411915 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405271    2532 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 18 19:13:53.411915 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405275    2532 flags.go:64] FLAG: --tls-min-version=""
Sep 18 19:13:53.411915 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405278    2532 flags.go:64] FLAG: --tls-private-key-file=""
Sep 18 19:13:53.411915 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405281    2532 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 18 19:13:53.411915 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405284    2532 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 18 19:13:53.411915 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405287    2532 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 18 19:13:53.411915 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405290    2532 flags.go:64] FLAG: --v="2"
Sep 18 19:13:53.411915 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405294    2532 flags.go:64] FLAG: --version="false"
Sep 18 19:13:53.411915 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405299    2532 flags.go:64] FLAG: --vmodule=""
Sep 18 19:13:53.411915 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405304    2532 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 18 19:13:53.411915 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.405307    2532 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 18 19:13:53.411915 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405414    2532 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:13:53.411915 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405418    2532 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:13:53.411915 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405421    2532 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:13:53.412560 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405426    2532 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:13:53.412560 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405429    2532 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:13:53.412560 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405432    2532 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:13:53.412560 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405435    2532 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:13:53.412560 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405439    2532 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:13:53.412560 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405441    2532 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:13:53.412560 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405444    2532 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:13:53.412560 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405447    2532 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:13:53.412560 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405449    2532 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:13:53.412560 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405452    2532 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:13:53.412560 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405455    2532 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:13:53.412560 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405457    2532 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:13:53.412560 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405460    2532 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:13:53.412560 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405463    2532 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:13:53.412560 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405466    2532 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:13:53.412560 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405468    2532 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:13:53.412560 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405471    2532 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:13:53.412560 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405474    2532 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:13:53.412560 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405477    2532 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:13:53.413027 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405479    2532 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:13:53.413027 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405482    2532 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:13:53.413027 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405485    2532 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:13:53.413027 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405487    2532 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:13:53.413027 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405490    2532 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:13:53.413027 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405492    2532 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:13:53.413027 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405495    2532 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:13:53.413027 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405499    2532 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:13:53.413027 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405502    2532 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:13:53.413027 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405505    2532 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:13:53.413027 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405507    2532 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:13:53.413027 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405510    2532 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:13:53.413027 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405513    2532 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:13:53.413027 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405515    2532 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:13:53.413027 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405518    2532 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:13:53.413027 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405521    2532 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:13:53.413027 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405524    2532 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:13:53.413027 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405527    2532 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:13:53.413027 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405530    2532 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:13:53.413027 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405533    2532 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:13:53.413546 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405535    2532 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:13:53.413546 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405538    2532 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:13:53.413546 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405541    2532 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:13:53.413546 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405543    2532 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:13:53.413546 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405546    2532 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:13:53.413546 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405549    2532 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:13:53.413546 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405553    2532 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:13:53.413546 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405557    2532 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:13:53.413546 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405560    2532 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:13:53.413546 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405563    2532 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:13:53.413546 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405566    2532 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:13:53.413546 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405569    2532 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:13:53.413546 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405572    2532 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:13:53.413546 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405575    2532 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:13:53.413546 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405578    2532 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:13:53.413546 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405581    2532 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:13:53.413546 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405584    2532 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:13:53.413546 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405586    2532 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:13:53.413546 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405589    2532 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:13:53.413546 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405593    2532 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:13:53.414040 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405596    2532 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:13:53.414040 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405599    2532 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:13:53.414040 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405603    2532 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:13:53.414040 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405605    2532 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:13:53.414040 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405608    2532 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:13:53.414040 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405611    2532 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:13:53.414040 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405614    2532 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:13:53.414040 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405617    2532 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:13:53.414040 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405620    2532 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:13:53.414040 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405624    2532 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:13:53.414040 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405627    2532 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:13:53.414040 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405630    2532 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:13:53.414040 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405633    2532 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:13:53.414040 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405635    2532 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:13:53.414040 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405638    2532 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:13:53.414040 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405641    2532 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:13:53.414040 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405643    2532 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:13:53.414040 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405646    2532 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:13:53.414040 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405648    2532 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:13:53.414040 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405651    2532 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:13:53.414621 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405654    2532 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:13:53.414621 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405657    2532 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:13:53.414621 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405660    2532 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:13:53.414621 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405663    2532 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:13:53.414621 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405667    2532 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:13:53.414621 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405670    2532 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:13:53.414621 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405673    2532 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:13:53.414621 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405676    2532 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:13:53.414621 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405678    2532 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:13:53.414621 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405681    2532 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:13:53.414621 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405684    2532 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:13:53.414621 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405687    2532 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:13:53.414621 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405690    2532 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:13:53.414621 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405693    2532 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:13:53.414621 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405697    2532 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:13:53.414621 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405701    2532 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:13:53.414621 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.405706    2532 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:13:53.415031 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.406609    2532 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 19:13:53.415031 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.414800    2532 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 18 19:13:53.415031 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.414815    2532 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 18 19:13:53.415712 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.415701    2532 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 18 19:13:53.415741 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.415715    2532 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 18 19:13:53.415780 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415772    2532 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:13:53.415780 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415779    2532 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:13:53.415830 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415783    2532 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:13:53.415830 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415786    2532 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:13:53.415830 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415789    2532 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:13:53.415830 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415792    2532 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:13:53.415830 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415795    2532 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:13:53.415830 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415798    2532 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:13:53.415830 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415800    2532 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:13:53.415830 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415803    2532 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:13:53.415830 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415805    2532 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:13:53.415830 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415808    2532 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:13:53.415830 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415810    2532 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:13:53.415830 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415813    2532 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:13:53.415830 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415816    2532 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:13:53.415830 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415819    2532 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:13:53.415830 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415821    2532 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:13:53.415830 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415824    2532 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:13:53.415830 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415826    2532 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:13:53.415830 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415829    2532 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:13:53.415830 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415832    2532 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:13:53.415830 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415834    2532 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:13:53.416333 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415839    2532 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:13:53.416333 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415843    2532 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:13:53.416333 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415846    2532 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:13:53.416333 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415849    2532 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:13:53.416333 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415853    2532 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:13:53.416333 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415855    2532 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:13:53.416333 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415858    2532 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:13:53.416333 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415861    2532 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:13:53.416333 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415865    2532 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:13:53.416333 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415867    2532 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:13:53.416333 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415870    2532 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:13:53.416333 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415872    2532 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:13:53.416333 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415875    2532 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:13:53.416333 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415878    2532 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:13:53.416333 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415880    2532 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:13:53.416333 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415883    2532 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:13:53.416333 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415886    2532 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:13:53.416333 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415888    2532 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:13:53.416333 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415891    2532 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:13:53.416333 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415893    2532 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:13:53.416846 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415896    2532 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:13:53.416846 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415898    2532 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:13:53.416846 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415901    2532 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:13:53.416846 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415903    2532 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:13:53.416846 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415906    2532 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:13:53.416846 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415909    2532 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:13:53.416846 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415911    2532 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:13:53.416846 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415914    2532 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:13:53.416846 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415916    2532 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:13:53.416846 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415919    2532 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:13:53.416846 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415921    2532 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:13:53.416846 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415924    2532 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:13:53.416846 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415927    2532 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:13:53.416846 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415930    2532 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:13:53.416846 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415933    2532 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:13:53.416846 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415936    2532 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:13:53.416846 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415938    2532 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:13:53.416846 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415941    2532 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:13:53.416846 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415943    2532 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:13:53.416846 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415945    2532 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:13:53.417353 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415949    2532 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:13:53.417353 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415952    2532 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:13:53.417353 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415954    2532 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:13:53.417353 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415957    2532 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:13:53.417353 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415959    2532 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:13:53.417353 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415962    2532 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:13:53.417353 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415964    2532 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:13:53.417353 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415967    2532 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:13:53.417353 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415969    2532 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:13:53.417353 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415972    2532 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:13:53.417353 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415974    2532 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:13:53.417353 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415977    2532 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:13:53.417353 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415980    2532 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:13:53.417353 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415982    2532 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:13:53.417353 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415985    2532 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:13:53.417353 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415987    2532 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:13:53.417353 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415990    2532 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:13:53.417353 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415992    2532 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:13:53.417353 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415995    2532 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:13:53.417353 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.415998    2532 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:13:53.417832 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416000    2532 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:13:53.417832 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416003    2532 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:13:53.417832 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416006    2532 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:13:53.417832 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416010    2532 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:13:53.417832 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416013    2532 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:13:53.417832 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416016    2532 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:13:53.417832 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416019    2532 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:13:53.417832 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416022    2532 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:13:53.417832 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416025    2532 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:13:53.417832 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416028    2532 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:13:53.417832 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416031    2532 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:13:53.417832 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416034    2532 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:13:53.417832 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416036    2532 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:13:53.417832 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416040    2532 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:13:53.417832 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416042    2532 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:13:53.417832 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416045    2532 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:13:53.417832 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416048    2532 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:13:53.418265 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.416067    2532 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 19:13:53.418265 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416176    2532 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:13:53.418265 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416181    2532 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:13:53.418265 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416184    2532 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:13:53.418265 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416187    2532 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:13:53.418265 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416191    2532 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:13:53.418265 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416195    2532 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:13:53.418265 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416197    2532 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:13:53.418265 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416200    2532 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:13:53.418265 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416203    2532 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:13:53.418265 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416206    2532 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:13:53.418265 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416209    2532 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:13:53.418265 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416212    2532 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:13:53.418265 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416215    2532 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:13:53.418265 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416218    2532 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:13:53.418636 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416221    2532 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:13:53.418636 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416224    2532 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:13:53.418636 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416227    2532 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:13:53.418636 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416230    2532 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:13:53.418636 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416232    2532 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:13:53.418636 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416236    2532 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:13:53.418636 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416239    2532 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:13:53.418636 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416242    2532 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:13:53.418636 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416244    2532 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:13:53.418636 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416247    2532 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:13:53.418636 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416249    2532 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:13:53.418636 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416252    2532 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:13:53.418636 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416255    2532 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:13:53.418636 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416257    2532 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:13:53.418636 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416260    2532 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:13:53.418636 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416263    2532 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:13:53.418636 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416265    2532 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:13:53.418636 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416268    2532 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:13:53.418636 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416270    2532 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:13:53.418636 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416273    2532 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:13:53.419123 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416275    2532 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:13:53.419123 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416278    2532 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:13:53.419123 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416281    2532 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:13:53.419123 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416285    2532 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:13:53.419123 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416288    2532 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:13:53.419123 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416291    2532 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:13:53.419123 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416294    2532 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:13:53.419123 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416296    2532 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:13:53.419123 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416299    2532 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:13:53.419123 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416301    2532 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:13:53.419123 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416304    2532 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:13:53.419123 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416306    2532 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:13:53.419123 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416310    2532 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:13:53.419123 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416312    2532 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:13:53.419123 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416315    2532 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:13:53.419123 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416317    2532 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:13:53.419123 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416320    2532 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:13:53.419123 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416322    2532 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:13:53.419123 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416325    2532 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:13:53.419578 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416327    2532 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:13:53.419578 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416330    2532 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:13:53.419578 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416332    2532 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:13:53.419578 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416335    2532 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:13:53.419578 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416338    2532 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:13:53.419578 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416341    2532 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:13:53.419578 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416343    2532 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:13:53.419578 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416346    2532 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:13:53.419578 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416349    2532 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:13:53.419578 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416352    2532 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:13:53.419578 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416354    2532 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:13:53.419578 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416357    2532 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:13:53.419578 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416359    2532 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:13:53.419578 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416362    2532 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:13:53.419578 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416365    2532 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:13:53.419578 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416367    2532 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:13:53.419578 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416370    2532 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:13:53.419578 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416372    2532 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:13:53.419578 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416375    2532 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:13:53.420025 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416378    2532 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:13:53.420025 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416380    2532 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:13:53.420025 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416383    2532 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:13:53.420025 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416385    2532 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:13:53.420025 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416388    2532 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:13:53.420025 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416390    2532 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:13:53.420025 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416392    2532 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:13:53.420025 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416395    2532 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:13:53.420025 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416399    2532 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:13:53.420025 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416402    2532 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:13:53.420025 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416405    2532 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:13:53.420025 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416409    2532 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:13:53.420025 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416412    2532 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:13:53.420025 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416414    2532 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:13:53.420025 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416417    2532 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:13:53.420025 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416420    2532 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:13:53.420025 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416422    2532 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:13:53.420025 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416425    2532 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:13:53.420025 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416427    2532 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:13:53.420025 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416430    2532 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:13:53.420527 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416432    2532 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:13:53.420527 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416435    2532 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:13:53.420527 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416438    2532 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:13:53.420527 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416440    2532 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:13:53.420527 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416443    2532 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:13:53.420527 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416445    2532 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:13:53.420527 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:53.416448    2532 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:13:53.420527 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.416453    2532 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 19:13:53.420527 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.416560    2532 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 18 19:13:53.420527 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.418951    2532 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 18 19:13:53.420527 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.420046    2532 server.go:1019] "Starting client certificate rotation"
Sep 18 19:13:53.420527 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.420152    2532 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 19:13:53.420527 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.420177    2532 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 19:13:53.422866 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.422852    2532 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 19:13:53.424015 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.423998    2532 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 19:13:53.434722 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.434698    2532 log.go:25] "Validated CRI v1 runtime API"
Sep 18 19:13:53.437840 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.437826    2532 log.go:25] "Validated CRI v1 image API"
Sep 18 19:13:53.438660 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.438644    2532 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 18 19:13:53.445017 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.445001    2532 fs.go:135] Filesystem UUIDs: map[5ed7b31c-f1e5-4883-9892-8f5c61b33a57:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 8bb75ec9-fe33-4154-9d6a-22f72aadc378:/dev/nvme0n1p4]
Sep 18 19:13:53.445082 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.445016    2532 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 18 19:13:53.450883 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.450777    2532 manager.go:216] Machine: {Timestamp:2026-09-18 19:13:53.449416859 +0000 UTC m=+0.369920707 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100691 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21fa1abdd7831dcddf7a65907744d9 SystemUUID:ec21fa1a-bdd7-831d-cddf-7a65907744d9 BootID:ca6d0f0e-9485-4b77-8d01-33d532465aa7 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:fb:f6:1f:5b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:fb:f6:1f:5b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:da:8e:29:71:b1:14 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 18 19:13:53.450883 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.450883    2532 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 18 19:13:53.450991 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.450960    2532 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 18 19:13:53.453885 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.453863    2532 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 18 19:13:53.454021 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.453888    2532 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-252.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 18 19:13:53.454735 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.454725    2532 topology_manager.go:138] "Creating topology manager with none policy"
Sep 18 19:13:53.454770 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.454738    2532 container_manager_linux.go:309] "Creating device plugin manager"
Sep 18 19:13:53.454770 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.454746    2532 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 19:13:53.454770 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.454760    2532 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 19:13:53.454852 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.454826    2532 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 18 19:13:53.457803 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.457789    2532 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 19:13:53.459609 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.459597    2532 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 19:13:53.459720 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.459711    2532 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 18 19:13:53.459752 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.459742    2532 kubelet.go:486] "Attempting to sync node with API server"
Sep 18 19:13:53.459781 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.459755    2532 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 18 19:13:53.459781 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.459767    2532 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 18 19:13:53.459781 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.459775    2532 kubelet.go:398] "Adding apiserver pod source"
Sep 18 19:13:53.459856 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.459790    2532 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 18 19:13:53.460157 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.460146    2532 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 19:13:53.460205 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.460165    2532 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 19:13:53.463106 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.463089    2532 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 18 19:13:53.466028 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.466015    2532 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 18 19:13:53.466092 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.466045    2532 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 18 19:13:53.467488 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.467475    2532 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 18 19:13:53.467533 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.467493    2532 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 18 19:13:53.467533 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.467499    2532 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 18 19:13:53.467533 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.467505    2532 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 18 19:13:53.467533 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.467513    2532 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 18 19:13:53.467533 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.467524    2532 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 18 19:13:53.467669 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.467538    2532 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 18 19:13:53.467669 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.467543    2532 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 18 19:13:53.467669 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.467548    2532 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 18 19:13:53.467669 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.467554    2532 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 18 19:13:53.467669 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.467565    2532 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 18 19:13:53.467669 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.467570    2532 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 18 19:13:53.469348 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.469337    2532 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 18 19:13:53.469348 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.469348    2532 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 18 19:13:53.473069 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.473041    2532 server.go:1268] "Started kubelet"
Sep 18 19:13:53.473144 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.473126    2532 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 18 19:13:53.473231 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.473131    2532 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 18 19:13:53.473284 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.473245    2532 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 18 19:13:53.473873 ip-10-0-138-252 systemd[1]: Started Kubernetes Kubelet.
Sep 18 19:13:53.477774 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.476974    2532 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 18 19:13:53.478810 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.478779    2532 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-252.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 19:13:53.479538 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:53.479492    2532 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-252.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 19:13:53.480547 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:53.480524    2532 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 18 19:13:53.480945 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.480917    2532 server.go:310] "Adding debug handlers to kubelet server"
Sep 18 19:13:53.481177 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:53.481122    2532 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 19:13:53.483005 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:53.481826    2532 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-252.ec2.internal.18d6802d2ec6ca89  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-252.ec2.internal,UID:ip-10-0-138-252.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-252.ec2.internal,},FirstTimestamp:2026-09-18 19:13:53.473014409 +0000 UTC m=+0.393518257,LastTimestamp:2026-09-18 19:13:53.473014409 +0000 UTC m=+0.393518257,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-252.ec2.internal,}"
Sep 18 19:13:53.490549 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.489581    2532 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 18 19:13:53.490549 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.490251    2532 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 18 19:13:53.490549 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:53.490418    2532 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-252.ec2.internal\" not found"
Sep 18 19:13:53.491111 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.491092    2532 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 18 19:13:53.491111 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.491112    2532 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 18 19:13:53.491240 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.491202    2532 reconstruct.go:97] "Volume reconstruction finished"
Sep 18 19:13:53.491240 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.491209    2532 reconciler.go:29] "Reconciler: start to sync state"
Sep 18 19:13:53.491240 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.491227    2532 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 18 19:13:53.491436 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.491416    2532 factory.go:153] Registering CRI-O factory
Sep 18 19:13:53.491436 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.491437    2532 factory.go:223] Registration of the crio container factory successfully
Sep 18 19:13:53.491581 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.491490    2532 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 18 19:13:53.491581 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.491499    2532 factory.go:55] Registering systemd factory
Sep 18 19:13:53.491581 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.491508    2532 factory.go:223] Registration of the systemd container factory successfully
Sep 18 19:13:53.491581 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.491525    2532 factory.go:103] Registering Raw factory
Sep 18 19:13:53.491581 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.491538    2532 manager.go:1196] Started watching for new ooms in manager
Sep 18 19:13:53.492030 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.492012    2532 manager.go:349] Starting recovery of all containers
Sep 18 19:13:53.497076 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:53.497024    2532 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-252.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 18 19:13:53.497166 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:53.497075    2532 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 19:13:53.503815 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.503799    2532 manager.go:354] Recovery completed
Sep 18 19:13:53.508288 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.508274    2532 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:13:53.510089 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.510070    2532 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:13:53.510191 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.510108    2532 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:13:53.510191 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.510119    2532 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:13:53.510629 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.510613    2532 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 18 19:13:53.510629 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.510627    2532 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 18 19:13:53.510770 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.510630    2532 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:13:53.510770 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.510644    2532 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 19:13:53.511325 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.511310    2532 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:13:53.511405 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.511336    2532 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:13:53.511405 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.511346    2532 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:13:53.512892 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.512878    2532 policy_none.go:49] "None policy: Start"
Sep 18 19:13:53.512954 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.512896    2532 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 18 19:13:53.512954 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.512909    2532 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 18 19:13:53.514906 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.514895    2532 policy_none.go:47] "Start"
Sep 18 19:13:53.526050 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.526028    2532 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-g5bhl"
Sep 18 19:13:53.549623 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.549610    2532 manager.go:337] "Starting Device Plugin manager"
Sep 18 19:13:53.566009 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:53.549641    2532 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 18 19:13:53.566009 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.549651    2532 server.go:87] "Starting device plugin registration server"
Sep 18 19:13:53.566009 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.549947    2532 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 18 19:13:53.566009 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.549957    2532 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 18 19:13:53.566009 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.550087    2532 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 18 19:13:53.566009 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.550511    2532 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 18 19:13:53.566009 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.550522    2532 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 18 19:13:53.566009 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:53.550652    2532 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 18 19:13:53.566009 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:53.550700    2532 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-252.ec2.internal\" not found"
Sep 18 19:13:53.566009 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.551201    2532 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-g5bhl"
Sep 18 19:13:53.566009 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:53.561202    2532 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-252.ec2.internal.18d6802d30fc7f72  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-252.ec2.internal,UID:ip-10-0-138-252.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-252.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-252.ec2.internal,},FirstTimestamp:2026-09-18 19:13:53.510088562 +0000 UTC m=+0.430592410,LastTimestamp:2026-09-18 19:13:53.510088562 +0000 UTC m=+0.430592410,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-252.ec2.internal,}"
Sep 18 19:13:53.614300 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.614276    2532 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 18 19:13:53.615432 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.615415    2532 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 18 19:13:53.615502 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.615468    2532 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 18 19:13:53.615502 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.615492    2532 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 18 19:13:53.615584 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:53.615535    2532 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 18 19:13:53.618667 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.618651    2532 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:13:53.650777 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.650724    2532 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:13:53.651451 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.651431    2532 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:13:53.651528 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.651457    2532 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:13:53.651528 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.651470    2532 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:13:53.651528 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.651489    2532 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-252.ec2.internal"
Sep 18 19:13:53.665829 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.665812    2532 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-252.ec2.internal"
Sep 18 19:13:53.665829 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:53.665830    2532 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-252.ec2.internal\": node \"ip-10-0-138-252.ec2.internal\" not found"
Sep 18 19:13:53.715725 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.715703    2532 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-138-252.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-252.ec2.internal"]
Sep 18 19:13:53.715811 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.715754    2532 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:13:53.716974 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.716952    2532 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:13:53.717077 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.716980    2532 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:13:53.717077 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.716990    2532 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:13:53.719244 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.719232    2532 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:13:53.719377 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.719363    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-252.ec2.internal"
Sep 18 19:13:53.719432 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.719393    2532 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:13:53.719878 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.719862    2532 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:13:53.719965 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.719886    2532 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:13:53.719965 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.719888    2532 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:13:53.719965 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.719898    2532 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:13:53.719965 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.719908    2532 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:13:53.719965 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.719930    2532 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:13:53.722176 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.722162    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-252.ec2.internal"
Sep 18 19:13:53.722236 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.722186    2532 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:13:53.722855 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.722836    2532 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:13:53.722938 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.722865    2532 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:13:53.722938 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.722881    2532 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:13:53.740603 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:53.740584    2532 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-252.ec2.internal\" not found" node="ip-10-0-138-252.ec2.internal"
Sep 18 19:13:53.744243 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:53.744229    2532 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-252.ec2.internal\" not found" node="ip-10-0-138-252.ec2.internal"
Sep 18 19:13:53.793616 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.793595    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a8d56d698902f1045998309609be899-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-252.ec2.internal\" (UID: \"0a8d56d698902f1045998309609be899\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-252.ec2.internal"
Sep 18 19:13:53.793705 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.793653    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2d5d27fed1ef1d219785d3f7c796a73d-config\") pod \"kube-apiserver-proxy-ip-10-0-138-252.ec2.internal\" (UID: \"2d5d27fed1ef1d219785d3f7c796a73d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-252.ec2.internal"
Sep 18 19:13:53.793705 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.793677    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0a8d56d698902f1045998309609be899-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-252.ec2.internal\" (UID: \"0a8d56d698902f1045998309609be899\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-252.ec2.internal"
Sep 18 19:13:53.796317 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:53.796303    2532 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-252.ec2.internal\" not found"
Sep 18 19:13:53.894638 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.894616    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2d5d27fed1ef1d219785d3f7c796a73d-config\") pod \"kube-apiserver-proxy-ip-10-0-138-252.ec2.internal\" (UID: \"2d5d27fed1ef1d219785d3f7c796a73d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-252.ec2.internal"
Sep 18 19:13:53.894734 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.894654    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2d5d27fed1ef1d219785d3f7c796a73d-config\") pod \"kube-apiserver-proxy-ip-10-0-138-252.ec2.internal\" (UID: \"2d5d27fed1ef1d219785d3f7c796a73d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-252.ec2.internal"
Sep 18 19:13:53.894734 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.894713    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0a8d56d698902f1045998309609be899-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-252.ec2.internal\" (UID: \"0a8d56d698902f1045998309609be899\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-252.ec2.internal"
Sep 18 19:13:53.894841 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.894747    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a8d56d698902f1045998309609be899-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-252.ec2.internal\" (UID: \"0a8d56d698902f1045998309609be899\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-252.ec2.internal"
Sep 18 19:13:53.894841 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.894801    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a8d56d698902f1045998309609be899-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-252.ec2.internal\" (UID: \"0a8d56d698902f1045998309609be899\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-252.ec2.internal"
Sep 18 19:13:53.894841 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:53.894803    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0a8d56d698902f1045998309609be899-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-252.ec2.internal\" (UID: \"0a8d56d698902f1045998309609be899\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-252.ec2.internal"
Sep 18 19:13:53.896709 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:53.896694    2532 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-252.ec2.internal\" not found"
Sep 18 19:13:53.996978 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:53.996929    2532 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-252.ec2.internal\" not found"
Sep 18 19:13:54.043790 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:54.043765    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-252.ec2.internal"
Sep 18 19:13:54.047273 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:54.047259    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-252.ec2.internal"
Sep 18 19:13:54.097909 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:54.097889    2532 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-252.ec2.internal\" not found"
Sep 18 19:13:54.198455 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:54.198428    2532 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-252.ec2.internal\" not found"
Sep 18 19:13:54.299449 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:54.299427    2532 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-252.ec2.internal\" not found"
Sep 18 19:13:54.399866 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:54.399845    2532 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-252.ec2.internal\" not found"
Sep 18 19:13:54.420543 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:54.420521    2532 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 18 19:13:54.420688 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:54.420662    2532 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 19:13:54.500941 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:54.500918    2532 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-252.ec2.internal\" not found"
Sep 18 19:13:54.558327 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:54.558292    2532 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-17 19:08:53 +0000 UTC" deadline="2028-03-05 07:27:47.960761818 +0000 UTC"
Sep 18 19:13:54.558440 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:54.558327    2532 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12804h13m53.402438216s"
Sep 18 19:13:54.601214 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:54.601183    2532 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-252.ec2.internal\" not found"
Sep 18 19:13:54.666758 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:54.666738    2532 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:13:54.671639 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:54.671622    2532 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:13:54.671713 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:54.671654    2532 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:13:54.671713 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:54.671674    2532 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:13:54.702189 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:54.702169    2532 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-252.ec2.internal\" not found"
Sep 18 19:13:54.718479 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:54.718438    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d5d27fed1ef1d219785d3f7c796a73d.slice/crio-e43e345dd89d5691389ef81a9551bbf53dccbdb2cf0906c6bf19a9554a66b745 WatchSource:0}: Error finding container e43e345dd89d5691389ef81a9551bbf53dccbdb2cf0906c6bf19a9554a66b745: Status 404 returned error can't find the container with id e43e345dd89d5691389ef81a9551bbf53dccbdb2cf0906c6bf19a9554a66b745
Sep 18 19:13:54.718958 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:54.718939    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a8d56d698902f1045998309609be899.slice/crio-38b1e693c8d4ce3125a03fa386258125f05efc823ba27e4c4295918cc55c082e WatchSource:0}: Error finding container 38b1e693c8d4ce3125a03fa386258125f05efc823ba27e4c4295918cc55c082e: Status 404 returned error can't find the container with id 38b1e693c8d4ce3125a03fa386258125f05efc823ba27e4c4295918cc55c082e
Sep 18 19:13:54.723883 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:54.723867    2532 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:13:54.803143 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:54.803094    2532 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-252.ec2.internal\" not found"
Sep 18 19:13:54.834484 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:54.834442    2532 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:13:54.892006 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:54.891968    2532 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-252.ec2.internal"
Sep 18 19:13:54.906866 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:54.906821    2532 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 19:13:54.908547 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:54.908531    2532 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-252.ec2.internal"
Sep 18 19:13:54.915041 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:54.915022    2532 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:13:54.934936 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:54.934910    2532 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 19:13:54.961781 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:54.961760    2532 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:13:55.462216 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.462185    2532 apiserver.go:52] "Watching apiserver"
Sep 18 19:13:55.469975 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.469951    2532 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 18 19:13:55.470400 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.470376    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-rc579","openshift-network-diagnostics/network-check-target-42kxn","openshift-network-operator/iptables-alerter-7jr5l","kube-system/global-pull-secret-syncer-d9rlj","kube-system/konnectivity-agent-226rm","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dr7lf","openshift-multus/multus-mmlqn","openshift-multus/network-metrics-daemon-ps7w2","openshift-ovn-kubernetes/ovnkube-node-sksw8","kube-system/kube-apiserver-proxy-ip-10-0-138-252.ec2.internal","openshift-cluster-node-tuning-operator/tuned-cclx5","openshift-dns/node-resolver-qbs4d","openshift-image-registry/node-ca-7s97w","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-252.ec2.internal"]
Sep 18 19:13:55.473048 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.473025    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mmlqn"
Sep 18 19:13:55.475346 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.475321    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7s97w"
Sep 18 19:13:55.477175 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.477149    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 18 19:13:55.477782 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.477758    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7jr5l"
Sep 18 19:13:55.477882 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.477808    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 18 19:13:55.477882 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.477858    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-5vbxn\""
Sep 18 19:13:55.478010 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.477767    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 18 19:13:55.478114 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.478098    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 18 19:13:55.479855 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.479835    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 18 19:13:55.480004 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.479985    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d9rlj"
Sep 18 19:13:55.480137 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:55.480088    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d9rlj" podUID="7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed"
Sep 18 19:13:55.480504 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.480466    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zvx4d\""
Sep 18 19:13:55.482405 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.482386    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-226rm"
Sep 18 19:13:55.483507 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.483486    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:13:55.483598 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.483533    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-cdw6b\""
Sep 18 19:13:55.483649 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.483618    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 18 19:13:55.483736 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.483720    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 18 19:13:55.483784 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.483773    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 18 19:13:55.484834 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.484815    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dr7lf"
Sep 18 19:13:55.486948 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.486927    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 18 19:13:55.487634 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.487612    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-hrvvn\""
Sep 18 19:13:55.487739 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.487700    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:13:55.488284 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.488258    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 18 19:13:55.489019 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.488638    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 18 19:13:55.490701 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.488974    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 18 19:13:55.490701 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.489670    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 18 19:13:55.490701 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.490516    2532 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 18 19:13:55.492015 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.491996    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rc579"
Sep 18 19:13:55.494438 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.494421    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42kxn"
Sep 18 19:13:55.494523 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:55.494478    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42kxn" podUID="7082df75-a3d1-4eba-90e5-7a02a6aef94b"
Sep 18 19:13:55.496124 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.496102    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jthc9\""
Sep 18 19:13:55.496318 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.496292    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 18 19:13:55.496492 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.496477    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 18 19:13:55.496930 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.496911    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ps7w2"
Sep 18 19:13:55.497019 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:55.496990    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ps7w2" podUID="05c957d5-da77-47c2-8b9c-e7cdc087120c"
Sep 18 19:13:55.499481 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.499462    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cclx5"
Sep 18 19:13:55.499826 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.499792    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 18 19:13:55.499953 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.499861    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ts62c\""
Sep 18 19:13:55.499953 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.499878    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 18 19:13:55.499953 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.499903    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 18 19:13:55.500534 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.499798    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 18 19:13:55.500534 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.499908    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 18 19:13:55.500534 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.500096    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 18 19:13:55.500534 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.500220    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8cjrm\""
Sep 18 19:13:55.500534 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.500302    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 18 19:13:55.502262 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.501960    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qbs4d"
Sep 18 19:13:55.505089 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.504985    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/08755e2e-960f-4099-8bc0-1de6acfdd0c2-etc-selinux\") pod \"aws-ebs-csi-driver-node-dr7lf\" (UID: \"08755e2e-960f-4099-8bc0-1de6acfdd0c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dr7lf"
Sep 18 19:13:55.505089 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.505042    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/210a5221-5e8c-4bf5-9bf1-abe8413e3897-multus-socket-dir-parent\") pod \"multus-mmlqn\" (UID: \"210a5221-5e8c-4bf5-9bf1-abe8413e3897\") " pod="openshift-multus/multus-mmlqn"
Sep 18 19:13:55.505311 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.505115    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/22a18a1f-f825-4aff-8e52-18ab52124ced-host-slash\") pod \"iptables-alerter-7jr5l\" (UID: \"22a18a1f-f825-4aff-8e52-18ab52124ced\") " pod="openshift-network-operator/iptables-alerter-7jr5l"
Sep 18 19:13:55.505311 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.505162    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2cef0f4e-e44f-4996-8561-da9d261036c7-etc-openvswitch\") pod \"ovnkube-node-sksw8\" (UID: \"2cef0f4e-e44f-4996-8561-da9d261036c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:13:55.505311 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.505190    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2cef0f4e-e44f-4996-8561-da9d261036c7-host-cni-bin\") pod \"ovnkube-node-sksw8\" (UID: \"2cef0f4e-e44f-4996-8561-da9d261036c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:13:55.505311 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.505257    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2cef0f4e-e44f-4996-8561-da9d261036c7-node-log\") pod \"ovnkube-node-sksw8\" (UID: \"2cef0f4e-e44f-4996-8561-da9d261036c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:13:55.505311 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.505301    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/08755e2e-960f-4099-8bc0-1de6acfdd0c2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dr7lf\" (UID: \"08755e2e-960f-4099-8bc0-1de6acfdd0c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dr7lf"
Sep 18 19:13:55.505598 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.505330    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/08755e2e-960f-4099-8bc0-1de6acfdd0c2-device-dir\") pod \"aws-ebs-csi-driver-node-dr7lf\" (UID: \"08755e2e-960f-4099-8bc0-1de6acfdd0c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dr7lf"
Sep 18 19:13:55.505598 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.505355    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f222\" (UniqueName: \"kubernetes.io/projected/812cd3c0-1c61-4d21-8dd7-77d98f0edd2d-kube-api-access-9f222\") pod \"node-ca-7s97w\" (UID: \"812cd3c0-1c61-4d21-8dd7-77d98f0edd2d\") " pod="openshift-image-registry/node-ca-7s97w"
Sep 18 19:13:55.505598 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.505402    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/22a18a1f-f825-4aff-8e52-18ab52124ced-iptables-alerter-script\") pod \"iptables-alerter-7jr5l\" (UID: \"22a18a1f-f825-4aff-8e52-18ab52124ced\") " pod="openshift-network-operator/iptables-alerter-7jr5l"
Sep 18 19:13:55.505598 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.505436    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2cef0f4e-e44f-4996-8561-da9d261036c7-host-kubelet\") pod \"ovnkube-node-sksw8\" (UID: \"2cef0f4e-e44f-4996-8561-da9d261036c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:13:55.505598 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.505460    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2cef0f4e-e44f-4996-8561-da9d261036c7-host-slash\") pod \"ovnkube-node-sksw8\" (UID: \"2cef0f4e-e44f-4996-8561-da9d261036c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:13:55.505598 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.505482    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2cef0f4e-e44f-4996-8561-da9d261036c7-run-systemd\") pod \"ovnkube-node-sksw8\" (UID: \"2cef0f4e-e44f-4996-8561-da9d261036c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:13:55.505598 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.505513    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2cef0f4e-e44f-4996-8561-da9d261036c7-log-socket\") pod \"ovnkube-node-sksw8\" (UID: \"2cef0f4e-e44f-4996-8561-da9d261036c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:13:55.505598 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.505537    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed-kubelet-config\") pod \"global-pull-secret-syncer-d9rlj\" (UID: \"7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed\") " pod="kube-system/global-pull-secret-syncer-d9rlj"
Sep 18 19:13:55.505598 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.505566    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed-dbus\") pod \"global-pull-secret-syncer-d9rlj\" (UID: \"7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed\") " pod="kube-system/global-pull-secret-syncer-d9rlj"
Sep 18 19:13:55.505598 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.505590    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twqgk\" (UniqueName: \"kubernetes.io/projected/08755e2e-960f-4099-8bc0-1de6acfdd0c2-kube-api-access-twqgk\") pod \"aws-ebs-csi-driver-node-dr7lf\" (UID: \"08755e2e-960f-4099-8bc0-1de6acfdd0c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dr7lf"
Sep 18 19:13:55.505598 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.505615    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/210a5221-5e8c-4bf5-9bf1-abe8413e3897-host-var-lib-cni-multus\") pod \"multus-mmlqn\" (UID: \"210a5221-5e8c-4bf5-9bf1-abe8413e3897\") " pod="openshift-multus/multus-mmlqn"
Sep 18 19:13:55.506148 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.505653    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7959688f-be33-494c-a8e9-08d33f2c2d3b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rc579\" (UID: \"7959688f-be33-494c-a8e9-08d33f2c2d3b\") " pod="openshift-multus/multus-additional-cni-plugins-rc579"
Sep 18 19:13:55.506148 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.505705    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/210a5221-5e8c-4bf5-9bf1-abe8413e3897-hostroot\") pod \"multus-mmlqn\" (UID: \"210a5221-5e8c-4bf5-9bf1-abe8413e3897\") " pod="openshift-multus/multus-mmlqn"
Sep 18 19:13:55.506148 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.505729    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/812cd3c0-1c61-4d21-8dd7-77d98f0edd2d-serviceca\") pod \"node-ca-7s97w\" (UID: \"812cd3c0-1c61-4d21-8dd7-77d98f0edd2d\") " pod="openshift-image-registry/node-ca-7s97w"
Sep 18 19:13:55.506148 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.505759    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2cef0f4e-e44f-4996-8561-da9d261036c7-ovnkube-config\") pod \"ovnkube-node-sksw8\" (UID: \"2cef0f4e-e44f-4996-8561-da9d261036c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:13:55.506148 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.505785    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/210a5221-5e8c-4bf5-9bf1-abe8413e3897-host-run-multus-certs\") pod \"multus-mmlqn\" (UID: \"210a5221-5e8c-4bf5-9bf1-abe8413e3897\") " pod="openshift-multus/multus-mmlqn"
Sep 18 19:13:55.506148 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.505815    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69mnl\" (UniqueName: \"kubernetes.io/projected/22a18a1f-f825-4aff-8e52-18ab52124ced-kube-api-access-69mnl\") pod \"iptables-alerter-7jr5l\" (UID: \"22a18a1f-f825-4aff-8e52-18ab52124ced\") " pod="openshift-network-operator/iptables-alerter-7jr5l"
Sep 18 19:13:55.506148 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.505853    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7959688f-be33-494c-a8e9-08d33f2c2d3b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rc579\" (UID: \"7959688f-be33-494c-a8e9-08d33f2c2d3b\") " pod="openshift-multus/multus-additional-cni-plugins-rc579"
Sep 18 19:13:55.506148 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.505879    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/210a5221-5e8c-4bf5-9bf1-abe8413e3897-multus-cni-dir\") pod \"multus-mmlqn\" (UID: \"210a5221-5e8c-4bf5-9bf1-abe8413e3897\") " pod="openshift-multus/multus-mmlqn"
Sep 18 19:13:55.506148 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.505905    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2cef0f4e-e44f-4996-8561-da9d261036c7-systemd-units\") pod \"ovnkube-node-sksw8\" (UID: \"2cef0f4e-e44f-4996-8561-da9d261036c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:13:55.506148 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.505936    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p99zh\" (UniqueName: \"kubernetes.io/projected/2cef0f4e-e44f-4996-8561-da9d261036c7-kube-api-access-p99zh\") pod \"ovnkube-node-sksw8\" (UID: \"2cef0f4e-e44f-4996-8561-da9d261036c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:13:55.506148 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.505960    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/210a5221-5e8c-4bf5-9bf1-abe8413e3897-cni-binary-copy\") pod \"multus-mmlqn\" (UID: \"210a5221-5e8c-4bf5-9bf1-abe8413e3897\") " pod="openshift-multus/multus-mmlqn"
Sep 18 19:13:55.506148 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.505986    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed-original-pull-secret\") pod \"global-pull-secret-syncer-d9rlj\" (UID: \"7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed\") " pod="kube-system/global-pull-secret-syncer-d9rlj"
Sep 18 19:13:55.506148 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.506010    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/210a5221-5e8c-4bf5-9bf1-abe8413e3897-system-cni-dir\") pod \"multus-mmlqn\" (UID: \"210a5221-5e8c-4bf5-9bf1-abe8413e3897\") " pod="openshift-multus/multus-mmlqn"
Sep 18 19:13:55.506148 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.506040    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/210a5221-5e8c-4bf5-9bf1-abe8413e3897-multus-daemon-config\") pod \"multus-mmlqn\" (UID: \"210a5221-5e8c-4bf5-9bf1-abe8413e3897\") " pod="openshift-multus/multus-mmlqn"
Sep 18 19:13:55.506148 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.506085    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/812cd3c0-1c61-4d21-8dd7-77d98f0edd2d-host\") pod \"node-ca-7s97w\" (UID: \"812cd3c0-1c61-4d21-8dd7-77d98f0edd2d\") " pod="openshift-image-registry/node-ca-7s97w"
Sep 18 19:13:55.506148 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.506113    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d45a88d4-e763-4a5e-87a3-91c0bb3bbf68-agent-certs\") pod \"konnectivity-agent-226rm\" (UID: \"d45a88d4-e763-4a5e-87a3-91c0bb3bbf68\") " pod="kube-system/konnectivity-agent-226rm"
Sep 18 19:13:55.507005 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.506170    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/08755e2e-960f-4099-8bc0-1de6acfdd0c2-sys-fs\") pod \"aws-ebs-csi-driver-node-dr7lf\" (UID: \"08755e2e-960f-4099-8bc0-1de6acfdd0c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dr7lf"
Sep 18 19:13:55.507005 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.506194    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7959688f-be33-494c-a8e9-08d33f2c2d3b-cnibin\") pod \"multus-additional-cni-plugins-rc579\" (UID: \"7959688f-be33-494c-a8e9-08d33f2c2d3b\") " pod="openshift-multus/multus-additional-cni-plugins-rc579"
Sep 18 19:13:55.507005 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.506225    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7959688f-be33-494c-a8e9-08d33f2c2d3b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rc579\" (UID: \"7959688f-be33-494c-a8e9-08d33f2c2d3b\") " pod="openshift-multus/multus-additional-cni-plugins-rc579"
Sep 18 19:13:55.507005 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.506249    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/210a5221-5e8c-4bf5-9bf1-abe8413e3897-etc-kubernetes\") pod \"multus-mmlqn\" (UID: \"210a5221-5e8c-4bf5-9bf1-abe8413e3897\") " pod="openshift-multus/multus-mmlqn"
Sep 18 19:13:55.507005 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.506273    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2cef0f4e-e44f-4996-8561-da9d261036c7-env-overrides\") pod \"ovnkube-node-sksw8\" (UID: \"2cef0f4e-e44f-4996-8561-da9d261036c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:13:55.507005 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.506306    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2cef0f4e-e44f-4996-8561-da9d261036c7-ovn-node-metrics-cert\") pod \"ovnkube-node-sksw8\" (UID: \"2cef0f4e-e44f-4996-8561-da9d261036c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:13:55.507005 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.506333    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/210a5221-5e8c-4bf5-9bf1-abe8413e3897-host-run-k8s-cni-cncf-io\") pod \"multus-mmlqn\" (UID: \"210a5221-5e8c-4bf5-9bf1-abe8413e3897\") " pod="openshift-multus/multus-mmlqn"
Sep 18 19:13:55.507005 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.506356    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2cef0f4e-e44f-4996-8561-da9d261036c7-host-run-netns\") pod \"ovnkube-node-sksw8\" (UID: \"2cef0f4e-e44f-4996-8561-da9d261036c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:13:55.507005 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.506380    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7959688f-be33-494c-a8e9-08d33f2c2d3b-system-cni-dir\") pod \"multus-additional-cni-plugins-rc579\" (UID: \"7959688f-be33-494c-a8e9-08d33f2c2d3b\") " pod="openshift-multus/multus-additional-cni-plugins-rc579"
Sep 18 19:13:55.507005 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.506420    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/210a5221-5e8c-4bf5-9bf1-abe8413e3897-cnibin\") pod \"multus-mmlqn\" (UID: \"210a5221-5e8c-4bf5-9bf1-abe8413e3897\") " pod="openshift-multus/multus-mmlqn"
Sep 18 19:13:55.507005 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.506472    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/210a5221-5e8c-4bf5-9bf1-abe8413e3897-host-var-lib-cni-bin\") pod \"multus-mmlqn\" (UID: \"210a5221-5e8c-4bf5-9bf1-abe8413e3897\") " pod="openshift-multus/multus-mmlqn"
Sep 18 19:13:55.507005 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.506497    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/210a5221-5e8c-4bf5-9bf1-abe8413e3897-host-var-lib-kubelet\") pod \"multus-mmlqn\" (UID: \"210a5221-5e8c-4bf5-9bf1-abe8413e3897\") " pod="openshift-multus/multus-mmlqn"
Sep 18 19:13:55.507005 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.506605    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/08755e2e-960f-4099-8bc0-1de6acfdd0c2-socket-dir\") pod \"aws-ebs-csi-driver-node-dr7lf\" (UID: \"08755e2e-960f-4099-8bc0-1de6acfdd0c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dr7lf"
Sep 18 19:13:55.507005 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.506634    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bblcj\" (UniqueName: \"kubernetes.io/projected/7959688f-be33-494c-a8e9-08d33f2c2d3b-kube-api-access-bblcj\") pod \"multus-additional-cni-plugins-rc579\" (UID: \"7959688f-be33-494c-a8e9-08d33f2c2d3b\") " pod="openshift-multus/multus-additional-cni-plugins-rc579"
Sep 18 19:13:55.507005 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.506664    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d45a88d4-e763-4a5e-87a3-91c0bb3bbf68-konnectivity-ca\") pod \"konnectivity-agent-226rm\" (UID: \"d45a88d4-e763-4a5e-87a3-91c0bb3bbf68\") " pod="kube-system/konnectivity-agent-226rm"
Sep 18 19:13:55.507005 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.506690    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2cef0f4e-e44f-4996-8561-da9d261036c7-host-run-ovn-kubernetes\") pod \"ovnkube-node-sksw8\" (UID: \"2cef0f4e-e44f-4996-8561-da9d261036c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:13:55.507731 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.506717    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlf4p\" (UniqueName: \"kubernetes.io/projected/210a5221-5e8c-4bf5-9bf1-abe8413e3897-kube-api-access-rlf4p\") pod \"multus-mmlqn\" (UID: \"210a5221-5e8c-4bf5-9bf1-abe8413e3897\") " pod="openshift-multus/multus-mmlqn"
Sep 18 19:13:55.507731 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.506774    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2cef0f4e-e44f-4996-8561-da9d261036c7-var-lib-openvswitch\") pod \"ovnkube-node-sksw8\" (UID: \"2cef0f4e-e44f-4996-8561-da9d261036c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:13:55.507731 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.506803    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2cef0f4e-e44f-4996-8561-da9d261036c7-run-openvswitch\") pod \"ovnkube-node-sksw8\" (UID: \"2cef0f4e-e44f-4996-8561-da9d261036c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:13:55.507731 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.506826    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2cef0f4e-e44f-4996-8561-da9d261036c7-run-ovn\") pod \"ovnkube-node-sksw8\" (UID: \"2cef0f4e-e44f-4996-8561-da9d261036c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:13:55.507731 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.506859    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2cef0f4e-e44f-4996-8561-da9d261036c7-ovnkube-script-lib\") pod \"ovnkube-node-sksw8\" (UID: \"2cef0f4e-e44f-4996-8561-da9d261036c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:13:55.507731 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.506961    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/210a5221-5e8c-4bf5-9bf1-abe8413e3897-host-run-netns\") pod \"multus-mmlqn\" (UID: \"210a5221-5e8c-4bf5-9bf1-abe8413e3897\") " pod="openshift-multus/multus-mmlqn"
Sep 18 19:13:55.507731 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.507001    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2cef0f4e-e44f-4996-8561-da9d261036c7-host-cni-netd\") pod \"ovnkube-node-sksw8\" (UID: \"2cef0f4e-e44f-4996-8561-da9d261036c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:13:55.507731 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.507041    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2cef0f4e-e44f-4996-8561-da9d261036c7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sksw8\" (UID: \"2cef0f4e-e44f-4996-8561-da9d261036c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:13:55.507731 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.507071    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/08755e2e-960f-4099-8bc0-1de6acfdd0c2-registration-dir\") pod \"aws-ebs-csi-driver-node-dr7lf\" (UID: \"08755e2e-960f-4099-8bc0-1de6acfdd0c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dr7lf"
Sep 18 19:13:55.507731 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.507100    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7959688f-be33-494c-a8e9-08d33f2c2d3b-cni-binary-copy\") pod \"multus-additional-cni-plugins-rc579\" (UID: \"7959688f-be33-494c-a8e9-08d33f2c2d3b\") " pod="openshift-multus/multus-additional-cni-plugins-rc579"
Sep 18 19:13:55.507731 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.507123    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:13:55.507731 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.507143    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/210a5221-5e8c-4bf5-9bf1-abe8413e3897-multus-conf-dir\") pod \"multus-mmlqn\" (UID: \"210a5221-5e8c-4bf5-9bf1-abe8413e3897\") " pod="openshift-multus/multus-mmlqn"
Sep 18 19:13:55.507731 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.507200    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 18 19:13:55.509201 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.509005    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-k7p47\""
Sep 18 19:13:55.509309 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.509217    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 18 19:13:55.509399 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.509382    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 18 19:13:55.509531 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.509510    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-s25tn\""
Sep 18 19:13:55.532776 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.532754    2532 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:13:55.533370 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.533349    2532 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 19:13:55.562636 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.562595    2532 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-hnrx5"
Sep 18 19:13:55.581375 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.581349    2532 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-hnrx5"
Sep 18 19:13:55.592716 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.592698    2532 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 18 19:13:55.607874 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.607848    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7959688f-be33-494c-a8e9-08d33f2c2d3b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rc579\" (UID: \"7959688f-be33-494c-a8e9-08d33f2c2d3b\") " pod="openshift-multus/multus-additional-cni-plugins-rc579"
Sep 18 19:13:55.607992 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.607887    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/210a5221-5e8c-4bf5-9bf1-abe8413e3897-hostroot\") pod \"multus-mmlqn\" (UID: \"210a5221-5e8c-4bf5-9bf1-abe8413e3897\") " pod="openshift-multus/multus-mmlqn"
Sep 18 19:13:55.607992 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.607913    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/812cd3c0-1c61-4d21-8dd7-77d98f0edd2d-serviceca\") pod \"node-ca-7s97w\" (UID: \"812cd3c0-1c61-4d21-8dd7-77d98f0edd2d\") " pod="openshift-image-registry/node-ca-7s97w"
Sep 18 19:13:55.607992 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.607938    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2cef0f4e-e44f-4996-8561-da9d261036c7-ovnkube-config\") pod \"ovnkube-node-sksw8\" (UID: \"2cef0f4e-e44f-4996-8561-da9d261036c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:13:55.607992 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.607967    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/04602990-ffaa-4d6f-ac9e-bb960fccfbef-tmp\") pod \"tuned-cclx5\" (UID: \"04602990-ffaa-4d6f-ac9e-bb960fccfbef\") " pod="openshift-cluster-node-tuning-operator/tuned-cclx5"
Sep 18 19:13:55.608223 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.607993    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b7hs\" (UniqueName: \"kubernetes.io/projected/7082df75-a3d1-4eba-90e5-7a02a6aef94b-kube-api-access-6b7hs\") pod \"network-check-target-42kxn\" (UID: \"7082df75-a3d1-4eba-90e5-7a02a6aef94b\") " pod="openshift-network-diagnostics/network-check-target-42kxn"
Sep 18 19:13:55.608223 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.608177    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7959688f-be33-494c-a8e9-08d33f2c2d3b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rc579\" (UID: \"7959688f-be33-494c-a8e9-08d33f2c2d3b\") " pod="openshift-multus/multus-additional-cni-plugins-rc579"
Sep 18 19:13:55.608338 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.608226    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/210a5221-5e8c-4bf5-9bf1-abe8413e3897-host-run-multus-certs\") pod \"multus-mmlqn\" (UID: \"210a5221-5e8c-4bf5-9bf1-abe8413e3897\") " pod="openshift-multus/multus-mmlqn"
Sep 18 19:13:55.608338 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.608277    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69mnl\" (UniqueName: \"kubernetes.io/projected/22a18a1f-f825-4aff-8e52-18ab52124ced-kube-api-access-69mnl\") pod \"iptables-alerter-7jr5l\" (UID: \"22a18a1f-f825-4aff-8e52-18ab52124ced\") " pod="openshift-network-operator/iptables-alerter-7jr5l"
Sep 18 19:13:55.608338 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.608308    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7959688f-be33-494c-a8e9-08d33f2c2d3b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rc579\" (UID: \"7959688f-be33-494c-a8e9-08d33f2c2d3b\") " pod="openshift-multus/multus-additional-cni-plugins-rc579"
Sep 18 19:13:55.608494 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.608336    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/210a5221-5e8c-4bf5-9bf1-abe8413e3897-multus-cni-dir\") pod \"multus-mmlqn\" (UID: \"210a5221-5e8c-4bf5-9bf1-abe8413e3897\") " pod="openshift-multus/multus-mmlqn"
Sep 18 19:13:55.608494 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.608373    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2cef0f4e-e44f-4996-8561-da9d261036c7-systemd-units\") pod \"ovnkube-node-sksw8\" (UID: \"2cef0f4e-e44f-4996-8561-da9d261036c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:13:55.608494 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.608399    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p99zh\" (UniqueName: \"kubernetes.io/projected/2cef0f4e-e44f-4996-8561-da9d261036c7-kube-api-access-p99zh\") pod \"ovnkube-node-sksw8\" (UID: \"2cef0f4e-e44f-4996-8561-da9d261036c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:13:55.608494 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.608429    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/210a5221-5e8c-4bf5-9bf1-abe8413e3897-cni-binary-copy\") pod \"multus-mmlqn\" (UID: \"210a5221-5e8c-4bf5-9bf1-abe8413e3897\") " pod="openshift-multus/multus-mmlqn"
Sep 18 19:13:55.608494 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.608454    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed-original-pull-secret\") pod \"global-pull-secret-syncer-d9rlj\" (UID: \"7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed\") " pod="kube-system/global-pull-secret-syncer-d9rlj"
Sep 18 19:13:55.608494 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.608482    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/210a5221-5e8c-4bf5-9bf1-abe8413e3897-system-cni-dir\") pod \"multus-mmlqn\" (UID: \"210a5221-5e8c-4bf5-9bf1-abe8413e3897\") " pod="openshift-multus/multus-mmlqn"
Sep 18 19:13:55.608804 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.608503    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/812cd3c0-1c61-4d21-8dd7-77d98f0edd2d-serviceca\") pod \"node-ca-7s97w\" (UID: \"812cd3c0-1c61-4d21-8dd7-77d98f0edd2d\") " pod="openshift-image-registry/node-ca-7s97w"
Sep 18 19:13:55.608804 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.608506    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/210a5221-5e8c-4bf5-9bf1-abe8413e3897-multus-daemon-config\") pod \"multus-mmlqn\" (UID: \"210a5221-5e8c-4bf5-9bf1-abe8413e3897\") " pod="openshift-multus/multus-mmlqn"
Sep 18 19:13:55.608804 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.608567    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/812cd3c0-1c61-4d21-8dd7-77d98f0edd2d-host\") pod \"node-ca-7s97w\" (UID: \"812cd3c0-1c61-4d21-8dd7-77d98f0edd2d\") " pod="openshift-image-registry/node-ca-7s97w"
Sep 18 19:13:55.608804 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.608595    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d45a88d4-e763-4a5e-87a3-91c0bb3bbf68-agent-certs\") pod \"konnectivity-agent-226rm\" (UID: \"d45a88d4-e763-4a5e-87a3-91c0bb3bbf68\") " pod="kube-system/konnectivity-agent-226rm"
Sep 18 19:13:55.608804 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.608623    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/04602990-ffaa-4d6f-ac9e-bb960fccfbef-etc-modprobe-d\") pod \"tuned-cclx5\" (UID: \"04602990-ffaa-4d6f-ac9e-bb960fccfbef\") " pod="openshift-cluster-node-tuning-operator/tuned-cclx5"
Sep 18 19:13:55.608804 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.608641    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2cef0f4e-e44f-4996-8561-da9d261036c7-ovnkube-config\") pod \"ovnkube-node-sksw8\" (UID: \"2cef0f4e-e44f-4996-8561-da9d261036c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:13:55.608804 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.608653    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt5x5\" (UniqueName: \"kubernetes.io/projected/851825cd-f968-471c-87c0-b7d2423c9c99-kube-api-access-zt5x5\") pod \"node-resolver-qbs4d\" (UID: \"851825cd-f968-471c-87c0-b7d2423c9c99\") " pod="openshift-dns/node-resolver-qbs4d"
Sep 18 19:13:55.608804 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.608696    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/08755e2e-960f-4099-8bc0-1de6acfdd0c2-sys-fs\") pod \"aws-ebs-csi-driver-node-dr7lf\" (UID: \"08755e2e-960f-4099-8bc0-1de6acfdd0c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dr7lf"
Sep 18 19:13:55.608804 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.608712    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/812cd3c0-1c61-4d21-8dd7-77d98f0edd2d-host\") pod \"node-ca-7s97w\" (UID: \"812cd3c0-1c61-4d21-8dd7-77d98f0edd2d\") " pod="openshift-image-registry/node-ca-7s97w"
Sep 18 19:13:55.608804 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.608726    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7959688f-be33-494c-a8e9-08d33f2c2d3b-cnibin\") pod \"multus-additional-cni-plugins-rc579\" (UID: \"7959688f-be33-494c-a8e9-08d33f2c2d3b\") " pod="openshift-multus/multus-additional-cni-plugins-rc579"
Sep 18 19:13:55.609372 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.608843    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/08755e2e-960f-4099-8bc0-1de6acfdd0c2-sys-fs\") pod \"aws-ebs-csi-driver-node-dr7lf\" (UID: \"08755e2e-960f-4099-8bc0-1de6acfdd0c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dr7lf"
Sep 18 19:13:55.609372 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.608911    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/210a5221-5e8c-4bf5-9bf1-abe8413e3897-multus-cni-dir\") pod \"multus-mmlqn\" (UID: \"210a5221-5e8c-4bf5-9bf1-abe8413e3897\") " pod="openshift-multus/multus-mmlqn"
Sep 18 19:13:55.609372 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.608952    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/210a5221-5e8c-4bf5-9bf1-abe8413e3897-hostroot\") pod \"multus-mmlqn\" (UID: \"210a5221-5e8c-4bf5-9bf1-abe8413e3897\") " pod="openshift-multus/multus-mmlqn"
Sep 18 19:13:55.609372 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.608993    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/210a5221-5e8c-4bf5-9bf1-abe8413e3897-host-run-multus-certs\") pod \"multus-mmlqn\" (UID: \"210a5221-5e8c-4bf5-9bf1-abe8413e3897\") " pod="openshift-multus/multus-mmlqn"
Sep 18 19:13:55.609372 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.609034    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/210a5221-5e8c-4bf5-9bf1-abe8413e3897-multus-daemon-config\") pod \"multus-mmlqn\" (UID: \"210a5221-5e8c-4bf5-9bf1-abe8413e3897\") " pod="openshift-multus/multus-mmlqn"
Sep 18 19:13:55.609372 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.609105    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2cef0f4e-e44f-4996-8561-da9d261036c7-systemd-units\") pod \"ovnkube-node-sksw8\" (UID: \"2cef0f4e-e44f-4996-8561-da9d261036c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:13:55.609645 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:55.609411    2532 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:13:55.609645 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.609433    2532 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 18 19:13:55.609645 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:55.609489    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed-original-pull-secret podName:7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed nodeName:}" failed. No retries permitted until 2026-09-18 19:13:56.109467223 +0000 UTC m=+3.029971072 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed-original-pull-secret") pod "global-pull-secret-syncer-d9rlj" (UID: "7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:13:55.609778 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.609544    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/210a5221-5e8c-4bf5-9bf1-abe8413e3897-system-cni-dir\") pod \"multus-mmlqn\" (UID: \"210a5221-5e8c-4bf5-9bf1-abe8413e3897\") " pod="openshift-multus/multus-mmlqn"
Sep 18 19:13:55.609894 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.609868    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7959688f-be33-494c-a8e9-08d33f2c2d3b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rc579\" (UID: \"7959688f-be33-494c-a8e9-08d33f2c2d3b\") " pod="openshift-multus/multus-additional-cni-plugins-rc579"
Sep 18 19:13:55.609977 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.609891    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7959688f-be33-494c-a8e9-08d33f2c2d3b-cnibin\") pod \"multus-additional-cni-plugins-rc579\" (UID: \"7959688f-be33-494c-a8e9-08d33f2c2d3b\") " pod="openshift-multus/multus-additional-cni-plugins-rc579"
Sep 18 19:13:55.609977 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.609946    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7959688f-be33-494c-a8e9-08d33f2c2d3b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rc579\" (UID: \"7959688f-be33-494c-a8e9-08d33f2c2d3b\") " pod="openshift-multus/multus-additional-cni-plugins-rc579"
Sep 18 19:13:55.609977 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.609974    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/210a5221-5e8c-4bf5-9bf1-abe8413e3897-etc-kubernetes\") pod \"multus-mmlqn\" (UID: \"210a5221-5e8c-4bf5-9bf1-abe8413e3897\") " pod="openshift-multus/multus-mmlqn"
Sep 18 19:13:55.610147 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.609996    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2cef0f4e-e44f-4996-8561-da9d261036c7-env-overrides\") pod \"ovnkube-node-sksw8\" (UID: \"2cef0f4e-e44f-4996-8561-da9d261036c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:13:55.610802 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.610697    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2cef0f4e-e44f-4996-8561-da9d261036c7-ovn-node-metrics-cert\") pod \"ovnkube-node-sksw8\" (UID: \"2cef0f4e-e44f-4996-8561-da9d261036c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:13:55.610802 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.610750    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/851825cd-f968-471c-87c0-b7d2423c9c99-hosts-file\") pod \"node-resolver-qbs4d\" (UID: \"851825cd-f968-471c-87c0-b7d2423c9c99\") " pod="openshift-dns/node-resolver-qbs4d"
Sep 18 19:13:55.610802 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.610786    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/210a5221-5e8c-4bf5-9bf1-abe8413e3897-host-run-k8s-cni-cncf-io\") pod \"multus-mmlqn\" (UID: \"210a5221-5e8c-4bf5-9bf1-abe8413e3897\") " pod="openshift-multus/multus-mmlqn"
Sep 18 19:13:55.611005 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.610817    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2cef0f4e-e44f-4996-8561-da9d261036c7-host-run-netns\") pod \"ovnkube-node-sksw8\" (UID: \"2cef0f4e-e44f-4996-8561-da9d261036c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:13:55.611005 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.610855    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/04602990-ffaa-4d6f-ac9e-bb960fccfbef-host\") pod \"tuned-cclx5\" (UID: \"04602990-ffaa-4d6f-ac9e-bb960fccfbef\") " pod="openshift-cluster-node-tuning-operator/tuned-cclx5"
Sep 18 19:13:55.611005 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.610888    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7959688f-be33-494c-a8e9-08d33f2c2d3b-system-cni-dir\") pod \"multus-additional-cni-plugins-rc579\" (UID: \"7959688f-be33-494c-a8e9-08d33f2c2d3b\") " pod="openshift-multus/multus-additional-cni-plugins-rc579"
Sep 18 19:13:55.611005 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.610932    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/210a5221-5e8c-4bf5-9bf1-abe8413e3897-cnibin\") pod \"multus-mmlqn\" (UID: \"210a5221-5e8c-4bf5-9bf1-abe8413e3897\") " pod="openshift-multus/multus-mmlqn"
Sep 18 19:13:55.611005 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.610964    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/210a5221-5e8c-4bf5-9bf1-abe8413e3897-host-var-lib-cni-bin\") pod \"multus-mmlqn\" (UID: \"210a5221-5e8c-4bf5-9bf1-abe8413e3897\") " pod="openshift-multus/multus-mmlqn"
Sep 18 19:13:55.611005 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.610993    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/210a5221-5e8c-4bf5-9bf1-abe8413e3897-host-var-lib-kubelet\") pod \"multus-mmlqn\" (UID: \"210a5221-5e8c-4bf5-9bf1-abe8413e3897\") " pod="openshift-multus/multus-mmlqn"
Sep 18 19:13:55.611314 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.611007    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/210a5221-5e8c-4bf5-9bf1-abe8413e3897-cni-binary-copy\") pod \"multus-mmlqn\" (UID: \"210a5221-5e8c-4bf5-9bf1-abe8413e3897\") " pod="openshift-multus/multus-mmlqn"
Sep 18 19:13:55.611314 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.611029    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/04602990-ffaa-4d6f-ac9e-bb960fccfbef-etc-tuned\") pod \"tuned-cclx5\" (UID: \"04602990-ffaa-4d6f-ac9e-bb960fccfbef\") " pod="openshift-cluster-node-tuning-operator/tuned-cclx5"
Sep 18 19:13:55.612486 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.611713    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvm7x\" (UniqueName: \"kubernetes.io/projected/04602990-ffaa-4d6f-ac9e-bb960fccfbef-kube-api-access-tvm7x\") pod \"tuned-cclx5\" (UID: \"04602990-ffaa-4d6f-ac9e-bb960fccfbef\") " pod="openshift-cluster-node-tuning-operator/tuned-cclx5"
Sep 18 19:13:55.612486 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.611808    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/08755e2e-960f-4099-8bc0-1de6acfdd0c2-socket-dir\") pod \"aws-ebs-csi-driver-node-dr7lf\" (UID: \"08755e2e-960f-4099-8bc0-1de6acfdd0c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dr7lf"
Sep 18 19:13:55.612486 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.611845    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bblcj\" (UniqueName: \"kubernetes.io/projected/7959688f-be33-494c-a8e9-08d33f2c2d3b-kube-api-access-bblcj\") pod \"multus-additional-cni-plugins-rc579\" (UID: \"7959688f-be33-494c-a8e9-08d33f2c2d3b\") " pod="openshift-multus/multus-additional-cni-plugins-rc579"
Sep 18 19:13:55.612486 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.611882    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d45a88d4-e763-4a5e-87a3-91c0bb3bbf68-konnectivity-ca\") pod \"konnectivity-agent-226rm\" (UID: \"d45a88d4-e763-4a5e-87a3-91c0bb3bbf68\") " pod="kube-system/konnectivity-agent-226rm"
Sep 18 19:13:55.612486 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.611916    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2cef0f4e-e44f-4996-8561-da9d261036c7-host-run-ovn-kubernetes\") pod \"ovnkube-node-sksw8\" (UID: \"2cef0f4e-e44f-4996-8561-da9d261036c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:13:55.612486 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.611949    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/04602990-ffaa-4d6f-ac9e-bb960fccfbef-etc-sysctl-d\") pod \"tuned-cclx5\" (UID: \"04602990-ffaa-4d6f-ac9e-bb960fccfbef\") " pod="openshift-cluster-node-tuning-operator/tuned-cclx5"
Sep 18 19:13:55.612486 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.611976    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/04602990-ffaa-4d6f-ac9e-bb960fccfbef-var-lib-kubelet\") pod \"tuned-cclx5\" (UID: \"04602990-ffaa-4d6f-ac9e-bb960fccfbef\") " pod="openshift-cluster-node-tuning-operator/tuned-cclx5"
Sep 18 19:13:55.612486 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.612008    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/851825cd-f968-471c-87c0-b7d2423c9c99-tmp-dir\") pod \"node-resolver-qbs4d\" (UID: \"851825cd-f968-471c-87c0-b7d2423c9c99\") " pod="openshift-dns/node-resolver-qbs4d"
Sep 18 19:13:55.612486 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.612040    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlf4p\" (UniqueName: \"kubernetes.io/projected/210a5221-5e8c-4bf5-9bf1-abe8413e3897-kube-api-access-rlf4p\") pod \"multus-mmlqn\" (UID: \"210a5221-5e8c-4bf5-9bf1-abe8413e3897\") " pod="openshift-multus/multus-mmlqn"
Sep 18 19:13:55.612486 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.612052    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7959688f-be33-494c-a8e9-08d33f2c2d3b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rc579\" (UID: \"7959688f-be33-494c-a8e9-08d33f2c2d3b\") " pod="openshift-multus/multus-additional-cni-plugins-rc579"
Sep 18 19:13:55.612486 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.612071    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2cef0f4e-e44f-4996-8561-da9d261036c7-var-lib-openvswitch\") pod \"ovnkube-node-sksw8\" (UID: \"2cef0f4e-e44f-4996-8561-da9d261036c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:13:55.612486 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.612117    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2cef0f4e-e44f-4996-8561-da9d261036c7-run-openvswitch\") pod \"ovnkube-node-sksw8\" (UID: \"2cef0f4e-e44f-4996-8561-da9d261036c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:13:55.612486 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.612150    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2cef0f4e-e44f-4996-8561-da9d261036c7-run-ovn\") pod \"ovnkube-node-sksw8\" (UID: \"2cef0f4e-e44f-4996-8561-da9d261036c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:13:55.612486 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.612184    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2cef0f4e-e44f-4996-8561-da9d261036c7-ovnkube-script-lib\") pod \"ovnkube-node-sksw8\" (UID: \"2cef0f4e-e44f-4996-8561-da9d261036c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:13:55.612486 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.612217    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/04602990-ffaa-4d6f-ac9e-bb960fccfbef-etc-sysconfig\") pod \"tuned-cclx5\" (UID: \"04602990-ffaa-4d6f-ac9e-bb960fccfbef\") " pod="openshift-cluster-node-tuning-operator/tuned-cclx5"
Sep 18 19:13:55.612486 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.612247    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/04602990-ffaa-4d6f-ac9e-bb960fccfbef-etc-systemd\") pod \"tuned-cclx5\" (UID: \"04602990-ffaa-4d6f-ac9e-bb960fccfbef\") " pod="openshift-cluster-node-tuning-operator/tuned-cclx5"
Sep 18 19:13:55.612486 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.612277    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/210a5221-5e8c-4bf5-9bf1-abe8413e3897-host-run-netns\") pod \"multus-mmlqn\" (UID: \"210a5221-5e8c-4bf5-9bf1-abe8413e3897\") " pod="openshift-multus/multus-mmlqn"
Sep 18 19:13:55.613391 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.612309    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2cef0f4e-e44f-4996-8561-da9d261036c7-host-cni-netd\") pod \"ovnkube-node-sksw8\" (UID: \"2cef0f4e-e44f-4996-8561-da9d261036c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:13:55.613391 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.612344    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2cef0f4e-e44f-4996-8561-da9d261036c7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sksw8\" (UID: \"2cef0f4e-e44f-4996-8561-da9d261036c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:13:55.613391 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.612377    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22jgj\" (UniqueName: \"kubernetes.io/projected/05c957d5-da77-47c2-8b9c-e7cdc087120c-kube-api-access-22jgj\") pod \"network-metrics-daemon-ps7w2\" (UID: \"05c957d5-da77-47c2-8b9c-e7cdc087120c\") " pod="openshift-multus/network-metrics-daemon-ps7w2"
Sep 18 19:13:55.613391 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.612410    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/08755e2e-960f-4099-8bc0-1de6acfdd0c2-registration-dir\") pod \"aws-ebs-csi-driver-node-dr7lf\" (UID: \"08755e2e-960f-4099-8bc0-1de6acfdd0c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dr7lf"
Sep 18 19:13:55.613391 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.612445    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7959688f-be33-494c-a8e9-08d33f2c2d3b-cni-binary-copy\") pod \"multus-additional-cni-plugins-rc579\" (UID: \"7959688f-be33-494c-a8e9-08d33f2c2d3b\") " pod="openshift-multus/multus-additional-cni-plugins-rc579"
Sep 18 19:13:55.613391 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.612476    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/210a5221-5e8c-4bf5-9bf1-abe8413e3897-multus-conf-dir\") pod \"multus-mmlqn\" (UID: \"210a5221-5e8c-4bf5-9bf1-abe8413e3897\") " pod="openshift-multus/multus-mmlqn"
Sep 18 19:13:55.613391 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.612503    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/08755e2e-960f-4099-8bc0-1de6acfdd0c2-etc-selinux\") pod \"aws-ebs-csi-driver-node-dr7lf\" (UID: \"08755e2e-960f-4099-8bc0-1de6acfdd0c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dr7lf"
Sep 18 19:13:55.613391 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.612536    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/210a5221-5e8c-4bf5-9bf1-abe8413e3897-multus-socket-dir-parent\") pod \"multus-mmlqn\" (UID: \"210a5221-5e8c-4bf5-9bf1-abe8413e3897\") " pod="openshift-multus/multus-mmlqn"
Sep 18 19:13:55.613391 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.612566    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/22a18a1f-f825-4aff-8e52-18ab52124ced-host-slash\") pod \"iptables-alerter-7jr5l\" (UID: \"22a18a1f-f825-4aff-8e52-18ab52124ced\") " pod="openshift-network-operator/iptables-alerter-7jr5l"
Sep 18 19:13:55.613391 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.612597    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2cef0f4e-e44f-4996-8561-da9d261036c7-etc-openvswitch\") pod \"ovnkube-node-sksw8\" (UID: \"2cef0f4e-e44f-4996-8561-da9d261036c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:13:55.613391 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.612612    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d45a88d4-e763-4a5e-87a3-91c0bb3bbf68-konnectivity-ca\") pod \"konnectivity-agent-226rm\" (UID: \"d45a88d4-e763-4a5e-87a3-91c0bb3bbf68\") " pod="kube-system/konnectivity-agent-226rm"
Sep 18 19:13:55.613391 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.612623    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2cef0f4e-e44f-4996-8561-da9d261036c7-host-cni-bin\") pod \"ovnkube-node-sksw8\" (UID: \"2cef0f4e-e44f-4996-8561-da9d261036c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:13:55.613391 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.612714    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2cef0f4e-e44f-4996-8561-da9d261036c7-host-run-ovn-kubernetes\") pod \"ovnkube-node-sksw8\" (UID: \"2cef0f4e-e44f-4996-8561-da9d261036c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:13:55.613391 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.611545    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7959688f-be33-494c-a8e9-08d33f2c2d3b-system-cni-dir\") pod \"multus-additional-cni-plugins-rc579\" (UID: \"7959688f-be33-494c-a8e9-08d33f2c2d3b\") " pod="openshift-multus/multus-additional-cni-plugins-rc579"
Sep 18 19:13:55.613391 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.611441    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/210a5221-5e8c-4bf5-9bf1-abe8413e3897-host-var-lib-kubelet\") pod \"multus-mmlqn\" (UID: \"210a5221-5e8c-4bf5-9bf1-abe8413e3897\") " pod="openshift-multus/multus-mmlqn"
Sep 18 19:13:55.613391 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.611476    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2cef0f4e-e44f-4996-8561-da9d261036c7-env-overrides\") pod \"ovnkube-node-sksw8\" (UID: \"2cef0f4e-e44f-4996-8561-da9d261036c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:13:55.613391 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.611501    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/210a5221-5e8c-4bf5-9bf1-abe8413e3897-host-run-k8s-cni-cncf-io\") pod \"multus-mmlqn\" (UID: \"210a5221-5e8c-4bf5-9bf1-abe8413e3897\") " pod="openshift-multus/multus-mmlqn"
Sep 18 19:13:55.614226 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.611529    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/210a5221-5e8c-4bf5-9bf1-abe8413e3897-cnibin\") pod \"multus-mmlqn\" (UID: \"210a5221-5e8c-4bf5-9bf1-abe8413e3897\") " pod="openshift-multus/multus-mmlqn"
Sep 18 19:13:55.614226 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.612670    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2cef0f4e-e44f-4996-8561-da9d261036c7-host-cni-bin\") pod \"ovnkube-node-sksw8\" (UID: \"2cef0f4e-e44f-4996-8561-da9d261036c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:13:55.614226 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.611568    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2cef0f4e-e44f-4996-8561-da9d261036c7-host-run-netns\") pod \"ovnkube-node-sksw8\" (UID: \"2cef0f4e-e44f-4996-8561-da9d261036c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:13:55.614226 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.611608    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/210a5221-5e8c-4bf5-9bf1-abe8413e3897-etc-kubernetes\") pod \"multus-mmlqn\" (UID: \"210a5221-5e8c-4bf5-9bf1-abe8413e3897\") " pod="openshift-multus/multus-mmlqn"
Sep 18 19:13:55.614226 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.611639    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/210a5221-5e8c-4bf5-9bf1-abe8413e3897-host-var-lib-cni-bin\") pod \"multus-mmlqn\" (UID: \"210a5221-5e8c-4bf5-9bf1-abe8413e3897\") " pod="openshift-multus/multus-mmlqn"
Sep 18 19:13:55.614226 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.613669    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/08755e2e-960f-4099-8bc0-1de6acfdd0c2-socket-dir\") pod \"aws-ebs-csi-driver-node-dr7lf\" (UID: \"08755e2e-960f-4099-8bc0-1de6acfdd0c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dr7lf"
Sep 18 19:13:55.614590 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.614422    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/210a5221-5e8c-4bf5-9bf1-abe8413e3897-multus-conf-dir\") pod \"multus-mmlqn\" (UID: \"210a5221-5e8c-4bf5-9bf1-abe8413e3897\") " pod="openshift-multus/multus-mmlqn"
Sep 18 19:13:55.614590 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.614433    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d45a88d4-e763-4a5e-87a3-91c0bb3bbf68-agent-certs\") pod \"konnectivity-agent-226rm\" (UID: \"d45a88d4-e763-4a5e-87a3-91c0bb3bbf68\") " pod="kube-system/konnectivity-agent-226rm"
Sep 18 19:13:55.614590 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.614441    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7959688f-be33-494c-a8e9-08d33f2c2d3b-cni-binary-copy\") pod \"multus-additional-cni-plugins-rc579\" (UID: \"7959688f-be33-494c-a8e9-08d33f2c2d3b\") " pod="openshift-multus/multus-additional-cni-plugins-rc579"
Sep 18 19:13:55.614590 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.614474    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/210a5221-5e8c-4bf5-9bf1-abe8413e3897-host-run-netns\") pod \"multus-mmlqn\" (UID: \"210a5221-5e8c-4bf5-9bf1-abe8413e3897\") " pod="openshift-multus/multus-mmlqn"
Sep 18 19:13:55.614590 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.614523    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2cef0f4e-e44f-4996-8561-da9d261036c7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sksw8\" (UID: \"2cef0f4e-e44f-4996-8561-da9d261036c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:13:55.614590 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.614534    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2cef0f4e-e44f-4996-8561-da9d261036c7-host-cni-netd\") pod \"ovnkube-node-sksw8\" (UID: \"2cef0f4e-e44f-4996-8561-da9d261036c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:13:55.614590 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.614553    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/08755e2e-960f-4099-8bc0-1de6acfdd0c2-etc-selinux\") pod \"aws-ebs-csi-driver-node-dr7lf\" (UID: \"08755e2e-960f-4099-8bc0-1de6acfdd0c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dr7lf"
Sep 18 19:13:55.614944 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.614622    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/210a5221-5e8c-4bf5-9bf1-abe8413e3897-multus-socket-dir-parent\") pod \"multus-mmlqn\" (UID: \"210a5221-5e8c-4bf5-9bf1-abe8413e3897\") " pod="openshift-multus/multus-mmlqn"
Sep 18 19:13:55.614944 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.614625    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2cef0f4e-e44f-4996-8561-da9d261036c7-run-ovn\") pod \"ovnkube-node-sksw8\" (UID: \"2cef0f4e-e44f-4996-8561-da9d261036c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:13:55.614944 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.614614    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/08755e2e-960f-4099-8bc0-1de6acfdd0c2-registration-dir\") pod \"aws-ebs-csi-driver-node-dr7lf\" (UID: \"08755e2e-960f-4099-8bc0-1de6acfdd0c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dr7lf"
Sep 18 19:13:55.614944 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.614674    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2cef0f4e-e44f-4996-8561-da9d261036c7-run-openvswitch\") pod \"ovnkube-node-sksw8\" (UID: \"2cef0f4e-e44f-4996-8561-da9d261036c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:13:55.614944 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.614716    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/22a18a1f-f825-4aff-8e52-18ab52124ced-host-slash\") pod \"iptables-alerter-7jr5l\" (UID: \"22a18a1f-f825-4aff-8e52-18ab52124ced\") " pod="openshift-network-operator/iptables-alerter-7jr5l"
Sep 18 19:13:55.614944 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.614749    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2cef0f4e-e44f-4996-8561-da9d261036c7-var-lib-openvswitch\") pod \"ovnkube-node-sksw8\" (UID: \"2cef0f4e-e44f-4996-8561-da9d261036c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:13:55.615271 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.614972    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2cef0f4e-e44f-4996-8561-da9d261036c7-etc-openvswitch\") pod \"ovnkube-node-sksw8\" (UID: \"2cef0f4e-e44f-4996-8561-da9d261036c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:13:55.615271 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.612733    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04602990-ffaa-4d6f-ac9e-bb960fccfbef-etc-kubernetes\") pod \"tuned-cclx5\" (UID: \"04602990-ffaa-4d6f-ac9e-bb960fccfbef\") " pod="openshift-cluster-node-tuning-operator/tuned-cclx5"
Sep 18 19:13:55.615271 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.615118    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/04602990-ffaa-4d6f-ac9e-bb960fccfbef-sys\") pod \"tuned-cclx5\" (UID: \"04602990-ffaa-4d6f-ac9e-bb960fccfbef\") " pod="openshift-cluster-node-tuning-operator/tuned-cclx5"
Sep 18 19:13:55.615271 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.615151    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2cef0f4e-e44f-4996-8561-da9d261036c7-node-log\") pod \"ovnkube-node-sksw8\" (UID: \"2cef0f4e-e44f-4996-8561-da9d261036c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:13:55.615271 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.615188    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/04602990-ffaa-4d6f-ac9e-bb960fccfbef-etc-sysctl-conf\") pod \"tuned-cclx5\" (UID: \"04602990-ffaa-4d6f-ac9e-bb960fccfbef\") " pod="openshift-cluster-node-tuning-operator/tuned-cclx5"
Sep 18 19:13:55.615271 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.615200    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2cef0f4e-e44f-4996-8561-da9d261036c7-ovnkube-script-lib\") pod \"ovnkube-node-sksw8\" (UID: \"2cef0f4e-e44f-4996-8561-da9d261036c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:13:55.615271 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.615225    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2cef0f4e-e44f-4996-8561-da9d261036c7-ovn-node-metrics-cert\") pod \"ovnkube-node-sksw8\" (UID: \"2cef0f4e-e44f-4996-8561-da9d261036c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:13:55.615629 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.615224    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/04602990-ffaa-4d6f-ac9e-bb960fccfbef-run\") pod \"tuned-cclx5\" (UID: \"04602990-ffaa-4d6f-ac9e-bb960fccfbef\") " pod="openshift-cluster-node-tuning-operator/tuned-cclx5"
Sep 18 19:13:55.615629 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.615323    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2cef0f4e-e44f-4996-8561-da9d261036c7-node-log\") pod \"ovnkube-node-sksw8\" (UID: \"2cef0f4e-e44f-4996-8561-da9d261036c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:13:55.615629 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.615370    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/08755e2e-960f-4099-8bc0-1de6acfdd0c2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dr7lf\" (UID: \"08755e2e-960f-4099-8bc0-1de6acfdd0c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dr7lf"
Sep 18 19:13:55.615629 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.615426    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/08755e2e-960f-4099-8bc0-1de6acfdd0c2-device-dir\") pod \"aws-ebs-csi-driver-node-dr7lf\" (UID: \"08755e2e-960f-4099-8bc0-1de6acfdd0c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dr7lf"
Sep 18 19:13:55.615629 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.615473    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f222\" (UniqueName: \"kubernetes.io/projected/812cd3c0-1c61-4d21-8dd7-77d98f0edd2d-kube-api-access-9f222\") pod \"node-ca-7s97w\" (UID: \"812cd3c0-1c61-4d21-8dd7-77d98f0edd2d\") " pod="openshift-image-registry/node-ca-7s97w"
Sep 18 19:13:55.615629 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.615479    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/08755e2e-960f-4099-8bc0-1de6acfdd0c2-device-dir\") pod \"aws-ebs-csi-driver-node-dr7lf\" (UID: \"08755e2e-960f-4099-8bc0-1de6acfdd0c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dr7lf"
Sep 18 19:13:55.615629 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.615506    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/22a18a1f-f825-4aff-8e52-18ab52124ced-iptables-alerter-script\") pod \"iptables-alerter-7jr5l\" (UID: \"22a18a1f-f825-4aff-8e52-18ab52124ced\") " pod="openshift-network-operator/iptables-alerter-7jr5l"
Sep 18 19:13:55.615629 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.615561    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2cef0f4e-e44f-4996-8561-da9d261036c7-host-kubelet\") pod \"ovnkube-node-sksw8\" (UID: \"2cef0f4e-e44f-4996-8561-da9d261036c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:13:55.615629 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.615599    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2cef0f4e-e44f-4996-8561-da9d261036c7-host-slash\") pod \"ovnkube-node-sksw8\" (UID: \"2cef0f4e-e44f-4996-8561-da9d261036c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:13:55.616078 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.615633    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2cef0f4e-e44f-4996-8561-da9d261036c7-run-systemd\") pod \"ovnkube-node-sksw8\" (UID: \"2cef0f4e-e44f-4996-8561-da9d261036c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:13:55.616078 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.615691    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2cef0f4e-e44f-4996-8561-da9d261036c7-log-socket\") pod \"ovnkube-node-sksw8\" (UID: \"2cef0f4e-e44f-4996-8561-da9d261036c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:13:55.616078 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.615720    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed-kubelet-config\") pod \"global-pull-secret-syncer-d9rlj\" (UID: \"7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed\") " pod="kube-system/global-pull-secret-syncer-d9rlj"
Sep 18 19:13:55.616078 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.615759    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed-dbus\") pod \"global-pull-secret-syncer-d9rlj\" (UID: \"7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed\") " pod="kube-system/global-pull-secret-syncer-d9rlj"
Sep 18 19:13:55.616078 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.615794    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twqgk\" (UniqueName: \"kubernetes.io/projected/08755e2e-960f-4099-8bc0-1de6acfdd0c2-kube-api-access-twqgk\") pod \"aws-ebs-csi-driver-node-dr7lf\" (UID: \"08755e2e-960f-4099-8bc0-1de6acfdd0c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dr7lf"
Sep 18 19:13:55.616078 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.615827    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/210a5221-5e8c-4bf5-9bf1-abe8413e3897-host-var-lib-cni-multus\") pod \"multus-mmlqn\" (UID: \"210a5221-5e8c-4bf5-9bf1-abe8413e3897\") " pod="openshift-multus/multus-mmlqn"
Sep 18 19:13:55.616078 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.615862    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/04602990-ffaa-4d6f-ac9e-bb960fccfbef-lib-modules\") pod \"tuned-cclx5\" (UID: \"04602990-ffaa-4d6f-ac9e-bb960fccfbef\") " pod="openshift-cluster-node-tuning-operator/tuned-cclx5"
Sep 18 19:13:55.616078 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.615961    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/210a5221-5e8c-4bf5-9bf1-abe8413e3897-host-var-lib-cni-multus\") pod \"multus-mmlqn\" (UID: \"210a5221-5e8c-4bf5-9bf1-abe8413e3897\") " pod="openshift-multus/multus-mmlqn"
Sep 18 19:13:55.616078 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.616002    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05c957d5-da77-47c2-8b9c-e7cdc087120c-metrics-certs\") pod \"network-metrics-daemon-ps7w2\" (UID: \"05c957d5-da77-47c2-8b9c-e7cdc087120c\") " pod="openshift-multus/network-metrics-daemon-ps7w2"
Sep 18 19:13:55.616573 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.616163    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/08755e2e-960f-4099-8bc0-1de6acfdd0c2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dr7lf\" (UID: \"08755e2e-960f-4099-8bc0-1de6acfdd0c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dr7lf"
Sep 18 19:13:55.616573 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.616237    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed-kubelet-config\") pod \"global-pull-secret-syncer-d9rlj\" (UID: \"7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed\") " pod="kube-system/global-pull-secret-syncer-d9rlj"
Sep 18 19:13:55.616573 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.616283    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2cef0f4e-e44f-4996-8561-da9d261036c7-host-slash\") pod \"ovnkube-node-sksw8\" (UID: \"2cef0f4e-e44f-4996-8561-da9d261036c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:13:55.616573 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.616322    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2cef0f4e-e44f-4996-8561-da9d261036c7-log-socket\") pod \"ovnkube-node-sksw8\" (UID: \"2cef0f4e-e44f-4996-8561-da9d261036c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:13:55.616573 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.616419    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed-dbus\") pod \"global-pull-secret-syncer-d9rlj\" (UID: \"7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed\") " pod="kube-system/global-pull-secret-syncer-d9rlj"
Sep 18 19:13:55.616805 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.616662    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/22a18a1f-f825-4aff-8e52-18ab52124ced-iptables-alerter-script\") pod \"iptables-alerter-7jr5l\" (UID: \"22a18a1f-f825-4aff-8e52-18ab52124ced\") " pod="openshift-network-operator/iptables-alerter-7jr5l"
Sep 18 19:13:55.616805 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.616745    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2cef0f4e-e44f-4996-8561-da9d261036c7-host-kubelet\") pod \"ovnkube-node-sksw8\" (UID: \"2cef0f4e-e44f-4996-8561-da9d261036c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:13:55.616805 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.616801    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2cef0f4e-e44f-4996-8561-da9d261036c7-run-systemd\") pod \"ovnkube-node-sksw8\" (UID: \"2cef0f4e-e44f-4996-8561-da9d261036c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:13:55.621482 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.621403    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-252.ec2.internal" event={"ID":"2d5d27fed1ef1d219785d3f7c796a73d","Type":"ContainerStarted","Data":"e43e345dd89d5691389ef81a9551bbf53dccbdb2cf0906c6bf19a9554a66b745"}
Sep 18 19:13:55.622987 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.622956    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-252.ec2.internal" event={"ID":"0a8d56d698902f1045998309609be899","Type":"ContainerStarted","Data":"38b1e693c8d4ce3125a03fa386258125f05efc823ba27e4c4295918cc55c082e"}
Sep 18 19:13:55.635991 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.635964    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlf4p\" (UniqueName: \"kubernetes.io/projected/210a5221-5e8c-4bf5-9bf1-abe8413e3897-kube-api-access-rlf4p\") pod \"multus-mmlqn\" (UID: \"210a5221-5e8c-4bf5-9bf1-abe8413e3897\") " pod="openshift-multus/multus-mmlqn"
Sep 18 19:13:55.638597 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.638539    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69mnl\" (UniqueName: \"kubernetes.io/projected/22a18a1f-f825-4aff-8e52-18ab52124ced-kube-api-access-69mnl\") pod \"iptables-alerter-7jr5l\" (UID: \"22a18a1f-f825-4aff-8e52-18ab52124ced\") " pod="openshift-network-operator/iptables-alerter-7jr5l"
Sep 18 19:13:55.639194 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.639144    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f222\" (UniqueName: \"kubernetes.io/projected/812cd3c0-1c61-4d21-8dd7-77d98f0edd2d-kube-api-access-9f222\") pod \"node-ca-7s97w\" (UID: \"812cd3c0-1c61-4d21-8dd7-77d98f0edd2d\") " pod="openshift-image-registry/node-ca-7s97w"
Sep 18 19:13:55.641840 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.641796    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p99zh\" (UniqueName: \"kubernetes.io/projected/2cef0f4e-e44f-4996-8561-da9d261036c7-kube-api-access-p99zh\") pod \"ovnkube-node-sksw8\" (UID: \"2cef0f4e-e44f-4996-8561-da9d261036c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:13:55.644487 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.644446    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bblcj\" (UniqueName: \"kubernetes.io/projected/7959688f-be33-494c-a8e9-08d33f2c2d3b-kube-api-access-bblcj\") pod \"multus-additional-cni-plugins-rc579\" (UID: \"7959688f-be33-494c-a8e9-08d33f2c2d3b\") " pod="openshift-multus/multus-additional-cni-plugins-rc579"
Sep 18 19:13:55.649422 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.649391    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twqgk\" (UniqueName: \"kubernetes.io/projected/08755e2e-960f-4099-8bc0-1de6acfdd0c2-kube-api-access-twqgk\") pod \"aws-ebs-csi-driver-node-dr7lf\" (UID: \"08755e2e-960f-4099-8bc0-1de6acfdd0c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dr7lf"
Sep 18 19:13:55.716624 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.716556    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/04602990-ffaa-4d6f-ac9e-bb960fccfbef-lib-modules\") pod \"tuned-cclx5\" (UID: \"04602990-ffaa-4d6f-ac9e-bb960fccfbef\") " pod="openshift-cluster-node-tuning-operator/tuned-cclx5"
Sep 18 19:13:55.716624 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.716602    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05c957d5-da77-47c2-8b9c-e7cdc087120c-metrics-certs\") pod \"network-metrics-daemon-ps7w2\" (UID: \"05c957d5-da77-47c2-8b9c-e7cdc087120c\") " pod="openshift-multus/network-metrics-daemon-ps7w2"
Sep 18 19:13:55.716777 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.716627    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/04602990-ffaa-4d6f-ac9e-bb960fccfbef-tmp\") pod \"tuned-cclx5\" (UID: \"04602990-ffaa-4d6f-ac9e-bb960fccfbef\") " pod="openshift-cluster-node-tuning-operator/tuned-cclx5"
Sep 18 19:13:55.716777 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.716649    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b7hs\" (UniqueName: \"kubernetes.io/projected/7082df75-a3d1-4eba-90e5-7a02a6aef94b-kube-api-access-6b7hs\") pod \"network-check-target-42kxn\" (UID: \"7082df75-a3d1-4eba-90e5-7a02a6aef94b\") " pod="openshift-network-diagnostics/network-check-target-42kxn"
Sep 18 19:13:55.716777 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.716692    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/04602990-ffaa-4d6f-ac9e-bb960fccfbef-etc-modprobe-d\") pod \"tuned-cclx5\" (UID: \"04602990-ffaa-4d6f-ac9e-bb960fccfbef\") " pod="openshift-cluster-node-tuning-operator/tuned-cclx5"
Sep 18 19:13:55.716777 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.716701    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/04602990-ffaa-4d6f-ac9e-bb960fccfbef-lib-modules\") pod \"tuned-cclx5\" (UID: \"04602990-ffaa-4d6f-ac9e-bb960fccfbef\") " pod="openshift-cluster-node-tuning-operator/tuned-cclx5"
Sep 18 19:13:55.716777 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.716717    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt5x5\" (UniqueName: \"kubernetes.io/projected/851825cd-f968-471c-87c0-b7d2423c9c99-kube-api-access-zt5x5\") pod \"node-resolver-qbs4d\" (UID: \"851825cd-f968-471c-87c0-b7d2423c9c99\") " pod="openshift-dns/node-resolver-qbs4d"
Sep 18 19:13:55.716777 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.716745    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/851825cd-f968-471c-87c0-b7d2423c9c99-hosts-file\") pod \"node-resolver-qbs4d\" (UID: \"851825cd-f968-471c-87c0-b7d2423c9c99\") " pod="openshift-dns/node-resolver-qbs4d"
Sep 18 19:13:55.716777 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:55.716752    2532 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:13:55.716777 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.716772    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/04602990-ffaa-4d6f-ac9e-bb960fccfbef-host\") pod \"tuned-cclx5\" (UID: \"04602990-ffaa-4d6f-ac9e-bb960fccfbef\") " pod="openshift-cluster-node-tuning-operator/tuned-cclx5"
Sep 18 19:13:55.717142 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.716798    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/04602990-ffaa-4d6f-ac9e-bb960fccfbef-etc-tuned\") pod \"tuned-cclx5\" (UID: \"04602990-ffaa-4d6f-ac9e-bb960fccfbef\") " pod="openshift-cluster-node-tuning-operator/tuned-cclx5"
Sep 18 19:13:55.717142 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:55.716819    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05c957d5-da77-47c2-8b9c-e7cdc087120c-metrics-certs podName:05c957d5-da77-47c2-8b9c-e7cdc087120c nodeName:}" failed. No retries permitted until 2026-09-18 19:13:56.21679929 +0000 UTC m=+3.137303127 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05c957d5-da77-47c2-8b9c-e7cdc087120c-metrics-certs") pod "network-metrics-daemon-ps7w2" (UID: "05c957d5-da77-47c2-8b9c-e7cdc087120c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:13:55.717142 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.716856    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvm7x\" (UniqueName: \"kubernetes.io/projected/04602990-ffaa-4d6f-ac9e-bb960fccfbef-kube-api-access-tvm7x\") pod \"tuned-cclx5\" (UID: \"04602990-ffaa-4d6f-ac9e-bb960fccfbef\") " pod="openshift-cluster-node-tuning-operator/tuned-cclx5"
Sep 18 19:13:55.717142 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.716890    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/04602990-ffaa-4d6f-ac9e-bb960fccfbef-etc-sysctl-d\") pod \"tuned-cclx5\" (UID: \"04602990-ffaa-4d6f-ac9e-bb960fccfbef\") " pod="openshift-cluster-node-tuning-operator/tuned-cclx5"
Sep 18 19:13:55.717142 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.716915    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/04602990-ffaa-4d6f-ac9e-bb960fccfbef-var-lib-kubelet\") pod \"tuned-cclx5\" (UID: \"04602990-ffaa-4d6f-ac9e-bb960fccfbef\") " pod="openshift-cluster-node-tuning-operator/tuned-cclx5"
Sep 18 19:13:55.717142 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.716951    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/851825cd-f968-471c-87c0-b7d2423c9c99-tmp-dir\") pod \"node-resolver-qbs4d\" (UID: \"851825cd-f968-471c-87c0-b7d2423c9c99\") " pod="openshift-dns/node-resolver-qbs4d"
Sep 18 19:13:55.717142 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.716981    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/04602990-ffaa-4d6f-ac9e-bb960fccfbef-etc-sysconfig\") pod \"tuned-cclx5\" (UID: \"04602990-ffaa-4d6f-ac9e-bb960fccfbef\") " pod="openshift-cluster-node-tuning-operator/tuned-cclx5"
Sep 18 19:13:55.717142 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.717014    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/04602990-ffaa-4d6f-ac9e-bb960fccfbef-etc-systemd\") pod \"tuned-cclx5\" (UID: \"04602990-ffaa-4d6f-ac9e-bb960fccfbef\") " pod="openshift-cluster-node-tuning-operator/tuned-cclx5"
Sep 18 19:13:55.717142 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.717062    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22jgj\" (UniqueName: \"kubernetes.io/projected/05c957d5-da77-47c2-8b9c-e7cdc087120c-kube-api-access-22jgj\") pod \"network-metrics-daemon-ps7w2\" (UID: \"05c957d5-da77-47c2-8b9c-e7cdc087120c\") " pod="openshift-multus/network-metrics-daemon-ps7w2"
Sep 18 19:13:55.717142 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.717115    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04602990-ffaa-4d6f-ac9e-bb960fccfbef-etc-kubernetes\") pod \"tuned-cclx5\" (UID: \"04602990-ffaa-4d6f-ac9e-bb960fccfbef\") " pod="openshift-cluster-node-tuning-operator/tuned-cclx5"
Sep 18 19:13:55.717602 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.717147    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/04602990-ffaa-4d6f-ac9e-bb960fccfbef-sys\") pod \"tuned-cclx5\" (UID: \"04602990-ffaa-4d6f-ac9e-bb960fccfbef\") " pod="openshift-cluster-node-tuning-operator/tuned-cclx5"
Sep 18 19:13:55.717602 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.717159    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/04602990-ffaa-4d6f-ac9e-bb960fccfbef-var-lib-kubelet\") pod \"tuned-cclx5\" (UID: \"04602990-ffaa-4d6f-ac9e-bb960fccfbef\") " pod="openshift-cluster-node-tuning-operator/tuned-cclx5"
Sep 18 19:13:55.717602 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.717173    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/04602990-ffaa-4d6f-ac9e-bb960fccfbef-etc-sysconfig\") pod \"tuned-cclx5\" (UID: \"04602990-ffaa-4d6f-ac9e-bb960fccfbef\") " pod="openshift-cluster-node-tuning-operator/tuned-cclx5"
Sep 18 19:13:55.717602 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.717202    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/04602990-ffaa-4d6f-ac9e-bb960fccfbef-etc-sysctl-d\") pod \"tuned-cclx5\" (UID: \"04602990-ffaa-4d6f-ac9e-bb960fccfbef\") " pod="openshift-cluster-node-tuning-operator/tuned-cclx5"
Sep 18 19:13:55.717602 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.717221    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/04602990-ffaa-4d6f-ac9e-bb960fccfbef-etc-modprobe-d\") pod \"tuned-cclx5\" (UID: \"04602990-ffaa-4d6f-ac9e-bb960fccfbef\") " pod="openshift-cluster-node-tuning-operator/tuned-cclx5"
Sep 18 19:13:55.717602 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.717231    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/04602990-ffaa-4d6f-ac9e-bb960fccfbef-sys\") pod \"tuned-cclx5\" (UID: \"04602990-ffaa-4d6f-ac9e-bb960fccfbef\") " pod="openshift-cluster-node-tuning-operator/tuned-cclx5"
Sep 18 19:13:55.717602 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.717226    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04602990-ffaa-4d6f-ac9e-bb960fccfbef-etc-kubernetes\") pod \"tuned-cclx5\" (UID: \"04602990-ffaa-4d6f-ac9e-bb960fccfbef\") " pod="openshift-cluster-node-tuning-operator/tuned-cclx5"
Sep 18 19:13:55.717602 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.717286    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/851825cd-f968-471c-87c0-b7d2423c9c99-hosts-file\") pod \"node-resolver-qbs4d\" (UID: \"851825cd-f968-471c-87c0-b7d2423c9c99\") " pod="openshift-dns/node-resolver-qbs4d"
Sep 18 19:13:55.717602 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.717294    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/04602990-ffaa-4d6f-ac9e-bb960fccfbef-host\") pod \"tuned-cclx5\" (UID: \"04602990-ffaa-4d6f-ac9e-bb960fccfbef\") " pod="openshift-cluster-node-tuning-operator/tuned-cclx5"
Sep 18 19:13:55.717602 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.717287    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/04602990-ffaa-4d6f-ac9e-bb960fccfbef-etc-systemd\") pod \"tuned-cclx5\" (UID: \"04602990-ffaa-4d6f-ac9e-bb960fccfbef\") " pod="openshift-cluster-node-tuning-operator/tuned-cclx5"
Sep 18 19:13:55.717602 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.717309    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/04602990-ffaa-4d6f-ac9e-bb960fccfbef-etc-sysctl-conf\") pod \"tuned-cclx5\" (UID: \"04602990-ffaa-4d6f-ac9e-bb960fccfbef\") " pod="openshift-cluster-node-tuning-operator/tuned-cclx5"
Sep 18 19:13:55.717602 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.717339    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/04602990-ffaa-4d6f-ac9e-bb960fccfbef-run\") pod \"tuned-cclx5\" (UID: \"04602990-ffaa-4d6f-ac9e-bb960fccfbef\") " pod="openshift-cluster-node-tuning-operator/tuned-cclx5"
Sep 18 19:13:55.717602 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.717407    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/04602990-ffaa-4d6f-ac9e-bb960fccfbef-run\") pod \"tuned-cclx5\" (UID: \"04602990-ffaa-4d6f-ac9e-bb960fccfbef\") " pod="openshift-cluster-node-tuning-operator/tuned-cclx5"
Sep 18 19:13:55.717602 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.717505    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/04602990-ffaa-4d6f-ac9e-bb960fccfbef-etc-sysctl-conf\") pod \"tuned-cclx5\" (UID: \"04602990-ffaa-4d6f-ac9e-bb960fccfbef\") " pod="openshift-cluster-node-tuning-operator/tuned-cclx5"
Sep 18 19:13:55.717602 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.717531    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/851825cd-f968-471c-87c0-b7d2423c9c99-tmp-dir\") pod \"node-resolver-qbs4d\" (UID: \"851825cd-f968-471c-87c0-b7d2423c9c99\") " pod="openshift-dns/node-resolver-qbs4d"
Sep 18 19:13:55.720131 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.720075    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/04602990-ffaa-4d6f-ac9e-bb960fccfbef-etc-tuned\") pod \"tuned-cclx5\" (UID: \"04602990-ffaa-4d6f-ac9e-bb960fccfbef\") " pod="openshift-cluster-node-tuning-operator/tuned-cclx5"
Sep 18 19:13:55.721151 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.721108    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/04602990-ffaa-4d6f-ac9e-bb960fccfbef-tmp\") pod \"tuned-cclx5\" (UID: \"04602990-ffaa-4d6f-ac9e-bb960fccfbef\") " pod="openshift-cluster-node-tuning-operator/tuned-cclx5"
Sep 18 19:13:55.730112 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:55.729886    2532 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:13:55.730112 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:55.729911    2532 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:13:55.730112 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:55.729925    2532 projected.go:196] Error preparing data for projected volume kube-api-access-6b7hs for pod openshift-network-diagnostics/network-check-target-42kxn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:13:55.730112 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:55.729985    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7082df75-a3d1-4eba-90e5-7a02a6aef94b-kube-api-access-6b7hs podName:7082df75-a3d1-4eba-90e5-7a02a6aef94b nodeName:}" failed. No retries permitted until 2026-09-18 19:13:56.229967391 +0000 UTC m=+3.150471234 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6b7hs" (UniqueName: "kubernetes.io/projected/7082df75-a3d1-4eba-90e5-7a02a6aef94b-kube-api-access-6b7hs") pod "network-check-target-42kxn" (UID: "7082df75-a3d1-4eba-90e5-7a02a6aef94b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:13:55.745772 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.745748    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvm7x\" (UniqueName: \"kubernetes.io/projected/04602990-ffaa-4d6f-ac9e-bb960fccfbef-kube-api-access-tvm7x\") pod \"tuned-cclx5\" (UID: \"04602990-ffaa-4d6f-ac9e-bb960fccfbef\") " pod="openshift-cluster-node-tuning-operator/tuned-cclx5"
Sep 18 19:13:55.746697 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.746677    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22jgj\" (UniqueName: \"kubernetes.io/projected/05c957d5-da77-47c2-8b9c-e7cdc087120c-kube-api-access-22jgj\") pod \"network-metrics-daemon-ps7w2\" (UID: \"05c957d5-da77-47c2-8b9c-e7cdc087120c\") " pod="openshift-multus/network-metrics-daemon-ps7w2"
Sep 18 19:13:55.750121 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.750039    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt5x5\" (UniqueName: \"kubernetes.io/projected/851825cd-f968-471c-87c0-b7d2423c9c99-kube-api-access-zt5x5\") pod \"node-resolver-qbs4d\" (UID: \"851825cd-f968-471c-87c0-b7d2423c9c99\") " pod="openshift-dns/node-resolver-qbs4d"
Sep 18 19:13:55.786386 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.786187    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mmlqn"
Sep 18 19:13:55.796563 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.796542    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7s97w"
Sep 18 19:13:55.807388 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.807367    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7jr5l"
Sep 18 19:13:55.821774 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.821751    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-226rm"
Sep 18 19:13:55.830697 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.830676    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dr7lf"
Sep 18 19:13:55.839636 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.839589    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:13:55.847454 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.847435    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rc579"
Sep 18 19:13:55.854340 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.854319    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cclx5"
Sep 18 19:13:55.861270 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:55.861227    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qbs4d"
Sep 18 19:13:56.056501 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:56.056464    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7959688f_be33_494c_a8e9_08d33f2c2d3b.slice/crio-e643eaf84966fa8319a3778ae311b4fa999dc6d73881a61f2865dd4e9c89ac25 WatchSource:0}: Error finding container e643eaf84966fa8319a3778ae311b4fa999dc6d73881a61f2865dd4e9c89ac25: Status 404 returned error can't find the container with id e643eaf84966fa8319a3778ae311b4fa999dc6d73881a61f2865dd4e9c89ac25
Sep 18 19:13:56.057947 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:56.057912    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod851825cd_f968_471c_87c0_b7d2423c9c99.slice/crio-1b5bf6a1bce0303e46dc702177bd1efccd99c3f69c212874d1758a63a47ac217 WatchSource:0}: Error finding container 1b5bf6a1bce0303e46dc702177bd1efccd99c3f69c212874d1758a63a47ac217: Status 404 returned error can't find the container with id 1b5bf6a1bce0303e46dc702177bd1efccd99c3f69c212874d1758a63a47ac217
Sep 18 19:13:56.060264 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:56.060175    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08755e2e_960f_4099_8bc0_1de6acfdd0c2.slice/crio-526b2cb3fc487ecaefc9de23a575b8cf2ad50d6d79add2f869bcbd8a5944afff WatchSource:0}: Error finding container 526b2cb3fc487ecaefc9de23a575b8cf2ad50d6d79add2f869bcbd8a5944afff: Status 404 returned error can't find the container with id 526b2cb3fc487ecaefc9de23a575b8cf2ad50d6d79add2f869bcbd8a5944afff
Sep 18 19:13:56.065799 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:56.065772    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22a18a1f_f825_4aff_8e52_18ab52124ced.slice/crio-7e2d2b70575354346067bd44136dae1fe6a1a1387c3d3865175099cc4c73296d WatchSource:0}: Error finding container 7e2d2b70575354346067bd44136dae1fe6a1a1387c3d3865175099cc4c73296d: Status 404 returned error can't find the container with id 7e2d2b70575354346067bd44136dae1fe6a1a1387c3d3865175099cc4c73296d
Sep 18 19:13:56.087314 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:56.087289    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod812cd3c0_1c61_4d21_8dd7_77d98f0edd2d.slice/crio-aebe0a3e610e59374e33812fd7534fb3124f9dba89e57563cdf6778b913d2221 WatchSource:0}: Error finding container aebe0a3e610e59374e33812fd7534fb3124f9dba89e57563cdf6778b913d2221: Status 404 returned error can't find the container with id aebe0a3e610e59374e33812fd7534fb3124f9dba89e57563cdf6778b913d2221
Sep 18 19:13:56.088237 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:56.088212    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cef0f4e_e44f_4996_8561_da9d261036c7.slice/crio-f2c36c789156425e939bbd17cf04501b2e43effbd4d9cd39b2ba55f400b25678 WatchSource:0}: Error finding container f2c36c789156425e939bbd17cf04501b2e43effbd4d9cd39b2ba55f400b25678: Status 404 returned error can't find the container with id f2c36c789156425e939bbd17cf04501b2e43effbd4d9cd39b2ba55f400b25678
Sep 18 19:13:56.088982 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:13:56.088965    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04602990_ffaa_4d6f_ac9e_bb960fccfbef.slice/crio-bd2af0627f63519628b11a99e9a9bdd95d6616034d036ea90aa66cab95f0ba08 WatchSource:0}: Error finding container bd2af0627f63519628b11a99e9a9bdd95d6616034d036ea90aa66cab95f0ba08: Status 404 returned error can't find the container with id bd2af0627f63519628b11a99e9a9bdd95d6616034d036ea90aa66cab95f0ba08
Sep 18 19:13:56.119601 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:56.119572    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed-original-pull-secret\") pod \"global-pull-secret-syncer-d9rlj\" (UID: \"7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed\") " pod="kube-system/global-pull-secret-syncer-d9rlj"
Sep 18 19:13:56.119722 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:56.119705    2532 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:13:56.119799 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:56.119775    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed-original-pull-secret podName:7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed nodeName:}" failed. No retries permitted until 2026-09-18 19:13:57.119754994 +0000 UTC m=+4.040258849 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed-original-pull-secret") pod "global-pull-secret-syncer-d9rlj" (UID: "7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:13:56.220630 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:56.220591    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05c957d5-da77-47c2-8b9c-e7cdc087120c-metrics-certs\") pod \"network-metrics-daemon-ps7w2\" (UID: \"05c957d5-da77-47c2-8b9c-e7cdc087120c\") " pod="openshift-multus/network-metrics-daemon-ps7w2"
Sep 18 19:13:56.220799 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:56.220777    2532 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:13:56.220866 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:56.220854    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05c957d5-da77-47c2-8b9c-e7cdc087120c-metrics-certs podName:05c957d5-da77-47c2-8b9c-e7cdc087120c nodeName:}" failed. No retries permitted until 2026-09-18 19:13:57.220832919 +0000 UTC m=+4.141336761 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05c957d5-da77-47c2-8b9c-e7cdc087120c-metrics-certs") pod "network-metrics-daemon-ps7w2" (UID: "05c957d5-da77-47c2-8b9c-e7cdc087120c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:13:56.321665 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:56.321565    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b7hs\" (UniqueName: \"kubernetes.io/projected/7082df75-a3d1-4eba-90e5-7a02a6aef94b-kube-api-access-6b7hs\") pod \"network-check-target-42kxn\" (UID: \"7082df75-a3d1-4eba-90e5-7a02a6aef94b\") " pod="openshift-network-diagnostics/network-check-target-42kxn"
Sep 18 19:13:56.321806 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:56.321752    2532 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:13:56.321806 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:56.321773    2532 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:13:56.321806 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:56.321786    2532 projected.go:196] Error preparing data for projected volume kube-api-access-6b7hs for pod openshift-network-diagnostics/network-check-target-42kxn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:13:56.321956 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:56.321842    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7082df75-a3d1-4eba-90e5-7a02a6aef94b-kube-api-access-6b7hs podName:7082df75-a3d1-4eba-90e5-7a02a6aef94b nodeName:}" failed. No retries permitted until 2026-09-18 19:13:57.321827952 +0000 UTC m=+4.242331794 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6b7hs" (UniqueName: "kubernetes.io/projected/7082df75-a3d1-4eba-90e5-7a02a6aef94b-kube-api-access-6b7hs") pod "network-check-target-42kxn" (UID: "7082df75-a3d1-4eba-90e5-7a02a6aef94b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:13:56.585269 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:56.585102    2532 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 19:08:55 +0000 UTC" deadline="2028-02-20 20:28:09.278409884 +0000 UTC"
Sep 18 19:13:56.585269 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:56.585219    2532 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12481h14m12.693196186s"
Sep 18 19:13:56.616562 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:56.616527    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ps7w2"
Sep 18 19:13:56.616717 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:56.616674    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ps7w2" podUID="05c957d5-da77-47c2-8b9c-e7cdc087120c"
Sep 18 19:13:56.626607 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:56.626578    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sksw8" event={"ID":"2cef0f4e-e44f-4996-8561-da9d261036c7","Type":"ContainerStarted","Data":"f2c36c789156425e939bbd17cf04501b2e43effbd4d9cd39b2ba55f400b25678"}
Sep 18 19:13:56.628242 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:56.628211    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7jr5l" event={"ID":"22a18a1f-f825-4aff-8e52-18ab52124ced","Type":"ContainerStarted","Data":"7e2d2b70575354346067bd44136dae1fe6a1a1387c3d3865175099cc4c73296d"}
Sep 18 19:13:56.629602 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:56.629552    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qbs4d" event={"ID":"851825cd-f968-471c-87c0-b7d2423c9c99","Type":"ContainerStarted","Data":"1b5bf6a1bce0303e46dc702177bd1efccd99c3f69c212874d1758a63a47ac217"}
Sep 18 19:13:56.631209 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:56.631180    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cclx5" event={"ID":"04602990-ffaa-4d6f-ac9e-bb960fccfbef","Type":"ContainerStarted","Data":"bd2af0627f63519628b11a99e9a9bdd95d6616034d036ea90aa66cab95f0ba08"}
Sep 18 19:13:56.632691 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:56.632660    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7s97w" event={"ID":"812cd3c0-1c61-4d21-8dd7-77d98f0edd2d","Type":"ContainerStarted","Data":"aebe0a3e610e59374e33812fd7534fb3124f9dba89e57563cdf6778b913d2221"}
Sep 18 19:13:56.634042 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:56.634002    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mmlqn" event={"ID":"210a5221-5e8c-4bf5-9bf1-abe8413e3897","Type":"ContainerStarted","Data":"791b6511205f1352f1b8f5041dc6aa87c03ae47e0f3d7bee5d2bd249ff2176aa"}
Sep 18 19:13:56.635395 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:56.635364    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-226rm" event={"ID":"d45a88d4-e763-4a5e-87a3-91c0bb3bbf68","Type":"ContainerStarted","Data":"67fe6a71c156742b1084e310fc4a99ff3b7a5631fbbdcf979eaa1db2c198fe59"}
Sep 18 19:13:56.636706 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:56.636658    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dr7lf" event={"ID":"08755e2e-960f-4099-8bc0-1de6acfdd0c2","Type":"ContainerStarted","Data":"526b2cb3fc487ecaefc9de23a575b8cf2ad50d6d79add2f869bcbd8a5944afff"}
Sep 18 19:13:56.638026 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:56.638001    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rc579" event={"ID":"7959688f-be33-494c-a8e9-08d33f2c2d3b","Type":"ContainerStarted","Data":"e643eaf84966fa8319a3778ae311b4fa999dc6d73881a61f2865dd4e9c89ac25"}
Sep 18 19:13:57.130050 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:57.129346    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed-original-pull-secret\") pod \"global-pull-secret-syncer-d9rlj\" (UID: \"7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed\") " pod="kube-system/global-pull-secret-syncer-d9rlj"
Sep 18 19:13:57.130050 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:57.129574    2532 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:13:57.130050 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:57.129639    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed-original-pull-secret podName:7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed nodeName:}" failed. No retries permitted until 2026-09-18 19:13:59.129620348 +0000 UTC m=+6.050124191 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed-original-pull-secret") pod "global-pull-secret-syncer-d9rlj" (UID: "7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:13:57.230985 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:57.230401    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05c957d5-da77-47c2-8b9c-e7cdc087120c-metrics-certs\") pod \"network-metrics-daemon-ps7w2\" (UID: \"05c957d5-da77-47c2-8b9c-e7cdc087120c\") " pod="openshift-multus/network-metrics-daemon-ps7w2"
Sep 18 19:13:57.230985 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:57.230577    2532 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:13:57.230985 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:57.230641    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05c957d5-da77-47c2-8b9c-e7cdc087120c-metrics-certs podName:05c957d5-da77-47c2-8b9c-e7cdc087120c nodeName:}" failed. No retries permitted until 2026-09-18 19:13:59.230622798 +0000 UTC m=+6.151126640 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05c957d5-da77-47c2-8b9c-e7cdc087120c-metrics-certs") pod "network-metrics-daemon-ps7w2" (UID: "05c957d5-da77-47c2-8b9c-e7cdc087120c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:13:57.331937 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:57.331286    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b7hs\" (UniqueName: \"kubernetes.io/projected/7082df75-a3d1-4eba-90e5-7a02a6aef94b-kube-api-access-6b7hs\") pod \"network-check-target-42kxn\" (UID: \"7082df75-a3d1-4eba-90e5-7a02a6aef94b\") " pod="openshift-network-diagnostics/network-check-target-42kxn"
Sep 18 19:13:57.331937 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:57.331472    2532 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:13:57.331937 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:57.331494    2532 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:13:57.331937 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:57.331508    2532 projected.go:196] Error preparing data for projected volume kube-api-access-6b7hs for pod openshift-network-diagnostics/network-check-target-42kxn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:13:57.331937 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:57.331566    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7082df75-a3d1-4eba-90e5-7a02a6aef94b-kube-api-access-6b7hs podName:7082df75-a3d1-4eba-90e5-7a02a6aef94b nodeName:}" failed. No retries permitted until 2026-09-18 19:13:59.331546782 +0000 UTC m=+6.252050634 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6b7hs" (UniqueName: "kubernetes.io/projected/7082df75-a3d1-4eba-90e5-7a02a6aef94b-kube-api-access-6b7hs") pod "network-check-target-42kxn" (UID: "7082df75-a3d1-4eba-90e5-7a02a6aef94b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:13:57.585838 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:57.585753    2532 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 19:08:55 +0000 UTC" deadline="2028-05-01 00:32:27.377888158 +0000 UTC"
Sep 18 19:13:57.585838 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:57.585797    2532 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14165h18m29.7920978s"
Sep 18 19:13:57.616821 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:57.616393    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42kxn"
Sep 18 19:13:57.616821 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:57.616393    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d9rlj"
Sep 18 19:13:57.616821 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:57.616505    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42kxn" podUID="7082df75-a3d1-4eba-90e5-7a02a6aef94b"
Sep 18 19:13:57.616821 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:57.616561    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d9rlj" podUID="7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed"
Sep 18 19:13:57.659498 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:57.659295    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-252.ec2.internal" event={"ID":"2d5d27fed1ef1d219785d3f7c796a73d","Type":"ContainerStarted","Data":"e880084e75b4c2817bec6298148bed48e2b6a156aa78bba4deb7ce3546183a9c"}
Sep 18 19:13:58.617296 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:58.616641    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ps7w2"
Sep 18 19:13:58.617296 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:58.616793    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ps7w2" podUID="05c957d5-da77-47c2-8b9c-e7cdc087120c"
Sep 18 19:13:58.682435 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:58.682397    2532 generic.go:356] "Generic (PLEG): container finished" podID="0a8d56d698902f1045998309609be899" containerID="51440fedc35b91d172cb61f149a2af986419114413aae37859b3ad464d95253e" exitCode=0
Sep 18 19:13:58.682613 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:58.682541    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-252.ec2.internal" event={"ID":"0a8d56d698902f1045998309609be899","Type":"ContainerDied","Data":"51440fedc35b91d172cb61f149a2af986419114413aae37859b3ad464d95253e"}
Sep 18 19:13:58.714467 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:58.714402    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-252.ec2.internal" podStartSLOduration=3.714378602 podStartE2EDuration="3.714378602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:13:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:13:57.685954743 +0000 UTC m=+4.606458604" watchObservedRunningTime="2026-09-18 19:13:58.714378602 +0000 UTC m=+5.634882463"
Sep 18 19:13:59.148153 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:59.147699    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed-original-pull-secret\") pod \"global-pull-secret-syncer-d9rlj\" (UID: \"7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed\") " pod="kube-system/global-pull-secret-syncer-d9rlj"
Sep 18 19:13:59.148153 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:59.148008    2532 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:13:59.148153 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:59.148089    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed-original-pull-secret podName:7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed nodeName:}" failed. No retries permitted until 2026-09-18 19:14:03.148068756 +0000 UTC m=+10.068572607 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed-original-pull-secret") pod "global-pull-secret-syncer-d9rlj" (UID: "7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:13:59.249295 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:59.249255    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05c957d5-da77-47c2-8b9c-e7cdc087120c-metrics-certs\") pod \"network-metrics-daemon-ps7w2\" (UID: \"05c957d5-da77-47c2-8b9c-e7cdc087120c\") " pod="openshift-multus/network-metrics-daemon-ps7w2"
Sep 18 19:13:59.249484 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:59.249468    2532 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:13:59.249559 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:59.249536    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05c957d5-da77-47c2-8b9c-e7cdc087120c-metrics-certs podName:05c957d5-da77-47c2-8b9c-e7cdc087120c nodeName:}" failed. No retries permitted until 2026-09-18 19:14:03.249515431 +0000 UTC m=+10.170019283 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05c957d5-da77-47c2-8b9c-e7cdc087120c-metrics-certs") pod "network-metrics-daemon-ps7w2" (UID: "05c957d5-da77-47c2-8b9c-e7cdc087120c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:13:59.349692 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:59.349655    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b7hs\" (UniqueName: \"kubernetes.io/projected/7082df75-a3d1-4eba-90e5-7a02a6aef94b-kube-api-access-6b7hs\") pod \"network-check-target-42kxn\" (UID: \"7082df75-a3d1-4eba-90e5-7a02a6aef94b\") " pod="openshift-network-diagnostics/network-check-target-42kxn"
Sep 18 19:13:59.349881 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:59.349866    2532 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:13:59.349942 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:59.349889    2532 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:13:59.349942 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:59.349901    2532 projected.go:196] Error preparing data for projected volume kube-api-access-6b7hs for pod openshift-network-diagnostics/network-check-target-42kxn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:13:59.350050 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:59.349954    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7082df75-a3d1-4eba-90e5-7a02a6aef94b-kube-api-access-6b7hs podName:7082df75-a3d1-4eba-90e5-7a02a6aef94b nodeName:}" failed. No retries permitted until 2026-09-18 19:14:03.349938267 +0000 UTC m=+10.270442107 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6b7hs" (UniqueName: "kubernetes.io/projected/7082df75-a3d1-4eba-90e5-7a02a6aef94b-kube-api-access-6b7hs") pod "network-check-target-42kxn" (UID: "7082df75-a3d1-4eba-90e5-7a02a6aef94b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:13:59.617232 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:59.617164    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d9rlj"
Sep 18 19:13:59.617389 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:59.617297    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d9rlj" podUID="7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed"
Sep 18 19:13:59.617752 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:59.617176    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42kxn"
Sep 18 19:13:59.617752 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:13:59.617685    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42kxn" podUID="7082df75-a3d1-4eba-90e5-7a02a6aef94b"
Sep 18 19:13:59.689971 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:59.689932    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-252.ec2.internal" event={"ID":"0a8d56d698902f1045998309609be899","Type":"ContainerStarted","Data":"60db49c14908d99704e950e8666712646e324ea00ac061503b54ade969156bfe"}
Sep 18 19:13:59.715179 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:13:59.715117    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-252.ec2.internal" podStartSLOduration=4.7150965540000005 podStartE2EDuration="4.715096554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:13:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:13:59.714625042 +0000 UTC m=+6.635128911" watchObservedRunningTime="2026-09-18 19:13:59.715096554 +0000 UTC m=+6.635600412"
Sep 18 19:14:00.615697 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:00.615662    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ps7w2"
Sep 18 19:14:00.615862 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:00.615813    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ps7w2" podUID="05c957d5-da77-47c2-8b9c-e7cdc087120c"
Sep 18 19:14:01.616321 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:01.616289    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d9rlj"
Sep 18 19:14:01.616729 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:01.616422    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d9rlj" podUID="7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed"
Sep 18 19:14:01.616806 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:01.616787    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42kxn"
Sep 18 19:14:01.616900 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:01.616881    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42kxn" podUID="7082df75-a3d1-4eba-90e5-7a02a6aef94b"
Sep 18 19:14:02.616699 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:02.616205    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ps7w2"
Sep 18 19:14:02.616699 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:02.616352    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ps7w2" podUID="05c957d5-da77-47c2-8b9c-e7cdc087120c"
Sep 18 19:14:03.183952 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:03.183909    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed-original-pull-secret\") pod \"global-pull-secret-syncer-d9rlj\" (UID: \"7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed\") " pod="kube-system/global-pull-secret-syncer-d9rlj"
Sep 18 19:14:03.184138 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:03.184102    2532 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:03.184184 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:03.184170    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed-original-pull-secret podName:7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed nodeName:}" failed. No retries permitted until 2026-09-18 19:14:11.184156612 +0000 UTC m=+18.104660446 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed-original-pull-secret") pod "global-pull-secret-syncer-d9rlj" (UID: "7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:03.284999 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:03.284955    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05c957d5-da77-47c2-8b9c-e7cdc087120c-metrics-certs\") pod \"network-metrics-daemon-ps7w2\" (UID: \"05c957d5-da77-47c2-8b9c-e7cdc087120c\") " pod="openshift-multus/network-metrics-daemon-ps7w2"
Sep 18 19:14:03.285198 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:03.285141    2532 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:03.285263 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:03.285233    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05c957d5-da77-47c2-8b9c-e7cdc087120c-metrics-certs podName:05c957d5-da77-47c2-8b9c-e7cdc087120c nodeName:}" failed. No retries permitted until 2026-09-18 19:14:11.285211631 +0000 UTC m=+18.205715477 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05c957d5-da77-47c2-8b9c-e7cdc087120c-metrics-certs") pod "network-metrics-daemon-ps7w2" (UID: "05c957d5-da77-47c2-8b9c-e7cdc087120c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:03.386236 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:03.386117    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b7hs\" (UniqueName: \"kubernetes.io/projected/7082df75-a3d1-4eba-90e5-7a02a6aef94b-kube-api-access-6b7hs\") pod \"network-check-target-42kxn\" (UID: \"7082df75-a3d1-4eba-90e5-7a02a6aef94b\") " pod="openshift-network-diagnostics/network-check-target-42kxn"
Sep 18 19:14:03.386433 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:03.386352    2532 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:14:03.386433 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:03.386377    2532 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:14:03.386433 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:03.386390    2532 projected.go:196] Error preparing data for projected volume kube-api-access-6b7hs for pod openshift-network-diagnostics/network-check-target-42kxn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:03.386607 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:03.386450    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7082df75-a3d1-4eba-90e5-7a02a6aef94b-kube-api-access-6b7hs podName:7082df75-a3d1-4eba-90e5-7a02a6aef94b nodeName:}" failed. No retries permitted until 2026-09-18 19:14:11.386432446 +0000 UTC m=+18.306936288 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6b7hs" (UniqueName: "kubernetes.io/projected/7082df75-a3d1-4eba-90e5-7a02a6aef94b-kube-api-access-6b7hs") pod "network-check-target-42kxn" (UID: "7082df75-a3d1-4eba-90e5-7a02a6aef94b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:03.616785 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:03.616700    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d9rlj"
Sep 18 19:14:03.617257 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:03.616807    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d9rlj" podUID="7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed"
Sep 18 19:14:03.617373 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:03.617281    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42kxn"
Sep 18 19:14:03.617439 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:03.617370    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42kxn" podUID="7082df75-a3d1-4eba-90e5-7a02a6aef94b"
Sep 18 19:14:04.615846 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:04.615811    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ps7w2"
Sep 18 19:14:04.616003 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:04.615955    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ps7w2" podUID="05c957d5-da77-47c2-8b9c-e7cdc087120c"
Sep 18 19:14:05.616735 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:05.616700    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42kxn"
Sep 18 19:14:05.616735 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:05.616727    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d9rlj"
Sep 18 19:14:05.617282 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:05.616844    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42kxn" podUID="7082df75-a3d1-4eba-90e5-7a02a6aef94b"
Sep 18 19:14:05.617282 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:05.616874    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d9rlj" podUID="7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed"
Sep 18 19:14:06.615833 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:06.615803    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ps7w2"
Sep 18 19:14:06.616047 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:06.615925    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ps7w2" podUID="05c957d5-da77-47c2-8b9c-e7cdc087120c"
Sep 18 19:14:07.615945 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:07.615913    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42kxn"
Sep 18 19:14:07.616410 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:07.616031    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42kxn" podUID="7082df75-a3d1-4eba-90e5-7a02a6aef94b"
Sep 18 19:14:07.616410 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:07.616098    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d9rlj"
Sep 18 19:14:07.616410 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:07.616202    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d9rlj" podUID="7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed"
Sep 18 19:14:08.615679 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:08.615642    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ps7w2"
Sep 18 19:14:08.615863 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:08.615776    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ps7w2" podUID="05c957d5-da77-47c2-8b9c-e7cdc087120c"
Sep 18 19:14:09.619014 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:09.618984    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d9rlj"
Sep 18 19:14:09.619455 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:09.618992    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42kxn"
Sep 18 19:14:09.619455 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:09.619124    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d9rlj" podUID="7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed"
Sep 18 19:14:09.619455 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:09.619177    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42kxn" podUID="7082df75-a3d1-4eba-90e5-7a02a6aef94b"
Sep 18 19:14:10.616481 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:10.616452    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ps7w2"
Sep 18 19:14:10.616670 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:10.616570    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ps7w2" podUID="05c957d5-da77-47c2-8b9c-e7cdc087120c"
Sep 18 19:14:11.237007 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:11.236967    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed-original-pull-secret\") pod \"global-pull-secret-syncer-d9rlj\" (UID: \"7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed\") " pod="kube-system/global-pull-secret-syncer-d9rlj"
Sep 18 19:14:11.237462 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:11.237096    2532 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:11.237462 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:11.237163    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed-original-pull-secret podName:7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed nodeName:}" failed. No retries permitted until 2026-09-18 19:14:27.237146078 +0000 UTC m=+34.157649915 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed-original-pull-secret") pod "global-pull-secret-syncer-d9rlj" (UID: "7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:11.337602 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:11.337561    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05c957d5-da77-47c2-8b9c-e7cdc087120c-metrics-certs\") pod \"network-metrics-daemon-ps7w2\" (UID: \"05c957d5-da77-47c2-8b9c-e7cdc087120c\") " pod="openshift-multus/network-metrics-daemon-ps7w2"
Sep 18 19:14:11.337758 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:11.337726    2532 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:11.337809 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:11.337801    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05c957d5-da77-47c2-8b9c-e7cdc087120c-metrics-certs podName:05c957d5-da77-47c2-8b9c-e7cdc087120c nodeName:}" failed. No retries permitted until 2026-09-18 19:14:27.337779992 +0000 UTC m=+34.258283849 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05c957d5-da77-47c2-8b9c-e7cdc087120c-metrics-certs") pod "network-metrics-daemon-ps7w2" (UID: "05c957d5-da77-47c2-8b9c-e7cdc087120c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:11.438465 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:11.438425    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b7hs\" (UniqueName: \"kubernetes.io/projected/7082df75-a3d1-4eba-90e5-7a02a6aef94b-kube-api-access-6b7hs\") pod \"network-check-target-42kxn\" (UID: \"7082df75-a3d1-4eba-90e5-7a02a6aef94b\") " pod="openshift-network-diagnostics/network-check-target-42kxn"
Sep 18 19:14:11.438622 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:11.438578    2532 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:14:11.438622 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:11.438597    2532 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:14:11.438622 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:11.438606    2532 projected.go:196] Error preparing data for projected volume kube-api-access-6b7hs for pod openshift-network-diagnostics/network-check-target-42kxn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:11.438764 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:11.438652    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7082df75-a3d1-4eba-90e5-7a02a6aef94b-kube-api-access-6b7hs podName:7082df75-a3d1-4eba-90e5-7a02a6aef94b nodeName:}" failed. No retries permitted until 2026-09-18 19:14:27.438639683 +0000 UTC m=+34.359143519 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6b7hs" (UniqueName: "kubernetes.io/projected/7082df75-a3d1-4eba-90e5-7a02a6aef94b-kube-api-access-6b7hs") pod "network-check-target-42kxn" (UID: "7082df75-a3d1-4eba-90e5-7a02a6aef94b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:11.619316 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:11.619235    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d9rlj"
Sep 18 19:14:11.619316 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:11.619251    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42kxn"
Sep 18 19:14:11.619535 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:11.619350    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d9rlj" podUID="7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed"
Sep 18 19:14:11.619535 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:11.619479    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42kxn" podUID="7082df75-a3d1-4eba-90e5-7a02a6aef94b"
Sep 18 19:14:12.615957 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:12.615922    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ps7w2"
Sep 18 19:14:12.616389 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:12.616044    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ps7w2" podUID="05c957d5-da77-47c2-8b9c-e7cdc087120c"
Sep 18 19:14:13.616912 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:13.616884    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d9rlj"
Sep 18 19:14:13.617386 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:13.616970    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d9rlj" podUID="7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed"
Sep 18 19:14:13.617386 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:13.617048    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42kxn"
Sep 18 19:14:13.617386 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:13.617163    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42kxn" podUID="7082df75-a3d1-4eba-90e5-7a02a6aef94b"
Sep 18 19:14:14.616540 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:14.616416    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ps7w2"
Sep 18 19:14:14.616655 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:14.616632    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ps7w2" podUID="05c957d5-da77-47c2-8b9c-e7cdc087120c"
Sep 18 19:14:14.721967 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:14.721946    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sksw8_2cef0f4e-e44f-4996-8561-da9d261036c7/ovn-acl-logging/0.log"
Sep 18 19:14:14.722673 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:14.722316    2532 generic.go:356] "Generic (PLEG): container finished" podID="2cef0f4e-e44f-4996-8561-da9d261036c7" containerID="9e32b5db7de6a14bb21e71c7e95ffc62810c8c8fe68dd5778abd278a29cb010a" exitCode=1
Sep 18 19:14:14.722673 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:14.722384    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sksw8" event={"ID":"2cef0f4e-e44f-4996-8561-da9d261036c7","Type":"ContainerStarted","Data":"92733f6df432ee3c179bc91facc14d8fa7ffacc18f3b0666e9934e7a02d46e88"}
Sep 18 19:14:14.722673 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:14.722461    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sksw8" event={"ID":"2cef0f4e-e44f-4996-8561-da9d261036c7","Type":"ContainerStarted","Data":"dcf6b90ba9c8c5b63f871b924120089e9f7d962343e148a4066efd5d451b04ea"}
Sep 18 19:14:14.722673 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:14.722504    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sksw8" event={"ID":"2cef0f4e-e44f-4996-8561-da9d261036c7","Type":"ContainerStarted","Data":"755fe81ab10bea2382e38418ff1e22bf2c6aa3a9b2c0e27c465986e94b26b4de"}
Sep 18 19:14:14.722673 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:14.722555    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sksw8" event={"ID":"2cef0f4e-e44f-4996-8561-da9d261036c7","Type":"ContainerDied","Data":"9e32b5db7de6a14bb21e71c7e95ffc62810c8c8fe68dd5778abd278a29cb010a"}
Sep 18 19:14:14.722673 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:14.722602    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sksw8" event={"ID":"2cef0f4e-e44f-4996-8561-da9d261036c7","Type":"ContainerStarted","Data":"e0200f3e90929cefc65b93846c780f2d1b55daaeb114e2037aef200085a5a3f2"}
Sep 18 19:14:14.727209 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:14.727182    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qbs4d" event={"ID":"851825cd-f968-471c-87c0-b7d2423c9c99","Type":"ContainerStarted","Data":"71c55b20d28abe2cbb72e8169e5def8146acb773a55c6ccde85dfa8fb05e4458"}
Sep 18 19:14:14.728790 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:14.728757    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cclx5" event={"ID":"04602990-ffaa-4d6f-ac9e-bb960fccfbef","Type":"ContainerStarted","Data":"f00da1473555d71c3f7a1e5022a8a5925aa26ce874439ed7d87e5dd4121b9e00"}
Sep 18 19:14:14.730481 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:14.730447    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7s97w" event={"ID":"812cd3c0-1c61-4d21-8dd7-77d98f0edd2d","Type":"ContainerStarted","Data":"5f86330cf5f939b32c3138cf75b31a9e1a04c6bbaeb87ed531f5766be19186ef"}
Sep 18 19:14:14.732110 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:14.732089    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mmlqn" event={"ID":"210a5221-5e8c-4bf5-9bf1-abe8413e3897","Type":"ContainerStarted","Data":"538bac3acc9cb38dc0ecebcbd14fe3276adb93e37416f0f0324d46d59f71164b"}
Sep 18 19:14:14.733879 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:14.733856    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-226rm" event={"ID":"d45a88d4-e763-4a5e-87a3-91c0bb3bbf68","Type":"ContainerStarted","Data":"ca52afb960692beb61a615e02fee682be1b15ef9cf333e361c302a7e64a778f0"}
Sep 18 19:14:14.735440 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:14.735421    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dr7lf" event={"ID":"08755e2e-960f-4099-8bc0-1de6acfdd0c2","Type":"ContainerStarted","Data":"b01ccfc5e29022ff1a006b380872dc9e687bb69efae65ffa773480ff263e89c4"}
Sep 18 19:14:14.736949 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:14.736930    2532 generic.go:356] "Generic (PLEG): container finished" podID="7959688f-be33-494c-a8e9-08d33f2c2d3b" containerID="36ac73c6aeaeb899d58c54299c568c5f016de5311d28875844ca99c65ce3dc14" exitCode=0
Sep 18 19:14:14.737038 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:14.736955    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rc579" event={"ID":"7959688f-be33-494c-a8e9-08d33f2c2d3b","Type":"ContainerDied","Data":"36ac73c6aeaeb899d58c54299c568c5f016de5311d28875844ca99c65ce3dc14"}
Sep 18 19:14:14.766513 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:14.766475    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-qbs4d" podStartSLOduration=2.6441167610000003 podStartE2EDuration="20.766462034s" totalImagesPullingTime="18.122345273s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:13:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:14.766143613 +0000 UTC m=+21.686647487" watchObservedRunningTime="2026-09-18 19:14:14.766462034 +0000 UTC m=+21.686965891"
Sep 18 19:14:14.811409 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:14.811364    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-mmlqn" podStartSLOduration=2.21669778 podStartE2EDuration="20.811339799s" totalImagesPullingTime="18.594642019s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:13:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:14.809770401 +0000 UTC m=+21.730274258" watchObservedRunningTime="2026-09-18 19:14:14.811339799 +0000 UTC m=+21.731843655"
Sep 18 19:14:14.895437 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:14.895199    2532 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-226rm"
Sep 18 19:14:14.895879 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:14.895858    2532 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-226rm"
Sep 18 19:14:14.900979 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:14.900939    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7s97w" podStartSLOduration=2.820863657 podStartE2EDuration="20.90092437s" totalImagesPullingTime="18.080060713s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:13:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:14.900260341 +0000 UTC m=+21.820764200" watchObservedRunningTime="2026-09-18 19:14:14.90092437 +0000 UTC m=+21.821428226"
Sep 18 19:14:14.941642 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:14.941605    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-226rm" podStartSLOduration=6.593653462 podStartE2EDuration="20.941590914s" totalImagesPullingTime="14.347937452s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:13:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:14.941380489 +0000 UTC m=+21.861884345" watchObservedRunningTime="2026-09-18 19:14:14.941590914 +0000 UTC m=+21.862094771"
Sep 18 19:14:15.166587 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:15.166473    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-cclx5" podStartSLOduration=3.071196821 podStartE2EDuration="21.166457649s" totalImagesPullingTime="18.095260828s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:13:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:15.011697393 +0000 UTC m=+21.932201253" watchObservedRunningTime="2026-09-18 19:14:15.166457649 +0000 UTC m=+22.086961506"
Sep 18 19:14:15.616599 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:15.616529    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d9rlj"
Sep 18 19:14:15.616790 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:15.616641    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d9rlj" podUID="7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed"
Sep 18 19:14:15.616790 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:15.616690    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42kxn"
Sep 18 19:14:15.616898 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:15.616809    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42kxn" podUID="7082df75-a3d1-4eba-90e5-7a02a6aef94b"
Sep 18 19:14:15.742362 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:15.742334    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sksw8_2cef0f4e-e44f-4996-8561-da9d261036c7/ovn-acl-logging/0.log"
Sep 18 19:14:15.742820 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:15.742785    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sksw8" event={"ID":"2cef0f4e-e44f-4996-8561-da9d261036c7","Type":"ContainerStarted","Data":"bd00c3ff4672d9f8785de1d2a6321bed8907cbb5611aaed9d0ebd6e1f56a15a1"}
Sep 18 19:14:15.744716 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:15.744691    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7jr5l" event={"ID":"22a18a1f-f825-4aff-8e52-18ab52124ced","Type":"ContainerStarted","Data":"37e940d864ed4adfaf56171f7a3f7818b574a83439797d398cc9a5f606742d35"}
Sep 18 19:14:15.844201 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:15.844166    2532 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 18 19:14:16.562070 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:16.561823    2532 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-18T19:14:15.844185928Z","UUID":"47506c97-d40c-46c9-a4f2-b56853836cbd","Handler":null,"Name":"","Endpoint":""}
Sep 18 19:14:16.563599 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:16.563574    2532 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 18 19:14:16.563723 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:16.563611    2532 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 18 19:14:16.615751 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:16.615727    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ps7w2"
Sep 18 19:14:16.615914 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:16.615856    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ps7w2" podUID="05c957d5-da77-47c2-8b9c-e7cdc087120c"
Sep 18 19:14:16.750137 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:16.750103    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dr7lf" event={"ID":"08755e2e-960f-4099-8bc0-1de6acfdd0c2","Type":"ContainerStarted","Data":"76e31e74d2ed59fe90f1d82e7fda2a29ef257793b2248f2a49b0e994fbc5a926"}
Sep 18 19:14:16.750575 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:16.750161    2532 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 18 19:14:17.615840 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:17.615811    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42kxn"
Sep 18 19:14:17.615840 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:17.615820    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d9rlj"
Sep 18 19:14:17.616137 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:17.615941    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42kxn" podUID="7082df75-a3d1-4eba-90e5-7a02a6aef94b"
Sep 18 19:14:17.616137 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:17.616068    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d9rlj" podUID="7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed"
Sep 18 19:14:17.756965 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:17.756938    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sksw8_2cef0f4e-e44f-4996-8561-da9d261036c7/ovn-acl-logging/0.log"
Sep 18 19:14:17.757443 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:17.757417    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sksw8" event={"ID":"2cef0f4e-e44f-4996-8561-da9d261036c7","Type":"ContainerStarted","Data":"71b69b55e0042e1c44b5acf8723f6130299b23875b927f6289053f3b2571bc68"}
Sep 18 19:14:17.759778 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:17.759750    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dr7lf" event={"ID":"08755e2e-960f-4099-8bc0-1de6acfdd0c2","Type":"ContainerStarted","Data":"0b27f546ce0b7943dd5b226cece61bcfe949158f1ccbf803d5c74085b943a7f6"}
Sep 18 19:14:17.786494 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:17.786448    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-7jr5l" podStartSLOduration=5.687536057 podStartE2EDuration="23.786430986s" totalImagesPullingTime="18.098894929s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:13:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:15.776955484 +0000 UTC m=+22.697459341" watchObservedRunningTime="2026-09-18 19:14:17.786430986 +0000 UTC m=+24.706934844"
Sep 18 19:14:17.786647 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:17.786624    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dr7lf" podStartSLOduration=3.122117773 podStartE2EDuration="23.786614994s" totalImagesPullingTime="20.664497221s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:13:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:17.786341559 +0000 UTC m=+24.706845438" watchObservedRunningTime="2026-09-18 19:14:17.786614994 +0000 UTC m=+24.707118852"
Sep 18 19:14:18.616369 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:18.616340    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ps7w2"
Sep 18 19:14:18.616530 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:18.616476    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ps7w2" podUID="05c957d5-da77-47c2-8b9c-e7cdc087120c"
Sep 18 19:14:19.372636 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:19.372407    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-226rm"
Sep 18 19:14:19.373241 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:19.372712    2532 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 18 19:14:19.373241 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:19.372976    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-226rm"
Sep 18 19:14:19.615797 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:19.615768    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d9rlj"
Sep 18 19:14:19.616236 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:19.615853    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d9rlj" podUID="7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed"
Sep 18 19:14:19.616236 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:19.615770    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42kxn"
Sep 18 19:14:19.616236 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:19.615925    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42kxn" podUID="7082df75-a3d1-4eba-90e5-7a02a6aef94b"
Sep 18 19:14:19.769366 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:19.769300    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sksw8_2cef0f4e-e44f-4996-8561-da9d261036c7/ovn-acl-logging/0.log"
Sep 18 19:14:19.769729 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:19.769706    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sksw8" event={"ID":"2cef0f4e-e44f-4996-8561-da9d261036c7","Type":"ContainerStarted","Data":"141455e04909eb5e9c8fb12a533ce6337a40cb50c3dee1af1915ad208dd56f0f"}
Sep 18 19:14:19.770307 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:19.769976    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:14:19.770307 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:19.770042    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:14:19.770307 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:19.770180    2532 scope.go:117] "RemoveContainer" containerID="9e32b5db7de6a14bb21e71c7e95ffc62810c8c8fe68dd5778abd278a29cb010a"
Sep 18 19:14:19.771903 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:19.771875    2532 generic.go:356] "Generic (PLEG): container finished" podID="7959688f-be33-494c-a8e9-08d33f2c2d3b" containerID="5fb430bd3c960a38fee656baaa1c7b7b2465f9d03719e34904bac4e0dd90c5b1" exitCode=0
Sep 18 19:14:19.772005 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:19.771973    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rc579" event={"ID":"7959688f-be33-494c-a8e9-08d33f2c2d3b","Type":"ContainerDied","Data":"5fb430bd3c960a38fee656baaa1c7b7b2465f9d03719e34904bac4e0dd90c5b1"}
Sep 18 19:14:19.785794 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:19.785774    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:14:19.786891 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:19.786872    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:14:20.616538 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:20.616511    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ps7w2"
Sep 18 19:14:20.617026 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:20.616611    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ps7w2" podUID="05c957d5-da77-47c2-8b9c-e7cdc087120c"
Sep 18 19:14:20.731871 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:20.731848    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:14:20.781626 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:20.781602    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sksw8_2cef0f4e-e44f-4996-8561-da9d261036c7/ovn-acl-logging/0.log"
Sep 18 19:14:20.781957 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:20.781935    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sksw8" event={"ID":"2cef0f4e-e44f-4996-8561-da9d261036c7","Type":"ContainerStarted","Data":"80c4d9772eaecdfbb86144a42959338c62e577dee06eb8fd0aadab418cbff11d"}
Sep 18 19:14:20.835954 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:20.835910    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-sksw8" podStartSLOduration=8.701490356 podStartE2EDuration="26.835897135s" totalImagesPullingTime="18.134406779s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:13:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:20.834972253 +0000 UTC m=+27.755476111" watchObservedRunningTime="2026-09-18 19:14:20.835897135 +0000 UTC m=+27.756400992"
Sep 18 19:14:21.211037 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:21.210782    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-42kxn"]
Sep 18 19:14:21.211182 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:21.211041    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42kxn"
Sep 18 19:14:21.211182 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:21.211132    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42kxn" podUID="7082df75-a3d1-4eba-90e5-7a02a6aef94b"
Sep 18 19:14:21.213342 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:21.213321    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-d9rlj"]
Sep 18 19:14:21.213448 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:21.213404    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d9rlj"
Sep 18 19:14:21.213485 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:21.213469    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d9rlj" podUID="7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed"
Sep 18 19:14:21.222333 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:21.222312    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ps7w2"]
Sep 18 19:14:21.222444 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:21.222403    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ps7w2"
Sep 18 19:14:21.222528 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:21.222509    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ps7w2" podUID="05c957d5-da77-47c2-8b9c-e7cdc087120c"
Sep 18 19:14:21.785860 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:21.785827    2532 generic.go:356] "Generic (PLEG): container finished" podID="7959688f-be33-494c-a8e9-08d33f2c2d3b" containerID="efa272fb9127e42ed567e158b534a1958ebf15384cf535dbdf5c46e69e14b220" exitCode=0
Sep 18 19:14:21.786245 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:21.785886    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rc579" event={"ID":"7959688f-be33-494c-a8e9-08d33f2c2d3b","Type":"ContainerDied","Data":"efa272fb9127e42ed567e158b534a1958ebf15384cf535dbdf5c46e69e14b220"}
Sep 18 19:14:22.616013 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:22.615985    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d9rlj"
Sep 18 19:14:22.616195 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:22.615988    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ps7w2"
Sep 18 19:14:22.616195 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:22.616124    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d9rlj" podUID="7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed"
Sep 18 19:14:22.616195 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:22.616168    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ps7w2" podUID="05c957d5-da77-47c2-8b9c-e7cdc087120c"
Sep 18 19:14:23.619070 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:23.619036    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42kxn"
Sep 18 19:14:23.619412 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:23.619166    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42kxn" podUID="7082df75-a3d1-4eba-90e5-7a02a6aef94b"
Sep 18 19:14:23.792664 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:23.792636    2532 generic.go:356] "Generic (PLEG): container finished" podID="7959688f-be33-494c-a8e9-08d33f2c2d3b" containerID="358e5de03124ad347d1df7e6c707e054e82af0dfbf892773a9bfdcbdcd8d70c9" exitCode=0
Sep 18 19:14:23.792809 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:23.792673    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rc579" event={"ID":"7959688f-be33-494c-a8e9-08d33f2c2d3b","Type":"ContainerDied","Data":"358e5de03124ad347d1df7e6c707e054e82af0dfbf892773a9bfdcbdcd8d70c9"}
Sep 18 19:14:24.616301 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:24.616270    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d9rlj"
Sep 18 19:14:24.616471 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:24.616380    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d9rlj" podUID="7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed"
Sep 18 19:14:24.616471 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:24.616431    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ps7w2"
Sep 18 19:14:24.616584 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:24.616519    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ps7w2" podUID="05c957d5-da77-47c2-8b9c-e7cdc087120c"
Sep 18 19:14:25.616228 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:25.616189    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42kxn"
Sep 18 19:14:25.616672 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:25.616308    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42kxn" podUID="7082df75-a3d1-4eba-90e5-7a02a6aef94b"
Sep 18 19:14:26.615879 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:26.615846    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ps7w2"
Sep 18 19:14:26.616179 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:26.615968    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ps7w2" podUID="05c957d5-da77-47c2-8b9c-e7cdc087120c"
Sep 18 19:14:26.616179 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:26.616019    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d9rlj"
Sep 18 19:14:26.616179 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:26.616121    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d9rlj" podUID="7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed"
Sep 18 19:14:27.257616 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.257585    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed-original-pull-secret\") pod \"global-pull-secret-syncer-d9rlj\" (UID: \"7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed\") " pod="kube-system/global-pull-secret-syncer-d9rlj"
Sep 18 19:14:27.257799 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:27.257713    2532 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:27.257799 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:27.257769    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed-original-pull-secret podName:7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed nodeName:}" failed. No retries permitted until 2026-09-18 19:14:59.257754887 +0000 UTC m=+66.178258723 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed-original-pull-secret") pod "global-pull-secret-syncer-d9rlj" (UID: "7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:27.358510 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.358413    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05c957d5-da77-47c2-8b9c-e7cdc087120c-metrics-certs\") pod \"network-metrics-daemon-ps7w2\" (UID: \"05c957d5-da77-47c2-8b9c-e7cdc087120c\") " pod="openshift-multus/network-metrics-daemon-ps7w2"
Sep 18 19:14:27.358663 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:27.358546    2532 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:27.358663 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:27.358612    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05c957d5-da77-47c2-8b9c-e7cdc087120c-metrics-certs podName:05c957d5-da77-47c2-8b9c-e7cdc087120c nodeName:}" failed. No retries permitted until 2026-09-18 19:14:59.358593995 +0000 UTC m=+66.279097859 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05c957d5-da77-47c2-8b9c-e7cdc087120c-metrics-certs") pod "network-metrics-daemon-ps7w2" (UID: "05c957d5-da77-47c2-8b9c-e7cdc087120c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:27.419914 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.419880    2532 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-252.ec2.internal" event="NodeReady"
Sep 18 19:14:27.420108 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.420033    2532 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 18 19:14:27.459800 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.459759    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b7hs\" (UniqueName: \"kubernetes.io/projected/7082df75-a3d1-4eba-90e5-7a02a6aef94b-kube-api-access-6b7hs\") pod \"network-check-target-42kxn\" (UID: \"7082df75-a3d1-4eba-90e5-7a02a6aef94b\") " pod="openshift-network-diagnostics/network-check-target-42kxn"
Sep 18 19:14:27.459978 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:27.459949    2532 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:14:27.459978 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:27.459977    2532 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:14:27.460128 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:27.459993    2532 projected.go:196] Error preparing data for projected volume kube-api-access-6b7hs for pod openshift-network-diagnostics/network-check-target-42kxn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:27.460128 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:27.460076    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7082df75-a3d1-4eba-90e5-7a02a6aef94b-kube-api-access-6b7hs podName:7082df75-a3d1-4eba-90e5-7a02a6aef94b nodeName:}" failed. No retries permitted until 2026-09-18 19:14:59.460039494 +0000 UTC m=+66.380543343 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-6b7hs" (UniqueName: "kubernetes.io/projected/7082df75-a3d1-4eba-90e5-7a02a6aef94b-kube-api-access-6b7hs") pod "network-check-target-42kxn" (UID: "7082df75-a3d1-4eba-90e5-7a02a6aef94b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:27.509722 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.509688    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764c5dfc6-m99t4"]
Sep 18 19:14:27.514530 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.514506    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6dbcf5f8b8-7f6cw"]
Sep 18 19:14:27.514683 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.514645    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764c5dfc6-m99t4"
Sep 18 19:14:27.517307 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.517287    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7c9c7469f5-nckkz"]
Sep 18 19:14:27.517431 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.517409    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6dbcf5f8b8-7f6cw"
Sep 18 19:14:27.520012 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.519993    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64fbd65ff6-m84r9"]
Sep 18 19:14:27.520152 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.520132    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c9c7469f5-nckkz"
Sep 18 19:14:27.521943 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.521924    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 18 19:14:27.522740 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.522657    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64fbd65ff6-m84r9"
Sep 18 19:14:27.529081 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.529045    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 18 19:14:27.529215 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.529196    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 18 19:14:27.529940 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.529921    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 18 19:14:27.529940 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.529928    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 18 19:14:27.530188 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.529989    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 18 19:14:27.530389 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.530375    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-zqrsb\""
Sep 18 19:14:27.530875 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.530858    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 18 19:14:27.530954 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.530933    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 18 19:14:27.531468 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.531453    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 18 19:14:27.540399 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.540378    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rzkv7\""
Sep 18 19:14:27.541001 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.540955    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 18 19:14:27.543525 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.543511    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 18 19:14:27.547645 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.547629    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 18 19:14:27.564107 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.564002    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 18 19:14:27.564215 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.564125    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764c5dfc6-m99t4"]
Sep 18 19:14:27.571591 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.571573    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c9c7469f5-nckkz"]
Sep 18 19:14:27.604885 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.604817    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64fbd65ff6-m84r9"]
Sep 18 19:14:27.605737 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.605707    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6dbcf5f8b8-7f6cw"]
Sep 18 19:14:27.616323 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.616267    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42kxn"
Sep 18 19:14:27.620686 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.620666    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 19:14:27.620795 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.620770    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 19:14:27.620861 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.620795    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lf5lk\""
Sep 18 19:14:27.629909 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.629883    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-54cns"]
Sep 18 19:14:27.635088 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.635042    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-54cns"
Sep 18 19:14:27.661178 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.661148    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8367afb3-7c5e-429d-8dc1-2750240a5b30-installation-pull-secrets\") pod \"image-registry-7c9c7469f5-nckkz\" (UID: \"8367afb3-7c5e-429d-8dc1-2750240a5b30\") " pod="openshift-image-registry/image-registry-7c9c7469f5-nckkz"
Sep 18 19:14:27.661296 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.661182    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/776f6d09-2964-4c91-95fb-e7cdfec3e6ac-klusterlet-config\") pod \"klusterlet-addon-workmgr-6dbcf5f8b8-7f6cw\" (UID: \"776f6d09-2964-4c91-95fb-e7cdfec3e6ac\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6dbcf5f8b8-7f6cw"
Sep 18 19:14:27.661296 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.661204    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8367afb3-7c5e-429d-8dc1-2750240a5b30-image-registry-private-configuration\") pod \"image-registry-7c9c7469f5-nckkz\" (UID: \"8367afb3-7c5e-429d-8dc1-2750240a5b30\") " pod="openshift-image-registry/image-registry-7c9c7469f5-nckkz"
Sep 18 19:14:27.661296 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.661227    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8367afb3-7c5e-429d-8dc1-2750240a5b30-registry-tls\") pod \"image-registry-7c9c7469f5-nckkz\" (UID: \"8367afb3-7c5e-429d-8dc1-2750240a5b30\") " pod="openshift-image-registry/image-registry-7c9c7469f5-nckkz"
Sep 18 19:14:27.661296 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.661248    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8367afb3-7c5e-429d-8dc1-2750240a5b30-trusted-ca\") pod \"image-registry-7c9c7469f5-nckkz\" (UID: \"8367afb3-7c5e-429d-8dc1-2750240a5b30\") " pod="openshift-image-registry/image-registry-7c9c7469f5-nckkz"
Sep 18 19:14:27.661296 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.661273    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0a2aba53-8251-4204-9845-f5572077030b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-764c5dfc6-m99t4\" (UID: \"0a2aba53-8251-4204-9845-f5572077030b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764c5dfc6-m99t4"
Sep 18 19:14:27.661527 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.661302    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0a2aba53-8251-4204-9845-f5572077030b-ca\") pod \"cluster-proxy-proxy-agent-764c5dfc6-m99t4\" (UID: \"0a2aba53-8251-4204-9845-f5572077030b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764c5dfc6-m99t4"
Sep 18 19:14:27.661527 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.661352    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlzl6\" (UniqueName: \"kubernetes.io/projected/0a2aba53-8251-4204-9845-f5572077030b-kube-api-access-tlzl6\") pod \"cluster-proxy-proxy-agent-764c5dfc6-m99t4\" (UID: \"0a2aba53-8251-4204-9845-f5572077030b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764c5dfc6-m99t4"
Sep 18 19:14:27.661527 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.661390    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/85a6cce8-df47-4abf-92c8-f6239d084c07-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-64fbd65ff6-m84r9\" (UID: \"85a6cce8-df47-4abf-92c8-f6239d084c07\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64fbd65ff6-m84r9"
Sep 18 19:14:27.661527 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.661415    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/776f6d09-2964-4c91-95fb-e7cdfec3e6ac-tmp\") pod \"klusterlet-addon-workmgr-6dbcf5f8b8-7f6cw\" (UID: \"776f6d09-2964-4c91-95fb-e7cdfec3e6ac\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6dbcf5f8b8-7f6cw"
Sep 18 19:14:27.661527 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.661446    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2ss8\" (UniqueName: \"kubernetes.io/projected/85a6cce8-df47-4abf-92c8-f6239d084c07-kube-api-access-x2ss8\") pod \"managed-serviceaccount-addon-agent-64fbd65ff6-m84r9\" (UID: \"85a6cce8-df47-4abf-92c8-f6239d084c07\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64fbd65ff6-m84r9"
Sep 18 19:14:27.661527 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.661519    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a2aba53-8251-4204-9845-f5572077030b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-764c5dfc6-m99t4\" (UID: \"0a2aba53-8251-4204-9845-f5572077030b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764c5dfc6-m99t4"
Sep 18 19:14:27.661754 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.661548    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pxhk\" (UniqueName: \"kubernetes.io/projected/8367afb3-7c5e-429d-8dc1-2750240a5b30-kube-api-access-9pxhk\") pod \"image-registry-7c9c7469f5-nckkz\" (UID: \"8367afb3-7c5e-429d-8dc1-2750240a5b30\") " pod="openshift-image-registry/image-registry-7c9c7469f5-nckkz"
Sep 18 19:14:27.661754 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.661583    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g445w\" (UniqueName: \"kubernetes.io/projected/776f6d09-2964-4c91-95fb-e7cdfec3e6ac-kube-api-access-g445w\") pod \"klusterlet-addon-workmgr-6dbcf5f8b8-7f6cw\" (UID: \"776f6d09-2964-4c91-95fb-e7cdfec3e6ac\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6dbcf5f8b8-7f6cw"
Sep 18 19:14:27.661754 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.661613    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8367afb3-7c5e-429d-8dc1-2750240a5b30-bound-sa-token\") pod \"image-registry-7c9c7469f5-nckkz\" (UID: \"8367afb3-7c5e-429d-8dc1-2750240a5b30\") " pod="openshift-image-registry/image-registry-7c9c7469f5-nckkz"
Sep 18 19:14:27.661754 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.661660    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0a2aba53-8251-4204-9845-f5572077030b-hub\") pod \"cluster-proxy-proxy-agent-764c5dfc6-m99t4\" (UID: \"0a2aba53-8251-4204-9845-f5572077030b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764c5dfc6-m99t4"
Sep 18 19:14:27.661754 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.661687    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0a2aba53-8251-4204-9845-f5572077030b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-764c5dfc6-m99t4\" (UID: \"0a2aba53-8251-4204-9845-f5572077030b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764c5dfc6-m99t4"
Sep 18 19:14:27.661754 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.661713    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8367afb3-7c5e-429d-8dc1-2750240a5b30-ca-trust-extracted\") pod \"image-registry-7c9c7469f5-nckkz\" (UID: \"8367afb3-7c5e-429d-8dc1-2750240a5b30\") " pod="openshift-image-registry/image-registry-7c9c7469f5-nckkz"
Sep 18 19:14:27.661754 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.661740    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8367afb3-7c5e-429d-8dc1-2750240a5b30-registry-certificates\") pod \"image-registry-7c9c7469f5-nckkz\" (UID: \"8367afb3-7c5e-429d-8dc1-2750240a5b30\") " pod="openshift-image-registry/image-registry-7c9c7469f5-nckkz"
Sep 18 19:14:27.662005 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.661951    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-54cns"]
Sep 18 19:14:27.666027 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.666007    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 18 19:14:27.666287 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.666273    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-sd5f2\""
Sep 18 19:14:27.666423 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.666399    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 18 19:14:27.724017 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.723984    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-wgxds"]
Sep 18 19:14:27.727034 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.727012    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wgxds"
Sep 18 19:14:27.731277 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.731253    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 18 19:14:27.731402 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.731305    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 18 19:14:27.731402 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.731316    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-lxlj8\""
Sep 18 19:14:27.731954 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.731938    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 18 19:14:27.761180 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.761148    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wgxds"]
Sep 18 19:14:27.762501 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.762472    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0a2aba53-8251-4204-9845-f5572077030b-ca\") pod \"cluster-proxy-proxy-agent-764c5dfc6-m99t4\" (UID: \"0a2aba53-8251-4204-9845-f5572077030b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764c5dfc6-m99t4"
Sep 18 19:14:27.762611 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.762518    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlzl6\" (UniqueName: \"kubernetes.io/projected/0a2aba53-8251-4204-9845-f5572077030b-kube-api-access-tlzl6\") pod \"cluster-proxy-proxy-agent-764c5dfc6-m99t4\" (UID: \"0a2aba53-8251-4204-9845-f5572077030b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764c5dfc6-m99t4"
Sep 18 19:14:27.762611 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.762549    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/85a6cce8-df47-4abf-92c8-f6239d084c07-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-64fbd65ff6-m84r9\" (UID: \"85a6cce8-df47-4abf-92c8-f6239d084c07\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64fbd65ff6-m84r9"
Sep 18 19:14:27.762611 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.762573    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/776f6d09-2964-4c91-95fb-e7cdfec3e6ac-tmp\") pod \"klusterlet-addon-workmgr-6dbcf5f8b8-7f6cw\" (UID: \"776f6d09-2964-4c91-95fb-e7cdfec3e6ac\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6dbcf5f8b8-7f6cw"
Sep 18 19:14:27.762611 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.762602    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd8vt\" (UniqueName: \"kubernetes.io/projected/ad80f379-b5dc-4d02-a61d-b8b2a956a551-kube-api-access-pd8vt\") pod \"dns-default-54cns\" (UID: \"ad80f379-b5dc-4d02-a61d-b8b2a956a551\") " pod="openshift-dns/dns-default-54cns"
Sep 18 19:14:27.762804 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.762634    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2ss8\" (UniqueName: \"kubernetes.io/projected/85a6cce8-df47-4abf-92c8-f6239d084c07-kube-api-access-x2ss8\") pod \"managed-serviceaccount-addon-agent-64fbd65ff6-m84r9\" (UID: \"85a6cce8-df47-4abf-92c8-f6239d084c07\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64fbd65ff6-m84r9"
Sep 18 19:14:27.762804 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.762679    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a2aba53-8251-4204-9845-f5572077030b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-764c5dfc6-m99t4\" (UID: \"0a2aba53-8251-4204-9845-f5572077030b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764c5dfc6-m99t4"
Sep 18 19:14:27.762804 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.762697    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pxhk\" (UniqueName: \"kubernetes.io/projected/8367afb3-7c5e-429d-8dc1-2750240a5b30-kube-api-access-9pxhk\") pod \"image-registry-7c9c7469f5-nckkz\" (UID: \"8367afb3-7c5e-429d-8dc1-2750240a5b30\") " pod="openshift-image-registry/image-registry-7c9c7469f5-nckkz"
Sep 18 19:14:27.762804 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.762736    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g445w\" (UniqueName: \"kubernetes.io/projected/776f6d09-2964-4c91-95fb-e7cdfec3e6ac-kube-api-access-g445w\") pod \"klusterlet-addon-workmgr-6dbcf5f8b8-7f6cw\" (UID: \"776f6d09-2964-4c91-95fb-e7cdfec3e6ac\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6dbcf5f8b8-7f6cw"
Sep 18 19:14:27.762804 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.762759    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ad80f379-b5dc-4d02-a61d-b8b2a956a551-config-volume\") pod \"dns-default-54cns\" (UID: \"ad80f379-b5dc-4d02-a61d-b8b2a956a551\") " pod="openshift-dns/dns-default-54cns"
Sep 18 19:14:27.762804 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.762778    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8367afb3-7c5e-429d-8dc1-2750240a5b30-bound-sa-token\") pod \"image-registry-7c9c7469f5-nckkz\" (UID: \"8367afb3-7c5e-429d-8dc1-2750240a5b30\") " pod="openshift-image-registry/image-registry-7c9c7469f5-nckkz"
Sep 18 19:14:27.762804 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.762792    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad80f379-b5dc-4d02-a61d-b8b2a956a551-metrics-tls\") pod \"dns-default-54cns\" (UID: \"ad80f379-b5dc-4d02-a61d-b8b2a956a551\") " pod="openshift-dns/dns-default-54cns"
Sep 18 19:14:27.763100 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.762822    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0a2aba53-8251-4204-9845-f5572077030b-hub\") pod \"cluster-proxy-proxy-agent-764c5dfc6-m99t4\" (UID: \"0a2aba53-8251-4204-9845-f5572077030b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764c5dfc6-m99t4"
Sep 18 19:14:27.763100 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.762838    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0a2aba53-8251-4204-9845-f5572077030b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-764c5dfc6-m99t4\" (UID: \"0a2aba53-8251-4204-9845-f5572077030b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764c5dfc6-m99t4"
Sep 18 19:14:27.763100 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.762860    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8367afb3-7c5e-429d-8dc1-2750240a5b30-ca-trust-extracted\") pod \"image-registry-7c9c7469f5-nckkz\" (UID: \"8367afb3-7c5e-429d-8dc1-2750240a5b30\") " pod="openshift-image-registry/image-registry-7c9c7469f5-nckkz"
Sep 18 19:14:27.763100 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.762875    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8367afb3-7c5e-429d-8dc1-2750240a5b30-registry-certificates\") pod \"image-registry-7c9c7469f5-nckkz\" (UID: \"8367afb3-7c5e-429d-8dc1-2750240a5b30\") " pod="openshift-image-registry/image-registry-7c9c7469f5-nckkz"
Sep 18 19:14:27.763100 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.762891    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8367afb3-7c5e-429d-8dc1-2750240a5b30-installation-pull-secrets\") pod \"image-registry-7c9c7469f5-nckkz\" (UID: \"8367afb3-7c5e-429d-8dc1-2750240a5b30\") " pod="openshift-image-registry/image-registry-7c9c7469f5-nckkz"
Sep 18 19:14:27.763100 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.762907    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/776f6d09-2964-4c91-95fb-e7cdfec3e6ac-klusterlet-config\") pod \"klusterlet-addon-workmgr-6dbcf5f8b8-7f6cw\" (UID: \"776f6d09-2964-4c91-95fb-e7cdfec3e6ac\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6dbcf5f8b8-7f6cw"
Sep 18 19:14:27.763100 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.762925    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8367afb3-7c5e-429d-8dc1-2750240a5b30-image-registry-private-configuration\") pod \"image-registry-7c9c7469f5-nckkz\" (UID: \"8367afb3-7c5e-429d-8dc1-2750240a5b30\") " pod="openshift-image-registry/image-registry-7c9c7469f5-nckkz"
Sep 18 19:14:27.763100 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.762947    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8367afb3-7c5e-429d-8dc1-2750240a5b30-registry-tls\") pod \"image-registry-7c9c7469f5-nckkz\" (UID: \"8367afb3-7c5e-429d-8dc1-2750240a5b30\") " pod="openshift-image-registry/image-registry-7c9c7469f5-nckkz"
Sep 18 19:14:27.763100 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.762963    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8367afb3-7c5e-429d-8dc1-2750240a5b30-trusted-ca\") pod \"image-registry-7c9c7469f5-nckkz\" (UID: \"8367afb3-7c5e-429d-8dc1-2750240a5b30\") " pod="openshift-image-registry/image-registry-7c9c7469f5-nckkz"
Sep 18 19:14:27.763100 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.762978    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad80f379-b5dc-4d02-a61d-b8b2a956a551-tmp-dir\") pod \"dns-default-54cns\" (UID: \"ad80f379-b5dc-4d02-a61d-b8b2a956a551\") " pod="openshift-dns/dns-default-54cns"
Sep 18 19:14:27.763100 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.762976    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/776f6d09-2964-4c91-95fb-e7cdfec3e6ac-tmp\") pod \"klusterlet-addon-workmgr-6dbcf5f8b8-7f6cw\" (UID: \"776f6d09-2964-4c91-95fb-e7cdfec3e6ac\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6dbcf5f8b8-7f6cw"
Sep 18 19:14:27.763730 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.763312    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0a2aba53-8251-4204-9845-f5572077030b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-764c5dfc6-m99t4\" (UID: \"0a2aba53-8251-4204-9845-f5572077030b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764c5dfc6-m99t4"
Sep 18 19:14:27.763796 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.763768    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a2aba53-8251-4204-9845-f5572077030b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-764c5dfc6-m99t4\" (UID: \"0a2aba53-8251-4204-9845-f5572077030b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764c5dfc6-m99t4"
Sep 18 19:14:27.764519 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.764318    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8367afb3-7c5e-429d-8dc1-2750240a5b30-registry-certificates\") pod \"image-registry-7c9c7469f5-nckkz\" (UID: \"8367afb3-7c5e-429d-8dc1-2750240a5b30\") " pod="openshift-image-registry/image-registry-7c9c7469f5-nckkz"
Sep 18 19:14:27.764619 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.764601    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8367afb3-7c5e-429d-8dc1-2750240a5b30-ca-trust-extracted\") pod \"image-registry-7c9c7469f5-nckkz\" (UID: \"8367afb3-7c5e-429d-8dc1-2750240a5b30\") " pod="openshift-image-registry/image-registry-7c9c7469f5-nckkz"
Sep 18 19:14:27.764730 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:27.764712    2532 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:14:27.764795 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:27.764735    2532 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c9c7469f5-nckkz: secret "image-registry-tls" not found
Sep 18 19:14:27.764844 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:27.764797    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8367afb3-7c5e-429d-8dc1-2750240a5b30-registry-tls podName:8367afb3-7c5e-429d-8dc1-2750240a5b30 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:28.264779645 +0000 UTC m=+35.185283481 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8367afb3-7c5e-429d-8dc1-2750240a5b30-registry-tls") pod "image-registry-7c9c7469f5-nckkz" (UID: "8367afb3-7c5e-429d-8dc1-2750240a5b30") : secret "image-registry-tls" not found
Sep 18 19:14:27.765560 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.765534    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8367afb3-7c5e-429d-8dc1-2750240a5b30-trusted-ca\") pod \"image-registry-7c9c7469f5-nckkz\" (UID: \"8367afb3-7c5e-429d-8dc1-2750240a5b30\") " pod="openshift-image-registry/image-registry-7c9c7469f5-nckkz"
Sep 18 19:14:27.768082 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.767859    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8367afb3-7c5e-429d-8dc1-2750240a5b30-image-registry-private-configuration\") pod \"image-registry-7c9c7469f5-nckkz\" (UID: \"8367afb3-7c5e-429d-8dc1-2750240a5b30\") " pod="openshift-image-registry/image-registry-7c9c7469f5-nckkz"
Sep 18 19:14:27.768082 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.767982    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/85a6cce8-df47-4abf-92c8-f6239d084c07-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-64fbd65ff6-m84r9\" (UID: \"85a6cce8-df47-4abf-92c8-f6239d084c07\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64fbd65ff6-m84r9"
Sep 18 19:14:27.768082 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.768000    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0a2aba53-8251-4204-9845-f5572077030b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-764c5dfc6-m99t4\" (UID: \"0a2aba53-8251-4204-9845-f5572077030b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764c5dfc6-m99t4"
Sep 18 19:14:27.768414 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.768394    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/776f6d09-2964-4c91-95fb-e7cdfec3e6ac-klusterlet-config\") pod \"klusterlet-addon-workmgr-6dbcf5f8b8-7f6cw\" (UID: \"776f6d09-2964-4c91-95fb-e7cdfec3e6ac\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6dbcf5f8b8-7f6cw"
Sep 18 19:14:27.768612 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.768594    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0a2aba53-8251-4204-9845-f5572077030b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-764c5dfc6-m99t4\" (UID: \"0a2aba53-8251-4204-9845-f5572077030b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764c5dfc6-m99t4"
Sep 18 19:14:27.768934 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.768912    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0a2aba53-8251-4204-9845-f5572077030b-ca\") pod \"cluster-proxy-proxy-agent-764c5dfc6-m99t4\" (UID: \"0a2aba53-8251-4204-9845-f5572077030b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764c5dfc6-m99t4"
Sep 18 19:14:27.769166 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.769146    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8367afb3-7c5e-429d-8dc1-2750240a5b30-installation-pull-secrets\") pod \"image-registry-7c9c7469f5-nckkz\" (UID: \"8367afb3-7c5e-429d-8dc1-2750240a5b30\") " pod="openshift-image-registry/image-registry-7c9c7469f5-nckkz"
Sep 18 19:14:27.769796 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.769780    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0a2aba53-8251-4204-9845-f5572077030b-hub\") pod \"cluster-proxy-proxy-agent-764c5dfc6-m99t4\" (UID: \"0a2aba53-8251-4204-9845-f5572077030b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764c5dfc6-m99t4"
Sep 18 19:14:27.786535 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.786508    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlzl6\" (UniqueName: \"kubernetes.io/projected/0a2aba53-8251-4204-9845-f5572077030b-kube-api-access-tlzl6\") pod \"cluster-proxy-proxy-agent-764c5dfc6-m99t4\" (UID: \"0a2aba53-8251-4204-9845-f5572077030b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764c5dfc6-m99t4"
Sep 18 19:14:27.801091 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.801044    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8367afb3-7c5e-429d-8dc1-2750240a5b30-bound-sa-token\") pod \"image-registry-7c9c7469f5-nckkz\" (UID: \"8367afb3-7c5e-429d-8dc1-2750240a5b30\") " pod="openshift-image-registry/image-registry-7c9c7469f5-nckkz"
Sep 18 19:14:27.801208 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.801154    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pxhk\" (UniqueName: \"kubernetes.io/projected/8367afb3-7c5e-429d-8dc1-2750240a5b30-kube-api-access-9pxhk\") pod \"image-registry-7c9c7469f5-nckkz\" (UID: \"8367afb3-7c5e-429d-8dc1-2750240a5b30\") " pod="openshift-image-registry/image-registry-7c9c7469f5-nckkz"
Sep 18 19:14:27.801594 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.801578    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2ss8\" (UniqueName: \"kubernetes.io/projected/85a6cce8-df47-4abf-92c8-f6239d084c07-kube-api-access-x2ss8\") pod \"managed-serviceaccount-addon-agent-64fbd65ff6-m84r9\" (UID: \"85a6cce8-df47-4abf-92c8-f6239d084c07\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64fbd65ff6-m84r9"
Sep 18 19:14:27.812225 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.812205    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g445w\" (UniqueName: \"kubernetes.io/projected/776f6d09-2964-4c91-95fb-e7cdfec3e6ac-kube-api-access-g445w\") pod \"klusterlet-addon-workmgr-6dbcf5f8b8-7f6cw\" (UID: \"776f6d09-2964-4c91-95fb-e7cdfec3e6ac\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6dbcf5f8b8-7f6cw"
Sep 18 19:14:27.832930 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.832903    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764c5dfc6-m99t4"
Sep 18 19:14:27.840420 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.840398    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6dbcf5f8b8-7f6cw"
Sep 18 19:14:27.853865 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.853842    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64fbd65ff6-m84r9"
Sep 18 19:14:27.864660 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.864640    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd8vt\" (UniqueName: \"kubernetes.io/projected/ad80f379-b5dc-4d02-a61d-b8b2a956a551-kube-api-access-pd8vt\") pod \"dns-default-54cns\" (UID: \"ad80f379-b5dc-4d02-a61d-b8b2a956a551\") " pod="openshift-dns/dns-default-54cns"
Sep 18 19:14:27.864756 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.864670    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/35de2d74-739d-411b-ad9b-dc99018ad770-cert\") pod \"ingress-canary-wgxds\" (UID: \"35de2d74-739d-411b-ad9b-dc99018ad770\") " pod="openshift-ingress-canary/ingress-canary-wgxds"
Sep 18 19:14:27.864756 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.864716    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ad80f379-b5dc-4d02-a61d-b8b2a956a551-config-volume\") pod \"dns-default-54cns\" (UID: \"ad80f379-b5dc-4d02-a61d-b8b2a956a551\") " pod="openshift-dns/dns-default-54cns"
Sep 18 19:14:27.864868 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.864782    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad80f379-b5dc-4d02-a61d-b8b2a956a551-metrics-tls\") pod \"dns-default-54cns\" (UID: \"ad80f379-b5dc-4d02-a61d-b8b2a956a551\") " pod="openshift-dns/dns-default-54cns"
Sep 18 19:14:27.864868 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.864860    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad80f379-b5dc-4d02-a61d-b8b2a956a551-tmp-dir\") pod \"dns-default-54cns\" (UID: \"ad80f379-b5dc-4d02-a61d-b8b2a956a551\") " pod="openshift-dns/dns-default-54cns"
Sep 18 19:14:27.864961 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:27.864916    2532 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:14:27.865013 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.864917    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq6xw\" (UniqueName: \"kubernetes.io/projected/35de2d74-739d-411b-ad9b-dc99018ad770-kube-api-access-nq6xw\") pod \"ingress-canary-wgxds\" (UID: \"35de2d74-739d-411b-ad9b-dc99018ad770\") " pod="openshift-ingress-canary/ingress-canary-wgxds"
Sep 18 19:14:27.865013 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:27.864981    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad80f379-b5dc-4d02-a61d-b8b2a956a551-metrics-tls podName:ad80f379-b5dc-4d02-a61d-b8b2a956a551 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:28.364965433 +0000 UTC m=+35.285469281 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ad80f379-b5dc-4d02-a61d-b8b2a956a551-metrics-tls") pod "dns-default-54cns" (UID: "ad80f379-b5dc-4d02-a61d-b8b2a956a551") : secret "dns-default-metrics-tls" not found
Sep 18 19:14:27.865141 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.865119    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad80f379-b5dc-4d02-a61d-b8b2a956a551-tmp-dir\") pod \"dns-default-54cns\" (UID: \"ad80f379-b5dc-4d02-a61d-b8b2a956a551\") " pod="openshift-dns/dns-default-54cns"
Sep 18 19:14:27.865254 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.865234    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ad80f379-b5dc-4d02-a61d-b8b2a956a551-config-volume\") pod \"dns-default-54cns\" (UID: \"ad80f379-b5dc-4d02-a61d-b8b2a956a551\") " pod="openshift-dns/dns-default-54cns"
Sep 18 19:14:27.889412 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.889355    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd8vt\" (UniqueName: \"kubernetes.io/projected/ad80f379-b5dc-4d02-a61d-b8b2a956a551-kube-api-access-pd8vt\") pod \"dns-default-54cns\" (UID: \"ad80f379-b5dc-4d02-a61d-b8b2a956a551\") " pod="openshift-dns/dns-default-54cns"
Sep 18 19:14:27.966030 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.965962    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq6xw\" (UniqueName: \"kubernetes.io/projected/35de2d74-739d-411b-ad9b-dc99018ad770-kube-api-access-nq6xw\") pod \"ingress-canary-wgxds\" (UID: \"35de2d74-739d-411b-ad9b-dc99018ad770\") " pod="openshift-ingress-canary/ingress-canary-wgxds"
Sep 18 19:14:27.966030 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.966022    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/35de2d74-739d-411b-ad9b-dc99018ad770-cert\") pod \"ingress-canary-wgxds\" (UID: \"35de2d74-739d-411b-ad9b-dc99018ad770\") " pod="openshift-ingress-canary/ingress-canary-wgxds"
Sep 18 19:14:27.968989 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.968962    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/35de2d74-739d-411b-ad9b-dc99018ad770-cert\") pod \"ingress-canary-wgxds\" (UID: \"35de2d74-739d-411b-ad9b-dc99018ad770\") " pod="openshift-ingress-canary/ingress-canary-wgxds"
Sep 18 19:14:27.984378 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:27.984349    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq6xw\" (UniqueName: \"kubernetes.io/projected/35de2d74-739d-411b-ad9b-dc99018ad770-kube-api-access-nq6xw\") pod \"ingress-canary-wgxds\" (UID: \"35de2d74-739d-411b-ad9b-dc99018ad770\") " pod="openshift-ingress-canary/ingress-canary-wgxds"
Sep 18 19:14:28.037232 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:28.037192    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wgxds"
Sep 18 19:14:28.268112 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:28.268074    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8367afb3-7c5e-429d-8dc1-2750240a5b30-registry-tls\") pod \"image-registry-7c9c7469f5-nckkz\" (UID: \"8367afb3-7c5e-429d-8dc1-2750240a5b30\") " pod="openshift-image-registry/image-registry-7c9c7469f5-nckkz"
Sep 18 19:14:28.268295 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:28.268244    2532 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:14:28.268295 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:28.268264    2532 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c9c7469f5-nckkz: secret "image-registry-tls" not found
Sep 18 19:14:28.268397 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:28.268333    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8367afb3-7c5e-429d-8dc1-2750240a5b30-registry-tls podName:8367afb3-7c5e-429d-8dc1-2750240a5b30 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:29.268312397 +0000 UTC m=+36.188816235 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8367afb3-7c5e-429d-8dc1-2750240a5b30-registry-tls") pod "image-registry-7c9c7469f5-nckkz" (UID: "8367afb3-7c5e-429d-8dc1-2750240a5b30") : secret "image-registry-tls" not found
Sep 18 19:14:28.369529 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:28.369493    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad80f379-b5dc-4d02-a61d-b8b2a956a551-metrics-tls\") pod \"dns-default-54cns\" (UID: \"ad80f379-b5dc-4d02-a61d-b8b2a956a551\") " pod="openshift-dns/dns-default-54cns"
Sep 18 19:14:28.369711 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:28.369662    2532 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:14:28.369775 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:28.369737    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad80f379-b5dc-4d02-a61d-b8b2a956a551-metrics-tls podName:ad80f379-b5dc-4d02-a61d-b8b2a956a551 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:29.369716727 +0000 UTC m=+36.290220576 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ad80f379-b5dc-4d02-a61d-b8b2a956a551-metrics-tls") pod "dns-default-54cns" (UID: "ad80f379-b5dc-4d02-a61d-b8b2a956a551") : secret "dns-default-metrics-tls" not found
Sep 18 19:14:28.616173 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:28.616150    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ps7w2"
Sep 18 19:14:28.616567 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:28.616309    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d9rlj"
Sep 18 19:14:28.620250 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:28.620231    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 19:14:28.620352 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:28.620303    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-95w6x\""
Sep 18 19:14:28.620352 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:28.620237    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 19:14:28.791275 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:28.791200    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764c5dfc6-m99t4"]
Sep 18 19:14:28.793865 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:28.793816    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6dbcf5f8b8-7f6cw"]
Sep 18 19:14:28.794807 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:28.794789    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64fbd65ff6-m84r9"]
Sep 18 19:14:28.795576 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:28.795558    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wgxds"]
Sep 18 19:14:28.797522 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:14:28.797498    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod776f6d09_2964_4c91_95fb_e7cdfec3e6ac.slice/crio-056a9cd3a3b9de0747cc1425459f5f560b7bcb5b4a44bcea5e02b985774224aa WatchSource:0}: Error finding container 056a9cd3a3b9de0747cc1425459f5f560b7bcb5b4a44bcea5e02b985774224aa: Status 404 returned error can't find the container with id 056a9cd3a3b9de0747cc1425459f5f560b7bcb5b4a44bcea5e02b985774224aa
Sep 18 19:14:28.798376 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:14:28.798344    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35de2d74_739d_411b_ad9b_dc99018ad770.slice/crio-2401d7c08e4b7eb3d88c1628ca340218f01495f6e8844d1ad83e5cec08cdfbc3 WatchSource:0}: Error finding container 2401d7c08e4b7eb3d88c1628ca340218f01495f6e8844d1ad83e5cec08cdfbc3: Status 404 returned error can't find the container with id 2401d7c08e4b7eb3d88c1628ca340218f01495f6e8844d1ad83e5cec08cdfbc3
Sep 18 19:14:28.799093 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:14:28.799071    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a2aba53_8251_4204_9845_f5572077030b.slice/crio-f15474aa50ac1e0b4170868a1b4955349de9d94465103ed547d2880b106bcaf9 WatchSource:0}: Error finding container f15474aa50ac1e0b4170868a1b4955349de9d94465103ed547d2880b106bcaf9: Status 404 returned error can't find the container with id f15474aa50ac1e0b4170868a1b4955349de9d94465103ed547d2880b106bcaf9
Sep 18 19:14:28.800016 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:14:28.799959    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod85a6cce8_df47_4abf_92c8_f6239d084c07.slice/crio-2d1f211158e144a4cc30b687245d0c946b3dc33ffd7744f3234cb4bfa28b01e7 WatchSource:0}: Error finding container 2d1f211158e144a4cc30b687245d0c946b3dc33ffd7744f3234cb4bfa28b01e7: Status 404 returned error can't find the container with id 2d1f211158e144a4cc30b687245d0c946b3dc33ffd7744f3234cb4bfa28b01e7
Sep 18 19:14:28.805908 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:28.805882    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64fbd65ff6-m84r9" event={"ID":"85a6cce8-df47-4abf-92c8-f6239d084c07","Type":"ContainerStarted","Data":"2d1f211158e144a4cc30b687245d0c946b3dc33ffd7744f3234cb4bfa28b01e7"}
Sep 18 19:14:28.807551 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:28.807503    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6dbcf5f8b8-7f6cw" event={"ID":"776f6d09-2964-4c91-95fb-e7cdfec3e6ac","Type":"ContainerStarted","Data":"056a9cd3a3b9de0747cc1425459f5f560b7bcb5b4a44bcea5e02b985774224aa"}
Sep 18 19:14:28.810850 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:28.810819    2532 generic.go:356] "Generic (PLEG): container finished" podID="7959688f-be33-494c-a8e9-08d33f2c2d3b" containerID="5009c622a2c4cd72ed1a43ae0032328f6be7452f24d66bdbe252f0846f3fd800" exitCode=0
Sep 18 19:14:28.810956 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:28.810924    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rc579" event={"ID":"7959688f-be33-494c-a8e9-08d33f2c2d3b","Type":"ContainerDied","Data":"5009c622a2c4cd72ed1a43ae0032328f6be7452f24d66bdbe252f0846f3fd800"}
Sep 18 19:14:28.812272 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:28.812251    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wgxds" event={"ID":"35de2d74-739d-411b-ad9b-dc99018ad770","Type":"ContainerStarted","Data":"2401d7c08e4b7eb3d88c1628ca340218f01495f6e8844d1ad83e5cec08cdfbc3"}
Sep 18 19:14:28.813972 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:28.813939    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764c5dfc6-m99t4" event={"ID":"0a2aba53-8251-4204-9845-f5572077030b","Type":"ContainerStarted","Data":"f15474aa50ac1e0b4170868a1b4955349de9d94465103ed547d2880b106bcaf9"}
Sep 18 19:14:29.278258 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:29.278039    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8367afb3-7c5e-429d-8dc1-2750240a5b30-registry-tls\") pod \"image-registry-7c9c7469f5-nckkz\" (UID: \"8367afb3-7c5e-429d-8dc1-2750240a5b30\") " pod="openshift-image-registry/image-registry-7c9c7469f5-nckkz"
Sep 18 19:14:29.278444 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:29.278214    2532 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:14:29.278444 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:29.278294    2532 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c9c7469f5-nckkz: secret "image-registry-tls" not found
Sep 18 19:14:29.278444 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:29.278342    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8367afb3-7c5e-429d-8dc1-2750240a5b30-registry-tls podName:8367afb3-7c5e-429d-8dc1-2750240a5b30 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:31.278327974 +0000 UTC m=+38.198831822 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8367afb3-7c5e-429d-8dc1-2750240a5b30-registry-tls") pod "image-registry-7c9c7469f5-nckkz" (UID: "8367afb3-7c5e-429d-8dc1-2750240a5b30") : secret "image-registry-tls" not found
Sep 18 19:14:29.378831 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:29.378792    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad80f379-b5dc-4d02-a61d-b8b2a956a551-metrics-tls\") pod \"dns-default-54cns\" (UID: \"ad80f379-b5dc-4d02-a61d-b8b2a956a551\") " pod="openshift-dns/dns-default-54cns"
Sep 18 19:14:29.378972 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:29.378919    2532 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:14:29.379012 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:29.378988    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad80f379-b5dc-4d02-a61d-b8b2a956a551-metrics-tls podName:ad80f379-b5dc-4d02-a61d-b8b2a956a551 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:31.378965493 +0000 UTC m=+38.299469332 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ad80f379-b5dc-4d02-a61d-b8b2a956a551-metrics-tls") pod "dns-default-54cns" (UID: "ad80f379-b5dc-4d02-a61d-b8b2a956a551") : secret "dns-default-metrics-tls" not found
Sep 18 19:14:29.825519 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:29.825483    2532 generic.go:356] "Generic (PLEG): container finished" podID="7959688f-be33-494c-a8e9-08d33f2c2d3b" containerID="af388e7491f0599ae2012d499bf3acc43feb3e8dc34634d99af6895a95cd30b9" exitCode=0
Sep 18 19:14:29.826028 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:29.825563    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rc579" event={"ID":"7959688f-be33-494c-a8e9-08d33f2c2d3b","Type":"ContainerDied","Data":"af388e7491f0599ae2012d499bf3acc43feb3e8dc34634d99af6895a95cd30b9"}
Sep 18 19:14:30.851980 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:30.851207    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rc579" event={"ID":"7959688f-be33-494c-a8e9-08d33f2c2d3b","Type":"ContainerStarted","Data":"e26b21fe4d29437d0d041086068fa89b38db178733538cee362b30ff81737b15"}
Sep 18 19:14:30.907929 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:30.907870    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-rc579" podStartSLOduration=7.78535869 podStartE2EDuration="36.907848394s" totalImagesPullingTime="29.122489704s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:13:54 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:30.907595021 +0000 UTC m=+37.828098882" watchObservedRunningTime="2026-09-18 19:14:30.907848394 +0000 UTC m=+37.828352252"
Sep 18 19:14:31.157500 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:31.157462    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:14:31.288994 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:31.288943    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:14:31.295916 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:31.295883    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8367afb3-7c5e-429d-8dc1-2750240a5b30-registry-tls\") pod \"image-registry-7c9c7469f5-nckkz\" (UID: \"8367afb3-7c5e-429d-8dc1-2750240a5b30\") " pod="openshift-image-registry/image-registry-7c9c7469f5-nckkz"
Sep 18 19:14:31.296119 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:31.296084    2532 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:14:31.296119 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:31.296105    2532 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c9c7469f5-nckkz: secret "image-registry-tls" not found
Sep 18 19:14:31.296238 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:31.296162    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8367afb3-7c5e-429d-8dc1-2750240a5b30-registry-tls podName:8367afb3-7c5e-429d-8dc1-2750240a5b30 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:35.296143924 +0000 UTC m=+42.216647763 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8367afb3-7c5e-429d-8dc1-2750240a5b30-registry-tls") pod "image-registry-7c9c7469f5-nckkz" (UID: "8367afb3-7c5e-429d-8dc1-2750240a5b30") : secret "image-registry-tls" not found
Sep 18 19:14:31.397367 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:31.397279    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad80f379-b5dc-4d02-a61d-b8b2a956a551-metrics-tls\") pod \"dns-default-54cns\" (UID: \"ad80f379-b5dc-4d02-a61d-b8b2a956a551\") " pod="openshift-dns/dns-default-54cns"
Sep 18 19:14:31.397570 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:31.397429    2532 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:14:31.397570 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:31.397508    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad80f379-b5dc-4d02-a61d-b8b2a956a551-metrics-tls podName:ad80f379-b5dc-4d02-a61d-b8b2a956a551 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:35.397487383 +0000 UTC m=+42.317991231 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ad80f379-b5dc-4d02-a61d-b8b2a956a551-metrics-tls") pod "dns-default-54cns" (UID: "ad80f379-b5dc-4d02-a61d-b8b2a956a551") : secret "dns-default-metrics-tls" not found
Sep 18 19:14:32.863500 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:32.863444    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:14:32.952187 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:32.952150    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:14:33.911978 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:33.911942    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:14:34.075114 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:34.074992    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:14:34.287253 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:34.287209    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:14:34.450716 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:34.450679    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:14:34.667373 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:34.667329    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:14:34.851518 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:34.851411    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:14:34.952761 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:34.952721    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:14:35.102985 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:35.102890    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:14:35.193884 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:35.193825    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:14:35.331403 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:35.331370    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8367afb3-7c5e-429d-8dc1-2750240a5b30-registry-tls\") pod \"image-registry-7c9c7469f5-nckkz\" (UID: \"8367afb3-7c5e-429d-8dc1-2750240a5b30\") " pod="openshift-image-registry/image-registry-7c9c7469f5-nckkz"
Sep 18 19:14:35.331561 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:35.331514    2532 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:14:35.331561 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:35.331530    2532 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c9c7469f5-nckkz: secret "image-registry-tls" not found
Sep 18 19:14:35.331673 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:35.331586    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8367afb3-7c5e-429d-8dc1-2750240a5b30-registry-tls podName:8367afb3-7c5e-429d-8dc1-2750240a5b30 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:43.331567734 +0000 UTC m=+50.252071576 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8367afb3-7c5e-429d-8dc1-2750240a5b30-registry-tls") pod "image-registry-7c9c7469f5-nckkz" (UID: "8367afb3-7c5e-429d-8dc1-2750240a5b30") : secret "image-registry-tls" not found
Sep 18 19:14:35.432263 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:35.432233    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad80f379-b5dc-4d02-a61d-b8b2a956a551-metrics-tls\") pod \"dns-default-54cns\" (UID: \"ad80f379-b5dc-4d02-a61d-b8b2a956a551\") " pod="openshift-dns/dns-default-54cns"
Sep 18 19:14:35.432403 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:35.432384    2532 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:14:35.432461 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:35.432450    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad80f379-b5dc-4d02-a61d-b8b2a956a551-metrics-tls podName:ad80f379-b5dc-4d02-a61d-b8b2a956a551 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:43.432430005 +0000 UTC m=+50.352933839 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ad80f379-b5dc-4d02-a61d-b8b2a956a551-metrics-tls") pod "dns-default-54cns" (UID: "ad80f379-b5dc-4d02-a61d-b8b2a956a551") : secret "dns-default-metrics-tls" not found
Sep 18 19:14:35.449610 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:35.449581    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:14:35.593749 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:35.593719    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:14:37.872560 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:37.872459    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6dbcf5f8b8-7f6cw" event={"ID":"776f6d09-2964-4c91-95fb-e7cdfec3e6ac","Type":"ContainerStarted","Data":"2151283f8d97fe44e93c0cdad8e9aa11618800f9aeb15d7a17a10c26d4717205"}
Sep 18 19:14:37.873024 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:37.872584    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6dbcf5f8b8-7f6cw"
Sep 18 19:14:37.874135 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:37.874109    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wgxds" event={"ID":"35de2d74-739d-411b-ad9b-dc99018ad770","Type":"ContainerStarted","Data":"d0dbb3996b2e0f62117f7bf0d0f1e0f86735b10b023591022da0bbe117bfe1e5"}
Sep 18 19:14:37.874559 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:37.874540    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6dbcf5f8b8-7f6cw"
Sep 18 19:14:37.876211 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:37.876183    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764c5dfc6-m99t4" event={"ID":"0a2aba53-8251-4204-9845-f5572077030b","Type":"ContainerStarted","Data":"99f911254e73e4264b4e3b8aaf6773d621022971d4d829f1e090f26f26d91447"}
Sep 18 19:14:37.876308 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:37.876215    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764c5dfc6-m99t4" event={"ID":"0a2aba53-8251-4204-9845-f5572077030b","Type":"ContainerStarted","Data":"2737ea34ab75428e72d99e6e2261bcbb5ae95dc54a0aec7db72d60f25e546168"}
Sep 18 19:14:37.876308 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:37.876232    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764c5dfc6-m99t4" event={"ID":"0a2aba53-8251-4204-9845-f5572077030b","Type":"ContainerStarted","Data":"95f07e62ee89f09244ec060c91ee46c1f7489fff9d33d505e44e56153c34ffdc"}
Sep 18 19:14:37.877449 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:37.877431    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64fbd65ff6-m84r9" event={"ID":"85a6cce8-df47-4abf-92c8-f6239d084c07","Type":"ContainerStarted","Data":"369828b71ee0c8f67d31fa1d7f19213b5484b99d1ea13e9f212bad8a8fa0395b"}
Sep 18 19:14:37.970330 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:37.970284    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6dbcf5f8b8-7f6cw" podStartSLOduration=19.536423967 podStartE2EDuration="27.97026954s" totalImagesPullingTime="8.433845573s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:14:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:37.928179952 +0000 UTC m=+44.848683820" watchObservedRunningTime="2026-09-18 19:14:37.97026954 +0000 UTC m=+44.890773397"
Sep 18 19:14:38.018949 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:38.018906    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64fbd65ff6-m84r9" podStartSLOduration=19.587265903 podStartE2EDuration="28.018892317s" totalImagesPullingTime="8.431626414s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:14:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:38.015788855 +0000 UTC m=+44.936292712" watchObservedRunningTime="2026-09-18 19:14:38.018892317 +0000 UTC m=+44.939396171"
Sep 18 19:14:38.070559 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:38.070512    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-wgxds" podStartSLOduration=2.6553360919999998 podStartE2EDuration="11.070496659s" totalImagesPullingTime="8.415160567s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:14:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:38.070414954 +0000 UTC m=+44.990918811" watchObservedRunningTime="2026-09-18 19:14:38.070496659 +0000 UTC m=+44.991000515"
Sep 18 19:14:38.114753 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:38.114707    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764c5dfc6-m99t4" podStartSLOduration=19.664045463 podStartE2EDuration="28.114690322s" totalImagesPullingTime="8.450644859s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:14:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:38.114051958 +0000 UTC m=+45.034555814" watchObservedRunningTime="2026-09-18 19:14:38.114690322 +0000 UTC m=+45.035194172"
Sep 18 19:14:40.066703 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:40.066662    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:14:40.229917 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:40.229877    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:14:40.364950 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:40.364873    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:14:40.488694 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:40.488660    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:14:40.607309 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:40.607277    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:14:40.705494 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:40.705458    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:14:43.337972 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:43.337938    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:14:43.390211 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:43.390180    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8367afb3-7c5e-429d-8dc1-2750240a5b30-registry-tls\") pod \"image-registry-7c9c7469f5-nckkz\" (UID: \"8367afb3-7c5e-429d-8dc1-2750240a5b30\") " pod="openshift-image-registry/image-registry-7c9c7469f5-nckkz"
Sep 18 19:14:43.390343 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:43.390328    2532 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:14:43.390392 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:43.390345    2532 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c9c7469f5-nckkz: secret "image-registry-tls" not found
Sep 18 19:14:43.390426 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:43.390410    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8367afb3-7c5e-429d-8dc1-2750240a5b30-registry-tls podName:8367afb3-7c5e-429d-8dc1-2750240a5b30 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:59.390394978 +0000 UTC m=+66.310898812 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8367afb3-7c5e-429d-8dc1-2750240a5b30-registry-tls") pod "image-registry-7c9c7469f5-nckkz" (UID: "8367afb3-7c5e-429d-8dc1-2750240a5b30") : secret "image-registry-tls" not found
Sep 18 19:14:43.491248 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:43.491220    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad80f379-b5dc-4d02-a61d-b8b2a956a551-metrics-tls\") pod \"dns-default-54cns\" (UID: \"ad80f379-b5dc-4d02-a61d-b8b2a956a551\") " pod="openshift-dns/dns-default-54cns"
Sep 18 19:14:43.491385 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:43.491333    2532 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:14:43.491424 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:43.491395    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad80f379-b5dc-4d02-a61d-b8b2a956a551-metrics-tls podName:ad80f379-b5dc-4d02-a61d-b8b2a956a551 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:59.491383276 +0000 UTC m=+66.411887112 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ad80f379-b5dc-4d02-a61d-b8b2a956a551-metrics-tls") pod "dns-default-54cns" (UID: "ad80f379-b5dc-4d02-a61d-b8b2a956a551") : secret "dns-default-metrics-tls" not found
Sep 18 19:14:44.887719 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:44.887684    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:14:51.796684 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:51.796652    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sksw8"
Sep 18 19:14:52.058023 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:52.057947    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:14:52.158645 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:52.158619    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:14:52.235521 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:52.235502    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:14:55.114559 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:55.114520    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:14:55.216398 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:55.216363    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:14:56.083502 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:56.083469    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:14:56.186139 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:56.186104    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:14:56.293015 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:56.292988    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:14:59.300879 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:59.300842    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed-original-pull-secret\") pod \"global-pull-secret-syncer-d9rlj\" (UID: \"7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed\") " pod="kube-system/global-pull-secret-syncer-d9rlj"
Sep 18 19:14:59.304015 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:59.303995    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 19:14:59.315234 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:59.315207    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed-original-pull-secret\") pod \"global-pull-secret-syncer-d9rlj\" (UID: \"7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed\") " pod="kube-system/global-pull-secret-syncer-d9rlj"
Sep 18 19:14:59.402198 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:59.402162    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05c957d5-da77-47c2-8b9c-e7cdc087120c-metrics-certs\") pod \"network-metrics-daemon-ps7w2\" (UID: \"05c957d5-da77-47c2-8b9c-e7cdc087120c\") " pod="openshift-multus/network-metrics-daemon-ps7w2"
Sep 18 19:14:59.402347 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:59.402226    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8367afb3-7c5e-429d-8dc1-2750240a5b30-registry-tls\") pod \"image-registry-7c9c7469f5-nckkz\" (UID: \"8367afb3-7c5e-429d-8dc1-2750240a5b30\") " pod="openshift-image-registry/image-registry-7c9c7469f5-nckkz"
Sep 18 19:14:59.402347 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:59.402321    2532 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:14:59.402347 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:59.402331    2532 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c9c7469f5-nckkz: secret "image-registry-tls" not found
Sep 18 19:14:59.402469 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:59.402387    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8367afb3-7c5e-429d-8dc1-2750240a5b30-registry-tls podName:8367afb3-7c5e-429d-8dc1-2750240a5b30 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:31.402373341 +0000 UTC m=+98.322877176 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8367afb3-7c5e-429d-8dc1-2750240a5b30-registry-tls") pod "image-registry-7c9c7469f5-nckkz" (UID: "8367afb3-7c5e-429d-8dc1-2750240a5b30") : secret "image-registry-tls" not found
Sep 18 19:14:59.405009 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:59.404987    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 19:14:59.413198 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:59.413182    2532 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 18 19:14:59.413263 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:59.413227    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05c957d5-da77-47c2-8b9c-e7cdc087120c-metrics-certs podName:05c957d5-da77-47c2-8b9c-e7cdc087120c nodeName:}" failed. No retries permitted until 2026-09-18 19:16:03.413213811 +0000 UTC m=+130.333717658 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05c957d5-da77-47c2-8b9c-e7cdc087120c-metrics-certs") pod "network-metrics-daemon-ps7w2" (UID: "05c957d5-da77-47c2-8b9c-e7cdc087120c") : secret "metrics-daemon-secret" not found
Sep 18 19:14:59.503270 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:59.503246    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b7hs\" (UniqueName: \"kubernetes.io/projected/7082df75-a3d1-4eba-90e5-7a02a6aef94b-kube-api-access-6b7hs\") pod \"network-check-target-42kxn\" (UID: \"7082df75-a3d1-4eba-90e5-7a02a6aef94b\") " pod="openshift-network-diagnostics/network-check-target-42kxn"
Sep 18 19:14:59.503361 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:59.503276    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad80f379-b5dc-4d02-a61d-b8b2a956a551-metrics-tls\") pod \"dns-default-54cns\" (UID: \"ad80f379-b5dc-4d02-a61d-b8b2a956a551\") " pod="openshift-dns/dns-default-54cns"
Sep 18 19:14:59.503402 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:59.503370    2532 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:14:59.503434 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:14:59.503410    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad80f379-b5dc-4d02-a61d-b8b2a956a551-metrics-tls podName:ad80f379-b5dc-4d02-a61d-b8b2a956a551 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:31.503397544 +0000 UTC m=+98.423901396 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ad80f379-b5dc-4d02-a61d-b8b2a956a551-metrics-tls") pod "dns-default-54cns" (UID: "ad80f379-b5dc-4d02-a61d-b8b2a956a551") : secret "dns-default-metrics-tls" not found
Sep 18 19:14:59.506033 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:59.506019    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 19:14:59.516539 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:59.516525    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 19:14:59.526779 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:59.526760    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b7hs\" (UniqueName: \"kubernetes.io/projected/7082df75-a3d1-4eba-90e5-7a02a6aef94b-kube-api-access-6b7hs\") pod \"network-check-target-42kxn\" (UID: \"7082df75-a3d1-4eba-90e5-7a02a6aef94b\") " pod="openshift-network-diagnostics/network-check-target-42kxn"
Sep 18 19:14:59.541163 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:59.541140    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d9rlj"
Sep 18 19:14:59.742545 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:59.742510    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lf5lk\""
Sep 18 19:14:59.749729 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:59.749709    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42kxn"
Sep 18 19:14:59.845752 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:59.845696    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-d9rlj"]
Sep 18 19:14:59.852737 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:14:59.852711    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cae3b8f_84ca_4cb2_9003_4f38a71ab6ed.slice/crio-e71010317d890af9bcbc24158d203283dcc3a258242c28210109ef0e705e1106 WatchSource:0}: Error finding container e71010317d890af9bcbc24158d203283dcc3a258242c28210109ef0e705e1106: Status 404 returned error can't find the container with id e71010317d890af9bcbc24158d203283dcc3a258242c28210109ef0e705e1106
Sep 18 19:14:59.942435 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:59.942409    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:14:59.943958 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:14:59.943937    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-d9rlj" event={"ID":"7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed","Type":"ContainerStarted","Data":"e71010317d890af9bcbc24158d203283dcc3a258242c28210109ef0e705e1106"}
Sep 18 19:15:00.044873 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:00.044847    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-42kxn"]
Sep 18 19:15:00.047963 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:15:00.047943    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7082df75_a3d1_4eba_90e5_7a02a6aef94b.slice/crio-4c6134ab50f40612d83e064af733429b307d01594edb343ed2c69f52bd5624f0 WatchSource:0}: Error finding container 4c6134ab50f40612d83e064af733429b307d01594edb343ed2c69f52bd5624f0: Status 404 returned error can't find the container with id 4c6134ab50f40612d83e064af733429b307d01594edb343ed2c69f52bd5624f0
Sep 18 19:15:00.938131 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:00.938091    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:00.950030 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:00.949997    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-42kxn" event={"ID":"7082df75-a3d1-4eba-90e5-7a02a6aef94b","Type":"ContainerStarted","Data":"4c6134ab50f40612d83e064af733429b307d01594edb343ed2c69f52bd5624f0"}
Sep 18 19:15:01.038330 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:01.038280    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:01.145155 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:01.145046    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:02.086454 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:02.086421    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:02.183188 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:02.183118    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:04.965112 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:04.965074    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-d9rlj" event={"ID":"7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed","Type":"ContainerStarted","Data":"d18cbe756c0d2baa5375af1064aab493cefe19c3ae7ed1fdcebdae5035c8e081"}
Sep 18 19:15:04.966578 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:04.966558    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-42kxn" event={"ID":"7082df75-a3d1-4eba-90e5-7a02a6aef94b","Type":"ContainerStarted","Data":"a321ffc6f4273f751f7c32bf34ee25677d0bd02ef1d9350a96bbd5ef1b479454"}
Sep 18 19:15:04.966679 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:04.966613    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-42kxn"
Sep 18 19:15:04.989447 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:04.989401    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-d9rlj" podStartSLOduration=66.415044238 podStartE2EDuration="1m10.989385206s" totalImagesPullingTime="4.574340968s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:13:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:15:04.987142009 +0000 UTC m=+71.907645878" watchObservedRunningTime="2026-09-18 19:15:04.989385206 +0000 UTC m=+71.909889062"
Sep 18 19:15:05.024167 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:05.024115    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-42kxn" podStartSLOduration=66.649470143 podStartE2EDuration="1m11.024097453s" totalImagesPullingTime="4.37462731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:13:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:15:05.02377217 +0000 UTC m=+71.944276027" watchObservedRunningTime="2026-09-18 19:15:05.024097453 +0000 UTC m=+71.944601309"
Sep 18 19:15:06.013353 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:06.013312    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:06.882374 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:06.882335    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:11.011811 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:11.011775    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:11.079284 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:11.079251    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:11.152526 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:11.152488    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:11.943745 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:11.943710    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:12.007149 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:12.007121    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:13.241043 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:13.241012    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:17.056410 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:17.056282    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:17.179507 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:17.179473    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:17.250656 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:17.250627    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:17.367456 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:17.367371    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:18.430654 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:18.430617    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:18.499462 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:18.499426    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:24.604963 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:24.604926    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:24.713119 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:24.713083    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:27.012149 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:27.012120    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:27.074507 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:27.074468    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:27.723223 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:27.723195    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:27.824382 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:27.824349    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:29.111160 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:29.111123    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-tz4cf"]
Sep 18 19:15:29.115668 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:29.115643    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tz4cf"
Sep 18 19:15:29.120237 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:29.118805    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 18 19:15:29.120237 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:29.119418    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 18 19:15:29.120237 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:29.119498    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 18 19:15:29.120237 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:29.119910    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 18 19:15:29.120237 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:29.119974    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wlcxl\""
Sep 18 19:15:29.125250 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:29.125217    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:29.134025 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:29.133999    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tz4cf"]
Sep 18 19:15:29.210587 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:29.210559    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3cd69cee-5d0e-41d9-8dc9-a0fb2898085d-crio-socket\") pod \"insights-runtime-extractor-tz4cf\" (UID: \"3cd69cee-5d0e-41d9-8dc9-a0fb2898085d\") " pod="openshift-insights/insights-runtime-extractor-tz4cf"
Sep 18 19:15:29.210681 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:29.210599    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmhzc\" (UniqueName: \"kubernetes.io/projected/3cd69cee-5d0e-41d9-8dc9-a0fb2898085d-kube-api-access-jmhzc\") pod \"insights-runtime-extractor-tz4cf\" (UID: \"3cd69cee-5d0e-41d9-8dc9-a0fb2898085d\") " pod="openshift-insights/insights-runtime-extractor-tz4cf"
Sep 18 19:15:29.210681 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:29.210666    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3cd69cee-5d0e-41d9-8dc9-a0fb2898085d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tz4cf\" (UID: \"3cd69cee-5d0e-41d9-8dc9-a0fb2898085d\") " pod="openshift-insights/insights-runtime-extractor-tz4cf"
Sep 18 19:15:29.210806 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:29.210742    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3cd69cee-5d0e-41d9-8dc9-a0fb2898085d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tz4cf\" (UID: \"3cd69cee-5d0e-41d9-8dc9-a0fb2898085d\") " pod="openshift-insights/insights-runtime-extractor-tz4cf"
Sep 18 19:15:29.210806 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:29.210777    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3cd69cee-5d0e-41d9-8dc9-a0fb2898085d-data-volume\") pod \"insights-runtime-extractor-tz4cf\" (UID: \"3cd69cee-5d0e-41d9-8dc9-a0fb2898085d\") " pod="openshift-insights/insights-runtime-extractor-tz4cf"
Sep 18 19:15:29.235190 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:29.235168    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:29.311472 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:29.311445    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3cd69cee-5d0e-41d9-8dc9-a0fb2898085d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tz4cf\" (UID: \"3cd69cee-5d0e-41d9-8dc9-a0fb2898085d\") " pod="openshift-insights/insights-runtime-extractor-tz4cf"
Sep 18 19:15:29.311550 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:29.311477    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3cd69cee-5d0e-41d9-8dc9-a0fb2898085d-data-volume\") pod \"insights-runtime-extractor-tz4cf\" (UID: \"3cd69cee-5d0e-41d9-8dc9-a0fb2898085d\") " pod="openshift-insights/insights-runtime-extractor-tz4cf"
Sep 18 19:15:29.311550 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:29.311499    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3cd69cee-5d0e-41d9-8dc9-a0fb2898085d-crio-socket\") pod \"insights-runtime-extractor-tz4cf\" (UID: \"3cd69cee-5d0e-41d9-8dc9-a0fb2898085d\") " pod="openshift-insights/insights-runtime-extractor-tz4cf"
Sep 18 19:15:29.311550 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:29.311520    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmhzc\" (UniqueName: \"kubernetes.io/projected/3cd69cee-5d0e-41d9-8dc9-a0fb2898085d-kube-api-access-jmhzc\") pod \"insights-runtime-extractor-tz4cf\" (UID: \"3cd69cee-5d0e-41d9-8dc9-a0fb2898085d\") " pod="openshift-insights/insights-runtime-extractor-tz4cf"
Sep 18 19:15:29.311709 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:29.311562    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3cd69cee-5d0e-41d9-8dc9-a0fb2898085d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tz4cf\" (UID: \"3cd69cee-5d0e-41d9-8dc9-a0fb2898085d\") " pod="openshift-insights/insights-runtime-extractor-tz4cf"
Sep 18 19:15:29.311709 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:15:29.311585    2532 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:15:29.311709 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:29.311599    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3cd69cee-5d0e-41d9-8dc9-a0fb2898085d-crio-socket\") pod \"insights-runtime-extractor-tz4cf\" (UID: \"3cd69cee-5d0e-41d9-8dc9-a0fb2898085d\") " pod="openshift-insights/insights-runtime-extractor-tz4cf"
Sep 18 19:15:29.311709 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:15:29.311646    2532 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:15:29.311709 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:15:29.311653    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cd69cee-5d0e-41d9-8dc9-a0fb2898085d-insights-runtime-extractor-tls podName:3cd69cee-5d0e-41d9-8dc9-a0fb2898085d nodeName:}" failed. No retries permitted until 2026-09-18 19:15:29.8116345 +0000 UTC m=+96.732138335 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3cd69cee-5d0e-41d9-8dc9-a0fb2898085d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tz4cf" (UID: "3cd69cee-5d0e-41d9-8dc9-a0fb2898085d") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:15:29.311709 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:15:29.311698    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3cd69cee-5d0e-41d9-8dc9-a0fb2898085d-kube-rbac-proxy-cm podName:3cd69cee-5d0e-41d9-8dc9-a0fb2898085d nodeName:}" failed. No retries permitted until 2026-09-18 19:15:29.811687365 +0000 UTC m=+96.732191200 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3cd69cee-5d0e-41d9-8dc9-a0fb2898085d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tz4cf" (UID: "3cd69cee-5d0e-41d9-8dc9-a0fb2898085d") : configmap "kube-rbac-proxy" not found
Sep 18 19:15:29.312008 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:29.311793    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3cd69cee-5d0e-41d9-8dc9-a0fb2898085d-data-volume\") pod \"insights-runtime-extractor-tz4cf\" (UID: \"3cd69cee-5d0e-41d9-8dc9-a0fb2898085d\") " pod="openshift-insights/insights-runtime-extractor-tz4cf"
Sep 18 19:15:29.312576 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:29.312551    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:29.322032 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:29.322012    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmhzc\" (UniqueName: \"kubernetes.io/projected/3cd69cee-5d0e-41d9-8dc9-a0fb2898085d-kube-api-access-jmhzc\") pod \"insights-runtime-extractor-tz4cf\" (UID: \"3cd69cee-5d0e-41d9-8dc9-a0fb2898085d\") " pod="openshift-insights/insights-runtime-extractor-tz4cf"
Sep 18 19:15:29.421937 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:29.421882    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:29.494735 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:29.494711    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:29.816106 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:29.816080    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3cd69cee-5d0e-41d9-8dc9-a0fb2898085d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tz4cf\" (UID: \"3cd69cee-5d0e-41d9-8dc9-a0fb2898085d\") " pod="openshift-insights/insights-runtime-extractor-tz4cf"
Sep 18 19:15:29.816246 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:29.816139    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3cd69cee-5d0e-41d9-8dc9-a0fb2898085d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tz4cf\" (UID: \"3cd69cee-5d0e-41d9-8dc9-a0fb2898085d\") " pod="openshift-insights/insights-runtime-extractor-tz4cf"
Sep 18 19:15:29.816246 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:15:29.816183    2532 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:15:29.816246 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:15:29.816244    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3cd69cee-5d0e-41d9-8dc9-a0fb2898085d-kube-rbac-proxy-cm podName:3cd69cee-5d0e-41d9-8dc9-a0fb2898085d nodeName:}" failed. No retries permitted until 2026-09-18 19:15:30.816226974 +0000 UTC m=+97.736730826 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3cd69cee-5d0e-41d9-8dc9-a0fb2898085d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tz4cf" (UID: "3cd69cee-5d0e-41d9-8dc9-a0fb2898085d") : configmap "kube-rbac-proxy" not found
Sep 18 19:15:29.816362 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:15:29.816252    2532 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:15:29.816362 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:15:29.816295    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cd69cee-5d0e-41d9-8dc9-a0fb2898085d-insights-runtime-extractor-tls podName:3cd69cee-5d0e-41d9-8dc9-a0fb2898085d nodeName:}" failed. No retries permitted until 2026-09-18 19:15:30.816283584 +0000 UTC m=+97.736787419 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3cd69cee-5d0e-41d9-8dc9-a0fb2898085d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tz4cf" (UID: "3cd69cee-5d0e-41d9-8dc9-a0fb2898085d") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:15:29.843763 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:29.843739    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:29.944473 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:29.944436    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:30.120515 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:30.120455    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:30.220750 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:30.220727    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:30.824084 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:30.824036    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3cd69cee-5d0e-41d9-8dc9-a0fb2898085d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tz4cf\" (UID: \"3cd69cee-5d0e-41d9-8dc9-a0fb2898085d\") " pod="openshift-insights/insights-runtime-extractor-tz4cf"
Sep 18 19:15:30.824187 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:30.824113    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3cd69cee-5d0e-41d9-8dc9-a0fb2898085d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tz4cf\" (UID: \"3cd69cee-5d0e-41d9-8dc9-a0fb2898085d\") " pod="openshift-insights/insights-runtime-extractor-tz4cf"
Sep 18 19:15:30.824187 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:15:30.824155    2532 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:15:30.824252 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:15:30.824207    2532 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:15:30.824252 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:15:30.824228    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3cd69cee-5d0e-41d9-8dc9-a0fb2898085d-kube-rbac-proxy-cm podName:3cd69cee-5d0e-41d9-8dc9-a0fb2898085d nodeName:}" failed. No retries permitted until 2026-09-18 19:15:32.824213327 +0000 UTC m=+99.744717161 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3cd69cee-5d0e-41d9-8dc9-a0fb2898085d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tz4cf" (UID: "3cd69cee-5d0e-41d9-8dc9-a0fb2898085d") : configmap "kube-rbac-proxy" not found
Sep 18 19:15:30.824252 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:15:30.824246    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cd69cee-5d0e-41d9-8dc9-a0fb2898085d-insights-runtime-extractor-tls podName:3cd69cee-5d0e-41d9-8dc9-a0fb2898085d nodeName:}" failed. No retries permitted until 2026-09-18 19:15:32.824235081 +0000 UTC m=+99.744738949 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3cd69cee-5d0e-41d9-8dc9-a0fb2898085d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tz4cf" (UID: "3cd69cee-5d0e-41d9-8dc9-a0fb2898085d") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:15:31.015043 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:31.014969    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qbs4d_851825cd-f968-471c-87c0-b7d2423c9c99/dns-node-resolver/0.log"
Sep 18 19:15:31.429403 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:31.429381    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8367afb3-7c5e-429d-8dc1-2750240a5b30-registry-tls\") pod \"image-registry-7c9c7469f5-nckkz\" (UID: \"8367afb3-7c5e-429d-8dc1-2750240a5b30\") " pod="openshift-image-registry/image-registry-7c9c7469f5-nckkz"
Sep 18 19:15:31.429742 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:15:31.429502    2532 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:15:31.429742 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:15:31.429515    2532 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c9c7469f5-nckkz: secret "image-registry-tls" not found
Sep 18 19:15:31.429742 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:15:31.429563    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8367afb3-7c5e-429d-8dc1-2750240a5b30-registry-tls podName:8367afb3-7c5e-429d-8dc1-2750240a5b30 nodeName:}" failed. No retries permitted until 2026-09-18 19:16:35.429550132 +0000 UTC m=+162.350053967 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8367afb3-7c5e-429d-8dc1-2750240a5b30-registry-tls") pod "image-registry-7c9c7469f5-nckkz" (UID: "8367afb3-7c5e-429d-8dc1-2750240a5b30") : secret "image-registry-tls" not found
Sep 18 19:15:31.530647 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:31.530616    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad80f379-b5dc-4d02-a61d-b8b2a956a551-metrics-tls\") pod \"dns-default-54cns\" (UID: \"ad80f379-b5dc-4d02-a61d-b8b2a956a551\") " pod="openshift-dns/dns-default-54cns"
Sep 18 19:15:31.530801 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:15:31.530750    2532 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:15:31.530851 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:15:31.530807    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad80f379-b5dc-4d02-a61d-b8b2a956a551-metrics-tls podName:ad80f379-b5dc-4d02-a61d-b8b2a956a551 nodeName:}" failed. No retries permitted until 2026-09-18 19:16:35.530793143 +0000 UTC m=+162.451296995 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ad80f379-b5dc-4d02-a61d-b8b2a956a551-metrics-tls") pod "dns-default-54cns" (UID: "ad80f379-b5dc-4d02-a61d-b8b2a956a551") : secret "dns-default-metrics-tls" not found
Sep 18 19:15:31.793642 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:31.793560    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:31.889726 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:31.889699    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:31.994960 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:31.994930    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:32.014955 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:32.014933    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7s97w_812cd3c0-1c61-4d21-8dd7-77d98f0edd2d/node-ca/0.log"
Sep 18 19:15:32.089431 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:32.089367    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:32.189287 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:32.189263    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:32.839540 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:32.839509    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3cd69cee-5d0e-41d9-8dc9-a0fb2898085d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tz4cf\" (UID: \"3cd69cee-5d0e-41d9-8dc9-a0fb2898085d\") " pod="openshift-insights/insights-runtime-extractor-tz4cf"
Sep 18 19:15:32.839991 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:32.839557    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3cd69cee-5d0e-41d9-8dc9-a0fb2898085d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tz4cf\" (UID: \"3cd69cee-5d0e-41d9-8dc9-a0fb2898085d\") " pod="openshift-insights/insights-runtime-extractor-tz4cf"
Sep 18 19:15:32.839991 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:15:32.839645    2532 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:15:32.839991 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:15:32.839724    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3cd69cee-5d0e-41d9-8dc9-a0fb2898085d-kube-rbac-proxy-cm podName:3cd69cee-5d0e-41d9-8dc9-a0fb2898085d nodeName:}" failed. No retries permitted until 2026-09-18 19:15:36.839704244 +0000 UTC m=+103.760208097 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3cd69cee-5d0e-41d9-8dc9-a0fb2898085d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tz4cf" (UID: "3cd69cee-5d0e-41d9-8dc9-a0fb2898085d") : configmap "kube-rbac-proxy" not found
Sep 18 19:15:32.839991 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:15:32.839655    2532 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:15:32.839991 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:15:32.839794    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cd69cee-5d0e-41d9-8dc9-a0fb2898085d-insights-runtime-extractor-tls podName:3cd69cee-5d0e-41d9-8dc9-a0fb2898085d nodeName:}" failed. No retries permitted until 2026-09-18 19:15:36.839781505 +0000 UTC m=+103.760285340 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3cd69cee-5d0e-41d9-8dc9-a0fb2898085d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tz4cf" (UID: "3cd69cee-5d0e-41d9-8dc9-a0fb2898085d") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:15:32.937402 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:32.937377    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:33.016950 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:33.016923    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wgxds_35de2d74-739d-411b-ad9b-dc99018ad770/serve-healthcheck-canary/0.log"
Sep 18 19:15:35.337322 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:35.337275    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:35.880156 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:35.880116    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:35.972447 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:35.972422    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-42kxn"
Sep 18 19:15:36.053530 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:36.053488    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:36.164993 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:36.164911    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:36.265665 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:36.265643    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:36.376101 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:36.376051    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:36.470350 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:36.470323    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:36.577487 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:36.577455    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:36.869192 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:36.869107    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3cd69cee-5d0e-41d9-8dc9-a0fb2898085d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tz4cf\" (UID: \"3cd69cee-5d0e-41d9-8dc9-a0fb2898085d\") " pod="openshift-insights/insights-runtime-extractor-tz4cf"
Sep 18 19:15:36.869192 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:36.869168    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3cd69cee-5d0e-41d9-8dc9-a0fb2898085d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tz4cf\" (UID: \"3cd69cee-5d0e-41d9-8dc9-a0fb2898085d\") " pod="openshift-insights/insights-runtime-extractor-tz4cf"
Sep 18 19:15:36.869369 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:15:36.869225    2532 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:15:36.869369 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:15:36.869265    2532 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:15:36.869369 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:15:36.869286    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3cd69cee-5d0e-41d9-8dc9-a0fb2898085d-kube-rbac-proxy-cm podName:3cd69cee-5d0e-41d9-8dc9-a0fb2898085d nodeName:}" failed. No retries permitted until 2026-09-18 19:15:44.869271166 +0000 UTC m=+111.789775002 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3cd69cee-5d0e-41d9-8dc9-a0fb2898085d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tz4cf" (UID: "3cd69cee-5d0e-41d9-8dc9-a0fb2898085d") : configmap "kube-rbac-proxy" not found
Sep 18 19:15:36.869369 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:15:36.869304    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cd69cee-5d0e-41d9-8dc9-a0fb2898085d-insights-runtime-extractor-tls podName:3cd69cee-5d0e-41d9-8dc9-a0fb2898085d nodeName:}" failed. No retries permitted until 2026-09-18 19:15:44.869292581 +0000 UTC m=+111.789796416 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3cd69cee-5d0e-41d9-8dc9-a0fb2898085d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tz4cf" (UID: "3cd69cee-5d0e-41d9-8dc9-a0fb2898085d") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:15:37.597530 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:37.597495    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:37.970625 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:37.970590    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:38.167642 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:38.167610    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:38.361258 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:38.361180    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:38.776203 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:38.776170    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:39.582501 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:39.582467    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:39.649613 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:39.649579    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:39.778384 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:39.778338    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:40.011250 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:40.011215    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:40.202751 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:40.202721    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:40.302503 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:40.302432    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:40.392088 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:40.392040    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:40.460765 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:40.460733    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:40.801325 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:40.801291    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:41.371515 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:41.371477    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:41.767395 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:41.767358    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:41.871220 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:41.871189    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:41.977119 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:41.977080    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:42.328480 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:42.328443    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:42.429163 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:42.429138    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:43.203231 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:43.203200    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:43.313699 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:43.313669    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:43.414148 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:43.414120    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:43.767586 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:43.767553    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:43.972515 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:43.972492    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:44.417510 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:44.417475    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:44.930754 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:44.930722    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3cd69cee-5d0e-41d9-8dc9-a0fb2898085d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tz4cf\" (UID: \"3cd69cee-5d0e-41d9-8dc9-a0fb2898085d\") " pod="openshift-insights/insights-runtime-extractor-tz4cf"
Sep 18 19:15:44.930945 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:44.930783    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3cd69cee-5d0e-41d9-8dc9-a0fb2898085d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tz4cf\" (UID: \"3cd69cee-5d0e-41d9-8dc9-a0fb2898085d\") " pod="openshift-insights/insights-runtime-extractor-tz4cf"
Sep 18 19:15:44.930945 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:15:44.930909    2532 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:15:44.931080 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:15:44.930980    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cd69cee-5d0e-41d9-8dc9-a0fb2898085d-insights-runtime-extractor-tls podName:3cd69cee-5d0e-41d9-8dc9-a0fb2898085d nodeName:}" failed. No retries permitted until 2026-09-18 19:16:00.93096321 +0000 UTC m=+127.851467045 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3cd69cee-5d0e-41d9-8dc9-a0fb2898085d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tz4cf" (UID: "3cd69cee-5d0e-41d9-8dc9-a0fb2898085d") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:15:44.931237 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:44.931215    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3cd69cee-5d0e-41d9-8dc9-a0fb2898085d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tz4cf\" (UID: \"3cd69cee-5d0e-41d9-8dc9-a0fb2898085d\") " pod="openshift-insights/insights-runtime-extractor-tz4cf"
Sep 18 19:15:45.196899 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:45.196821    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:45.394175 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:45.394141    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:45.796744 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:45.796712    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:46.040708 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:46.040676    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:46.392509 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:46.392484    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:46.617996 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:46.617962    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:47.834358 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:47.834309    2532 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764c5dfc6-m99t4" podUID="0a2aba53-8251-4204-9845-f5572077030b" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 18 19:15:50.782320 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:50.782281    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:50.888515 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:50.888480    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:51.009618 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:51.009585    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:51.110198 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:51.110120    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:51.216762 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:51.216732    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:51.600873 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:51.600840    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:51.721807 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:51.721779    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:52.009470 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:52.009433    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:52.157004 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:52.156978    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:52.241798 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:52.241767    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:52.372449 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:52.372385    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:52.811630 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:52.811604    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:52.911777 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:52.911749    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:53.023932 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:53.023902    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:53.132926 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:53.132864    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:53.592800 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:53.592770    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:54.586834 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:54.586797    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:57.834216 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:57.834176    2532 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764c5dfc6-m99t4" podUID="0a2aba53-8251-4204-9845-f5572077030b" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 18 19:15:59.058114 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:59.058082    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:59.164383 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:59.164349    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:59.295127 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:59.295099    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:59.704071 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:15:59.704026    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:00.022260 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:00.022178    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:00.100382 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:00.100356    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:00.944537 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:00.944500    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3cd69cee-5d0e-41d9-8dc9-a0fb2898085d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tz4cf\" (UID: \"3cd69cee-5d0e-41d9-8dc9-a0fb2898085d\") " pod="openshift-insights/insights-runtime-extractor-tz4cf"
Sep 18 19:16:00.946916 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:00.946896    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3cd69cee-5d0e-41d9-8dc9-a0fb2898085d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tz4cf\" (UID: \"3cd69cee-5d0e-41d9-8dc9-a0fb2898085d\") " pod="openshift-insights/insights-runtime-extractor-tz4cf"
Sep 18 19:16:01.230012 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:01.229941    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wlcxl\""
Sep 18 19:16:01.238264 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:01.238247    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tz4cf"
Sep 18 19:16:01.517481 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:01.517406    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tz4cf"]
Sep 18 19:16:01.520770 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:16:01.520746    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cd69cee_5d0e_41d9_8dc9_a0fb2898085d.slice/crio-1160bcb82dedf46c0713a3bec8e37bacc4da5545b817bc514871c6d35fed89a7 WatchSource:0}: Error finding container 1160bcb82dedf46c0713a3bec8e37bacc4da5545b817bc514871c6d35fed89a7: Status 404 returned error can't find the container with id 1160bcb82dedf46c0713a3bec8e37bacc4da5545b817bc514871c6d35fed89a7
Sep 18 19:16:02.127024 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:02.126960    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tz4cf" event={"ID":"3cd69cee-5d0e-41d9-8dc9-a0fb2898085d","Type":"ContainerStarted","Data":"76a3727ae0b64c7519aae4278500796b1a831413ca0df4d80f3547b424d2242d"}
Sep 18 19:16:02.127024 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:02.127030    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tz4cf" event={"ID":"3cd69cee-5d0e-41d9-8dc9-a0fb2898085d","Type":"ContainerStarted","Data":"1160bcb82dedf46c0713a3bec8e37bacc4da5545b817bc514871c6d35fed89a7"}
Sep 18 19:16:02.229271 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:02.229244    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:02.332911 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:02.332874    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:02.401083 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:02.400986    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:02.509491 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:02.509447    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:03.132338 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:03.132304    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tz4cf" event={"ID":"3cd69cee-5d0e-41d9-8dc9-a0fb2898085d","Type":"ContainerStarted","Data":"20afc42046b05c09c3f871faec503bdcbf02124e5cf5453b929bfeb150b8b212"}
Sep 18 19:16:03.464264 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:03.464239    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05c957d5-da77-47c2-8b9c-e7cdc087120c-metrics-certs\") pod \"network-metrics-daemon-ps7w2\" (UID: \"05c957d5-da77-47c2-8b9c-e7cdc087120c\") " pod="openshift-multus/network-metrics-daemon-ps7w2"
Sep 18 19:16:03.466633 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:03.466607    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05c957d5-da77-47c2-8b9c-e7cdc087120c-metrics-certs\") pod \"network-metrics-daemon-ps7w2\" (UID: \"05c957d5-da77-47c2-8b9c-e7cdc087120c\") " pod="openshift-multus/network-metrics-daemon-ps7w2"
Sep 18 19:16:03.739910 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:03.739831    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-95w6x\""
Sep 18 19:16:03.748001 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:03.747970    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ps7w2"
Sep 18 19:16:04.029344 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:04.029263    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ps7w2"]
Sep 18 19:16:04.032850 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:16:04.032826    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05c957d5_da77_47c2_8b9c_e7cdc087120c.slice/crio-1571fd54deba086519b0693c018dca637ef5afddd11f5a8c89d7362be736503f WatchSource:0}: Error finding container 1571fd54deba086519b0693c018dca637ef5afddd11f5a8c89d7362be736503f: Status 404 returned error can't find the container with id 1571fd54deba086519b0693c018dca637ef5afddd11f5a8c89d7362be736503f
Sep 18 19:16:04.139144 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:04.139111    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tz4cf" event={"ID":"3cd69cee-5d0e-41d9-8dc9-a0fb2898085d","Type":"ContainerStarted","Data":"10af56b93c139b6690cb49ab23cf0ec48c5a82e636783b401f725bb4d704dd48"}
Sep 18 19:16:04.140257 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:04.140238    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ps7w2" event={"ID":"05c957d5-da77-47c2-8b9c-e7cdc087120c","Type":"ContainerStarted","Data":"1571fd54deba086519b0693c018dca637ef5afddd11f5a8c89d7362be736503f"}
Sep 18 19:16:04.166613 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:04.166568    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-tz4cf" podStartSLOduration=33.344501697 podStartE2EDuration="35.166552762s" totalImagesPullingTime="1.822051065s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:15:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:16:04.163911275 +0000 UTC m=+131.084415132" watchObservedRunningTime="2026-09-18 19:16:04.166552762 +0000 UTC m=+131.087056619"
Sep 18 19:16:05.145111 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:05.145021    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ps7w2" event={"ID":"05c957d5-da77-47c2-8b9c-e7cdc087120c","Type":"ContainerStarted","Data":"b13e2f478ecf86412883b5bc5be3037c8ee1c5af60e90ae94a2b14dc25cf8b93"}
Sep 18 19:16:05.145111 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:05.145090    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ps7w2" event={"ID":"05c957d5-da77-47c2-8b9c-e7cdc087120c","Type":"ContainerStarted","Data":"9c1bffdd9669bb6c08a597ab3fbb1c0a391062d66d32299ed7ed8de1abe0e0ab"}
Sep 18 19:16:05.172198 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:05.172157    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-ps7w2" podStartSLOduration=130.348999864 podStartE2EDuration="2m11.172142805s" totalImagesPullingTime="823.142941ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:13:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:16:05.170329961 +0000 UTC m=+132.090833817" watchObservedRunningTime="2026-09-18 19:16:05.172142805 +0000 UTC m=+132.092646661"
Sep 18 19:16:05.469855 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:05.469829    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:05.589142 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:05.589113    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:05.680775 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:05.680747    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:05.811698 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:05.811622    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:05.896859 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:05.896834    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:07.834463 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:07.834426    2532 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764c5dfc6-m99t4" podUID="0a2aba53-8251-4204-9845-f5572077030b" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 18 19:16:07.834899 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:07.834491    2532 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764c5dfc6-m99t4"
Sep 18 19:16:07.835000 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:07.834913    2532 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"99f911254e73e4264b4e3b8aaf6773d621022971d4d829f1e090f26f26d91447"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764c5dfc6-m99t4" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 18 19:16:07.835000 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:07.834960    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764c5dfc6-m99t4" podUID="0a2aba53-8251-4204-9845-f5572077030b" containerName="service-proxy" containerID="cri-o://99f911254e73e4264b4e3b8aaf6773d621022971d4d829f1e090f26f26d91447" gracePeriod=30
Sep 18 19:16:08.156674 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:08.156609    2532 generic.go:356] "Generic (PLEG): container finished" podID="0a2aba53-8251-4204-9845-f5572077030b" containerID="99f911254e73e4264b4e3b8aaf6773d621022971d4d829f1e090f26f26d91447" exitCode=2
Sep 18 19:16:08.156791 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:08.156686    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764c5dfc6-m99t4" event={"ID":"0a2aba53-8251-4204-9845-f5572077030b","Type":"ContainerDied","Data":"99f911254e73e4264b4e3b8aaf6773d621022971d4d829f1e090f26f26d91447"}
Sep 18 19:16:08.156791 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:08.156730    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764c5dfc6-m99t4" event={"ID":"0a2aba53-8251-4204-9845-f5572077030b","Type":"ContainerStarted","Data":"97b0f5a02366b40a1611c71312d6b1aecef7a6fa5a112a738704caa8bfd5a574"}
Sep 18 19:16:08.221252 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:08.221221    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:08.320311 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:08.320280    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:10.170178 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:10.170146    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:10.323933 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:10.323895    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:13.086670 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:13.086636    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7c9c7469f5-nckkz"]
Sep 18 19:16:13.087093 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:16:13.086836    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-7c9c7469f5-nckkz" podUID="8367afb3-7c5e-429d-8dc1-2750240a5b30"
Sep 18 19:16:13.174342 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:13.174311    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c9c7469f5-nckkz"
Sep 18 19:16:13.180999 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:13.180975    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c9c7469f5-nckkz"
Sep 18 19:16:13.239282 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:13.239247    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:13.328183 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:13.328152    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:13.333209 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:13.333190    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8367afb3-7c5e-429d-8dc1-2750240a5b30-ca-trust-extracted\") pod \"8367afb3-7c5e-429d-8dc1-2750240a5b30\" (UID: \"8367afb3-7c5e-429d-8dc1-2750240a5b30\") "
Sep 18 19:16:13.333292 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:13.333218    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9pxhk\" (UniqueName: \"kubernetes.io/projected/8367afb3-7c5e-429d-8dc1-2750240a5b30-kube-api-access-9pxhk\") pod \"8367afb3-7c5e-429d-8dc1-2750240a5b30\" (UID: \"8367afb3-7c5e-429d-8dc1-2750240a5b30\") "
Sep 18 19:16:13.333292 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:13.333236    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8367afb3-7c5e-429d-8dc1-2750240a5b30-registry-certificates\") pod \"8367afb3-7c5e-429d-8dc1-2750240a5b30\" (UID: \"8367afb3-7c5e-429d-8dc1-2750240a5b30\") "
Sep 18 19:16:13.333292 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:13.333253    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8367afb3-7c5e-429d-8dc1-2750240a5b30-trusted-ca\") pod \"8367afb3-7c5e-429d-8dc1-2750240a5b30\" (UID: \"8367afb3-7c5e-429d-8dc1-2750240a5b30\") "
Sep 18 19:16:13.333292 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:13.333287    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8367afb3-7c5e-429d-8dc1-2750240a5b30-image-registry-private-configuration\") pod \"8367afb3-7c5e-429d-8dc1-2750240a5b30\" (UID: \"8367afb3-7c5e-429d-8dc1-2750240a5b30\") "
Sep 18 19:16:13.333482 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:13.333307    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8367afb3-7c5e-429d-8dc1-2750240a5b30-bound-sa-token\") pod \"8367afb3-7c5e-429d-8dc1-2750240a5b30\" (UID: \"8367afb3-7c5e-429d-8dc1-2750240a5b30\") "
Sep 18 19:16:13.333482 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:13.333338    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8367afb3-7c5e-429d-8dc1-2750240a5b30-installation-pull-secrets\") pod \"8367afb3-7c5e-429d-8dc1-2750240a5b30\" (UID: \"8367afb3-7c5e-429d-8dc1-2750240a5b30\") "
Sep 18 19:16:13.333482 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:13.333464    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8367afb3-7c5e-429d-8dc1-2750240a5b30-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "8367afb3-7c5e-429d-8dc1-2750240a5b30" (UID: "8367afb3-7c5e-429d-8dc1-2750240a5b30"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:16:13.333652 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:13.333628    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8367afb3-7c5e-429d-8dc1-2750240a5b30-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "8367afb3-7c5e-429d-8dc1-2750240a5b30" (UID: "8367afb3-7c5e-429d-8dc1-2750240a5b30"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:16:13.333808 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:13.333766    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8367afb3-7c5e-429d-8dc1-2750240a5b30-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "8367afb3-7c5e-429d-8dc1-2750240a5b30" (UID: "8367afb3-7c5e-429d-8dc1-2750240a5b30"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:16:13.335664 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:13.335633    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8367afb3-7c5e-429d-8dc1-2750240a5b30-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "8367afb3-7c5e-429d-8dc1-2750240a5b30" (UID: "8367afb3-7c5e-429d-8dc1-2750240a5b30"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:16:13.335776 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:13.335717    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8367afb3-7c5e-429d-8dc1-2750240a5b30-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "8367afb3-7c5e-429d-8dc1-2750240a5b30" (UID: "8367afb3-7c5e-429d-8dc1-2750240a5b30"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:16:13.335832 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:13.335767    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8367afb3-7c5e-429d-8dc1-2750240a5b30-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "8367afb3-7c5e-429d-8dc1-2750240a5b30" (UID: "8367afb3-7c5e-429d-8dc1-2750240a5b30"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:16:13.335870 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:13.335822    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8367afb3-7c5e-429d-8dc1-2750240a5b30-kube-api-access-9pxhk" (OuterVolumeSpecName: "kube-api-access-9pxhk") pod "8367afb3-7c5e-429d-8dc1-2750240a5b30" (UID: "8367afb3-7c5e-429d-8dc1-2750240a5b30"). InnerVolumeSpecName "kube-api-access-9pxhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:16:13.433921 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:13.433898    2532 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8367afb3-7c5e-429d-8dc1-2750240a5b30-ca-trust-extracted\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:16:13.433921 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:13.433918    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pxhk\" (UniqueName: \"kubernetes.io/projected/8367afb3-7c5e-429d-8dc1-2750240a5b30-kube-api-access-9pxhk\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:16:13.434030 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:13.433928    2532 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8367afb3-7c5e-429d-8dc1-2750240a5b30-registry-certificates\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:16:13.434030 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:13.433937    2532 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8367afb3-7c5e-429d-8dc1-2750240a5b30-trusted-ca\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:16:13.434030 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:13.433946    2532 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8367afb3-7c5e-429d-8dc1-2750240a5b30-image-registry-private-configuration\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:16:13.434030 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:13.433954    2532 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8367afb3-7c5e-429d-8dc1-2750240a5b30-bound-sa-token\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:16:13.434030 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:13.433962    2532 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8367afb3-7c5e-429d-8dc1-2750240a5b30-installation-pull-secrets\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:16:14.177229 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:14.177194    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c9c7469f5-nckkz"
Sep 18 19:16:14.292236 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:14.292207    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7c9c7469f5-nckkz"]
Sep 18 19:16:14.308712 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:14.308685    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7c9c7469f5-nckkz"]
Sep 18 19:16:14.440216 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:14.440193    2532 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8367afb3-7c5e-429d-8dc1-2750240a5b30-registry-tls\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:16:14.959231 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:14.959199    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:15.112160 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:15.112128    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:15.620332 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:15.620301    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8367afb3-7c5e-429d-8dc1-2750240a5b30" path="/var/lib/kubelet/pods/8367afb3-7c5e-429d-8dc1-2750240a5b30/volumes"
Sep 18 19:16:15.891743 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:15.891662    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:16.033292 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:16.033263    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:17.673899 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:17.673864    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:17.868587 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:17.868551    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:18.242071 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:18.242036    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:18.453437 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:18.453409    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:18.614930 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:18.614844    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:18.782642 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:18.782603    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-fcd5d"]
Sep 18 19:16:18.785884 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:18.785862    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fcd5d"
Sep 18 19:16:18.794131 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:18.794110    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 18 19:16:18.794238 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:18.794112    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 18 19:16:18.794999 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:18.794983    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 18 19:16:18.795108 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:18.795090    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-w8xns\""
Sep 18 19:16:18.795175 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:18.795095    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 18 19:16:18.795175 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:18.795115    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 18 19:16:18.795254 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:18.795221    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 18 19:16:18.839645 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:18.839617    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:18.974833 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:18.974805    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a661c12a-c025-4e31-9c98-ba3d6a66089e-root\") pod \"node-exporter-fcd5d\" (UID: \"a661c12a-c025-4e31-9c98-ba3d6a66089e\") " pod="openshift-monitoring/node-exporter-fcd5d"
Sep 18 19:16:18.974979 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:18.974843    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a661c12a-c025-4e31-9c98-ba3d6a66089e-node-exporter-wtmp\") pod \"node-exporter-fcd5d\" (UID: \"a661c12a-c025-4e31-9c98-ba3d6a66089e\") " pod="openshift-monitoring/node-exporter-fcd5d"
Sep 18 19:16:18.974979 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:18.974897    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a661c12a-c025-4e31-9c98-ba3d6a66089e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fcd5d\" (UID: \"a661c12a-c025-4e31-9c98-ba3d6a66089e\") " pod="openshift-monitoring/node-exporter-fcd5d"
Sep 18 19:16:18.974979 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:18.974923    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a661c12a-c025-4e31-9c98-ba3d6a66089e-metrics-client-ca\") pod \"node-exporter-fcd5d\" (UID: \"a661c12a-c025-4e31-9c98-ba3d6a66089e\") " pod="openshift-monitoring/node-exporter-fcd5d"
Sep 18 19:16:18.974979 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:18.974952    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a661c12a-c025-4e31-9c98-ba3d6a66089e-node-exporter-accelerators-collector-config\") pod \"node-exporter-fcd5d\" (UID: \"a661c12a-c025-4e31-9c98-ba3d6a66089e\") " pod="openshift-monitoring/node-exporter-fcd5d"
Sep 18 19:16:18.975135 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:18.974985    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpcxr\" (UniqueName: \"kubernetes.io/projected/a661c12a-c025-4e31-9c98-ba3d6a66089e-kube-api-access-qpcxr\") pod \"node-exporter-fcd5d\" (UID: \"a661c12a-c025-4e31-9c98-ba3d6a66089e\") " pod="openshift-monitoring/node-exporter-fcd5d"
Sep 18 19:16:18.975135 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:18.975007    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a661c12a-c025-4e31-9c98-ba3d6a66089e-node-exporter-tls\") pod \"node-exporter-fcd5d\" (UID: \"a661c12a-c025-4e31-9c98-ba3d6a66089e\") " pod="openshift-monitoring/node-exporter-fcd5d"
Sep 18 19:16:18.975135 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:18.975038    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a661c12a-c025-4e31-9c98-ba3d6a66089e-sys\") pod \"node-exporter-fcd5d\" (UID: \"a661c12a-c025-4e31-9c98-ba3d6a66089e\") " pod="openshift-monitoring/node-exporter-fcd5d"
Sep 18 19:16:18.975135 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:18.975076    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a661c12a-c025-4e31-9c98-ba3d6a66089e-node-exporter-textfile\") pod \"node-exporter-fcd5d\" (UID: \"a661c12a-c025-4e31-9c98-ba3d6a66089e\") " pod="openshift-monitoring/node-exporter-fcd5d"
Sep 18 19:16:18.982400 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:18.982368    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:19.076011 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:19.075981    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a661c12a-c025-4e31-9c98-ba3d6a66089e-node-exporter-accelerators-collector-config\") pod \"node-exporter-fcd5d\" (UID: \"a661c12a-c025-4e31-9c98-ba3d6a66089e\") " pod="openshift-monitoring/node-exporter-fcd5d"
Sep 18 19:16:19.076144 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:19.076023    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpcxr\" (UniqueName: \"kubernetes.io/projected/a661c12a-c025-4e31-9c98-ba3d6a66089e-kube-api-access-qpcxr\") pod \"node-exporter-fcd5d\" (UID: \"a661c12a-c025-4e31-9c98-ba3d6a66089e\") " pod="openshift-monitoring/node-exporter-fcd5d"
Sep 18 19:16:19.076144 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:19.076040    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a661c12a-c025-4e31-9c98-ba3d6a66089e-node-exporter-tls\") pod \"node-exporter-fcd5d\" (UID: \"a661c12a-c025-4e31-9c98-ba3d6a66089e\") " pod="openshift-monitoring/node-exporter-fcd5d"
Sep 18 19:16:19.076144 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:19.076071    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a661c12a-c025-4e31-9c98-ba3d6a66089e-sys\") pod \"node-exporter-fcd5d\" (UID: \"a661c12a-c025-4e31-9c98-ba3d6a66089e\") " pod="openshift-monitoring/node-exporter-fcd5d"
Sep 18 19:16:19.076144 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:19.076102    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a661c12a-c025-4e31-9c98-ba3d6a66089e-node-exporter-textfile\") pod \"node-exporter-fcd5d\" (UID: \"a661c12a-c025-4e31-9c98-ba3d6a66089e\") " pod="openshift-monitoring/node-exporter-fcd5d"
Sep 18 19:16:19.076144 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:19.076137    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a661c12a-c025-4e31-9c98-ba3d6a66089e-root\") pod \"node-exporter-fcd5d\" (UID: \"a661c12a-c025-4e31-9c98-ba3d6a66089e\") " pod="openshift-monitoring/node-exporter-fcd5d"
Sep 18 19:16:19.076390 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:19.076169    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a661c12a-c025-4e31-9c98-ba3d6a66089e-node-exporter-wtmp\") pod \"node-exporter-fcd5d\" (UID: \"a661c12a-c025-4e31-9c98-ba3d6a66089e\") " pod="openshift-monitoring/node-exporter-fcd5d"
Sep 18 19:16:19.076390 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:19.076197    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a661c12a-c025-4e31-9c98-ba3d6a66089e-sys\") pod \"node-exporter-fcd5d\" (UID: \"a661c12a-c025-4e31-9c98-ba3d6a66089e\") " pod="openshift-monitoring/node-exporter-fcd5d"
Sep 18 19:16:19.076390 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:19.076205    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a661c12a-c025-4e31-9c98-ba3d6a66089e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fcd5d\" (UID: \"a661c12a-c025-4e31-9c98-ba3d6a66089e\") " pod="openshift-monitoring/node-exporter-fcd5d"
Sep 18 19:16:19.076390 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:19.076225    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a661c12a-c025-4e31-9c98-ba3d6a66089e-root\") pod \"node-exporter-fcd5d\" (UID: \"a661c12a-c025-4e31-9c98-ba3d6a66089e\") " pod="openshift-monitoring/node-exporter-fcd5d"
Sep 18 19:16:19.076390 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:19.076257    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a661c12a-c025-4e31-9c98-ba3d6a66089e-metrics-client-ca\") pod \"node-exporter-fcd5d\" (UID: \"a661c12a-c025-4e31-9c98-ba3d6a66089e\") " pod="openshift-monitoring/node-exporter-fcd5d"
Sep 18 19:16:19.076390 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:19.076326    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a661c12a-c025-4e31-9c98-ba3d6a66089e-node-exporter-wtmp\") pod \"node-exporter-fcd5d\" (UID: \"a661c12a-c025-4e31-9c98-ba3d6a66089e\") " pod="openshift-monitoring/node-exporter-fcd5d"
Sep 18 19:16:19.076681 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:19.076523    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a661c12a-c025-4e31-9c98-ba3d6a66089e-node-exporter-textfile\") pod \"node-exporter-fcd5d\" (UID: \"a661c12a-c025-4e31-9c98-ba3d6a66089e\") " pod="openshift-monitoring/node-exporter-fcd5d"
Sep 18 19:16:19.076681 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:19.076657    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a661c12a-c025-4e31-9c98-ba3d6a66089e-node-exporter-accelerators-collector-config\") pod \"node-exporter-fcd5d\" (UID: \"a661c12a-c025-4e31-9c98-ba3d6a66089e\") " pod="openshift-monitoring/node-exporter-fcd5d"
Sep 18 19:16:19.076819 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:19.076800    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a661c12a-c025-4e31-9c98-ba3d6a66089e-metrics-client-ca\") pod \"node-exporter-fcd5d\" (UID: \"a661c12a-c025-4e31-9c98-ba3d6a66089e\") " pod="openshift-monitoring/node-exporter-fcd5d"
Sep 18 19:16:19.078407 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:19.078388    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a661c12a-c025-4e31-9c98-ba3d6a66089e-node-exporter-tls\") pod \"node-exporter-fcd5d\" (UID: \"a661c12a-c025-4e31-9c98-ba3d6a66089e\") " pod="openshift-monitoring/node-exporter-fcd5d"
Sep 18 19:16:19.078495 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:19.078418    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a661c12a-c025-4e31-9c98-ba3d6a66089e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fcd5d\" (UID: \"a661c12a-c025-4e31-9c98-ba3d6a66089e\") " pod="openshift-monitoring/node-exporter-fcd5d"
Sep 18 19:16:19.095748 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:19.095730    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpcxr\" (UniqueName: \"kubernetes.io/projected/a661c12a-c025-4e31-9c98-ba3d6a66089e-kube-api-access-qpcxr\") pod \"node-exporter-fcd5d\" (UID: \"a661c12a-c025-4e31-9c98-ba3d6a66089e\") " pod="openshift-monitoring/node-exporter-fcd5d"
Sep 18 19:16:19.394553 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:19.394476    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fcd5d"
Sep 18 19:16:19.405205 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:16:19.405179    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda661c12a_c025_4e31_9c98_ba3d6a66089e.slice/crio-2a3fc15c3498b16a0d4f048b42c29c4f0cd36df2fad07df0d6b17404a74011fb WatchSource:0}: Error finding container 2a3fc15c3498b16a0d4f048b42c29c4f0cd36df2fad07df0d6b17404a74011fb: Status 404 returned error can't find the container with id 2a3fc15c3498b16a0d4f048b42c29c4f0cd36df2fad07df0d6b17404a74011fb
Sep 18 19:16:19.574821 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:19.574786    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:19.703660 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:19.703611    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:20.061742 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:20.061666    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:20.177846 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:20.177800    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:20.197819 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:20.197794    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fcd5d" event={"ID":"a661c12a-c025-4e31-9c98-ba3d6a66089e","Type":"ContainerStarted","Data":"2a3fc15c3498b16a0d4f048b42c29c4f0cd36df2fad07df0d6b17404a74011fb"}
Sep 18 19:16:21.202544 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:21.202505    2532 generic.go:356] "Generic (PLEG): container finished" podID="a661c12a-c025-4e31-9c98-ba3d6a66089e" containerID="30787987c0d2210e61d9a0d2476dd61441f8d82403b216613c3acf87f735ce5e" exitCode=0
Sep 18 19:16:21.202975 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:21.202599    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fcd5d" event={"ID":"a661c12a-c025-4e31-9c98-ba3d6a66089e","Type":"ContainerDied","Data":"30787987c0d2210e61d9a0d2476dd61441f8d82403b216613c3acf87f735ce5e"}
Sep 18 19:16:22.018235 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:22.018200    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:22.208046 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:22.208015    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fcd5d" event={"ID":"a661c12a-c025-4e31-9c98-ba3d6a66089e","Type":"ContainerStarted","Data":"4388ad519cd77ddb31400bf4b424b5d92ef5dcc829b4883c50f4b804e931c4c9"}
Sep 18 19:16:22.208412 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:22.208080    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fcd5d" event={"ID":"a661c12a-c025-4e31-9c98-ba3d6a66089e","Type":"ContainerStarted","Data":"75c9d202ef477ede217a79fe86e365dd26b7d1c874d8562270821c876d696c74"}
Sep 18 19:16:22.453104 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:22.453049    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:22.556904 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:22.556871    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:22.671432 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:22.671397    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:22.857812 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:22.857736    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:23.054749 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:23.054720    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:23.405809 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:23.405771    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:23.506686 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:23.506661    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:23.632353 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:23.632308    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-fcd5d" podStartSLOduration=4.766190128 podStartE2EDuration="5.632289153s" totalImagesPullingTime="806.587731ms" totalInitContainerRuntime="59.511294ms" isStatefulPod=true podCreationTimestamp="2026-09-18 19:16:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:16:22.260318194 +0000 UTC m=+149.180822052" watchObservedRunningTime="2026-09-18 19:16:23.632289153 +0000 UTC m=+150.552793012"
Sep 18 19:16:23.632753 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:23.632723    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5dfbb88bb-tpbp8"]
Sep 18 19:16:23.635698 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:23.635679    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5dfbb88bb-tpbp8"
Sep 18 19:16:23.638909 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:23.638892    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 18 19:16:23.639077 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:23.639045    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dc0boov2aqeb9\""
Sep 18 19:16:23.641853 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:23.641629    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 18 19:16:23.641918 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:23.641905    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 18 19:16:23.641978 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:23.641957    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-8x2tg\""
Sep 18 19:16:23.642217 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:23.642202    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 18 19:16:23.669464 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:23.669418    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5dfbb88bb-tpbp8"]
Sep 18 19:16:23.706110 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:23.706085    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/4eba974d-5929-4fe6-8dec-4e3d7fab0e0e-secret-metrics-server-client-certs\") pod \"metrics-server-5dfbb88bb-tpbp8\" (UID: \"4eba974d-5929-4fe6-8dec-4e3d7fab0e0e\") " pod="openshift-monitoring/metrics-server-5dfbb88bb-tpbp8"
Sep 18 19:16:23.706202 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:23.706121    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grtjz\" (UniqueName: \"kubernetes.io/projected/4eba974d-5929-4fe6-8dec-4e3d7fab0e0e-kube-api-access-grtjz\") pod \"metrics-server-5dfbb88bb-tpbp8\" (UID: \"4eba974d-5929-4fe6-8dec-4e3d7fab0e0e\") " pod="openshift-monitoring/metrics-server-5dfbb88bb-tpbp8"
Sep 18 19:16:23.706202 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:23.706146    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4eba974d-5929-4fe6-8dec-4e3d7fab0e0e-client-ca-bundle\") pod \"metrics-server-5dfbb88bb-tpbp8\" (UID: \"4eba974d-5929-4fe6-8dec-4e3d7fab0e0e\") " pod="openshift-monitoring/metrics-server-5dfbb88bb-tpbp8"
Sep 18 19:16:23.706284 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:23.706209    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/4eba974d-5929-4fe6-8dec-4e3d7fab0e0e-secret-metrics-server-tls\") pod \"metrics-server-5dfbb88bb-tpbp8\" (UID: \"4eba974d-5929-4fe6-8dec-4e3d7fab0e0e\") " pod="openshift-monitoring/metrics-server-5dfbb88bb-tpbp8"
Sep 18 19:16:23.706284 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:23.706258    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/4eba974d-5929-4fe6-8dec-4e3d7fab0e0e-audit-log\") pod \"metrics-server-5dfbb88bb-tpbp8\" (UID: \"4eba974d-5929-4fe6-8dec-4e3d7fab0e0e\") " pod="openshift-monitoring/metrics-server-5dfbb88bb-tpbp8"
Sep 18 19:16:23.706284 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:23.706278    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/4eba974d-5929-4fe6-8dec-4e3d7fab0e0e-metrics-server-audit-profiles\") pod \"metrics-server-5dfbb88bb-tpbp8\" (UID: \"4eba974d-5929-4fe6-8dec-4e3d7fab0e0e\") " pod="openshift-monitoring/metrics-server-5dfbb88bb-tpbp8"
Sep 18 19:16:23.706390 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:23.706345    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4eba974d-5929-4fe6-8dec-4e3d7fab0e0e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5dfbb88bb-tpbp8\" (UID: \"4eba974d-5929-4fe6-8dec-4e3d7fab0e0e\") " pod="openshift-monitoring/metrics-server-5dfbb88bb-tpbp8"
Sep 18 19:16:23.807082 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:23.807043    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grtjz\" (UniqueName: \"kubernetes.io/projected/4eba974d-5929-4fe6-8dec-4e3d7fab0e0e-kube-api-access-grtjz\") pod \"metrics-server-5dfbb88bb-tpbp8\" (UID: \"4eba974d-5929-4fe6-8dec-4e3d7fab0e0e\") " pod="openshift-monitoring/metrics-server-5dfbb88bb-tpbp8"
Sep 18 19:16:23.807161 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:23.807091    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4eba974d-5929-4fe6-8dec-4e3d7fab0e0e-client-ca-bundle\") pod \"metrics-server-5dfbb88bb-tpbp8\" (UID: \"4eba974d-5929-4fe6-8dec-4e3d7fab0e0e\") " pod="openshift-monitoring/metrics-server-5dfbb88bb-tpbp8"
Sep 18 19:16:23.807161 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:23.807148    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/4eba974d-5929-4fe6-8dec-4e3d7fab0e0e-secret-metrics-server-tls\") pod \"metrics-server-5dfbb88bb-tpbp8\" (UID: \"4eba974d-5929-4fe6-8dec-4e3d7fab0e0e\") " pod="openshift-monitoring/metrics-server-5dfbb88bb-tpbp8"
Sep 18 19:16:23.807233 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:23.807198    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/4eba974d-5929-4fe6-8dec-4e3d7fab0e0e-audit-log\") pod \"metrics-server-5dfbb88bb-tpbp8\" (UID: \"4eba974d-5929-4fe6-8dec-4e3d7fab0e0e\") " pod="openshift-monitoring/metrics-server-5dfbb88bb-tpbp8"
Sep 18 19:16:23.807233 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:23.807218    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/4eba974d-5929-4fe6-8dec-4e3d7fab0e0e-metrics-server-audit-profiles\") pod \"metrics-server-5dfbb88bb-tpbp8\" (UID: \"4eba974d-5929-4fe6-8dec-4e3d7fab0e0e\") " pod="openshift-monitoring/metrics-server-5dfbb88bb-tpbp8"
Sep 18 19:16:23.807392 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:23.807369    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4eba974d-5929-4fe6-8dec-4e3d7fab0e0e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5dfbb88bb-tpbp8\" (UID: \"4eba974d-5929-4fe6-8dec-4e3d7fab0e0e\") " pod="openshift-monitoring/metrics-server-5dfbb88bb-tpbp8"
Sep 18 19:16:23.807464 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:23.807428    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/4eba974d-5929-4fe6-8dec-4e3d7fab0e0e-secret-metrics-server-client-certs\") pod \"metrics-server-5dfbb88bb-tpbp8\" (UID: \"4eba974d-5929-4fe6-8dec-4e3d7fab0e0e\") " pod="openshift-monitoring/metrics-server-5dfbb88bb-tpbp8"
Sep 18 19:16:23.807653 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:23.807633    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/4eba974d-5929-4fe6-8dec-4e3d7fab0e0e-audit-log\") pod \"metrics-server-5dfbb88bb-tpbp8\" (UID: \"4eba974d-5929-4fe6-8dec-4e3d7fab0e0e\") " pod="openshift-monitoring/metrics-server-5dfbb88bb-tpbp8"
Sep 18 19:16:23.808102 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:23.808051    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4eba974d-5929-4fe6-8dec-4e3d7fab0e0e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5dfbb88bb-tpbp8\" (UID: \"4eba974d-5929-4fe6-8dec-4e3d7fab0e0e\") " pod="openshift-monitoring/metrics-server-5dfbb88bb-tpbp8"
Sep 18 19:16:23.808454 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:23.808433    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/4eba974d-5929-4fe6-8dec-4e3d7fab0e0e-metrics-server-audit-profiles\") pod \"metrics-server-5dfbb88bb-tpbp8\" (UID: \"4eba974d-5929-4fe6-8dec-4e3d7fab0e0e\") " pod="openshift-monitoring/metrics-server-5dfbb88bb-tpbp8"
Sep 18 19:16:23.809736 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:23.809708    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/4eba974d-5929-4fe6-8dec-4e3d7fab0e0e-secret-metrics-server-tls\") pod \"metrics-server-5dfbb88bb-tpbp8\" (UID: \"4eba974d-5929-4fe6-8dec-4e3d7fab0e0e\") " pod="openshift-monitoring/metrics-server-5dfbb88bb-tpbp8"
Sep 18 19:16:23.809736 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:23.809724    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/4eba974d-5929-4fe6-8dec-4e3d7fab0e0e-secret-metrics-server-client-certs\") pod \"metrics-server-5dfbb88bb-tpbp8\" (UID: \"4eba974d-5929-4fe6-8dec-4e3d7fab0e0e\") " pod="openshift-monitoring/metrics-server-5dfbb88bb-tpbp8"
Sep 18 19:16:23.809885 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:23.809753    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4eba974d-5929-4fe6-8dec-4e3d7fab0e0e-client-ca-bundle\") pod \"metrics-server-5dfbb88bb-tpbp8\" (UID: \"4eba974d-5929-4fe6-8dec-4e3d7fab0e0e\") " pod="openshift-monitoring/metrics-server-5dfbb88bb-tpbp8"
Sep 18 19:16:23.816732 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:23.816709    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grtjz\" (UniqueName: \"kubernetes.io/projected/4eba974d-5929-4fe6-8dec-4e3d7fab0e0e-kube-api-access-grtjz\") pod \"metrics-server-5dfbb88bb-tpbp8\" (UID: \"4eba974d-5929-4fe6-8dec-4e3d7fab0e0e\") " pod="openshift-monitoring/metrics-server-5dfbb88bb-tpbp8"
Sep 18 19:16:23.943890 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:23.943865    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5dfbb88bb-tpbp8"
Sep 18 19:16:24.232021 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:24.231958    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5dfbb88bb-tpbp8"]
Sep 18 19:16:24.238947 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:16:24.238927    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eba974d_5929_4fe6_8dec_4e3d7fab0e0e.slice/crio-20f1f243e5322e2a9edca9cbef2d982b1f77bf1fc4ff76eb31ec4b558d24f3c3 WatchSource:0}: Error finding container 20f1f243e5322e2a9edca9cbef2d982b1f77bf1fc4ff76eb31ec4b558d24f3c3: Status 404 returned error can't find the container with id 20f1f243e5322e2a9edca9cbef2d982b1f77bf1fc4ff76eb31ec4b558d24f3c3
Sep 18 19:16:24.271696 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:24.271666    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:24.374282 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:24.374255    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:25.200690 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:25.200647    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:25.220143 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:25.220098    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5dfbb88bb-tpbp8" event={"ID":"4eba974d-5929-4fe6-8dec-4e3d7fab0e0e","Type":"ContainerStarted","Data":"20f1f243e5322e2a9edca9cbef2d982b1f77bf1fc4ff76eb31ec4b558d24f3c3"}
Sep 18 19:16:25.306573 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:25.306533    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:26.224641 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:26.224611    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5dfbb88bb-tpbp8" event={"ID":"4eba974d-5929-4fe6-8dec-4e3d7fab0e0e","Type":"ContainerStarted","Data":"da2b01e18bcf655c6a2ed5eaf8adf1a0a8dedc4734d311aff14fa21556c207fc"}
Sep 18 19:16:26.244264 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:26.244222    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5dfbb88bb-tpbp8" podStartSLOduration=1.966291046 podStartE2EDuration="3.244208197s" totalImagesPullingTime="1.277917151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:16:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:16:26.242658704 +0000 UTC m=+153.163162562" watchObservedRunningTime="2026-09-18 19:16:26.244208197 +0000 UTC m=+153.164712097"
Sep 18 19:16:28.175470 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:28.175430    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:28.275749 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:28.275723    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:30.645976 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:16:30.645928    2532 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-54cns" podUID="ad80f379-b5dc-4d02-a61d-b8b2a956a551"
Sep 18 19:16:31.240430 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:31.240395    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-54cns"
Sep 18 19:16:35.450122 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:35.450089    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:35.553457 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:35.553423    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:35.582546 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:35.582508    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad80f379-b5dc-4d02-a61d-b8b2a956a551-metrics-tls\") pod \"dns-default-54cns\" (UID: \"ad80f379-b5dc-4d02-a61d-b8b2a956a551\") " pod="openshift-dns/dns-default-54cns"
Sep 18 19:16:35.584745 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:35.584725    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad80f379-b5dc-4d02-a61d-b8b2a956a551-metrics-tls\") pod \"dns-default-54cns\" (UID: \"ad80f379-b5dc-4d02-a61d-b8b2a956a551\") " pod="openshift-dns/dns-default-54cns"
Sep 18 19:16:35.663669 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:35.663645    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:35.743374 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:35.743313    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-sd5f2\""
Sep 18 19:16:35.751292 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:35.751275    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-54cns"
Sep 18 19:16:36.029198 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:36.029132    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-54cns"]
Sep 18 19:16:36.033838 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:16:36.033814    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad80f379_b5dc_4d02_a61d_b8b2a956a551.slice/crio-8240cd7cd0daaac44407045e8a763ee18f92889edb6d127232dc15de6c8b9cc7 WatchSource:0}: Error finding container 8240cd7cd0daaac44407045e8a763ee18f92889edb6d127232dc15de6c8b9cc7: Status 404 returned error can't find the container with id 8240cd7cd0daaac44407045e8a763ee18f92889edb6d127232dc15de6c8b9cc7
Sep 18 19:16:36.257434 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:36.257404    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-54cns" event={"ID":"ad80f379-b5dc-4d02-a61d-b8b2a956a551","Type":"ContainerStarted","Data":"8240cd7cd0daaac44407045e8a763ee18f92889edb6d127232dc15de6c8b9cc7"}
Sep 18 19:16:38.145615 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:38.145539    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:38.265957 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:38.265926    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-54cns" event={"ID":"ad80f379-b5dc-4d02-a61d-b8b2a956a551","Type":"ContainerStarted","Data":"c1644242fa0a872c532de3f39624f78a344dd8b6ea0d7f3ebb282a32b0a01aa8"}
Sep 18 19:16:38.266113 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:38.265975    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-54cns" event={"ID":"ad80f379-b5dc-4d02-a61d-b8b2a956a551","Type":"ContainerStarted","Data":"7442340cbd9ca87992fbcb59638d8a7f46392c140c189b738dafeeb09f273ded"}
Sep 18 19:16:38.266113 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:38.266097    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-54cns"
Sep 18 19:16:38.300356 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:38.300314    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-54cns" podStartSLOduration=129.528141025 podStartE2EDuration="2m11.300302037s" totalImagesPullingTime="1.772161012s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:14:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:16:38.299285147 +0000 UTC m=+165.219789015" watchObservedRunningTime="2026-09-18 19:16:38.300302037 +0000 UTC m=+165.220805894"
Sep 18 19:16:38.328938 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:38.328916    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:38.432824 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:38.432788    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:38.526532 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:38.526502    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:43.290182 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:43.290151    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:43.398320 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:43.398291    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:43.944243 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:43.944216    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5dfbb88bb-tpbp8"
Sep 18 19:16:43.944393 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:43.944258    2532 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5dfbb88bb-tpbp8"
Sep 18 19:16:46.560263 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:46.560171    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:46.663253 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:46.663222    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:48.271050 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:48.271025    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-54cns"
Sep 18 19:16:57.670179 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:57.670144    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:58.331609 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:58.331572    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:58.438427 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:16:58.438394    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:17:03.949699 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:17:03.949671    2532 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5dfbb88bb-tpbp8"
Sep 18 19:17:03.954578 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:17:03.954559    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5dfbb88bb-tpbp8"
Sep 18 19:17:08.045492 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:17:08.045462    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:17:08.240348 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:17:08.240312    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:18:20.658568 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:18:20.658477    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:18:20.752285 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:18:20.752257    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:18:25.239964 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:18:25.239934    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:18:41.149231 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:18:41.149188    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:18:53.496028 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:18:53.495997    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sksw8_2cef0f4e-e44f-4996-8561-da9d261036c7/ovn-acl-logging/0.log"
Sep 18 19:18:53.503186 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:18:53.503165    2532 kubelet.go:1648] "Image garbage collection succeeded"
Sep 18 19:18:53.503883 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:18:53.503865    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sksw8_2cef0f4e-e44f-4996-8561-da9d261036c7/ovn-acl-logging/0.log"
Sep 18 19:19:11.130681 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:19:11.130642    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:21:23.788544 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:21:23.788512    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:21:25.185078 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:21:25.185029    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:21:27.023098 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:21:27.023039    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:21:27.106033 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:21:27.105998    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:21:27.254282 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:21:27.254245    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:21:27.356021 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:21:27.355940    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:21:30.517226 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:21:30.517195    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:21:32.748174 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:21:32.748137    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:21:32.828113 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:21:32.828073    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:21:43.650108 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:21:43.650048    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:21:43.778042 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:21:43.778001    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:22:40.287521 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:22:40.287476    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:22:40.386977 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:22:40.386945    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:22:49.561320 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:22:49.561237    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:22:52.242016 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:22:52.241981    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:22:52.310960 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:22:52.310924    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:22:58.303226 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:22:58.303186    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:22:58.392931 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:22:58.392899    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:22:58.477338 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:22:58.477305    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:23:00.889387 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:23:00.889346    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:23:00.989286 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:23:00.989259    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:23:01.061526 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:23:01.061492    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:23:45.256029 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:23:45.255990    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:23:45.356395 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:23:45.356354    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:23:45.457153 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:23:45.457120    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:23:53.516721 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:23:53.516693    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sksw8_2cef0f4e-e44f-4996-8561-da9d261036c7/ovn-acl-logging/0.log"
Sep 18 19:23:53.521909 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:23:53.521885    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sksw8_2cef0f4e-e44f-4996-8561-da9d261036c7/ovn-acl-logging/0.log"
Sep 18 19:24:08.343041 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:24:08.343002    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:24:08.469847 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:24:08.469813    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:24:15.275498 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:24:15.275461    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:24:15.489031 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:24:15.488975    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:24:15.637515 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:24:15.637435    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:24:15.832193 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:24:15.832160    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:24:15.907176 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:24:15.907096    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:24:16.444940 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:24:16.444902    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:24:16.543330 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:24:16.543305    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-nrbbq"]
Sep 18 19:24:16.546158 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:24:16.546135    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-nrbbq"
Sep 18 19:24:16.549738 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:24:16.549710    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 18 19:24:16.549865 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:24:16.549822    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-4lfp8\""
Sep 18 19:24:16.550360 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:24:16.550341    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 18 19:24:16.551176 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:24:16.551155    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 18 19:24:16.559594 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:24:16.559577    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 18 19:24:16.576928 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:24:16.576906    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-nrbbq"]
Sep 18 19:24:16.586624 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:24:16.586601    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:24:16.662432 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:24:16.662394    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:24:16.717419 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:24:16.717363    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6956\" (UniqueName: \"kubernetes.io/projected/3f1be2c2-cd25-42cb-9331-9fa443006813-kube-api-access-c6956\") pod \"keda-admission-6c4d864b86-nrbbq\" (UID: \"3f1be2c2-cd25-42cb-9331-9fa443006813\") " pod="openshift-keda/keda-admission-6c4d864b86-nrbbq"
Sep 18 19:24:16.717500 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:24:16.717417    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3f1be2c2-cd25-42cb-9331-9fa443006813-certificates\") pod \"keda-admission-6c4d864b86-nrbbq\" (UID: \"3f1be2c2-cd25-42cb-9331-9fa443006813\") " pod="openshift-keda/keda-admission-6c4d864b86-nrbbq"
Sep 18 19:24:16.761351 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:24:16.761328    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:24:16.818498 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:24:16.818473    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3f1be2c2-cd25-42cb-9331-9fa443006813-certificates\") pod \"keda-admission-6c4d864b86-nrbbq\" (UID: \"3f1be2c2-cd25-42cb-9331-9fa443006813\") " pod="openshift-keda/keda-admission-6c4d864b86-nrbbq"
Sep 18 19:24:16.818588 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:24:16.818541    2532 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Sep 18 19:24:16.818588 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:24:16.818568    2532 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-6c4d864b86-nrbbq: secret "keda-admission-webhooks-certs" not found
Sep 18 19:24:16.818699 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:24:16.818612    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f1be2c2-cd25-42cb-9331-9fa443006813-certificates podName:3f1be2c2-cd25-42cb-9331-9fa443006813 nodeName:}" failed. No retries permitted until 2026-09-18 19:24:17.318597454 +0000 UTC m=+624.239101288 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/3f1be2c2-cd25-42cb-9331-9fa443006813-certificates") pod "keda-admission-6c4d864b86-nrbbq" (UID: "3f1be2c2-cd25-42cb-9331-9fa443006813") : secret "keda-admission-webhooks-certs" not found
Sep 18 19:24:16.818699 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:24:16.818562    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6956\" (UniqueName: \"kubernetes.io/projected/3f1be2c2-cd25-42cb-9331-9fa443006813-kube-api-access-c6956\") pod \"keda-admission-6c4d864b86-nrbbq\" (UID: \"3f1be2c2-cd25-42cb-9331-9fa443006813\") " pod="openshift-keda/keda-admission-6c4d864b86-nrbbq"
Sep 18 19:24:16.827662 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:24:16.827638    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6956\" (UniqueName: \"kubernetes.io/projected/3f1be2c2-cd25-42cb-9331-9fa443006813-kube-api-access-c6956\") pod \"keda-admission-6c4d864b86-nrbbq\" (UID: \"3f1be2c2-cd25-42cb-9331-9fa443006813\") " pod="openshift-keda/keda-admission-6c4d864b86-nrbbq"
Sep 18 19:24:16.969040 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:24:16.968976    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:24:17.185911 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:24:17.185871    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:24:17.301213 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:24:17.301136    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:24:17.321208 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:24:17.321184    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3f1be2c2-cd25-42cb-9331-9fa443006813-certificates\") pod \"keda-admission-6c4d864b86-nrbbq\" (UID: \"3f1be2c2-cd25-42cb-9331-9fa443006813\") " pod="openshift-keda/keda-admission-6c4d864b86-nrbbq"
Sep 18 19:24:17.323659 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:24:17.323632    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3f1be2c2-cd25-42cb-9331-9fa443006813-certificates\") pod \"keda-admission-6c4d864b86-nrbbq\" (UID: \"3f1be2c2-cd25-42cb-9331-9fa443006813\") " pod="openshift-keda/keda-admission-6c4d864b86-nrbbq"
Sep 18 19:24:17.455605 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:24:17.455581    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-nrbbq"
Sep 18 19:24:17.517827 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:24:17.517797    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:24:17.611654 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:24:17.611586    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:24:17.735194 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:24:17.735166    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-nrbbq"]
Sep 18 19:24:17.739670 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:24:17.739638    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f1be2c2_cd25_42cb_9331_9fa443006813.slice/crio-746a735607ee33ec18df2a8e7c1623f71f98b866e44971b06ca7797061046e66 WatchSource:0}: Error finding container 746a735607ee33ec18df2a8e7c1623f71f98b866e44971b06ca7797061046e66: Status 404 returned error can't find the container with id 746a735607ee33ec18df2a8e7c1623f71f98b866e44971b06ca7797061046e66
Sep 18 19:24:17.741166 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:24:17.741147    2532 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:24:18.735345 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:24:18.735279    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-nrbbq" event={"ID":"3f1be2c2-cd25-42cb-9331-9fa443006813","Type":"ContainerStarted","Data":"746a735607ee33ec18df2a8e7c1623f71f98b866e44971b06ca7797061046e66"}
Sep 18 19:24:20.743488 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:24:20.743455    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-nrbbq" event={"ID":"3f1be2c2-cd25-42cb-9331-9fa443006813","Type":"ContainerStarted","Data":"3e83c5a8521f020bbead7ea3e7a26746b219b57670db2b590cc27c25ab2891db"}
Sep 18 19:24:20.743845 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:24:20.743539    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-nrbbq"
Sep 18 19:24:20.761689 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:24:20.761646    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-nrbbq" podStartSLOduration=2.167206449 podStartE2EDuration="4.761629211s" totalImagesPullingTime="2.594422762s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:24:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:24:20.760648414 +0000 UTC m=+627.681152270" watchObservedRunningTime="2026-09-18 19:24:20.761629211 +0000 UTC m=+627.682133070"
Sep 18 19:24:26.660629 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:24:26.660590    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:24:41.749365 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:24:41.749331    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-nrbbq"
Sep 18 19:24:45.814285 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:24:45.814251    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:24:45.883611 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:24:45.883560    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:25:07.577666 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:25:07.577629    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:25:07.728920 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:25:07.728886    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:25:22.011329 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:25:22.011295    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:25:22.177897 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:25:22.177872    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:25:40.362610 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:25:40.362567    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:25:40.468374 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:25:40.468341    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:25:40.495190 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:25:40.495159    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-2kcjk"]
Sep 18 19:25:40.498238 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:25:40.498218    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-2kcjk"
Sep 18 19:25:40.500718 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:25:40.500695    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 18 19:25:40.500718 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:25:40.500708    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 18 19:25:40.501220 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:25:40.501204    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 18 19:25:40.501353 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:25:40.501210    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-k2cgf\""
Sep 18 19:25:40.510892 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:25:40.510868    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-2kcjk"]
Sep 18 19:25:40.558689 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:25:40.558656    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8v6x\" (UniqueName: \"kubernetes.io/projected/e9d74016-32fe-4df7-9403-062ad106dd6d-kube-api-access-b8v6x\") pod \"seaweedfs-64568bcd49-2kcjk\" (UID: \"e9d74016-32fe-4df7-9403-062ad106dd6d\") " pod="kserve/seaweedfs-64568bcd49-2kcjk"
Sep 18 19:25:40.558828 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:25:40.558713    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e9d74016-32fe-4df7-9403-062ad106dd6d-data\") pod \"seaweedfs-64568bcd49-2kcjk\" (UID: \"e9d74016-32fe-4df7-9403-062ad106dd6d\") " pod="kserve/seaweedfs-64568bcd49-2kcjk"
Sep 18 19:25:40.653109 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:25:40.653009    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:25:40.660067 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:25:40.660037    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8v6x\" (UniqueName: \"kubernetes.io/projected/e9d74016-32fe-4df7-9403-062ad106dd6d-kube-api-access-b8v6x\") pod \"seaweedfs-64568bcd49-2kcjk\" (UID: \"e9d74016-32fe-4df7-9403-062ad106dd6d\") " pod="kserve/seaweedfs-64568bcd49-2kcjk"
Sep 18 19:25:40.660161 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:25:40.660146    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e9d74016-32fe-4df7-9403-062ad106dd6d-data\") pod \"seaweedfs-64568bcd49-2kcjk\" (UID: \"e9d74016-32fe-4df7-9403-062ad106dd6d\") " pod="kserve/seaweedfs-64568bcd49-2kcjk"
Sep 18 19:25:40.660616 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:25:40.660596    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e9d74016-32fe-4df7-9403-062ad106dd6d-data\") pod \"seaweedfs-64568bcd49-2kcjk\" (UID: \"e9d74016-32fe-4df7-9403-062ad106dd6d\") " pod="kserve/seaweedfs-64568bcd49-2kcjk"
Sep 18 19:25:40.670411 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:25:40.670390    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8v6x\" (UniqueName: \"kubernetes.io/projected/e9d74016-32fe-4df7-9403-062ad106dd6d-kube-api-access-b8v6x\") pod \"seaweedfs-64568bcd49-2kcjk\" (UID: \"e9d74016-32fe-4df7-9403-062ad106dd6d\") " pod="kserve/seaweedfs-64568bcd49-2kcjk"
Sep 18 19:25:40.752608 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:25:40.752579    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:25:40.807659 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:25:40.807643    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-2kcjk"
Sep 18 19:25:41.084240 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:25:41.084205    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-2kcjk"]
Sep 18 19:25:41.087721 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:25:41.087692    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9d74016_32fe_4df7_9403_062ad106dd6d.slice/crio-2eab7b07d9bd3f1f275c7cc75eaf4aa2007933325951d4077f76524a0c9c79f6 WatchSource:0}: Error finding container 2eab7b07d9bd3f1f275c7cc75eaf4aa2007933325951d4077f76524a0c9c79f6: Status 404 returned error can't find the container with id 2eab7b07d9bd3f1f275c7cc75eaf4aa2007933325951d4077f76524a0c9c79f6
Sep 18 19:25:41.175360 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:25:41.175320    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:25:41.284948 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:25:41.284922    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:25:41.471002 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:25:41.470980    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:25:42.008477 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:25:42.008443    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-2kcjk" event={"ID":"e9d74016-32fe-4df7-9403-062ad106dd6d","Type":"ContainerStarted","Data":"2eab7b07d9bd3f1f275c7cc75eaf4aa2007933325951d4077f76524a0c9c79f6"}
Sep 18 19:25:44.016876 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:25:44.016843    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-2kcjk" event={"ID":"e9d74016-32fe-4df7-9403-062ad106dd6d","Type":"ContainerStarted","Data":"5a4cdaa3292ef950a64eb12e1291c66834ca9768a666a54144382f7c7adcbcde"}
Sep 18 19:25:44.017235 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:25:44.016959    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-2kcjk"
Sep 18 19:25:44.034700 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:25:44.034652    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-2kcjk" podStartSLOduration=1.313804452 podStartE2EDuration="4.034638706s" totalImagesPullingTime="2.720834254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:25:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:25:44.0337633 +0000 UTC m=+710.954267158" watchObservedRunningTime="2026-09-18 19:25:44.034638706 +0000 UTC m=+710.955142563"
Sep 18 19:25:50.023434 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:25:50.023357    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-2kcjk"
Sep 18 19:26:02.744517 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:26:02.744482    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:26:02.818418 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:26:02.818390    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:26:10.834825 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:26:10.834791    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:26:10.897285 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:26:10.897252    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:26:19.128118 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:26:19.128081    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:26:30.044316 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:26:30.044277    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:26:31.527649 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:26:31.527611    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:27:02.997136 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:27:02.997101    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:27:03.066477 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:27:03.066444    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-cg6kz"]
Sep 18 19:27:03.069529 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:27:03.069513    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-cg6kz"
Sep 18 19:27:03.072018 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:27:03.071987    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 18 19:27:03.072162 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:27:03.072112    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-r482j\""
Sep 18 19:27:03.072923 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:27:03.072906    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 18 19:27:03.087088 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:27:03.087041    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-cg6kz"]
Sep 18 19:27:03.091837 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:27:03.091809    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:27:03.097523 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:27:03.097498    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhgxq\" (UniqueName: \"kubernetes.io/projected/6632b05b-6137-43b0-90d6-c2faf0a9c31f-kube-api-access-mhgxq\") pod \"odh-model-controller-7b966d7984-cg6kz\" (UID: \"6632b05b-6137-43b0-90d6-c2faf0a9c31f\") " pod="kserve/odh-model-controller-7b966d7984-cg6kz"
Sep 18 19:27:03.097632 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:27:03.097543    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6632b05b-6137-43b0-90d6-c2faf0a9c31f-metrics-certs\") pod \"odh-model-controller-7b966d7984-cg6kz\" (UID: \"6632b05b-6137-43b0-90d6-c2faf0a9c31f\") " pod="kserve/odh-model-controller-7b966d7984-cg6kz"
Sep 18 19:27:03.097632 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:27:03.097579    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6632b05b-6137-43b0-90d6-c2faf0a9c31f-cert\") pod \"odh-model-controller-7b966d7984-cg6kz\" (UID: \"6632b05b-6137-43b0-90d6-c2faf0a9c31f\") " pod="kserve/odh-model-controller-7b966d7984-cg6kz"
Sep 18 19:27:03.186448 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:27:03.186415    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:27:03.198609 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:27:03.198589    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6632b05b-6137-43b0-90d6-c2faf0a9c31f-metrics-certs\") pod \"odh-model-controller-7b966d7984-cg6kz\" (UID: \"6632b05b-6137-43b0-90d6-c2faf0a9c31f\") " pod="kserve/odh-model-controller-7b966d7984-cg6kz"
Sep 18 19:27:03.198747 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:27:03.198626    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6632b05b-6137-43b0-90d6-c2faf0a9c31f-cert\") pod \"odh-model-controller-7b966d7984-cg6kz\" (UID: \"6632b05b-6137-43b0-90d6-c2faf0a9c31f\") " pod="kserve/odh-model-controller-7b966d7984-cg6kz"
Sep 18 19:27:03.198747 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:27:03.198655    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhgxq\" (UniqueName: \"kubernetes.io/projected/6632b05b-6137-43b0-90d6-c2faf0a9c31f-kube-api-access-mhgxq\") pod \"odh-model-controller-7b966d7984-cg6kz\" (UID: \"6632b05b-6137-43b0-90d6-c2faf0a9c31f\") " pod="kserve/odh-model-controller-7b966d7984-cg6kz"
Sep 18 19:27:03.198849 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:27:03.198754    2532 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 18 19:27:03.198849 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:27:03.198830    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6632b05b-6137-43b0-90d6-c2faf0a9c31f-cert podName:6632b05b-6137-43b0-90d6-c2faf0a9c31f nodeName:}" failed. No retries permitted until 2026-09-18 19:27:03.698810824 +0000 UTC m=+790.619314674 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/6632b05b-6137-43b0-90d6-c2faf0a9c31f-cert") pod "odh-model-controller-7b966d7984-cg6kz" (UID: "6632b05b-6137-43b0-90d6-c2faf0a9c31f") : secret "odh-model-controller-webhook-cert" not found
Sep 18 19:27:03.201215 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:27:03.201194    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6632b05b-6137-43b0-90d6-c2faf0a9c31f-metrics-certs\") pod \"odh-model-controller-7b966d7984-cg6kz\" (UID: \"6632b05b-6137-43b0-90d6-c2faf0a9c31f\") " pod="kserve/odh-model-controller-7b966d7984-cg6kz"
Sep 18 19:27:03.207465 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:27:03.207444    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhgxq\" (UniqueName: \"kubernetes.io/projected/6632b05b-6137-43b0-90d6-c2faf0a9c31f-kube-api-access-mhgxq\") pod \"odh-model-controller-7b966d7984-cg6kz\" (UID: \"6632b05b-6137-43b0-90d6-c2faf0a9c31f\") " pod="kserve/odh-model-controller-7b966d7984-cg6kz"
Sep 18 19:27:03.361777 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:27:03.361685    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:27:03.701186 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:27:03.701159    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6632b05b-6137-43b0-90d6-c2faf0a9c31f-cert\") pod \"odh-model-controller-7b966d7984-cg6kz\" (UID: \"6632b05b-6137-43b0-90d6-c2faf0a9c31f\") " pod="kserve/odh-model-controller-7b966d7984-cg6kz"
Sep 18 19:27:03.701356 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:27:03.701296    2532 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 18 19:27:03.701441 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:27:03.701378    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6632b05b-6137-43b0-90d6-c2faf0a9c31f-cert podName:6632b05b-6137-43b0-90d6-c2faf0a9c31f nodeName:}" failed. No retries permitted until 2026-09-18 19:27:04.701357651 +0000 UTC m=+791.621861500 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/6632b05b-6137-43b0-90d6-c2faf0a9c31f-cert") pod "odh-model-controller-7b966d7984-cg6kz" (UID: "6632b05b-6137-43b0-90d6-c2faf0a9c31f") : secret "odh-model-controller-webhook-cert" not found
Sep 18 19:27:03.768117 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:27:03.768047    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:27:04.573527 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:27:04.573485    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:27:04.675010 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:27:04.674964    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:27:04.707959 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:27:04.707937    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6632b05b-6137-43b0-90d6-c2faf0a9c31f-cert\") pod \"odh-model-controller-7b966d7984-cg6kz\" (UID: \"6632b05b-6137-43b0-90d6-c2faf0a9c31f\") " pod="kserve/odh-model-controller-7b966d7984-cg6kz"
Sep 18 19:27:04.710383 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:27:04.710355    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6632b05b-6137-43b0-90d6-c2faf0a9c31f-cert\") pod \"odh-model-controller-7b966d7984-cg6kz\" (UID: \"6632b05b-6137-43b0-90d6-c2faf0a9c31f\") " pod="kserve/odh-model-controller-7b966d7984-cg6kz"
Sep 18 19:27:04.883006 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:27:04.882930    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-cg6kz"
Sep 18 19:27:05.159359 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:27:05.159161    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-cg6kz"]
Sep 18 19:27:05.161811 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:27:05.161788    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6632b05b_6137_43b0_90d6_c2faf0a9c31f.slice/crio-79d097e3971ca939aff306471dda8fa6871d3fb48e01c23a0586550215a99510 WatchSource:0}: Error finding container 79d097e3971ca939aff306471dda8fa6871d3fb48e01c23a0586550215a99510: Status 404 returned error can't find the container with id 79d097e3971ca939aff306471dda8fa6871d3fb48e01c23a0586550215a99510
Sep 18 19:27:05.283821 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:27:05.283793    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-cg6kz" event={"ID":"6632b05b-6137-43b0-90d6-c2faf0a9c31f","Type":"ContainerStarted","Data":"79d097e3971ca939aff306471dda8fa6871d3fb48e01c23a0586550215a99510"}
Sep 18 19:27:08.295665 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:27:08.295626    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-cg6kz" event={"ID":"6632b05b-6137-43b0-90d6-c2faf0a9c31f","Type":"ContainerStarted","Data":"74e0777c2795d48cea4139a62f9f68c50e6aa00825d200a6ba4c37bd638807bb"}
Sep 18 19:27:08.296046 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:27:08.295691    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-cg6kz"
Sep 18 19:27:08.322695 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:27:08.322649    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-cg6kz" podStartSLOduration=2.687279831 podStartE2EDuration="5.32263242s" totalImagesPullingTime="2.635352589s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:27:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:27:08.321589524 +0000 UTC m=+795.242093381" watchObservedRunningTime="2026-09-18 19:27:08.32263242 +0000 UTC m=+795.243136279"
Sep 18 19:27:19.301396 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:27:19.301313    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-cg6kz"
Sep 18 19:27:20.237733 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:27:20.237699    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-bwxwl"]
Sep 18 19:27:20.242321 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:27:20.242300    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-bwxwl"
Sep 18 19:27:20.244777 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:27:20.244759    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 18 19:27:20.252784 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:27:20.252759    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-bwxwl"]
Sep 18 19:27:20.317220 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:27:20.317197    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12b53e18-63de-43fc-85e6-2e3508ea739b-model-cache\") pod \"s3-init-bwxwl\" (UID: \"12b53e18-63de-43fc-85e6-2e3508ea739b\") " pod="kserve/s3-init-bwxwl"
Sep 18 19:27:20.317546 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:27:20.317248    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjtmd\" (UniqueName: \"kubernetes.io/projected/12b53e18-63de-43fc-85e6-2e3508ea739b-kube-api-access-fjtmd\") pod \"s3-init-bwxwl\" (UID: \"12b53e18-63de-43fc-85e6-2e3508ea739b\") " pod="kserve/s3-init-bwxwl"
Sep 18 19:27:20.417872 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:27:20.417841    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjtmd\" (UniqueName: \"kubernetes.io/projected/12b53e18-63de-43fc-85e6-2e3508ea739b-kube-api-access-fjtmd\") pod \"s3-init-bwxwl\" (UID: \"12b53e18-63de-43fc-85e6-2e3508ea739b\") " pod="kserve/s3-init-bwxwl"
Sep 18 19:27:20.418001 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:27:20.417900    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12b53e18-63de-43fc-85e6-2e3508ea739b-model-cache\") pod \"s3-init-bwxwl\" (UID: \"12b53e18-63de-43fc-85e6-2e3508ea739b\") " pod="kserve/s3-init-bwxwl"
Sep 18 19:27:20.418247 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:27:20.418233    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12b53e18-63de-43fc-85e6-2e3508ea739b-model-cache\") pod \"s3-init-bwxwl\" (UID: \"12b53e18-63de-43fc-85e6-2e3508ea739b\") " pod="kserve/s3-init-bwxwl"
Sep 18 19:27:20.427006 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:27:20.426980    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjtmd\" (UniqueName: \"kubernetes.io/projected/12b53e18-63de-43fc-85e6-2e3508ea739b-kube-api-access-fjtmd\") pod \"s3-init-bwxwl\" (UID: \"12b53e18-63de-43fc-85e6-2e3508ea739b\") " pod="kserve/s3-init-bwxwl"
Sep 18 19:27:20.551309 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:27:20.551254    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-bwxwl"
Sep 18 19:27:20.824165 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:27:20.824099    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-bwxwl"]
Sep 18 19:27:20.827605 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:27:20.827577    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod12b53e18_63de_43fc_85e6_2e3508ea739b.slice/crio-a179cae2560bc80430961971f77cd6b781652b9b84192fd40d5f5f48a0ecb5f1 WatchSource:0}: Error finding container a179cae2560bc80430961971f77cd6b781652b9b84192fd40d5f5f48a0ecb5f1: Status 404 returned error can't find the container with id a179cae2560bc80430961971f77cd6b781652b9b84192fd40d5f5f48a0ecb5f1
Sep 18 19:27:21.342973 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:27:21.342940    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-bwxwl" event={"ID":"12b53e18-63de-43fc-85e6-2e3508ea739b","Type":"ContainerStarted","Data":"a179cae2560bc80430961971f77cd6b781652b9b84192fd40d5f5f48a0ecb5f1"}
Sep 18 19:27:24.356580 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:27:24.356548    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-bwxwl" event={"ID":"12b53e18-63de-43fc-85e6-2e3508ea739b","Type":"ContainerStarted","Data":"95d9351708d408d8740f2eacd8cd9f30abbdcf98c6fb41fe52d8b9d4b411fc82"}
Sep 18 19:27:26.366373 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:27:26.366341    2532 generic.go:356] "Generic (PLEG): container finished" podID="12b53e18-63de-43fc-85e6-2e3508ea739b" containerID="95d9351708d408d8740f2eacd8cd9f30abbdcf98c6fb41fe52d8b9d4b411fc82" exitCode=0
Sep 18 19:27:26.366717 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:27:26.366381    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-bwxwl" event={"ID":"12b53e18-63de-43fc-85e6-2e3508ea739b","Type":"ContainerDied","Data":"95d9351708d408d8740f2eacd8cd9f30abbdcf98c6fb41fe52d8b9d4b411fc82"}
Sep 18 19:27:31.390114 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:27:31.390080    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-bwxwl" event={"ID":"12b53e18-63de-43fc-85e6-2e3508ea739b","Type":"ContainerStarted","Data":"73f31d6de56ec9e705a272a14eef9dc4d12755d26a93b2714ff1d0cc1af7e34d"}
Sep 18 19:27:31.411494 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:27:31.411448    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-bwxwl" podStartSLOduration=1.843822868 podStartE2EDuration="11.411432684s" totalImagesPullingTime="7.696767992s" totalInitContainerRuntime="1.870841824s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:27:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:27:31.410136775 +0000 UTC m=+818.330640632" watchObservedRunningTime="2026-09-18 19:27:31.411432684 +0000 UTC m=+818.331936540"
Sep 18 19:27:34.611457 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:27:34.611421    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:28:13.587554 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:13.587516    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:28:35.611014 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:35.610986    2532 generic.go:356] "Generic (PLEG): container finished" podID="12b53e18-63de-43fc-85e6-2e3508ea739b" containerID="73f31d6de56ec9e705a272a14eef9dc4d12755d26a93b2714ff1d0cc1af7e34d" exitCode=0
Sep 18 19:28:35.611368 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:35.611066    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-bwxwl" event={"ID":"12b53e18-63de-43fc-85e6-2e3508ea739b","Type":"ContainerDied","Data":"73f31d6de56ec9e705a272a14eef9dc4d12755d26a93b2714ff1d0cc1af7e34d"}
Sep 18 19:28:36.728917 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:36.728895    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-bwxwl"
Sep 18 19:28:36.842802 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:36.842770    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fjtmd\" (UniqueName: \"kubernetes.io/projected/12b53e18-63de-43fc-85e6-2e3508ea739b-kube-api-access-fjtmd\") pod \"12b53e18-63de-43fc-85e6-2e3508ea739b\" (UID: \"12b53e18-63de-43fc-85e6-2e3508ea739b\") "
Sep 18 19:28:36.842961 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:36.842859    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12b53e18-63de-43fc-85e6-2e3508ea739b-model-cache\") pod \"12b53e18-63de-43fc-85e6-2e3508ea739b\" (UID: \"12b53e18-63de-43fc-85e6-2e3508ea739b\") "
Sep 18 19:28:36.844943 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:36.844913    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12b53e18-63de-43fc-85e6-2e3508ea739b-kube-api-access-fjtmd" (OuterVolumeSpecName: "kube-api-access-fjtmd") pod "12b53e18-63de-43fc-85e6-2e3508ea739b" (UID: "12b53e18-63de-43fc-85e6-2e3508ea739b"). InnerVolumeSpecName "kube-api-access-fjtmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:28:36.861912 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:36.861883    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12b53e18-63de-43fc-85e6-2e3508ea739b-model-cache" (OuterVolumeSpecName: "model-cache") pod "12b53e18-63de-43fc-85e6-2e3508ea739b" (UID: "12b53e18-63de-43fc-85e6-2e3508ea739b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:28:36.943574 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:36.943545    2532 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12b53e18-63de-43fc-85e6-2e3508ea739b-model-cache\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:28:36.943574 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:36.943573    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjtmd\" (UniqueName: \"kubernetes.io/projected/12b53e18-63de-43fc-85e6-2e3508ea739b-kube-api-access-fjtmd\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:28:37.619449 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:37.619421    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-bwxwl"
Sep 18 19:28:37.619824 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:37.619800    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-bwxwl" event={"ID":"12b53e18-63de-43fc-85e6-2e3508ea739b","Type":"ContainerDied","Data":"a179cae2560bc80430961971f77cd6b781652b9b84192fd40d5f5f48a0ecb5f1"}
Sep 18 19:28:37.619899 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:37.619835    2532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a179cae2560bc80430961971f77cd6b781652b9b84192fd40d5f5f48a0ecb5f1"
Sep 18 19:28:38.000511 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:38.000471    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:28:38.101318 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:38.101290    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:28:50.369138 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.369040    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:28:50.562562 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.562529    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2"]
Sep 18 19:28:50.562778 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.562767    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12b53e18-63de-43fc-85e6-2e3508ea739b" containerName="download-hf-model"
Sep 18 19:28:50.562821 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.562779    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="12b53e18-63de-43fc-85e6-2e3508ea739b" containerName="download-hf-model"
Sep 18 19:28:50.562821 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.562791    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12b53e18-63de-43fc-85e6-2e3508ea739b" containerName="s3-init"
Sep 18 19:28:50.562821 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.562797    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="12b53e18-63de-43fc-85e6-2e3508ea739b" containerName="s3-init"
Sep 18 19:28:50.562909 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.562840    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="12b53e18-63de-43fc-85e6-2e3508ea739b" containerName="s3-init"
Sep 18 19:28:50.565770 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.565752    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2"
Sep 18 19:28:50.568296 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.568271    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-ct9wb\""
Sep 18 19:28:50.568847 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.568826    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 19:28:50.569137 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.569120    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 19:28:50.569238 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.569154    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 19:28:50.569445 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.569421    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-fb30d-kube-rbac-proxy-sar-config\""
Sep 18 19:28:50.569531 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.569480    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 19:28:50.569591 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.569541    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-fb30d-predictor-serving-cert\""
Sep 18 19:28:50.578360 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.578335    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2"]
Sep 18 19:28:50.581649 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.581625    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:28:50.663943 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.663814    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:28:50.664882 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.664862    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb"]
Sep 18 19:28:50.668019 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.668002    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb"
Sep 18 19:28:50.670226 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.670209    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 18 19:28:50.670329 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.670274    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Sep 18 19:28:50.678384 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.678364    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb"]
Sep 18 19:28:50.702341 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.702319    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-tflm2"]
Sep 18 19:28:50.705461 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.705447    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-tflm2"
Sep 18 19:28:50.708393 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.708376    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 18 19:28:50.716896 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.716875    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-tflm2"]
Sep 18 19:28:50.737437 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.737410    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:28:50.741371 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.741353    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b801f0ad-e919-4d3d-bf92-2dfd46d0029e-proxy-tls\") pod \"isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2\" (UID: \"b801f0ad-e919-4d3d-bf92-2dfd46d0029e\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2"
Sep 18 19:28:50.741458 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.741381    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b801f0ad-e919-4d3d-bf92-2dfd46d0029e-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2\" (UID: \"b801f0ad-e919-4d3d-bf92-2dfd46d0029e\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2"
Sep 18 19:28:50.741458 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.741403    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b801f0ad-e919-4d3d-bf92-2dfd46d0029e-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2\" (UID: \"b801f0ad-e919-4d3d-bf92-2dfd46d0029e\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2"
Sep 18 19:28:50.741458 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.741434    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b801f0ad-e919-4d3d-bf92-2dfd46d0029e-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2\" (UID: \"b801f0ad-e919-4d3d-bf92-2dfd46d0029e\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2"
Sep 18 19:28:50.741589 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.741490    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bnd6\" (UniqueName: \"kubernetes.io/projected/b801f0ad-e919-4d3d-bf92-2dfd46d0029e-kube-api-access-2bnd6\") pod \"isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2\" (UID: \"b801f0ad-e919-4d3d-bf92-2dfd46d0029e\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2"
Sep 18 19:28:50.842737 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.842708    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b282eb09-4598-4a77-8682-a3170ca1ccd2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb\" (UID: \"b282eb09-4598-4a77-8682-a3170ca1ccd2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb"
Sep 18 19:28:50.842867 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.842740    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1372611d-d215-4500-9e09-0b4fae36e5da-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-794fd55d6c-tflm2\" (UID: \"1372611d-d215-4500-9e09-0b4fae36e5da\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-tflm2"
Sep 18 19:28:50.842867 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.842761    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b282eb09-4598-4a77-8682-a3170ca1ccd2-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb\" (UID: \"b282eb09-4598-4a77-8682-a3170ca1ccd2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb"
Sep 18 19:28:50.842867 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.842791    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bnd6\" (UniqueName: \"kubernetes.io/projected/b801f0ad-e919-4d3d-bf92-2dfd46d0029e-kube-api-access-2bnd6\") pod \"isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2\" (UID: \"b801f0ad-e919-4d3d-bf92-2dfd46d0029e\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2"
Sep 18 19:28:50.843006 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.842881    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b282eb09-4598-4a77-8682-a3170ca1ccd2-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb\" (UID: \"b282eb09-4598-4a77-8682-a3170ca1ccd2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb"
Sep 18 19:28:50.843006 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.842916    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1372611d-d215-4500-9e09-0b4fae36e5da-cabundle-cert\") pod \"isvc-canary-rollback-predictor-794fd55d6c-tflm2\" (UID: \"1372611d-d215-4500-9e09-0b4fae36e5da\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-tflm2"
Sep 18 19:28:50.843006 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.842934    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b801f0ad-e919-4d3d-bf92-2dfd46d0029e-proxy-tls\") pod \"isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2\" (UID: \"b801f0ad-e919-4d3d-bf92-2dfd46d0029e\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2"
Sep 18 19:28:50.843006 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.842954    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b801f0ad-e919-4d3d-bf92-2dfd46d0029e-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2\" (UID: \"b801f0ad-e919-4d3d-bf92-2dfd46d0029e\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2"
Sep 18 19:28:50.843006 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.842980    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr88h\" (UniqueName: \"kubernetes.io/projected/1372611d-d215-4500-9e09-0b4fae36e5da-kube-api-access-pr88h\") pod \"isvc-canary-rollback-predictor-794fd55d6c-tflm2\" (UID: \"1372611d-d215-4500-9e09-0b4fae36e5da\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-tflm2"
Sep 18 19:28:50.843006 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.842997    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b282eb09-4598-4a77-8682-a3170ca1ccd2-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb\" (UID: \"b282eb09-4598-4a77-8682-a3170ca1ccd2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb"
Sep 18 19:28:50.843298 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.843015    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm6pt\" (UniqueName: \"kubernetes.io/projected/b282eb09-4598-4a77-8682-a3170ca1ccd2-kube-api-access-sm6pt\") pod \"isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb\" (UID: \"b282eb09-4598-4a77-8682-a3170ca1ccd2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb"
Sep 18 19:28:50.843298 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.843074    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b801f0ad-e919-4d3d-bf92-2dfd46d0029e-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2\" (UID: \"b801f0ad-e919-4d3d-bf92-2dfd46d0029e\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2"
Sep 18 19:28:50.843298 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.843164    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1372611d-d215-4500-9e09-0b4fae36e5da-proxy-tls\") pod \"isvc-canary-rollback-predictor-794fd55d6c-tflm2\" (UID: \"1372611d-d215-4500-9e09-0b4fae36e5da\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-tflm2"
Sep 18 19:28:50.843298 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.843188    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1372611d-d215-4500-9e09-0b4fae36e5da-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-794fd55d6c-tflm2\" (UID: \"1372611d-d215-4500-9e09-0b4fae36e5da\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-tflm2"
Sep 18 19:28:50.843298 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.843234    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b801f0ad-e919-4d3d-bf92-2dfd46d0029e-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2\" (UID: \"b801f0ad-e919-4d3d-bf92-2dfd46d0029e\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2"
Sep 18 19:28:50.843500 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.843428    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b801f0ad-e919-4d3d-bf92-2dfd46d0029e-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2\" (UID: \"b801f0ad-e919-4d3d-bf92-2dfd46d0029e\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2"
Sep 18 19:28:50.843851 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.843829    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b801f0ad-e919-4d3d-bf92-2dfd46d0029e-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2\" (UID: \"b801f0ad-e919-4d3d-bf92-2dfd46d0029e\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2"
Sep 18 19:28:50.843893 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.843874    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b801f0ad-e919-4d3d-bf92-2dfd46d0029e-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2\" (UID: \"b801f0ad-e919-4d3d-bf92-2dfd46d0029e\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2"
Sep 18 19:28:50.845396 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.845376    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b801f0ad-e919-4d3d-bf92-2dfd46d0029e-proxy-tls\") pod \"isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2\" (UID: \"b801f0ad-e919-4d3d-bf92-2dfd46d0029e\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2"
Sep 18 19:28:50.851834 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.851813    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bnd6\" (UniqueName: \"kubernetes.io/projected/b801f0ad-e919-4d3d-bf92-2dfd46d0029e-kube-api-access-2bnd6\") pod \"isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2\" (UID: \"b801f0ad-e919-4d3d-bf92-2dfd46d0029e\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2"
Sep 18 19:28:50.876402 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.876384    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2"
Sep 18 19:28:50.943813 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.943786    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b282eb09-4598-4a77-8682-a3170ca1ccd2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb\" (UID: \"b282eb09-4598-4a77-8682-a3170ca1ccd2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb"
Sep 18 19:28:50.943919 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.943821    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1372611d-d215-4500-9e09-0b4fae36e5da-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-794fd55d6c-tflm2\" (UID: \"1372611d-d215-4500-9e09-0b4fae36e5da\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-tflm2"
Sep 18 19:28:50.943919 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.943837    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b282eb09-4598-4a77-8682-a3170ca1ccd2-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb\" (UID: \"b282eb09-4598-4a77-8682-a3170ca1ccd2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb"
Sep 18 19:28:50.943919 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.943859    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b282eb09-4598-4a77-8682-a3170ca1ccd2-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb\" (UID: \"b282eb09-4598-4a77-8682-a3170ca1ccd2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb"
Sep 18 19:28:50.943919 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.943884    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1372611d-d215-4500-9e09-0b4fae36e5da-cabundle-cert\") pod \"isvc-canary-rollback-predictor-794fd55d6c-tflm2\" (UID: \"1372611d-d215-4500-9e09-0b4fae36e5da\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-tflm2"
Sep 18 19:28:50.943919 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.943903    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr88h\" (UniqueName: \"kubernetes.io/projected/1372611d-d215-4500-9e09-0b4fae36e5da-kube-api-access-pr88h\") pod \"isvc-canary-rollback-predictor-794fd55d6c-tflm2\" (UID: \"1372611d-d215-4500-9e09-0b4fae36e5da\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-tflm2"
Sep 18 19:28:50.944129 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.943921    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b282eb09-4598-4a77-8682-a3170ca1ccd2-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb\" (UID: \"b282eb09-4598-4a77-8682-a3170ca1ccd2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb"
Sep 18 19:28:50.944129 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.943937    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm6pt\" (UniqueName: \"kubernetes.io/projected/b282eb09-4598-4a77-8682-a3170ca1ccd2-kube-api-access-sm6pt\") pod \"isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb\" (UID: \"b282eb09-4598-4a77-8682-a3170ca1ccd2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb"
Sep 18 19:28:50.944129 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.943965    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1372611d-d215-4500-9e09-0b4fae36e5da-proxy-tls\") pod \"isvc-canary-rollback-predictor-794fd55d6c-tflm2\" (UID: \"1372611d-d215-4500-9e09-0b4fae36e5da\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-tflm2"
Sep 18 19:28:50.944129 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.943981    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1372611d-d215-4500-9e09-0b4fae36e5da-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-794fd55d6c-tflm2\" (UID: \"1372611d-d215-4500-9e09-0b4fae36e5da\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-tflm2"
Sep 18 19:28:50.944585 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.944564    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1372611d-d215-4500-9e09-0b4fae36e5da-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-794fd55d6c-tflm2\" (UID: \"1372611d-d215-4500-9e09-0b4fae36e5da\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-tflm2"
Sep 18 19:28:50.944675 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:28:50.944646    2532 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 18 19:28:50.944731 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:28:50.944697    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1372611d-d215-4500-9e09-0b4fae36e5da-proxy-tls podName:1372611d-d215-4500-9e09-0b4fae36e5da nodeName:}" failed. No retries permitted until 2026-09-18 19:28:51.444682389 +0000 UTC m=+898.365186224 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1372611d-d215-4500-9e09-0b4fae36e5da-proxy-tls") pod "isvc-canary-rollback-predictor-794fd55d6c-tflm2" (UID: "1372611d-d215-4500-9e09-0b4fae36e5da") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 18 19:28:50.944731 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.944716    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1372611d-d215-4500-9e09-0b4fae36e5da-cabundle-cert\") pod \"isvc-canary-rollback-predictor-794fd55d6c-tflm2\" (UID: \"1372611d-d215-4500-9e09-0b4fae36e5da\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-tflm2"
Sep 18 19:28:50.944847 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.944785    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b282eb09-4598-4a77-8682-a3170ca1ccd2-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb\" (UID: \"b282eb09-4598-4a77-8682-a3170ca1ccd2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb"
Sep 18 19:28:50.945265 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.945244    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b282eb09-4598-4a77-8682-a3170ca1ccd2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb\" (UID: \"b282eb09-4598-4a77-8682-a3170ca1ccd2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb"
Sep 18 19:28:50.945343 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.945324    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1372611d-d215-4500-9e09-0b4fae36e5da-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-794fd55d6c-tflm2\" (UID: \"1372611d-d215-4500-9e09-0b4fae36e5da\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-tflm2"
Sep 18 19:28:50.945428 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.945408    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b282eb09-4598-4a77-8682-a3170ca1ccd2-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb\" (UID: \"b282eb09-4598-4a77-8682-a3170ca1ccd2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb"
Sep 18 19:28:50.946962 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.946946    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b282eb09-4598-4a77-8682-a3170ca1ccd2-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb\" (UID: \"b282eb09-4598-4a77-8682-a3170ca1ccd2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb"
Sep 18 19:28:50.953974 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.953952    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm6pt\" (UniqueName: \"kubernetes.io/projected/b282eb09-4598-4a77-8682-a3170ca1ccd2-kube-api-access-sm6pt\") pod \"isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb\" (UID: \"b282eb09-4598-4a77-8682-a3170ca1ccd2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb"
Sep 18 19:28:50.954043 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.953976    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr88h\" (UniqueName: \"kubernetes.io/projected/1372611d-d215-4500-9e09-0b4fae36e5da-kube-api-access-pr88h\") pod \"isvc-canary-rollback-predictor-794fd55d6c-tflm2\" (UID: \"1372611d-d215-4500-9e09-0b4fae36e5da\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-tflm2"
Sep 18 19:28:50.977362 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.977335    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb"
Sep 18 19:28:50.978367 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:50.978341    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:28:51.079694 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:51.079666    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:28:51.158261 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:51.157951    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2"]
Sep 18 19:28:51.160328 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:28:51.160300    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb801f0ad_e919_4d3d_bf92_2dfd46d0029e.slice/crio-4168ba2bb49634058e9c11043e97cdff7566d86469d728fdf04890a3d55c7e6c WatchSource:0}: Error finding container 4168ba2bb49634058e9c11043e97cdff7566d86469d728fdf04890a3d55c7e6c: Status 404 returned error can't find the container with id 4168ba2bb49634058e9c11043e97cdff7566d86469d728fdf04890a3d55c7e6c
Sep 18 19:28:51.252315 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:51.252290    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb"]
Sep 18 19:28:51.254602 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:28:51.254581    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb282eb09_4598_4a77_8682_a3170ca1ccd2.slice/crio-022739a00001d51b08d25ecc0df1dd8f3a03dd379ad8006ac16b51498d57ee9e WatchSource:0}: Error finding container 022739a00001d51b08d25ecc0df1dd8f3a03dd379ad8006ac16b51498d57ee9e: Status 404 returned error can't find the container with id 022739a00001d51b08d25ecc0df1dd8f3a03dd379ad8006ac16b51498d57ee9e
Sep 18 19:28:51.448640 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:51.448605    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1372611d-d215-4500-9e09-0b4fae36e5da-proxy-tls\") pod \"isvc-canary-rollback-predictor-794fd55d6c-tflm2\" (UID: \"1372611d-d215-4500-9e09-0b4fae36e5da\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-tflm2"
Sep 18 19:28:51.451238 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:51.451207    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1372611d-d215-4500-9e09-0b4fae36e5da-proxy-tls\") pod \"isvc-canary-rollback-predictor-794fd55d6c-tflm2\" (UID: \"1372611d-d215-4500-9e09-0b4fae36e5da\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-tflm2"
Sep 18 19:28:51.614564 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:51.614524    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-tflm2"
Sep 18 19:28:51.671495 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:51.671414    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb" event={"ID":"b282eb09-4598-4a77-8682-a3170ca1ccd2","Type":"ContainerStarted","Data":"960f684d652d48f0c83f23e3bae5c1f67d823880675e880a8d50329a3f2828f3"}
Sep 18 19:28:51.671495 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:51.671486    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb" event={"ID":"b282eb09-4598-4a77-8682-a3170ca1ccd2","Type":"ContainerStarted","Data":"022739a00001d51b08d25ecc0df1dd8f3a03dd379ad8006ac16b51498d57ee9e"}
Sep 18 19:28:51.677740 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:51.677711    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2" event={"ID":"b801f0ad-e919-4d3d-bf92-2dfd46d0029e","Type":"ContainerStarted","Data":"fe61008c07b8ef22e5a2f46854e80667c4726205d0d75d3caf7a1caed942a5f0"}
Sep 18 19:28:51.677855 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:51.677785    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2" event={"ID":"b801f0ad-e919-4d3d-bf92-2dfd46d0029e","Type":"ContainerStarted","Data":"4168ba2bb49634058e9c11043e97cdff7566d86469d728fdf04890a3d55c7e6c"}
Sep 18 19:28:51.902418 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:51.902386    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-tflm2"]
Sep 18 19:28:51.903558 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:28:51.903534    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1372611d_d215_4500_9e09_0b4fae36e5da.slice/crio-7efb035c1d2610ab60a9ecdc13b2804941ca311ccbf72e436313fc643f90ff1a WatchSource:0}: Error finding container 7efb035c1d2610ab60a9ecdc13b2804941ca311ccbf72e436313fc643f90ff1a: Status 404 returned error can't find the container with id 7efb035c1d2610ab60a9ecdc13b2804941ca311ccbf72e436313fc643f90ff1a
Sep 18 19:28:52.685145 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:52.685102    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-tflm2" event={"ID":"1372611d-d215-4500-9e09-0b4fae36e5da","Type":"ContainerStarted","Data":"9fa789fdc0c9d0fafeac106f9258b9113256c3301eb955a5d22ba16c53896407"}
Sep 18 19:28:52.685615 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:52.685170    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-tflm2" event={"ID":"1372611d-d215-4500-9e09-0b4fae36e5da","Type":"ContainerStarted","Data":"7efb035c1d2610ab60a9ecdc13b2804941ca311ccbf72e436313fc643f90ff1a"}
Sep 18 19:28:53.538706 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:53.538678    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sksw8_2cef0f4e-e44f-4996-8561-da9d261036c7/ovn-acl-logging/0.log"
Sep 18 19:28:53.543509 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:53.543485    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sksw8_2cef0f4e-e44f-4996-8561-da9d261036c7/ovn-acl-logging/0.log"
Sep 18 19:28:55.695566 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:55.695533    2532 generic.go:356] "Generic (PLEG): container finished" podID="b801f0ad-e919-4d3d-bf92-2dfd46d0029e" containerID="fe61008c07b8ef22e5a2f46854e80667c4726205d0d75d3caf7a1caed942a5f0" exitCode=0
Sep 18 19:28:55.695902 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:55.695605    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2" event={"ID":"b801f0ad-e919-4d3d-bf92-2dfd46d0029e","Type":"ContainerDied","Data":"fe61008c07b8ef22e5a2f46854e80667c4726205d0d75d3caf7a1caed942a5f0"}
Sep 18 19:28:55.697242 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:55.697212    2532 generic.go:356] "Generic (PLEG): container finished" podID="b282eb09-4598-4a77-8682-a3170ca1ccd2" containerID="960f684d652d48f0c83f23e3bae5c1f67d823880675e880a8d50329a3f2828f3" exitCode=0
Sep 18 19:28:55.697332 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:55.697262    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb" event={"ID":"b282eb09-4598-4a77-8682-a3170ca1ccd2","Type":"ContainerDied","Data":"960f684d652d48f0c83f23e3bae5c1f67d823880675e880a8d50329a3f2828f3"}
Sep 18 19:28:55.698770 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:55.698747    2532 generic.go:356] "Generic (PLEG): container finished" podID="1372611d-d215-4500-9e09-0b4fae36e5da" containerID="9fa789fdc0c9d0fafeac106f9258b9113256c3301eb955a5d22ba16c53896407" exitCode=0
Sep 18 19:28:55.698856 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:28:55.698792    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-tflm2" event={"ID":"1372611d-d215-4500-9e09-0b4fae36e5da","Type":"ContainerDied","Data":"9fa789fdc0c9d0fafeac106f9258b9113256c3301eb955a5d22ba16c53896407"}
Sep 18 19:29:16.895164 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:29:16.895126    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb" event={"ID":"b282eb09-4598-4a77-8682-a3170ca1ccd2","Type":"ContainerStarted","Data":"e08573a7ae815390a1cfbfa27370149ba98b67f43516d4bd2e427e5ed99ddc31"}
Sep 18 19:29:16.898198 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:29:16.898117    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2" event={"ID":"b801f0ad-e919-4d3d-bf92-2dfd46d0029e","Type":"ContainerStarted","Data":"369a4fca571f306aa44f258d39d446a396157d5ea66299bd8d856889d6d11665"}
Sep 18 19:29:18.739927 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:29:18.739892    2532 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:29:18.909456 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:29:18.909367    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2" event={"ID":"b801f0ad-e919-4d3d-bf92-2dfd46d0029e","Type":"ContainerStarted","Data":"12533130ee96c78e1c1296da5c4d36518f0aa36ec3683e8ff61fd1e482801db7"}
Sep 18 19:29:18.911550 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:29:18.911526    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb" event={"ID":"b282eb09-4598-4a77-8682-a3170ca1ccd2","Type":"ContainerStarted","Data":"6d37ac6813b81ba0c0ba851ff896233c7dfc9c6cc2308cefed3f8bb9ad806d7b"}
Sep 18 19:29:18.911673 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:29:18.911645    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb"
Sep 18 19:29:18.913585 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:29:18.913561    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-tflm2" event={"ID":"1372611d-d215-4500-9e09-0b4fae36e5da","Type":"ContainerStarted","Data":"8395fefa0cadab14b649f8eb82f47a375f1b4905a696ce7684d5b0b75cfb949d"}
Sep 18 19:29:18.913670 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:29:18.913604    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-tflm2" event={"ID":"1372611d-d215-4500-9e09-0b4fae36e5da","Type":"ContainerStarted","Data":"e963b206e879c6e7de5ad40e0606237812b622ec5c6fd3cd4a8689d09511cb39"}
Sep 18 19:29:18.913974 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:29:18.913958    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-tflm2"
Sep 18 19:29:18.914020 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:29:18.914003    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-tflm2"
Sep 18 19:29:18.915153 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:29:18.915127    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-tflm2" podUID="1372611d-d215-4500-9e09-0b4fae36e5da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 18 19:29:18.939205 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:29:18.939148    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb" podStartSLOduration=2.456995875 podStartE2EDuration="28.939129432s" totalImagesPullingTime="22.85391158s" totalInitContainerRuntime="3.628221977s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:28:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:29:18.937907539 +0000 UTC m=+925.858411397" watchObservedRunningTime="2026-09-18 19:29:18.939129432 +0000 UTC m=+925.859633291"
Sep 18 19:29:18.960809 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:29:18.960766    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-tflm2" podStartSLOduration=2.431686377 podStartE2EDuration="28.960752588s" totalImagesPullingTime="22.908972502s" totalInitContainerRuntime="3.620093709s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:28:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:29:18.960530996 +0000 UTC m=+925.881034847" watchObservedRunningTime="2026-09-18 19:29:18.960752588 +0000 UTC m=+925.881256444"
Sep 18 19:29:19.922860 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:29:19.922813    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-tflm2" podUID="1372611d-d215-4500-9e09-0b4fae36e5da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 18 19:29:19.923242 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:29:19.922977    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb"
Sep 18 19:29:19.924244 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:29:19.924212    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb" podUID="b282eb09-4598-4a77-8682-a3170ca1ccd2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 18 19:29:20.927804 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:29:20.927654    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb" podUID="b282eb09-4598-4a77-8682-a3170ca1ccd2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 18 19:29:21.936116 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:29:21.936075    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2" event={"ID":"b801f0ad-e919-4d3d-bf92-2dfd46d0029e","Type":"ContainerStarted","Data":"367a5c4d5af63d363a61e5e532fd27fb4fb5e65d797ab0e6b190746f03270c3b"}
Sep 18 19:29:21.936569 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:29:21.936257    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2"
Sep 18 19:29:21.963638 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:29:21.963588    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2" podStartSLOduration=2.812816971 podStartE2EDuration="31.963573596s" totalImagesPullingTime="25.239837462s" totalInitContainerRuntime="3.910919163s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:28:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:29:21.962625941 +0000 UTC m=+928.883129795" watchObservedRunningTime="2026-09-18 19:29:21.963573596 +0000 UTC m=+928.884077454"
Sep 18 19:29:22.940485 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:29:22.940440    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2"
Sep 18 19:29:22.940921 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:29:22.940509    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2"
Sep 18 19:29:22.941887 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:29:22.941856    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2" podUID="b801f0ad-e919-4d3d-bf92-2dfd46d0029e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 18 19:29:22.942569 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:29:22.942546    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2" podUID="b801f0ad-e919-4d3d-bf92-2dfd46d0029e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:29:23.944563 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:29:23.944509    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2" podUID="b801f0ad-e919-4d3d-bf92-2dfd46d0029e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 18 19:29:23.945028 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:29:23.945005    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2" podUID="b801f0ad-e919-4d3d-bf92-2dfd46d0029e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:29:23.949369 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:29:23.949344    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2"
Sep 18 19:29:24.927324 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:29:24.927298    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-tflm2"
Sep 18 19:29:24.927794 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:29:24.927768    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-tflm2" podUID="1372611d-d215-4500-9e09-0b4fae36e5da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 18 19:29:24.951246 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:29:24.951214    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2" podUID="b801f0ad-e919-4d3d-bf92-2dfd46d0029e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 18 19:29:24.951683 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:29:24.951515    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2" podUID="b801f0ad-e919-4d3d-bf92-2dfd46d0029e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:29:25.933443 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:29:25.933404    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb"
Sep 18 19:29:25.934225 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:29:25.934196    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb" podUID="b282eb09-4598-4a77-8682-a3170ca1ccd2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 18 19:29:34.927776 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:29:34.927738    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-tflm2" podUID="1372611d-d215-4500-9e09-0b4fae36e5da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 18 19:29:34.951849 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:29:34.951805    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2" podUID="b801f0ad-e919-4d3d-bf92-2dfd46d0029e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 18 19:29:34.952315 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:29:34.952290    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2" podUID="b801f0ad-e919-4d3d-bf92-2dfd46d0029e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:29:35.935327 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:29:35.935288    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb" podUID="b282eb09-4598-4a77-8682-a3170ca1ccd2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 18 19:29:44.928785 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:29:44.928740    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-tflm2" podUID="1372611d-d215-4500-9e09-0b4fae36e5da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 18 19:29:44.951378 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:29:44.951336    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2" podUID="b801f0ad-e919-4d3d-bf92-2dfd46d0029e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 18 19:29:44.951768 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:29:44.951729    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2" podUID="b801f0ad-e919-4d3d-bf92-2dfd46d0029e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:29:45.935135 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:29:45.935089    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb" podUID="b282eb09-4598-4a77-8682-a3170ca1ccd2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 18 19:29:54.927762 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:29:54.927722    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-tflm2" podUID="1372611d-d215-4500-9e09-0b4fae36e5da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 18 19:29:54.952142 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:29:54.952100    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2" podUID="b801f0ad-e919-4d3d-bf92-2dfd46d0029e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 18 19:29:54.952586 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:29:54.952559    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2" podUID="b801f0ad-e919-4d3d-bf92-2dfd46d0029e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:29:55.934786 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:29:55.934741    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb" podUID="b282eb09-4598-4a77-8682-a3170ca1ccd2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 18 19:30:04.927862 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:04.927820    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-tflm2" podUID="1372611d-d215-4500-9e09-0b4fae36e5da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 18 19:30:04.951529 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:04.951485    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2" podUID="b801f0ad-e919-4d3d-bf92-2dfd46d0029e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 18 19:30:04.952016 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:04.951990    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2" podUID="b801f0ad-e919-4d3d-bf92-2dfd46d0029e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:30:05.934363 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:05.934317    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb" podUID="b282eb09-4598-4a77-8682-a3170ca1ccd2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 18 19:30:14.928738 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:14.928692    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-tflm2" podUID="1372611d-d215-4500-9e09-0b4fae36e5da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 18 19:30:14.952130 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:14.952089    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2" podUID="b801f0ad-e919-4d3d-bf92-2dfd46d0029e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 18 19:30:14.952589 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:14.952560    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2" podUID="b801f0ad-e919-4d3d-bf92-2dfd46d0029e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:30:15.934993 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:15.934943    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb" podUID="b282eb09-4598-4a77-8682-a3170ca1ccd2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 18 19:30:24.927794 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:24.927757    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-tflm2" podUID="1372611d-d215-4500-9e09-0b4fae36e5da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 18 19:30:24.951765 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:24.951729    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2" podUID="b801f0ad-e919-4d3d-bf92-2dfd46d0029e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 18 19:30:24.952212 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:24.952194    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2" podUID="b801f0ad-e919-4d3d-bf92-2dfd46d0029e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:30:25.934478 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:25.934431    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb" podUID="b282eb09-4598-4a77-8682-a3170ca1ccd2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 18 19:30:33.618844 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:33.618794    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-tflm2" podUID="1372611d-d215-4500-9e09-0b4fae36e5da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 18 19:30:34.952224 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:34.952196    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2"
Sep 18 19:30:34.952617 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:34.952484    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2"
Sep 18 19:30:35.935070 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:35.935027    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb"
Sep 18 19:30:36.086730 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:36.086701    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-tflm2"]
Sep 18 19:30:36.087197 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:36.086965    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-tflm2" podUID="1372611d-d215-4500-9e09-0b4fae36e5da" containerName="kserve-container" containerID="cri-o://e963b206e879c6e7de5ad40e0606237812b622ec5c6fd3cd4a8689d09511cb39" gracePeriod=30
Sep 18 19:30:36.087197 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:36.086993    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-tflm2" podUID="1372611d-d215-4500-9e09-0b4fae36e5da" containerName="kube-rbac-proxy" containerID="cri-o://8395fefa0cadab14b649f8eb82f47a375f1b4905a696ce7684d5b0b75cfb949d" gracePeriod=30
Sep 18 19:30:36.087923 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:36.087906    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-tflm2"
Sep 18 19:30:36.215752 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:36.215728    2532 generic.go:356] "Generic (PLEG): container finished" podID="1372611d-d215-4500-9e09-0b4fae36e5da" containerID="8395fefa0cadab14b649f8eb82f47a375f1b4905a696ce7684d5b0b75cfb949d" exitCode=2
Sep 18 19:30:36.215844 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:36.215768    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-tflm2" event={"ID":"1372611d-d215-4500-9e09-0b4fae36e5da","Type":"ContainerDied","Data":"8395fefa0cadab14b649f8eb82f47a375f1b4905a696ce7684d5b0b75cfb949d"}
Sep 18 19:30:39.923566 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:39.923529    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-tflm2" podUID="1372611d-d215-4500-9e09-0b4fae36e5da" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
Sep 18 19:30:40.355785 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:40.355750    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb"]
Sep 18 19:30:40.356002 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:40.355978    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb" podUID="b282eb09-4598-4a77-8682-a3170ca1ccd2" containerName="kserve-container" containerID="cri-o://e08573a7ae815390a1cfbfa27370149ba98b67f43516d4bd2e427e5ed99ddc31" gracePeriod=30
Sep 18 19:30:40.356083 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:40.356024    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb" podUID="b282eb09-4598-4a77-8682-a3170ca1ccd2" containerName="kube-rbac-proxy" containerID="cri-o://6d37ac6813b81ba0c0ba851ff896233c7dfc9c6cc2308cefed3f8bb9ad806d7b" gracePeriod=30
Sep 18 19:30:40.461099 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:40.461036    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:30:40.470980 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:40.470954    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-xm89q"]
Sep 18 19:30:40.474223 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:40.474206    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-xm89q"
Sep 18 19:30:40.492662 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:40.492640    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-xm89q"]
Sep 18 19:30:40.618460 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:40.618377    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/901ea860-4642-47d2-a1a4-7bd463389d8c-proxy-tls\") pod \"isvc-canary-rollback-predictor-794fd55d6c-xm89q\" (UID: \"901ea860-4642-47d2-a1a4-7bd463389d8c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-xm89q"
Sep 18 19:30:40.618460 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:40.618421    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/901ea860-4642-47d2-a1a4-7bd463389d8c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-794fd55d6c-xm89q\" (UID: \"901ea860-4642-47d2-a1a4-7bd463389d8c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-xm89q"
Sep 18 19:30:40.618640 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:40.618461    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/901ea860-4642-47d2-a1a4-7bd463389d8c-cabundle-cert\") pod \"isvc-canary-rollback-predictor-794fd55d6c-xm89q\" (UID: \"901ea860-4642-47d2-a1a4-7bd463389d8c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-xm89q"
Sep 18 19:30:40.618640 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:40.618497    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrpdh\" (UniqueName: \"kubernetes.io/projected/901ea860-4642-47d2-a1a4-7bd463389d8c-kube-api-access-mrpdh\") pod \"isvc-canary-rollback-predictor-794fd55d6c-xm89q\" (UID: \"901ea860-4642-47d2-a1a4-7bd463389d8c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-xm89q"
Sep 18 19:30:40.618640 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:40.618553    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/901ea860-4642-47d2-a1a4-7bd463389d8c-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-794fd55d6c-xm89q\" (UID: \"901ea860-4642-47d2-a1a4-7bd463389d8c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-xm89q"
Sep 18 19:30:40.719553 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:40.719522    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/901ea860-4642-47d2-a1a4-7bd463389d8c-proxy-tls\") pod \"isvc-canary-rollback-predictor-794fd55d6c-xm89q\" (UID: \"901ea860-4642-47d2-a1a4-7bd463389d8c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-xm89q"
Sep 18 19:30:40.719687 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:40.719562    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/901ea860-4642-47d2-a1a4-7bd463389d8c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-794fd55d6c-xm89q\" (UID: \"901ea860-4642-47d2-a1a4-7bd463389d8c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-xm89q"
Sep 18 19:30:40.719687 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:40.719583    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/901ea860-4642-47d2-a1a4-7bd463389d8c-cabundle-cert\") pod \"isvc-canary-rollback-predictor-794fd55d6c-xm89q\" (UID: \"901ea860-4642-47d2-a1a4-7bd463389d8c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-xm89q"
Sep 18 19:30:40.719687 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:40.719620    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrpdh\" (UniqueName: \"kubernetes.io/projected/901ea860-4642-47d2-a1a4-7bd463389d8c-kube-api-access-mrpdh\") pod \"isvc-canary-rollback-predictor-794fd55d6c-xm89q\" (UID: \"901ea860-4642-47d2-a1a4-7bd463389d8c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-xm89q"
Sep 18 19:30:40.719687 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:40.719654    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/901ea860-4642-47d2-a1a4-7bd463389d8c-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-794fd55d6c-xm89q\" (UID: \"901ea860-4642-47d2-a1a4-7bd463389d8c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-xm89q"
Sep 18 19:30:40.720175 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:40.720150    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/901ea860-4642-47d2-a1a4-7bd463389d8c-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-794fd55d6c-xm89q\" (UID: \"901ea860-4642-47d2-a1a4-7bd463389d8c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-xm89q"
Sep 18 19:30:40.720293 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:40.720275    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/901ea860-4642-47d2-a1a4-7bd463389d8c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-794fd55d6c-xm89q\" (UID: \"901ea860-4642-47d2-a1a4-7bd463389d8c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-xm89q"
Sep 18 19:30:40.720368 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:40.720352    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/901ea860-4642-47d2-a1a4-7bd463389d8c-cabundle-cert\") pod \"isvc-canary-rollback-predictor-794fd55d6c-xm89q\" (UID: \"901ea860-4642-47d2-a1a4-7bd463389d8c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-xm89q"
Sep 18 19:30:40.721919 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:40.721903    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/901ea860-4642-47d2-a1a4-7bd463389d8c-proxy-tls\") pod \"isvc-canary-rollback-predictor-794fd55d6c-xm89q\" (UID: \"901ea860-4642-47d2-a1a4-7bd463389d8c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-xm89q"
Sep 18 19:30:40.728818 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:40.728798    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrpdh\" (UniqueName: \"kubernetes.io/projected/901ea860-4642-47d2-a1a4-7bd463389d8c-kube-api-access-mrpdh\") pod \"isvc-canary-rollback-predictor-794fd55d6c-xm89q\" (UID: \"901ea860-4642-47d2-a1a4-7bd463389d8c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-xm89q"
Sep 18 19:30:40.783257 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:40.783234    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-xm89q"
Sep 18 19:30:40.928446 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:40.928418    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb" podUID="b282eb09-4598-4a77-8682-a3170ca1ccd2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.19:8643/healthz\": dial tcp 10.132.0.19:8643: connect: connection refused"
Sep 18 19:30:41.060963 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:41.060938    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-xm89q"]
Sep 18 19:30:41.076885 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:30:41.076859    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod901ea860_4642_47d2_a1a4_7bd463389d8c.slice/crio-e583d4c0e3abf332d36a14479bf76bf19c36830e4aa0b0a46d2997d38aaf2b85 WatchSource:0}: Error finding container e583d4c0e3abf332d36a14479bf76bf19c36830e4aa0b0a46d2997d38aaf2b85: Status 404 returned error can't find the container with id e583d4c0e3abf332d36a14479bf76bf19c36830e4aa0b0a46d2997d38aaf2b85
Sep 18 19:30:41.218689 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:41.218662    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-tflm2"
Sep 18 19:30:41.238565 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:41.238537    2532 generic.go:356] "Generic (PLEG): container finished" podID="b282eb09-4598-4a77-8682-a3170ca1ccd2" containerID="6d37ac6813b81ba0c0ba851ff896233c7dfc9c6cc2308cefed3f8bb9ad806d7b" exitCode=2
Sep 18 19:30:41.238676 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:41.238611    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb" event={"ID":"b282eb09-4598-4a77-8682-a3170ca1ccd2","Type":"ContainerDied","Data":"6d37ac6813b81ba0c0ba851ff896233c7dfc9c6cc2308cefed3f8bb9ad806d7b"}
Sep 18 19:30:41.241207 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:41.241175    2532 generic.go:356] "Generic (PLEG): container finished" podID="1372611d-d215-4500-9e09-0b4fae36e5da" containerID="e963b206e879c6e7de5ad40e0606237812b622ec5c6fd3cd4a8689d09511cb39" exitCode=0
Sep 18 19:30:41.241437 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:41.241327    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-tflm2"
Sep 18 19:30:41.241514 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:41.241435    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-tflm2" event={"ID":"1372611d-d215-4500-9e09-0b4fae36e5da","Type":"ContainerDied","Data":"e963b206e879c6e7de5ad40e0606237812b622ec5c6fd3cd4a8689d09511cb39"}
Sep 18 19:30:41.241569 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:41.241506    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-tflm2" event={"ID":"1372611d-d215-4500-9e09-0b4fae36e5da","Type":"ContainerDied","Data":"7efb035c1d2610ab60a9ecdc13b2804941ca311ccbf72e436313fc643f90ff1a"}
Sep 18 19:30:41.241616 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:41.241575    2532 scope.go:117] "RemoveContainer" containerID="8395fefa0cadab14b649f8eb82f47a375f1b4905a696ce7684d5b0b75cfb949d"
Sep 18 19:30:41.244858 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:41.244814    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-xm89q" event={"ID":"901ea860-4642-47d2-a1a4-7bd463389d8c","Type":"ContainerStarted","Data":"fa8b53dc6b8a991b79078830495dfd1d5a18e7854076206b4d5bea94760a1850"}
Sep 18 19:30:41.244966 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:41.244909    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-xm89q" event={"ID":"901ea860-4642-47d2-a1a4-7bd463389d8c","Type":"ContainerStarted","Data":"e583d4c0e3abf332d36a14479bf76bf19c36830e4aa0b0a46d2997d38aaf2b85"}
Sep 18 19:30:41.251005 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:41.250989    2532 scope.go:117] "RemoveContainer" containerID="e963b206e879c6e7de5ad40e0606237812b622ec5c6fd3cd4a8689d09511cb39"
Sep 18 19:30:41.258854 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:41.258821    2532 scope.go:117] "RemoveContainer" containerID="9fa789fdc0c9d0fafeac106f9258b9113256c3301eb955a5d22ba16c53896407"
Sep 18 19:30:41.267449 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:41.267386    2532 scope.go:117] "RemoveContainer" containerID="8395fefa0cadab14b649f8eb82f47a375f1b4905a696ce7684d5b0b75cfb949d"
Sep 18 19:30:41.267873 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:30:41.267836    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8395fefa0cadab14b649f8eb82f47a375f1b4905a696ce7684d5b0b75cfb949d\": container with ID starting with 8395fefa0cadab14b649f8eb82f47a375f1b4905a696ce7684d5b0b75cfb949d not found: ID does not exist" containerID="8395fefa0cadab14b649f8eb82f47a375f1b4905a696ce7684d5b0b75cfb949d"
Sep 18 19:30:41.267956 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:41.267885    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8395fefa0cadab14b649f8eb82f47a375f1b4905a696ce7684d5b0b75cfb949d"} err="failed to get container status \"8395fefa0cadab14b649f8eb82f47a375f1b4905a696ce7684d5b0b75cfb949d\": rpc error: code = NotFound desc = could not find container \"8395fefa0cadab14b649f8eb82f47a375f1b4905a696ce7684d5b0b75cfb949d\": container with ID starting with 8395fefa0cadab14b649f8eb82f47a375f1b4905a696ce7684d5b0b75cfb949d not found: ID does not exist"
Sep 18 19:30:41.267956 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:41.267911    2532 scope.go:117] "RemoveContainer" containerID="e963b206e879c6e7de5ad40e0606237812b622ec5c6fd3cd4a8689d09511cb39"
Sep 18 19:30:41.268330 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:30:41.268310    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e963b206e879c6e7de5ad40e0606237812b622ec5c6fd3cd4a8689d09511cb39\": container with ID starting with e963b206e879c6e7de5ad40e0606237812b622ec5c6fd3cd4a8689d09511cb39 not found: ID does not exist" containerID="e963b206e879c6e7de5ad40e0606237812b622ec5c6fd3cd4a8689d09511cb39"
Sep 18 19:30:41.268401 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:41.268336    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e963b206e879c6e7de5ad40e0606237812b622ec5c6fd3cd4a8689d09511cb39"} err="failed to get container status \"e963b206e879c6e7de5ad40e0606237812b622ec5c6fd3cd4a8689d09511cb39\": rpc error: code = NotFound desc = could not find container \"e963b206e879c6e7de5ad40e0606237812b622ec5c6fd3cd4a8689d09511cb39\": container with ID starting with e963b206e879c6e7de5ad40e0606237812b622ec5c6fd3cd4a8689d09511cb39 not found: ID does not exist"
Sep 18 19:30:41.268401 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:41.268351    2532 scope.go:117] "RemoveContainer" containerID="9fa789fdc0c9d0fafeac106f9258b9113256c3301eb955a5d22ba16c53896407"
Sep 18 19:30:41.268656 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:30:41.268635    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fa789fdc0c9d0fafeac106f9258b9113256c3301eb955a5d22ba16c53896407\": container with ID starting with 9fa789fdc0c9d0fafeac106f9258b9113256c3301eb955a5d22ba16c53896407 not found: ID does not exist" containerID="9fa789fdc0c9d0fafeac106f9258b9113256c3301eb955a5d22ba16c53896407"
Sep 18 19:30:41.268732 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:41.268661    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fa789fdc0c9d0fafeac106f9258b9113256c3301eb955a5d22ba16c53896407"} err="failed to get container status \"9fa789fdc0c9d0fafeac106f9258b9113256c3301eb955a5d22ba16c53896407\": rpc error: code = NotFound desc = could not find container \"9fa789fdc0c9d0fafeac106f9258b9113256c3301eb955a5d22ba16c53896407\": container with ID starting with 9fa789fdc0c9d0fafeac106f9258b9113256c3301eb955a5d22ba16c53896407 not found: ID does not exist"
Sep 18 19:30:41.323651 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:41.323630    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1372611d-d215-4500-9e09-0b4fae36e5da-kube-rbac-proxy-sar-config\") pod \"1372611d-d215-4500-9e09-0b4fae36e5da\" (UID: \"1372611d-d215-4500-9e09-0b4fae36e5da\") "
Sep 18 19:30:41.323745 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:41.323678    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1372611d-d215-4500-9e09-0b4fae36e5da-cabundle-cert\") pod \"1372611d-d215-4500-9e09-0b4fae36e5da\" (UID: \"1372611d-d215-4500-9e09-0b4fae36e5da\") "
Sep 18 19:30:41.323745 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:41.323707    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1372611d-d215-4500-9e09-0b4fae36e5da-proxy-tls\") pod \"1372611d-d215-4500-9e09-0b4fae36e5da\" (UID: \"1372611d-d215-4500-9e09-0b4fae36e5da\") "
Sep 18 19:30:41.323827 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:41.323743    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1372611d-d215-4500-9e09-0b4fae36e5da-kserve-provision-location\") pod \"1372611d-d215-4500-9e09-0b4fae36e5da\" (UID: \"1372611d-d215-4500-9e09-0b4fae36e5da\") "
Sep 18 19:30:41.323827 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:41.323818    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pr88h\" (UniqueName: \"kubernetes.io/projected/1372611d-d215-4500-9e09-0b4fae36e5da-kube-api-access-pr88h\") pod \"1372611d-d215-4500-9e09-0b4fae36e5da\" (UID: \"1372611d-d215-4500-9e09-0b4fae36e5da\") "
Sep 18 19:30:41.324304 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:41.324024    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1372611d-d215-4500-9e09-0b4fae36e5da-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1372611d-d215-4500-9e09-0b4fae36e5da" (UID: "1372611d-d215-4500-9e09-0b4fae36e5da"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:30:41.324304 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:41.324102    2532 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1372611d-d215-4500-9e09-0b4fae36e5da-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:30:41.324304 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:41.324142    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1372611d-d215-4500-9e09-0b4fae36e5da-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1372611d-d215-4500-9e09-0b4fae36e5da" (UID: "1372611d-d215-4500-9e09-0b4fae36e5da"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:30:41.324304 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:41.324157    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1372611d-d215-4500-9e09-0b4fae36e5da-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1372611d-d215-4500-9e09-0b4fae36e5da" (UID: "1372611d-d215-4500-9e09-0b4fae36e5da"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:30:41.325923 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:41.325896    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1372611d-d215-4500-9e09-0b4fae36e5da-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1372611d-d215-4500-9e09-0b4fae36e5da" (UID: "1372611d-d215-4500-9e09-0b4fae36e5da"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:30:41.326002 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:41.325961    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1372611d-d215-4500-9e09-0b4fae36e5da-kube-api-access-pr88h" (OuterVolumeSpecName: "kube-api-access-pr88h") pod "1372611d-d215-4500-9e09-0b4fae36e5da" (UID: "1372611d-d215-4500-9e09-0b4fae36e5da"). InnerVolumeSpecName "kube-api-access-pr88h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:30:41.425109 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:41.425077    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pr88h\" (UniqueName: \"kubernetes.io/projected/1372611d-d215-4500-9e09-0b4fae36e5da-kube-api-access-pr88h\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:30:41.425206 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:41.425112    2532 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1372611d-d215-4500-9e09-0b4fae36e5da-cabundle-cert\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:30:41.425206 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:41.425128    2532 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1372611d-d215-4500-9e09-0b4fae36e5da-proxy-tls\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:30:41.425206 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:41.425141    2532 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1372611d-d215-4500-9e09-0b4fae36e5da-kserve-provision-location\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:30:41.567609 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:41.567576    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-tflm2"]
Sep 18 19:30:41.574042 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:41.574019    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-tflm2"]
Sep 18 19:30:41.624115 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:41.621873    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1372611d-d215-4500-9e09-0b4fae36e5da" path="/var/lib/kubelet/pods/1372611d-d215-4500-9e09-0b4fae36e5da/volumes"
Sep 18 19:30:44.894402 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:44.894379    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb"
Sep 18 19:30:45.051921 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:45.051895    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b282eb09-4598-4a77-8682-a3170ca1ccd2-kserve-provision-location\") pod \"b282eb09-4598-4a77-8682-a3170ca1ccd2\" (UID: \"b282eb09-4598-4a77-8682-a3170ca1ccd2\") "
Sep 18 19:30:45.052023 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:45.051938    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b282eb09-4598-4a77-8682-a3170ca1ccd2-cabundle-cert\") pod \"b282eb09-4598-4a77-8682-a3170ca1ccd2\" (UID: \"b282eb09-4598-4a77-8682-a3170ca1ccd2\") "
Sep 18 19:30:45.052023 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:45.051965    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b282eb09-4598-4a77-8682-a3170ca1ccd2-kube-rbac-proxy-sar-config\") pod \"b282eb09-4598-4a77-8682-a3170ca1ccd2\" (UID: \"b282eb09-4598-4a77-8682-a3170ca1ccd2\") "
Sep 18 19:30:45.052023 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:45.051991    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b282eb09-4598-4a77-8682-a3170ca1ccd2-proxy-tls\") pod \"b282eb09-4598-4a77-8682-a3170ca1ccd2\" (UID: \"b282eb09-4598-4a77-8682-a3170ca1ccd2\") "
Sep 18 19:30:45.052201 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:45.052031    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sm6pt\" (UniqueName: \"kubernetes.io/projected/b282eb09-4598-4a77-8682-a3170ca1ccd2-kube-api-access-sm6pt\") pod \"b282eb09-4598-4a77-8682-a3170ca1ccd2\" (UID: \"b282eb09-4598-4a77-8682-a3170ca1ccd2\") "
Sep 18 19:30:45.052259 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:45.052231    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b282eb09-4598-4a77-8682-a3170ca1ccd2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b282eb09-4598-4a77-8682-a3170ca1ccd2" (UID: "b282eb09-4598-4a77-8682-a3170ca1ccd2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:30:45.052312 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:45.052268    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b282eb09-4598-4a77-8682-a3170ca1ccd2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b282eb09-4598-4a77-8682-a3170ca1ccd2" (UID: "b282eb09-4598-4a77-8682-a3170ca1ccd2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:30:45.052312 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:45.052289    2532 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b282eb09-4598-4a77-8682-a3170ca1ccd2-kserve-provision-location\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:30:45.052407 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:45.052370    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b282eb09-4598-4a77-8682-a3170ca1ccd2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b282eb09-4598-4a77-8682-a3170ca1ccd2" (UID: "b282eb09-4598-4a77-8682-a3170ca1ccd2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:30:45.053930 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:45.053908    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b282eb09-4598-4a77-8682-a3170ca1ccd2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b282eb09-4598-4a77-8682-a3170ca1ccd2" (UID: "b282eb09-4598-4a77-8682-a3170ca1ccd2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:30:45.054034 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:45.054016    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b282eb09-4598-4a77-8682-a3170ca1ccd2-kube-api-access-sm6pt" (OuterVolumeSpecName: "kube-api-access-sm6pt") pod "b282eb09-4598-4a77-8682-a3170ca1ccd2" (UID: "b282eb09-4598-4a77-8682-a3170ca1ccd2"). InnerVolumeSpecName "kube-api-access-sm6pt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:30:45.153245 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:45.153213    2532 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b282eb09-4598-4a77-8682-a3170ca1ccd2-cabundle-cert\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:30:45.153245 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:45.153245    2532 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b282eb09-4598-4a77-8682-a3170ca1ccd2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:30:45.153368 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:45.153256    2532 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b282eb09-4598-4a77-8682-a3170ca1ccd2-proxy-tls\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:30:45.153368 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:45.153265    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sm6pt\" (UniqueName: \"kubernetes.io/projected/b282eb09-4598-4a77-8682-a3170ca1ccd2-kube-api-access-sm6pt\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:30:45.261568 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:45.261500    2532 generic.go:356] "Generic (PLEG): container finished" podID="901ea860-4642-47d2-a1a4-7bd463389d8c" containerID="fa8b53dc6b8a991b79078830495dfd1d5a18e7854076206b4d5bea94760a1850" exitCode=0
Sep 18 19:30:45.261686 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:45.261573    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-xm89q" event={"ID":"901ea860-4642-47d2-a1a4-7bd463389d8c","Type":"ContainerDied","Data":"fa8b53dc6b8a991b79078830495dfd1d5a18e7854076206b4d5bea94760a1850"}
Sep 18 19:30:45.263523 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:45.263502    2532 generic.go:356] "Generic (PLEG): container finished" podID="b282eb09-4598-4a77-8682-a3170ca1ccd2" containerID="e08573a7ae815390a1cfbfa27370149ba98b67f43516d4bd2e427e5ed99ddc31" exitCode=0
Sep 18 19:30:45.263612 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:45.263536    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb" event={"ID":"b282eb09-4598-4a77-8682-a3170ca1ccd2","Type":"ContainerDied","Data":"e08573a7ae815390a1cfbfa27370149ba98b67f43516d4bd2e427e5ed99ddc31"}
Sep 18 19:30:45.263612 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:45.263556    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb"
Sep 18 19:30:45.263612 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:45.263588    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb" event={"ID":"b282eb09-4598-4a77-8682-a3170ca1ccd2","Type":"ContainerDied","Data":"022739a00001d51b08d25ecc0df1dd8f3a03dd379ad8006ac16b51498d57ee9e"}
Sep 18 19:30:45.263711 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:45.263624    2532 scope.go:117] "RemoveContainer" containerID="6d37ac6813b81ba0c0ba851ff896233c7dfc9c6cc2308cefed3f8bb9ad806d7b"
Sep 18 19:30:45.271636 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:45.271574    2532 scope.go:117] "RemoveContainer" containerID="e08573a7ae815390a1cfbfa27370149ba98b67f43516d4bd2e427e5ed99ddc31"
Sep 18 19:30:45.279176 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:45.279158    2532 scope.go:117] "RemoveContainer" containerID="960f684d652d48f0c83f23e3bae5c1f67d823880675e880a8d50329a3f2828f3"
Sep 18 19:30:45.287114 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:45.287097    2532 scope.go:117] "RemoveContainer" containerID="6d37ac6813b81ba0c0ba851ff896233c7dfc9c6cc2308cefed3f8bb9ad806d7b"
Sep 18 19:30:45.287374 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:30:45.287359    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d37ac6813b81ba0c0ba851ff896233c7dfc9c6cc2308cefed3f8bb9ad806d7b\": container with ID starting with 6d37ac6813b81ba0c0ba851ff896233c7dfc9c6cc2308cefed3f8bb9ad806d7b not found: ID does not exist" containerID="6d37ac6813b81ba0c0ba851ff896233c7dfc9c6cc2308cefed3f8bb9ad806d7b"
Sep 18 19:30:45.287424 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:45.287382    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d37ac6813b81ba0c0ba851ff896233c7dfc9c6cc2308cefed3f8bb9ad806d7b"} err="failed to get container status \"6d37ac6813b81ba0c0ba851ff896233c7dfc9c6cc2308cefed3f8bb9ad806d7b\": rpc error: code = NotFound desc = could not find container \"6d37ac6813b81ba0c0ba851ff896233c7dfc9c6cc2308cefed3f8bb9ad806d7b\": container with ID starting with 6d37ac6813b81ba0c0ba851ff896233c7dfc9c6cc2308cefed3f8bb9ad806d7b not found: ID does not exist"
Sep 18 19:30:45.287424 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:45.287404    2532 scope.go:117] "RemoveContainer" containerID="e08573a7ae815390a1cfbfa27370149ba98b67f43516d4bd2e427e5ed99ddc31"
Sep 18 19:30:45.287629 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:30:45.287610    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e08573a7ae815390a1cfbfa27370149ba98b67f43516d4bd2e427e5ed99ddc31\": container with ID starting with e08573a7ae815390a1cfbfa27370149ba98b67f43516d4bd2e427e5ed99ddc31 not found: ID does not exist" containerID="e08573a7ae815390a1cfbfa27370149ba98b67f43516d4bd2e427e5ed99ddc31"
Sep 18 19:30:45.287670 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:45.287635    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e08573a7ae815390a1cfbfa27370149ba98b67f43516d4bd2e427e5ed99ddc31"} err="failed to get container status \"e08573a7ae815390a1cfbfa27370149ba98b67f43516d4bd2e427e5ed99ddc31\": rpc error: code = NotFound desc = could not find container \"e08573a7ae815390a1cfbfa27370149ba98b67f43516d4bd2e427e5ed99ddc31\": container with ID starting with e08573a7ae815390a1cfbfa27370149ba98b67f43516d4bd2e427e5ed99ddc31 not found: ID does not exist"
Sep 18 19:30:45.287670 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:45.287654    2532 scope.go:117] "RemoveContainer" containerID="960f684d652d48f0c83f23e3bae5c1f67d823880675e880a8d50329a3f2828f3"
Sep 18 19:30:45.287864 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:30:45.287846    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"960f684d652d48f0c83f23e3bae5c1f67d823880675e880a8d50329a3f2828f3\": container with ID starting with 960f684d652d48f0c83f23e3bae5c1f67d823880675e880a8d50329a3f2828f3 not found: ID does not exist" containerID="960f684d652d48f0c83f23e3bae5c1f67d823880675e880a8d50329a3f2828f3"
Sep 18 19:30:45.287900 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:45.287870    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"960f684d652d48f0c83f23e3bae5c1f67d823880675e880a8d50329a3f2828f3"} err="failed to get container status \"960f684d652d48f0c83f23e3bae5c1f67d823880675e880a8d50329a3f2828f3\": rpc error: code = NotFound desc = could not find container \"960f684d652d48f0c83f23e3bae5c1f67d823880675e880a8d50329a3f2828f3\": container with ID starting with 960f684d652d48f0c83f23e3bae5c1f67d823880675e880a8d50329a3f2828f3 not found: ID does not exist"
Sep 18 19:30:45.296641 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:45.296623    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb"]
Sep 18 19:30:45.300323 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:45.300304    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-q2dtb"]
Sep 18 19:30:45.620839 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:45.620758    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b282eb09-4598-4a77-8682-a3170ca1ccd2" path="/var/lib/kubelet/pods/b282eb09-4598-4a77-8682-a3170ca1ccd2/volumes"
Sep 18 19:30:45.828640 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:45.828604    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2"]
Sep 18 19:30:45.828886 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:45.828854    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2" podUID="b801f0ad-e919-4d3d-bf92-2dfd46d0029e" containerName="kserve-container" containerID="cri-o://369a4fca571f306aa44f258d39d446a396157d5ea66299bd8d856889d6d11665" gracePeriod=30
Sep 18 19:30:45.829032 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:45.828914    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2" podUID="b801f0ad-e919-4d3d-bf92-2dfd46d0029e" containerName="kube-rbac-proxy" containerID="cri-o://12533130ee96c78e1c1296da5c4d36518f0aa36ec3683e8ff61fd1e482801db7" gracePeriod=30
Sep 18 19:30:45.829032 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:45.828855    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2" podUID="b801f0ad-e919-4d3d-bf92-2dfd46d0029e" containerName="agent" containerID="cri-o://367a5c4d5af63d363a61e5e532fd27fb4fb5e65d797ab0e6b190746f03270c3b" gracePeriod=30
Sep 18 19:30:45.843788 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:45.843768    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:30:45.969474 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:45.969449    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:30:46.028156 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:46.028126    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv"]
Sep 18 19:30:46.028421 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:46.028407    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1372611d-d215-4500-9e09-0b4fae36e5da" containerName="kube-rbac-proxy"
Sep 18 19:30:46.028421 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:46.028421    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="1372611d-d215-4500-9e09-0b4fae36e5da" containerName="kube-rbac-proxy"
Sep 18 19:30:46.028550 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:46.028431    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1372611d-d215-4500-9e09-0b4fae36e5da" containerName="kserve-container"
Sep 18 19:30:46.028550 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:46.028437    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="1372611d-d215-4500-9e09-0b4fae36e5da" containerName="kserve-container"
Sep 18 19:30:46.028550 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:46.028448    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b282eb09-4598-4a77-8682-a3170ca1ccd2" containerName="kube-rbac-proxy"
Sep 18 19:30:46.028550 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:46.028453    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="b282eb09-4598-4a77-8682-a3170ca1ccd2" containerName="kube-rbac-proxy"
Sep 18 19:30:46.028550 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:46.028465    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b282eb09-4598-4a77-8682-a3170ca1ccd2" containerName="storage-initializer"
Sep 18 19:30:46.028550 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:46.028470    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="b282eb09-4598-4a77-8682-a3170ca1ccd2" containerName="storage-initializer"
Sep 18 19:30:46.028550 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:46.028476    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1372611d-d215-4500-9e09-0b4fae36e5da" containerName="storage-initializer"
Sep 18 19:30:46.028550 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:46.028482    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="1372611d-d215-4500-9e09-0b4fae36e5da" containerName="storage-initializer"
Sep 18 19:30:46.028550 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:46.028489    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b282eb09-4598-4a77-8682-a3170ca1ccd2" containerName="kserve-container"
Sep 18 19:30:46.028550 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:46.028494    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="b282eb09-4598-4a77-8682-a3170ca1ccd2" containerName="kserve-container"
Sep 18 19:30:46.028550 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:46.028535    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="b282eb09-4598-4a77-8682-a3170ca1ccd2" containerName="kserve-container"
Sep 18 19:30:46.028550 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:46.028544    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="1372611d-d215-4500-9e09-0b4fae36e5da" containerName="kube-rbac-proxy"
Sep 18 19:30:46.028550 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:46.028549    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="b282eb09-4598-4a77-8682-a3170ca1ccd2" containerName="kube-rbac-proxy"
Sep 18 19:30:46.028550 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:46.028556    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="1372611d-d215-4500-9e09-0b4fae36e5da" containerName="kserve-container"
Sep 18 19:30:46.031503 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:46.031480    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv"
Sep 18 19:30:46.034306 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:46.034282    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-4a391-predictor-serving-cert\""
Sep 18 19:30:46.034761 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:46.034746    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-4a391-kube-rbac-proxy-sar-config\""
Sep 18 19:30:46.066521 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:46.066498    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv"]
Sep 18 19:30:46.152086 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:46.152040    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:30:46.161829 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:46.161810    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd73555e-e1dd-4509-8607-b23a135b0734-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv\" (UID: \"bd73555e-e1dd-4509-8607-b23a135b0734\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv"
Sep 18 19:30:46.161918 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:46.161837    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd73555e-e1dd-4509-8607-b23a135b0734-proxy-tls\") pod \"isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv\" (UID: \"bd73555e-e1dd-4509-8607-b23a135b0734\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv"
Sep 18 19:30:46.161956 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:46.161918    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbmfq\" (UniqueName: \"kubernetes.io/projected/bd73555e-e1dd-4509-8607-b23a135b0734-kube-api-access-tbmfq\") pod \"isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv\" (UID: \"bd73555e-e1dd-4509-8607-b23a135b0734\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv"
Sep 18 19:30:46.161956 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:46.161941    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd73555e-e1dd-4509-8607-b23a135b0734-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv\" (UID: \"bd73555e-e1dd-4509-8607-b23a135b0734\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv"
Sep 18 19:30:46.162018 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:46.161971    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd73555e-e1dd-4509-8607-b23a135b0734-cabundle-cert\") pod \"isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv\" (UID: \"bd73555e-e1dd-4509-8607-b23a135b0734\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv"
Sep 18 19:30:46.262805 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:46.262734    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbmfq\" (UniqueName: \"kubernetes.io/projected/bd73555e-e1dd-4509-8607-b23a135b0734-kube-api-access-tbmfq\") pod \"isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv\" (UID: \"bd73555e-e1dd-4509-8607-b23a135b0734\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv"
Sep 18 19:30:46.262805 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:46.262766    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd73555e-e1dd-4509-8607-b23a135b0734-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv\" (UID: \"bd73555e-e1dd-4509-8607-b23a135b0734\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv"
Sep 18 19:30:46.262805 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:46.262791    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd73555e-e1dd-4509-8607-b23a135b0734-cabundle-cert\") pod \"isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv\" (UID: \"bd73555e-e1dd-4509-8607-b23a135b0734\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv"
Sep 18 19:30:46.263030 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:46.262883    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd73555e-e1dd-4509-8607-b23a135b0734-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv\" (UID: \"bd73555e-e1dd-4509-8607-b23a135b0734\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv"
Sep 18 19:30:46.263030 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:46.262914    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd73555e-e1dd-4509-8607-b23a135b0734-proxy-tls\") pod \"isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv\" (UID: \"bd73555e-e1dd-4509-8607-b23a135b0734\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv"
Sep 18 19:30:46.263284 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:46.263264    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd73555e-e1dd-4509-8607-b23a135b0734-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv\" (UID: \"bd73555e-e1dd-4509-8607-b23a135b0734\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv"
Sep 18 19:30:46.263419 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:46.263401    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd73555e-e1dd-4509-8607-b23a135b0734-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv\" (UID: \"bd73555e-e1dd-4509-8607-b23a135b0734\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv"
Sep 18 19:30:46.263534 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:46.263508    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd73555e-e1dd-4509-8607-b23a135b0734-cabundle-cert\") pod \"isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv\" (UID: \"bd73555e-e1dd-4509-8607-b23a135b0734\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv"
Sep 18 19:30:46.265573 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:46.265555    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd73555e-e1dd-4509-8607-b23a135b0734-proxy-tls\") pod \"isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv\" (UID: \"bd73555e-e1dd-4509-8607-b23a135b0734\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv"
Sep 18 19:30:46.269800 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:46.269780    2532 generic.go:356] "Generic (PLEG): container finished" podID="b801f0ad-e919-4d3d-bf92-2dfd46d0029e" containerID="12533130ee96c78e1c1296da5c4d36518f0aa36ec3683e8ff61fd1e482801db7" exitCode=2
Sep 18 19:30:46.269899 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:46.269853    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2" event={"ID":"b801f0ad-e919-4d3d-bf92-2dfd46d0029e","Type":"ContainerDied","Data":"12533130ee96c78e1c1296da5c4d36518f0aa36ec3683e8ff61fd1e482801db7"}
Sep 18 19:30:46.272513 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:46.272491    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbmfq\" (UniqueName: \"kubernetes.io/projected/bd73555e-e1dd-4509-8607-b23a135b0734-kube-api-access-tbmfq\") pod \"isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv\" (UID: \"bd73555e-e1dd-4509-8607-b23a135b0734\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv"
Sep 18 19:30:46.273352 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:46.273328    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-xm89q" event={"ID":"901ea860-4642-47d2-a1a4-7bd463389d8c","Type":"ContainerStarted","Data":"8cffe138d2a7cba8c88e7f76b6b75c42cb55ab032b33d4295162bdbd218afcd2"}
Sep 18 19:30:46.273428 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:46.273390    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-xm89q" event={"ID":"901ea860-4642-47d2-a1a4-7bd463389d8c","Type":"ContainerStarted","Data":"8c19238a049d7b53a85c221c720f40710699e529c3972c5ddb654150fd801015"}
Sep 18 19:30:46.273610 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:46.273594    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-xm89q"
Sep 18 19:30:46.288016 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:46.287998    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:30:46.301741 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:46.301699    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-xm89q" podStartSLOduration=2.445325783 podStartE2EDuration="6.301685405s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.856359622s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:30:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:30:46.300664196 +0000 UTC m=+1013.221168053" watchObservedRunningTime="2026-09-18 19:30:46.301685405 +0000 UTC m=+1013.222189283"
Sep 18 19:30:46.342896 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:46.342874    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv"
Sep 18 19:30:46.374322 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:46.374290    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:30:46.628929 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:46.628901    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv"]
Sep 18 19:30:46.630724 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:30:46.630697    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd73555e_e1dd_4509_8607_b23a135b0734.slice/crio-a8aff1d38753db5e992590c42d211d6159ccf5598f5ce8905bb8b8c1e1baff5e WatchSource:0}: Error finding container a8aff1d38753db5e992590c42d211d6159ccf5598f5ce8905bb8b8c1e1baff5e: Status 404 returned error can't find the container with id a8aff1d38753db5e992590c42d211d6159ccf5598f5ce8905bb8b8c1e1baff5e
Sep 18 19:30:47.278918 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:47.278878    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv" event={"ID":"bd73555e-e1dd-4509-8607-b23a135b0734","Type":"ContainerStarted","Data":"15922ce1f626aaef8683e7833deefe2967b010b2ebe683519f8f440cb5492f41"}
Sep 18 19:30:47.279352 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:47.278949    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv" event={"ID":"bd73555e-e1dd-4509-8607-b23a135b0734","Type":"ContainerStarted","Data":"a8aff1d38753db5e992590c42d211d6159ccf5598f5ce8905bb8b8c1e1baff5e"}
Sep 18 19:30:47.279352 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:47.279205    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-xm89q"
Sep 18 19:30:47.280737 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:47.280711    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-xm89q" podUID="901ea860-4642-47d2-a1a4-7bd463389d8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 18 19:30:48.283814 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:48.283772    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-xm89q" podUID="901ea860-4642-47d2-a1a4-7bd463389d8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 18 19:30:48.945804 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:48.945764    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2" podUID="b801f0ad-e919-4d3d-bf92-2dfd46d0029e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.18:8643/healthz\": dial tcp 10.132.0.18:8643: connect: connection refused"
Sep 18 19:30:51.296860 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:51.296825    2532 generic.go:356] "Generic (PLEG): container finished" podID="bd73555e-e1dd-4509-8607-b23a135b0734" containerID="15922ce1f626aaef8683e7833deefe2967b010b2ebe683519f8f440cb5492f41" exitCode=0
Sep 18 19:30:51.297281 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:51.296897    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv" event={"ID":"bd73555e-e1dd-4509-8607-b23a135b0734","Type":"ContainerDied","Data":"15922ce1f626aaef8683e7833deefe2967b010b2ebe683519f8f440cb5492f41"}
Sep 18 19:30:51.299482 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:51.299419    2532 generic.go:356] "Generic (PLEG): container finished" podID="b801f0ad-e919-4d3d-bf92-2dfd46d0029e" containerID="369a4fca571f306aa44f258d39d446a396157d5ea66299bd8d856889d6d11665" exitCode=0
Sep 18 19:30:51.299482 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:51.299454    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2" event={"ID":"b801f0ad-e919-4d3d-bf92-2dfd46d0029e","Type":"ContainerDied","Data":"369a4fca571f306aa44f258d39d446a396157d5ea66299bd8d856889d6d11665"}
Sep 18 19:30:52.306000 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:52.305970    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv" event={"ID":"bd73555e-e1dd-4509-8607-b23a135b0734","Type":"ContainerStarted","Data":"d25ac0df380161ed7c3578a037d30a57fdfb84adf8364e488a997c554bd2ece0"}
Sep 18 19:30:52.306351 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:52.306030    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv" event={"ID":"bd73555e-e1dd-4509-8607-b23a135b0734","Type":"ContainerStarted","Data":"3aa8a9fbe9a0ce9ea6d7c5373260ba8eb4b6721770b20253805c8b7831e05f81"}
Sep 18 19:30:52.306351 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:52.306247    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv"
Sep 18 19:30:52.332764 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:52.332719    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv" podStartSLOduration=3.492933488 podStartE2EDuration="7.332705143s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.839771655s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:30:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:30:52.33116318 +0000 UTC m=+1019.251667050" watchObservedRunningTime="2026-09-18 19:30:52.332705143 +0000 UTC m=+1019.253209000"
Sep 18 19:30:53.288519 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:53.288489    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-xm89q"
Sep 18 19:30:53.289075 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:53.289029    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-xm89q" podUID="901ea860-4642-47d2-a1a4-7bd463389d8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 18 19:30:53.310174 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:53.310149    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv"
Sep 18 19:30:53.311245 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:53.311219    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv" podUID="bd73555e-e1dd-4509-8607-b23a135b0734" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 18 19:30:53.945251 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:53.945213    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2" podUID="b801f0ad-e919-4d3d-bf92-2dfd46d0029e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.18:8643/healthz\": dial tcp 10.132.0.18:8643: connect: connection refused"
Sep 18 19:30:54.313550 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:54.313453    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv" podUID="bd73555e-e1dd-4509-8607-b23a135b0734" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 18 19:30:54.952049 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:54.952007    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2" podUID="b801f0ad-e919-4d3d-bf92-2dfd46d0029e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 18 19:30:54.952377 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:54.952355    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2" podUID="b801f0ad-e919-4d3d-bf92-2dfd46d0029e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:30:58.944965 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:58.944922    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2" podUID="b801f0ad-e919-4d3d-bf92-2dfd46d0029e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.18:8643/healthz\": dial tcp 10.132.0.18:8643: connect: connection refused"
Sep 18 19:30:58.945476 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:58.945119    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2"
Sep 18 19:30:59.319214 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:59.319138    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv"
Sep 18 19:30:59.319642 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:30:59.319617    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv" podUID="bd73555e-e1dd-4509-8607-b23a135b0734" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 18 19:31:03.289106 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:31:03.289051    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-xm89q" podUID="901ea860-4642-47d2-a1a4-7bd463389d8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 18 19:31:03.944692 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:31:03.944655    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2" podUID="b801f0ad-e919-4d3d-bf92-2dfd46d0029e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.18:8643/healthz\": dial tcp 10.132.0.18:8643: connect: connection refused"
Sep 18 19:31:04.951462 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:31:04.951423    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2" podUID="b801f0ad-e919-4d3d-bf92-2dfd46d0029e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 18 19:31:04.951896 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:31:04.951734    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2" podUID="b801f0ad-e919-4d3d-bf92-2dfd46d0029e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:31:08.945648 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:31:08.945612    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2" podUID="b801f0ad-e919-4d3d-bf92-2dfd46d0029e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.18:8643/healthz\": dial tcp 10.132.0.18:8643: connect: connection refused"
Sep 18 19:31:09.319617 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:31:09.319534    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv" podUID="bd73555e-e1dd-4509-8607-b23a135b0734" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 18 19:31:13.289881 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:31:13.289836    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-xm89q" podUID="901ea860-4642-47d2-a1a4-7bd463389d8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 18 19:31:13.944925 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:31:13.944883    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2" podUID="b801f0ad-e919-4d3d-bf92-2dfd46d0029e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.18:8643/healthz\": dial tcp 10.132.0.18:8643: connect: connection refused"
Sep 18 19:31:14.952038 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:31:14.951998    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2" podUID="b801f0ad-e919-4d3d-bf92-2dfd46d0029e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 18 19:31:14.952516 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:31:14.952163    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2"
Sep 18 19:31:14.952516 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:31:14.952266    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2" podUID="b801f0ad-e919-4d3d-bf92-2dfd46d0029e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:31:14.952516 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:31:14.952363    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2"
Sep 18 19:31:16.399952 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:31:16.399924    2532 generic.go:356] "Generic (PLEG): container finished" podID="b801f0ad-e919-4d3d-bf92-2dfd46d0029e" containerID="367a5c4d5af63d363a61e5e532fd27fb4fb5e65d797ab0e6b190746f03270c3b" exitCode=0
Sep 18 19:31:16.400367 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:31:16.399965    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2" event={"ID":"b801f0ad-e919-4d3d-bf92-2dfd46d0029e","Type":"ContainerDied","Data":"367a5c4d5af63d363a61e5e532fd27fb4fb5e65d797ab0e6b190746f03270c3b"}
Sep 18 19:31:16.479780 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:31:16.479760    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2"
Sep 18 19:31:16.484716 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:31:16.484696    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2bnd6\" (UniqueName: \"kubernetes.io/projected/b801f0ad-e919-4d3d-bf92-2dfd46d0029e-kube-api-access-2bnd6\") pod \"b801f0ad-e919-4d3d-bf92-2dfd46d0029e\" (UID: \"b801f0ad-e919-4d3d-bf92-2dfd46d0029e\") "
Sep 18 19:31:16.484814 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:31:16.484732    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b801f0ad-e919-4d3d-bf92-2dfd46d0029e-kube-rbac-proxy-sar-config\") pod \"b801f0ad-e919-4d3d-bf92-2dfd46d0029e\" (UID: \"b801f0ad-e919-4d3d-bf92-2dfd46d0029e\") "
Sep 18 19:31:16.484814 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:31:16.484776    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b801f0ad-e919-4d3d-bf92-2dfd46d0029e-cabundle-cert\") pod \"b801f0ad-e919-4d3d-bf92-2dfd46d0029e\" (UID: \"b801f0ad-e919-4d3d-bf92-2dfd46d0029e\") "
Sep 18 19:31:16.484926 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:31:16.484840    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b801f0ad-e919-4d3d-bf92-2dfd46d0029e-proxy-tls\") pod \"b801f0ad-e919-4d3d-bf92-2dfd46d0029e\" (UID: \"b801f0ad-e919-4d3d-bf92-2dfd46d0029e\") "
Sep 18 19:31:16.484926 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:31:16.484884    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b801f0ad-e919-4d3d-bf92-2dfd46d0029e-kserve-provision-location\") pod \"b801f0ad-e919-4d3d-bf92-2dfd46d0029e\" (UID: \"b801f0ad-e919-4d3d-bf92-2dfd46d0029e\") "
Sep 18 19:31:16.485242 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:31:16.485214    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b801f0ad-e919-4d3d-bf92-2dfd46d0029e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b801f0ad-e919-4d3d-bf92-2dfd46d0029e" (UID: "b801f0ad-e919-4d3d-bf92-2dfd46d0029e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:31:16.485242 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:31:16.485233    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b801f0ad-e919-4d3d-bf92-2dfd46d0029e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b801f0ad-e919-4d3d-bf92-2dfd46d0029e" (UID: "b801f0ad-e919-4d3d-bf92-2dfd46d0029e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:31:16.485399 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:31:16.485214    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b801f0ad-e919-4d3d-bf92-2dfd46d0029e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b801f0ad-e919-4d3d-bf92-2dfd46d0029e" (UID: "b801f0ad-e919-4d3d-bf92-2dfd46d0029e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:31:16.486677 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:31:16.486646    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b801f0ad-e919-4d3d-bf92-2dfd46d0029e-kube-api-access-2bnd6" (OuterVolumeSpecName: "kube-api-access-2bnd6") pod "b801f0ad-e919-4d3d-bf92-2dfd46d0029e" (UID: "b801f0ad-e919-4d3d-bf92-2dfd46d0029e"). InnerVolumeSpecName "kube-api-access-2bnd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:31:16.486802 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:31:16.486786    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b801f0ad-e919-4d3d-bf92-2dfd46d0029e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b801f0ad-e919-4d3d-bf92-2dfd46d0029e" (UID: "b801f0ad-e919-4d3d-bf92-2dfd46d0029e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:31:16.585865 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:31:16.585838    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bnd6\" (UniqueName: \"kubernetes.io/projected/b801f0ad-e919-4d3d-bf92-2dfd46d0029e-kube-api-access-2bnd6\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:31:16.585865 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:31:16.585861    2532 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b801f0ad-e919-4d3d-bf92-2dfd46d0029e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:31:16.586008 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:31:16.585872    2532 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b801f0ad-e919-4d3d-bf92-2dfd46d0029e-cabundle-cert\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:31:16.586008 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:31:16.585881    2532 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b801f0ad-e919-4d3d-bf92-2dfd46d0029e-proxy-tls\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:31:16.586008 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:31:16.585889    2532 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b801f0ad-e919-4d3d-bf92-2dfd46d0029e-kserve-provision-location\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:31:17.406268 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:31:17.406235    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2" event={"ID":"b801f0ad-e919-4d3d-bf92-2dfd46d0029e","Type":"ContainerDied","Data":"4168ba2bb49634058e9c11043e97cdff7566d86469d728fdf04890a3d55c7e6c"}
Sep 18 19:31:17.406268 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:31:17.406260    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2"
Sep 18 19:31:17.406713 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:31:17.406336    2532 scope.go:117] "RemoveContainer" containerID="367a5c4d5af63d363a61e5e532fd27fb4fb5e65d797ab0e6b190746f03270c3b"
Sep 18 19:31:17.414193 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:31:17.414172    2532 scope.go:117] "RemoveContainer" containerID="12533130ee96c78e1c1296da5c4d36518f0aa36ec3683e8ff61fd1e482801db7"
Sep 18 19:31:17.421094 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:31:17.421074    2532 scope.go:117] "RemoveContainer" containerID="369a4fca571f306aa44f258d39d446a396157d5ea66299bd8d856889d6d11665"
Sep 18 19:31:17.427919 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:31:17.427904    2532 scope.go:117] "RemoveContainer" containerID="fe61008c07b8ef22e5a2f46854e80667c4726205d0d75d3caf7a1caed942a5f0"
Sep 18 19:31:17.469650 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:31:17.469630    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2"]
Sep 18 19:31:17.482733 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:31:17.482710    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fb30d-predictor-5f9f7c4cf4-tgxl2"]
Sep 18 19:31:17.621411 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:31:17.621381    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b801f0ad-e919-4d3d-bf92-2dfd46d0029e" path="/var/lib/kubelet/pods/b801f0ad-e919-4d3d-bf92-2dfd46d0029e/volumes"
Sep 18 19:31:19.320391 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:31:19.320354    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv" podUID="bd73555e-e1dd-4509-8607-b23a135b0734" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 18 19:31:23.289612 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:31:23.289569    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-xm89q" podUID="901ea860-4642-47d2-a1a4-7bd463389d8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 18 19:31:29.319691 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:31:29.319650    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv" podUID="bd73555e-e1dd-4509-8607-b23a135b0734" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 18 19:31:30.147009 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:31:30.146973    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:31:33.289965 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:31:33.289925    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-xm89q" podUID="901ea860-4642-47d2-a1a4-7bd463389d8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 18 19:31:39.320000 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:31:39.319958    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv" podUID="bd73555e-e1dd-4509-8607-b23a135b0734" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 18 19:31:43.289609 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:31:43.289563    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-xm89q" podUID="901ea860-4642-47d2-a1a4-7bd463389d8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 18 19:31:49.319883 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:31:49.319807    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv" podUID="bd73555e-e1dd-4509-8607-b23a135b0734" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 18 19:31:53.289759 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:31:53.289723    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-xm89q" podUID="901ea860-4642-47d2-a1a4-7bd463389d8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 18 19:31:53.618487 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:31:53.618455    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-xm89q" podUID="901ea860-4642-47d2-a1a4-7bd463389d8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 18 19:31:59.320491 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:31:59.320453    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv" podUID="bd73555e-e1dd-4509-8607-b23a135b0734" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 18 19:32:00.616396 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:00.616356    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv" podUID="bd73555e-e1dd-4509-8607-b23a135b0734" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 18 19:32:03.620399 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:03.620371    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-xm89q"
Sep 18 19:32:10.479220 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:10.479185    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-xm89q"]
Sep 18 19:32:10.479615 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:10.479501    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-xm89q" podUID="901ea860-4642-47d2-a1a4-7bd463389d8c" containerName="kserve-container" containerID="cri-o://8c19238a049d7b53a85c221c720f40710699e529c3972c5ddb654150fd801015" gracePeriod=30
Sep 18 19:32:10.479615 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:10.479544    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-xm89q" podUID="901ea860-4642-47d2-a1a4-7bd463389d8c" containerName="kube-rbac-proxy" containerID="cri-o://8cffe138d2a7cba8c88e7f76b6b75c42cb55ab032b33d4295162bdbd218afcd2" gracePeriod=30
Sep 18 19:32:10.519515 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:10.519486    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:32:10.603449 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:10.603426    2532 generic.go:356] "Generic (PLEG): container finished" podID="901ea860-4642-47d2-a1a4-7bd463389d8c" containerID="8cffe138d2a7cba8c88e7f76b6b75c42cb55ab032b33d4295162bdbd218afcd2" exitCode=2
Sep 18 19:32:10.603528 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:10.603502    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-xm89q" event={"ID":"901ea860-4642-47d2-a1a4-7bd463389d8c","Type":"ContainerDied","Data":"8cffe138d2a7cba8c88e7f76b6b75c42cb55ab032b33d4295162bdbd218afcd2"}
Sep 18 19:32:10.617199 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:10.617183    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv"
Sep 18 19:32:13.284583 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:13.284540    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-xm89q" podUID="901ea860-4642-47d2-a1a4-7bd463389d8c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.21:8643/healthz\": dial tcp 10.132.0.21:8643: connect: connection refused"
Sep 18 19:32:13.619098 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:13.618988    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-xm89q" podUID="901ea860-4642-47d2-a1a4-7bd463389d8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 18 19:32:14.818963 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:14.818941    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-xm89q"
Sep 18 19:32:14.907109 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:14.907021    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/901ea860-4642-47d2-a1a4-7bd463389d8c-proxy-tls\") pod \"901ea860-4642-47d2-a1a4-7bd463389d8c\" (UID: \"901ea860-4642-47d2-a1a4-7bd463389d8c\") "
Sep 18 19:32:14.907109 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:14.907086    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/901ea860-4642-47d2-a1a4-7bd463389d8c-kube-rbac-proxy-sar-config\") pod \"901ea860-4642-47d2-a1a4-7bd463389d8c\" (UID: \"901ea860-4642-47d2-a1a4-7bd463389d8c\") "
Sep 18 19:32:14.907301 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:14.907187    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/901ea860-4642-47d2-a1a4-7bd463389d8c-kserve-provision-location\") pod \"901ea860-4642-47d2-a1a4-7bd463389d8c\" (UID: \"901ea860-4642-47d2-a1a4-7bd463389d8c\") "
Sep 18 19:32:14.907301 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:14.907264    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mrpdh\" (UniqueName: \"kubernetes.io/projected/901ea860-4642-47d2-a1a4-7bd463389d8c-kube-api-access-mrpdh\") pod \"901ea860-4642-47d2-a1a4-7bd463389d8c\" (UID: \"901ea860-4642-47d2-a1a4-7bd463389d8c\") "
Sep 18 19:32:14.907301 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:14.907292    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/901ea860-4642-47d2-a1a4-7bd463389d8c-cabundle-cert\") pod \"901ea860-4642-47d2-a1a4-7bd463389d8c\" (UID: \"901ea860-4642-47d2-a1a4-7bd463389d8c\") "
Sep 18 19:32:14.907470 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:14.907433    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/901ea860-4642-47d2-a1a4-7bd463389d8c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "901ea860-4642-47d2-a1a4-7bd463389d8c" (UID: "901ea860-4642-47d2-a1a4-7bd463389d8c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:32:14.907512 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:14.907482    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/901ea860-4642-47d2-a1a4-7bd463389d8c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "901ea860-4642-47d2-a1a4-7bd463389d8c" (UID: "901ea860-4642-47d2-a1a4-7bd463389d8c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:32:14.907724 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:14.907701    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/901ea860-4642-47d2-a1a4-7bd463389d8c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "901ea860-4642-47d2-a1a4-7bd463389d8c" (UID: "901ea860-4642-47d2-a1a4-7bd463389d8c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:32:14.909103 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:14.909083    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/901ea860-4642-47d2-a1a4-7bd463389d8c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "901ea860-4642-47d2-a1a4-7bd463389d8c" (UID: "901ea860-4642-47d2-a1a4-7bd463389d8c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:32:14.909235 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:14.909215    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/901ea860-4642-47d2-a1a4-7bd463389d8c-kube-api-access-mrpdh" (OuterVolumeSpecName: "kube-api-access-mrpdh") pod "901ea860-4642-47d2-a1a4-7bd463389d8c" (UID: "901ea860-4642-47d2-a1a4-7bd463389d8c"). InnerVolumeSpecName "kube-api-access-mrpdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:32:15.008101 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.008052    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrpdh\" (UniqueName: \"kubernetes.io/projected/901ea860-4642-47d2-a1a4-7bd463389d8c-kube-api-access-mrpdh\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:32:15.008101 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.008099    2532 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/901ea860-4642-47d2-a1a4-7bd463389d8c-cabundle-cert\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:32:15.008101 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.008109    2532 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/901ea860-4642-47d2-a1a4-7bd463389d8c-proxy-tls\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:32:15.008256 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.008118    2532 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/901ea860-4642-47d2-a1a4-7bd463389d8c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:32:15.008256 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.008127    2532 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/901ea860-4642-47d2-a1a4-7bd463389d8c-kserve-provision-location\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:32:15.623809 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.623775    2532 generic.go:356] "Generic (PLEG): container finished" podID="901ea860-4642-47d2-a1a4-7bd463389d8c" containerID="8c19238a049d7b53a85c221c720f40710699e529c3972c5ddb654150fd801015" exitCode=0
Sep 18 19:32:15.623967 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.623828    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-xm89q"
Sep 18 19:32:15.623967 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.623849    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-xm89q" event={"ID":"901ea860-4642-47d2-a1a4-7bd463389d8c","Type":"ContainerDied","Data":"8c19238a049d7b53a85c221c720f40710699e529c3972c5ddb654150fd801015"}
Sep 18 19:32:15.623967 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.623904    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-xm89q" event={"ID":"901ea860-4642-47d2-a1a4-7bd463389d8c","Type":"ContainerDied","Data":"e583d4c0e3abf332d36a14479bf76bf19c36830e4aa0b0a46d2997d38aaf2b85"}
Sep 18 19:32:15.623967 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.623931    2532 scope.go:117] "RemoveContainer" containerID="8cffe138d2a7cba8c88e7f76b6b75c42cb55ab032b33d4295162bdbd218afcd2"
Sep 18 19:32:15.633095 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.633075    2532 scope.go:117] "RemoveContainer" containerID="8c19238a049d7b53a85c221c720f40710699e529c3972c5ddb654150fd801015"
Sep 18 19:32:15.640336 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.640321    2532 scope.go:117] "RemoveContainer" containerID="fa8b53dc6b8a991b79078830495dfd1d5a18e7854076206b4d5bea94760a1850"
Sep 18 19:32:15.649892 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.649856    2532 scope.go:117] "RemoveContainer" containerID="8cffe138d2a7cba8c88e7f76b6b75c42cb55ab032b33d4295162bdbd218afcd2"
Sep 18 19:32:15.650340 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:32:15.650314    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cffe138d2a7cba8c88e7f76b6b75c42cb55ab032b33d4295162bdbd218afcd2\": container with ID starting with 8cffe138d2a7cba8c88e7f76b6b75c42cb55ab032b33d4295162bdbd218afcd2 not found: ID does not exist" containerID="8cffe138d2a7cba8c88e7f76b6b75c42cb55ab032b33d4295162bdbd218afcd2"
Sep 18 19:32:15.650442 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.650345    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cffe138d2a7cba8c88e7f76b6b75c42cb55ab032b33d4295162bdbd218afcd2"} err="failed to get container status \"8cffe138d2a7cba8c88e7f76b6b75c42cb55ab032b33d4295162bdbd218afcd2\": rpc error: code = NotFound desc = could not find container \"8cffe138d2a7cba8c88e7f76b6b75c42cb55ab032b33d4295162bdbd218afcd2\": container with ID starting with 8cffe138d2a7cba8c88e7f76b6b75c42cb55ab032b33d4295162bdbd218afcd2 not found: ID does not exist"
Sep 18 19:32:15.650442 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.650361    2532 scope.go:117] "RemoveContainer" containerID="8c19238a049d7b53a85c221c720f40710699e529c3972c5ddb654150fd801015"
Sep 18 19:32:15.650630 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:32:15.650604    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c19238a049d7b53a85c221c720f40710699e529c3972c5ddb654150fd801015\": container with ID starting with 8c19238a049d7b53a85c221c720f40710699e529c3972c5ddb654150fd801015 not found: ID does not exist" containerID="8c19238a049d7b53a85c221c720f40710699e529c3972c5ddb654150fd801015"
Sep 18 19:32:15.650680 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.650634    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c19238a049d7b53a85c221c720f40710699e529c3972c5ddb654150fd801015"} err="failed to get container status \"8c19238a049d7b53a85c221c720f40710699e529c3972c5ddb654150fd801015\": rpc error: code = NotFound desc = could not find container \"8c19238a049d7b53a85c221c720f40710699e529c3972c5ddb654150fd801015\": container with ID starting with 8c19238a049d7b53a85c221c720f40710699e529c3972c5ddb654150fd801015 not found: ID does not exist"
Sep 18 19:32:15.650680 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.650647    2532 scope.go:117] "RemoveContainer" containerID="fa8b53dc6b8a991b79078830495dfd1d5a18e7854076206b4d5bea94760a1850"
Sep 18 19:32:15.650880 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:32:15.650863    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa8b53dc6b8a991b79078830495dfd1d5a18e7854076206b4d5bea94760a1850\": container with ID starting with fa8b53dc6b8a991b79078830495dfd1d5a18e7854076206b4d5bea94760a1850 not found: ID does not exist" containerID="fa8b53dc6b8a991b79078830495dfd1d5a18e7854076206b4d5bea94760a1850"
Sep 18 19:32:15.650927 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.650885    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa8b53dc6b8a991b79078830495dfd1d5a18e7854076206b4d5bea94760a1850"} err="failed to get container status \"fa8b53dc6b8a991b79078830495dfd1d5a18e7854076206b4d5bea94760a1850\": rpc error: code = NotFound desc = could not find container \"fa8b53dc6b8a991b79078830495dfd1d5a18e7854076206b4d5bea94760a1850\": container with ID starting with fa8b53dc6b8a991b79078830495dfd1d5a18e7854076206b4d5bea94760a1850 not found: ID does not exist"
Sep 18 19:32:15.662197 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.662174    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-xm89q"]
Sep 18 19:32:15.668675 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.668653    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-xm89q"]
Sep 18 19:32:15.676627 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.676600    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:32:15.705155 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.705134    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4"]
Sep 18 19:32:15.705403 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.705391    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b801f0ad-e919-4d3d-bf92-2dfd46d0029e" containerName="storage-initializer"
Sep 18 19:32:15.705446 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.705405    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="b801f0ad-e919-4d3d-bf92-2dfd46d0029e" containerName="storage-initializer"
Sep 18 19:32:15.705446 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.705420    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="901ea860-4642-47d2-a1a4-7bd463389d8c" containerName="kserve-container"
Sep 18 19:32:15.705446 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.705425    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="901ea860-4642-47d2-a1a4-7bd463389d8c" containerName="kserve-container"
Sep 18 19:32:15.705446 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.705432    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b801f0ad-e919-4d3d-bf92-2dfd46d0029e" containerName="kube-rbac-proxy"
Sep 18 19:32:15.705446 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.705438    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="b801f0ad-e919-4d3d-bf92-2dfd46d0029e" containerName="kube-rbac-proxy"
Sep 18 19:32:15.705446 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.705444    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="901ea860-4642-47d2-a1a4-7bd463389d8c" containerName="storage-initializer"
Sep 18 19:32:15.705621 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.705450    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="901ea860-4642-47d2-a1a4-7bd463389d8c" containerName="storage-initializer"
Sep 18 19:32:15.705621 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.705458    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b801f0ad-e919-4d3d-bf92-2dfd46d0029e" containerName="agent"
Sep 18 19:32:15.705621 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.705463    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="b801f0ad-e919-4d3d-bf92-2dfd46d0029e" containerName="agent"
Sep 18 19:32:15.705621 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.705467    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="901ea860-4642-47d2-a1a4-7bd463389d8c" containerName="kube-rbac-proxy"
Sep 18 19:32:15.705621 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.705472    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="901ea860-4642-47d2-a1a4-7bd463389d8c" containerName="kube-rbac-proxy"
Sep 18 19:32:15.705621 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.705478    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b801f0ad-e919-4d3d-bf92-2dfd46d0029e" containerName="kserve-container"
Sep 18 19:32:15.705621 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.705483    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="b801f0ad-e919-4d3d-bf92-2dfd46d0029e" containerName="kserve-container"
Sep 18 19:32:15.705621 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.705526    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="b801f0ad-e919-4d3d-bf92-2dfd46d0029e" containerName="kserve-container"
Sep 18 19:32:15.705621 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.705534    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="b801f0ad-e919-4d3d-bf92-2dfd46d0029e" containerName="kube-rbac-proxy"
Sep 18 19:32:15.705621 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.705540    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="b801f0ad-e919-4d3d-bf92-2dfd46d0029e" containerName="agent"
Sep 18 19:32:15.705621 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.705546    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="901ea860-4642-47d2-a1a4-7bd463389d8c" containerName="kserve-container"
Sep 18 19:32:15.705621 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.705551    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="901ea860-4642-47d2-a1a4-7bd463389d8c" containerName="kube-rbac-proxy"
Sep 18 19:32:15.708367 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.708352    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4"
Sep 18 19:32:15.711203 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.711188    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Sep 18 19:32:15.711650 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.711632    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 18 19:32:15.724093 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.724070    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4"]
Sep 18 19:32:15.775681 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.775656    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8"]
Sep 18 19:32:15.778834 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.778819    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8"
Sep 18 19:32:15.779850 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.779823    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:32:15.781528 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.781513    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 18 19:32:15.814359 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.814341    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcjk4\" (UniqueName: \"kubernetes.io/projected/f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207-kube-api-access-mcjk4\") pod \"isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4\" (UID: \"f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4"
Sep 18 19:32:15.814474 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.814375    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4\" (UID: \"f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4"
Sep 18 19:32:15.814474 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.814399    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4\" (UID: \"f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4"
Sep 18 19:32:15.814474 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.814439    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4\" (UID: \"f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4"
Sep 18 19:32:15.814639 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.814506    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4\" (UID: \"f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4"
Sep 18 19:32:15.857500 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.857478    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8"]
Sep 18 19:32:15.898106 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.898027    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:32:15.915161 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.915138    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67fs9\" (UniqueName: \"kubernetes.io/projected/4da5ab65-7c8d-41b7-b723-ac3108fc43e2-kube-api-access-67fs9\") pod \"isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8\" (UID: \"4da5ab65-7c8d-41b7-b723-ac3108fc43e2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8"
Sep 18 19:32:15.915260 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.915173    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4\" (UID: \"f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4"
Sep 18 19:32:15.915260 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.915204    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcjk4\" (UniqueName: \"kubernetes.io/projected/f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207-kube-api-access-mcjk4\") pod \"isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4\" (UID: \"f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4"
Sep 18 19:32:15.915260 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.915221    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4da5ab65-7c8d-41b7-b723-ac3108fc43e2-proxy-tls\") pod \"isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8\" (UID: \"4da5ab65-7c8d-41b7-b723-ac3108fc43e2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8"
Sep 18 19:32:15.915260 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.915239    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4da5ab65-7c8d-41b7-b723-ac3108fc43e2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8\" (UID: \"4da5ab65-7c8d-41b7-b723-ac3108fc43e2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8"
Sep 18 19:32:15.915260 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.915262    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4\" (UID: \"f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4"
Sep 18 19:32:15.915516 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.915285    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4\" (UID: \"f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4"
Sep 18 19:32:15.915516 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.915299    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4da5ab65-7c8d-41b7-b723-ac3108fc43e2-cabundle-cert\") pod \"isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8\" (UID: \"4da5ab65-7c8d-41b7-b723-ac3108fc43e2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8"
Sep 18 19:32:15.915516 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.915379    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4\" (UID: \"f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4"
Sep 18 19:32:15.915516 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.915437    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4da5ab65-7c8d-41b7-b723-ac3108fc43e2-kserve-provision-location\") pod \"isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8\" (UID: \"4da5ab65-7c8d-41b7-b723-ac3108fc43e2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8"
Sep 18 19:32:15.915662 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.915580    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4\" (UID: \"f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4"
Sep 18 19:32:15.915857 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.915836    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4\" (UID: \"f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4"
Sep 18 19:32:15.915957 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.915946    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4\" (UID: \"f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4"
Sep 18 19:32:15.917690 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.917673    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4\" (UID: \"f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4"
Sep 18 19:32:15.924509 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:15.924488    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcjk4\" (UniqueName: \"kubernetes.io/projected/f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207-kube-api-access-mcjk4\") pod \"isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4\" (UID: \"f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4"
Sep 18 19:32:16.016306 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:16.016276    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4da5ab65-7c8d-41b7-b723-ac3108fc43e2-cabundle-cert\") pod \"isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8\" (UID: \"4da5ab65-7c8d-41b7-b723-ac3108fc43e2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8"
Sep 18 19:32:16.016495 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:16.016314    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4da5ab65-7c8d-41b7-b723-ac3108fc43e2-kserve-provision-location\") pod \"isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8\" (UID: \"4da5ab65-7c8d-41b7-b723-ac3108fc43e2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8"
Sep 18 19:32:16.016495 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:16.016338    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67fs9\" (UniqueName: \"kubernetes.io/projected/4da5ab65-7c8d-41b7-b723-ac3108fc43e2-kube-api-access-67fs9\") pod \"isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8\" (UID: \"4da5ab65-7c8d-41b7-b723-ac3108fc43e2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8"
Sep 18 19:32:16.016495 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:16.016373    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4da5ab65-7c8d-41b7-b723-ac3108fc43e2-proxy-tls\") pod \"isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8\" (UID: \"4da5ab65-7c8d-41b7-b723-ac3108fc43e2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8"
Sep 18 19:32:16.016495 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:16.016391    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4da5ab65-7c8d-41b7-b723-ac3108fc43e2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8\" (UID: \"4da5ab65-7c8d-41b7-b723-ac3108fc43e2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8"
Sep 18 19:32:16.016651 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:32:16.016522    2532 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-predictor-serving-cert: secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 18 19:32:16.016651 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:32:16.016594    2532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4da5ab65-7c8d-41b7-b723-ac3108fc43e2-proxy-tls podName:4da5ab65-7c8d-41b7-b723-ac3108fc43e2 nodeName:}" failed. No retries permitted until 2026-09-18 19:32:16.516572097 +0000 UTC m=+1103.437075950 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4da5ab65-7c8d-41b7-b723-ac3108fc43e2-proxy-tls") pod "isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8" (UID: "4da5ab65-7c8d-41b7-b723-ac3108fc43e2") : secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 18 19:32:16.016741 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:16.016720    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4da5ab65-7c8d-41b7-b723-ac3108fc43e2-kserve-provision-location\") pod \"isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8\" (UID: \"4da5ab65-7c8d-41b7-b723-ac3108fc43e2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8"
Sep 18 19:32:16.016959 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:16.016944    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4da5ab65-7c8d-41b7-b723-ac3108fc43e2-cabundle-cert\") pod \"isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8\" (UID: \"4da5ab65-7c8d-41b7-b723-ac3108fc43e2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8"
Sep 18 19:32:16.017033 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:16.017017    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4da5ab65-7c8d-41b7-b723-ac3108fc43e2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8\" (UID: \"4da5ab65-7c8d-41b7-b723-ac3108fc43e2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8"
Sep 18 19:32:16.018227 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:16.018207    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4"
Sep 18 19:32:16.027228 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:16.027211    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67fs9\" (UniqueName: \"kubernetes.io/projected/4da5ab65-7c8d-41b7-b723-ac3108fc43e2-kube-api-access-67fs9\") pod \"isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8\" (UID: \"4da5ab65-7c8d-41b7-b723-ac3108fc43e2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8"
Sep 18 19:32:16.089154 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:16.089083    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:32:16.293677 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:16.293644    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4"]
Sep 18 19:32:16.296558 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:32:16.296534    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4c80a9f_e8b4_4df5_ba16_c84bcdf0e207.slice/crio-aa24c74db10de4eb560c879d3628e31d554c3550c2c630efa7a85f7a0d35a719 WatchSource:0}: Error finding container aa24c74db10de4eb560c879d3628e31d554c3550c2c630efa7a85f7a0d35a719: Status 404 returned error can't find the container with id aa24c74db10de4eb560c879d3628e31d554c3550c2c630efa7a85f7a0d35a719
Sep 18 19:32:16.521103 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:16.521048    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4da5ab65-7c8d-41b7-b723-ac3108fc43e2-proxy-tls\") pod \"isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8\" (UID: \"4da5ab65-7c8d-41b7-b723-ac3108fc43e2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8"
Sep 18 19:32:16.523366 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:16.523341    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4da5ab65-7c8d-41b7-b723-ac3108fc43e2-proxy-tls\") pod \"isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8\" (UID: \"4da5ab65-7c8d-41b7-b723-ac3108fc43e2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8"
Sep 18 19:32:16.630321 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:16.630249    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4" event={"ID":"f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207","Type":"ContainerStarted","Data":"a609864be33d7b4e0caa7a775273e5c5bdc2afdcc6478f2dec3506c54ed139e7"}
Sep 18 19:32:16.630321 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:16.630310    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4" event={"ID":"f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207","Type":"ContainerStarted","Data":"aa24c74db10de4eb560c879d3628e31d554c3550c2c630efa7a85f7a0d35a719"}
Sep 18 19:32:16.688260 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:16.688228    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8"
Sep 18 19:32:16.981490 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:16.981466    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8"]
Sep 18 19:32:16.983379 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:32:16.983353    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4da5ab65_7c8d_41b7_b723_ac3108fc43e2.slice/crio-e18edc40305188ff1aa03ccea2952e082479c078c2159c1b51ceacc3765becf4 WatchSource:0}: Error finding container e18edc40305188ff1aa03ccea2952e082479c078c2159c1b51ceacc3765becf4: Status 404 returned error can't find the container with id e18edc40305188ff1aa03ccea2952e082479c078c2159c1b51ceacc3765becf4
Sep 18 19:32:17.621421 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:17.621388    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="901ea860-4642-47d2-a1a4-7bd463389d8c" path="/var/lib/kubelet/pods/901ea860-4642-47d2-a1a4-7bd463389d8c/volumes"
Sep 18 19:32:17.635578 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:17.635549    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8" event={"ID":"4da5ab65-7c8d-41b7-b723-ac3108fc43e2","Type":"ContainerStarted","Data":"f7886435c039d0e397221152b038db52dceb0c7d2ae0cd27e9c695bde577c4e4"}
Sep 18 19:32:17.635720 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:17.635592    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8" event={"ID":"4da5ab65-7c8d-41b7-b723-ac3108fc43e2","Type":"ContainerStarted","Data":"e18edc40305188ff1aa03ccea2952e082479c078c2159c1b51ceacc3765becf4"}
Sep 18 19:32:20.653414 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:20.653377    2532 generic.go:356] "Generic (PLEG): container finished" podID="f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207" containerID="a609864be33d7b4e0caa7a775273e5c5bdc2afdcc6478f2dec3506c54ed139e7" exitCode=0
Sep 18 19:32:20.653881 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:20.653456    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4" event={"ID":"f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207","Type":"ContainerDied","Data":"a609864be33d7b4e0caa7a775273e5c5bdc2afdcc6478f2dec3506c54ed139e7"}
Sep 18 19:32:21.657932 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:21.657903    2532 generic.go:356] "Generic (PLEG): container finished" podID="4da5ab65-7c8d-41b7-b723-ac3108fc43e2" containerID="f7886435c039d0e397221152b038db52dceb0c7d2ae0cd27e9c695bde577c4e4" exitCode=0
Sep 18 19:32:21.658417 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:21.657978    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8" event={"ID":"4da5ab65-7c8d-41b7-b723-ac3108fc43e2","Type":"ContainerDied","Data":"f7886435c039d0e397221152b038db52dceb0c7d2ae0cd27e9c695bde577c4e4"}
Sep 18 19:32:21.660051 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:21.660032    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4" event={"ID":"f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207","Type":"ContainerStarted","Data":"3e563d87953aed5c400e667573d9d48a02cabc605198c0d2ce2f19b3bd2c6bf6"}
Sep 18 19:32:21.660146 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:21.660097    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4" event={"ID":"f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207","Type":"ContainerStarted","Data":"dcd704c0066a0f787c6361dab9ecb37c2a1b7d5d0e8263e55ba45f2ea415196f"}
Sep 18 19:32:21.660293 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:21.660275    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4"
Sep 18 19:32:21.702908 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:21.702854    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4" podStartSLOduration=3.061440562 podStartE2EDuration="6.702836765s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.641396203s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:32:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:32:21.701580443 +0000 UTC m=+1108.622084300" watchObservedRunningTime="2026-09-18 19:32:21.702836765 +0000 UTC m=+1108.623340623"
Sep 18 19:32:22.666046 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:22.666010    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8" event={"ID":"4da5ab65-7c8d-41b7-b723-ac3108fc43e2","Type":"ContainerStarted","Data":"6eb035007ac764145f50cd934a31d4ec67286ab9ef36c7aca53a0159b93f3596"}
Sep 18 19:32:22.666472 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:22.666081    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8" event={"ID":"4da5ab65-7c8d-41b7-b723-ac3108fc43e2","Type":"ContainerStarted","Data":"f6e63ec8965c60d077fd17cbe05328f619245d8f4a52d159e527b97853ea29ae"}
Sep 18 19:32:22.666472 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:22.666347    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4"
Sep 18 19:32:22.666472 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:22.666406    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8"
Sep 18 19:32:22.666472 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:22.666438    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8"
Sep 18 19:32:22.667418 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:22.667384    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4" podUID="f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 18 19:32:22.667418 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:22.667405    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8" podUID="4da5ab65-7c8d-41b7-b723-ac3108fc43e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 18 19:32:22.692969 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:22.692927    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8" podStartSLOduration=3.854719147 podStartE2EDuration="7.69291474s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.838195593s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:32:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:32:22.69101236 +0000 UTC m=+1109.611516217" watchObservedRunningTime="2026-09-18 19:32:22.69291474 +0000 UTC m=+1109.613418651"
Sep 18 19:32:23.670035 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:23.669992    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8" podUID="4da5ab65-7c8d-41b7-b723-ac3108fc43e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 18 19:32:23.670426 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:23.670140    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4" podUID="f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 18 19:32:26.019170 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:26.019136    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:32:26.154833 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:26.154803    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:32:28.674715 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:28.674676    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4"
Sep 18 19:32:28.675186 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:28.675144    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8"
Sep 18 19:32:28.675264 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:28.675233    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4" podUID="f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 18 19:32:28.675677 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:28.675651    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8" podUID="4da5ab65-7c8d-41b7-b723-ac3108fc43e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 18 19:32:36.102880 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:36.102841    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:32:36.185341 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:36.185303    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv"]
Sep 18 19:32:36.185583 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:36.185557    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv" podUID="bd73555e-e1dd-4509-8607-b23a135b0734" containerName="kserve-container" containerID="cri-o://3aa8a9fbe9a0ce9ea6d7c5373260ba8eb4b6721770b20253805c8b7831e05f81" gracePeriod=30
Sep 18 19:32:36.185642 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:36.185577    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv" podUID="bd73555e-e1dd-4509-8607-b23a135b0734" containerName="kube-rbac-proxy" containerID="cri-o://d25ac0df380161ed7c3578a037d30a57fdfb84adf8364e488a997c554bd2ece0" gracePeriod=30
Sep 18 19:32:36.237759 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:36.237711    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:32:36.343820 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:36.343791    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:32:36.452553 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:36.452525    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:32:36.587997 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:36.587957    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:32:36.689109 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:36.689045    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:32:36.723678 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:36.723601    2532 generic.go:356] "Generic (PLEG): container finished" podID="bd73555e-e1dd-4509-8607-b23a135b0734" containerID="d25ac0df380161ed7c3578a037d30a57fdfb84adf8364e488a997c554bd2ece0" exitCode=2
Sep 18 19:32:36.723678 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:36.723638    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv" event={"ID":"bd73555e-e1dd-4509-8607-b23a135b0734","Type":"ContainerDied","Data":"d25ac0df380161ed7c3578a037d30a57fdfb84adf8364e488a997c554bd2ece0"}
Sep 18 19:32:36.930740 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:36.930702    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:32:38.675914 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:38.675882    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8" podUID="4da5ab65-7c8d-41b7-b723-ac3108fc43e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 18 19:32:38.676297 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:38.675880    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4" podUID="f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 18 19:32:39.314482 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:39.314438    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv" podUID="bd73555e-e1dd-4509-8607-b23a135b0734" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
Sep 18 19:32:40.616697 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:40.616658    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv" podUID="bd73555e-e1dd-4509-8607-b23a135b0734" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 18 19:32:40.927744 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:40.927718    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv"
Sep 18 19:32:41.109127 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:41.109098    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbmfq\" (UniqueName: \"kubernetes.io/projected/bd73555e-e1dd-4509-8607-b23a135b0734-kube-api-access-tbmfq\") pod \"bd73555e-e1dd-4509-8607-b23a135b0734\" (UID: \"bd73555e-e1dd-4509-8607-b23a135b0734\") "
Sep 18 19:32:41.109311 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:41.109136    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd73555e-e1dd-4509-8607-b23a135b0734-proxy-tls\") pod \"bd73555e-e1dd-4509-8607-b23a135b0734\" (UID: \"bd73555e-e1dd-4509-8607-b23a135b0734\") "
Sep 18 19:32:41.109311 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:41.109179    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd73555e-e1dd-4509-8607-b23a135b0734-kube-rbac-proxy-sar-config\") pod \"bd73555e-e1dd-4509-8607-b23a135b0734\" (UID: \"bd73555e-e1dd-4509-8607-b23a135b0734\") "
Sep 18 19:32:41.109311 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:41.109216    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd73555e-e1dd-4509-8607-b23a135b0734-kserve-provision-location\") pod \"bd73555e-e1dd-4509-8607-b23a135b0734\" (UID: \"bd73555e-e1dd-4509-8607-b23a135b0734\") "
Sep 18 19:32:41.109311 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:41.109231    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd73555e-e1dd-4509-8607-b23a135b0734-cabundle-cert\") pod \"bd73555e-e1dd-4509-8607-b23a135b0734\" (UID: \"bd73555e-e1dd-4509-8607-b23a135b0734\") "
Sep 18 19:32:41.109594 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:41.109567    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd73555e-e1dd-4509-8607-b23a135b0734-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bd73555e-e1dd-4509-8607-b23a135b0734" (UID: "bd73555e-e1dd-4509-8607-b23a135b0734"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:32:41.109653 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:41.109592    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd73555e-e1dd-4509-8607-b23a135b0734-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bd73555e-e1dd-4509-8607-b23a135b0734" (UID: "bd73555e-e1dd-4509-8607-b23a135b0734"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:32:41.109653 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:41.109630    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd73555e-e1dd-4509-8607-b23a135b0734-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bd73555e-e1dd-4509-8607-b23a135b0734" (UID: "bd73555e-e1dd-4509-8607-b23a135b0734"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:32:41.111244 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:41.111210    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd73555e-e1dd-4509-8607-b23a135b0734-kube-api-access-tbmfq" (OuterVolumeSpecName: "kube-api-access-tbmfq") pod "bd73555e-e1dd-4509-8607-b23a135b0734" (UID: "bd73555e-e1dd-4509-8607-b23a135b0734"). InnerVolumeSpecName "kube-api-access-tbmfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:32:41.111346 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:41.111300    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd73555e-e1dd-4509-8607-b23a135b0734-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bd73555e-e1dd-4509-8607-b23a135b0734" (UID: "bd73555e-e1dd-4509-8607-b23a135b0734"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:32:41.209903 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:41.209877    2532 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd73555e-e1dd-4509-8607-b23a135b0734-kserve-provision-location\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:32:41.209903 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:41.209901    2532 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd73555e-e1dd-4509-8607-b23a135b0734-cabundle-cert\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:32:41.210075 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:41.209911    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbmfq\" (UniqueName: \"kubernetes.io/projected/bd73555e-e1dd-4509-8607-b23a135b0734-kube-api-access-tbmfq\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:32:41.210075 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:41.209919    2532 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd73555e-e1dd-4509-8607-b23a135b0734-proxy-tls\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:32:41.210075 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:41.209929    2532 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd73555e-e1dd-4509-8607-b23a135b0734-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:32:41.746358 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:41.746327    2532 generic.go:356] "Generic (PLEG): container finished" podID="bd73555e-e1dd-4509-8607-b23a135b0734" containerID="3aa8a9fbe9a0ce9ea6d7c5373260ba8eb4b6721770b20253805c8b7831e05f81" exitCode=0
Sep 18 19:32:41.746721 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:41.746384    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv"
Sep 18 19:32:41.746721 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:41.746384    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv" event={"ID":"bd73555e-e1dd-4509-8607-b23a135b0734","Type":"ContainerDied","Data":"3aa8a9fbe9a0ce9ea6d7c5373260ba8eb4b6721770b20253805c8b7831e05f81"}
Sep 18 19:32:41.746721 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:41.746457    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv" event={"ID":"bd73555e-e1dd-4509-8607-b23a135b0734","Type":"ContainerDied","Data":"a8aff1d38753db5e992590c42d211d6159ccf5598f5ce8905bb8b8c1e1baff5e"}
Sep 18 19:32:41.746721 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:41.746496    2532 scope.go:117] "RemoveContainer" containerID="d25ac0df380161ed7c3578a037d30a57fdfb84adf8364e488a997c554bd2ece0"
Sep 18 19:32:41.754159 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:41.754134    2532 scope.go:117] "RemoveContainer" containerID="3aa8a9fbe9a0ce9ea6d7c5373260ba8eb4b6721770b20253805c8b7831e05f81"
Sep 18 19:32:41.760978 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:41.760962    2532 scope.go:117] "RemoveContainer" containerID="15922ce1f626aaef8683e7833deefe2967b010b2ebe683519f8f440cb5492f41"
Sep 18 19:32:41.767711 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:41.767697    2532 scope.go:117] "RemoveContainer" containerID="d25ac0df380161ed7c3578a037d30a57fdfb84adf8364e488a997c554bd2ece0"
Sep 18 19:32:41.767956 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:32:41.767937    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d25ac0df380161ed7c3578a037d30a57fdfb84adf8364e488a997c554bd2ece0\": container with ID starting with d25ac0df380161ed7c3578a037d30a57fdfb84adf8364e488a997c554bd2ece0 not found: ID does not exist" containerID="d25ac0df380161ed7c3578a037d30a57fdfb84adf8364e488a997c554bd2ece0"
Sep 18 19:32:41.768000 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:41.767965    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d25ac0df380161ed7c3578a037d30a57fdfb84adf8364e488a997c554bd2ece0"} err="failed to get container status \"d25ac0df380161ed7c3578a037d30a57fdfb84adf8364e488a997c554bd2ece0\": rpc error: code = NotFound desc = could not find container \"d25ac0df380161ed7c3578a037d30a57fdfb84adf8364e488a997c554bd2ece0\": container with ID starting with d25ac0df380161ed7c3578a037d30a57fdfb84adf8364e488a997c554bd2ece0 not found: ID does not exist"
Sep 18 19:32:41.768000 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:41.767980    2532 scope.go:117] "RemoveContainer" containerID="3aa8a9fbe9a0ce9ea6d7c5373260ba8eb4b6721770b20253805c8b7831e05f81"
Sep 18 19:32:41.768253 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:32:41.768236    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3aa8a9fbe9a0ce9ea6d7c5373260ba8eb4b6721770b20253805c8b7831e05f81\": container with ID starting with 3aa8a9fbe9a0ce9ea6d7c5373260ba8eb4b6721770b20253805c8b7831e05f81 not found: ID does not exist" containerID="3aa8a9fbe9a0ce9ea6d7c5373260ba8eb4b6721770b20253805c8b7831e05f81"
Sep 18 19:32:41.768319 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:41.768266    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3aa8a9fbe9a0ce9ea6d7c5373260ba8eb4b6721770b20253805c8b7831e05f81"} err="failed to get container status \"3aa8a9fbe9a0ce9ea6d7c5373260ba8eb4b6721770b20253805c8b7831e05f81\": rpc error: code = NotFound desc = could not find container \"3aa8a9fbe9a0ce9ea6d7c5373260ba8eb4b6721770b20253805c8b7831e05f81\": container with ID starting with 3aa8a9fbe9a0ce9ea6d7c5373260ba8eb4b6721770b20253805c8b7831e05f81 not found: ID does not exist"
Sep 18 19:32:41.768319 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:41.768288    2532 scope.go:117] "RemoveContainer" containerID="15922ce1f626aaef8683e7833deefe2967b010b2ebe683519f8f440cb5492f41"
Sep 18 19:32:41.768555 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:32:41.768538    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15922ce1f626aaef8683e7833deefe2967b010b2ebe683519f8f440cb5492f41\": container with ID starting with 15922ce1f626aaef8683e7833deefe2967b010b2ebe683519f8f440cb5492f41 not found: ID does not exist" containerID="15922ce1f626aaef8683e7833deefe2967b010b2ebe683519f8f440cb5492f41"
Sep 18 19:32:41.768594 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:41.768560    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15922ce1f626aaef8683e7833deefe2967b010b2ebe683519f8f440cb5492f41"} err="failed to get container status \"15922ce1f626aaef8683e7833deefe2967b010b2ebe683519f8f440cb5492f41\": rpc error: code = NotFound desc = could not find container \"15922ce1f626aaef8683e7833deefe2967b010b2ebe683519f8f440cb5492f41\": container with ID starting with 15922ce1f626aaef8683e7833deefe2967b010b2ebe683519f8f440cb5492f41 not found: ID does not exist"
Sep 18 19:32:41.774541 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:41.774522    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv"]
Sep 18 19:32:41.781413 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:41.781394    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-4a391-predictor-65d4767d94-5tfcv"]
Sep 18 19:32:43.620569 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:43.620539    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd73555e-e1dd-4509-8607-b23a135b0734" path="/var/lib/kubelet/pods/bd73555e-e1dd-4509-8607-b23a135b0734/volumes"
Sep 18 19:32:48.676042 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:48.676000    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8" podUID="4da5ab65-7c8d-41b7-b723-ac3108fc43e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 18 19:32:48.676428 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:48.676000    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4" podUID="f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 18 19:32:58.676241 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:58.676206    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4" podUID="f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 18 19:32:58.676606 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:32:58.676206    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8" podUID="4da5ab65-7c8d-41b7-b723-ac3108fc43e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 18 19:33:05.524205 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:05.524174    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:33:08.676210 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:08.676176    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4" podUID="f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 18 19:33:08.676699 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:08.676176    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8" podUID="4da5ab65-7c8d-41b7-b723-ac3108fc43e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 18 19:33:18.675606 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:18.675525    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8" podUID="4da5ab65-7c8d-41b7-b723-ac3108fc43e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 18 19:33:18.675944 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:18.675525    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4" podUID="f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 18 19:33:28.675404 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:28.675366    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4" podUID="f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 18 19:33:28.675775 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:28.675615    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8" podUID="4da5ab65-7c8d-41b7-b723-ac3108fc43e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 18 19:33:36.617235 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:36.617200    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8"
Sep 18 19:33:38.676136 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:38.676108    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4"
Sep 18 19:33:40.441509 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:40.441475    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:33:46.182448 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:46.182416    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4"]
Sep 18 19:33:46.182812 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:46.182678    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4" podUID="f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207" containerName="kserve-container" containerID="cri-o://dcd704c0066a0f787c6361dab9ecb37c2a1b7d5d0e8263e55ba45f2ea415196f" gracePeriod=30
Sep 18 19:33:46.182812 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:46.182685    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4" podUID="f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207" containerName="kube-rbac-proxy" containerID="cri-o://3e563d87953aed5c400e667573d9d48a02cabc605198c0d2ce2f19b3bd2c6bf6" gracePeriod=30
Sep 18 19:33:46.201974 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:46.201948    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:33:46.231012 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:46.230988    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8"]
Sep 18 19:33:46.231286 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:46.231247    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8" podUID="4da5ab65-7c8d-41b7-b723-ac3108fc43e2" containerName="kserve-container" containerID="cri-o://f6e63ec8965c60d077fd17cbe05328f619245d8f4a52d159e527b97853ea29ae" gracePeriod=30
Sep 18 19:33:46.231367 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:46.231319    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8" podUID="4da5ab65-7c8d-41b7-b723-ac3108fc43e2" containerName="kube-rbac-proxy" containerID="cri-o://6eb035007ac764145f50cd934a31d4ec67286ab9ef36c7aca53a0159b93f3596" gracePeriod=30
Sep 18 19:33:46.278632 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:46.278613    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:33:46.617343 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:46.617263    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8" podUID="4da5ab65-7c8d-41b7-b723-ac3108fc43e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 18 19:33:46.990259 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:46.990226    2532 generic.go:356] "Generic (PLEG): container finished" podID="f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207" containerID="3e563d87953aed5c400e667573d9d48a02cabc605198c0d2ce2f19b3bd2c6bf6" exitCode=2
Sep 18 19:33:46.990414 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:46.990297    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4" event={"ID":"f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207","Type":"ContainerDied","Data":"3e563d87953aed5c400e667573d9d48a02cabc605198c0d2ce2f19b3bd2c6bf6"}
Sep 18 19:33:46.992273 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:46.992252    2532 generic.go:356] "Generic (PLEG): container finished" podID="4da5ab65-7c8d-41b7-b723-ac3108fc43e2" containerID="6eb035007ac764145f50cd934a31d4ec67286ab9ef36c7aca53a0159b93f3596" exitCode=2
Sep 18 19:33:46.992375 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:46.992281    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8" event={"ID":"4da5ab65-7c8d-41b7-b723-ac3108fc43e2","Type":"ContainerDied","Data":"6eb035007ac764145f50cd934a31d4ec67286ab9ef36c7aca53a0159b93f3596"}
Sep 18 19:33:48.670537 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:48.670497    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4" podUID="f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
Sep 18 19:33:48.670537 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:48.670512    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8" podUID="4da5ab65-7c8d-41b7-b723-ac3108fc43e2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
Sep 18 19:33:48.675867 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:48.675841    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4" podUID="f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 18 19:33:50.776793 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:50.776771    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4"
Sep 18 19:33:50.780449 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:50.780431    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8"
Sep 18 19:33:50.894310 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:50.894234    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207-kserve-provision-location\") pod \"f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207\" (UID: \"f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207\") "
Sep 18 19:33:50.894310 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:50.894275    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207-cabundle-cert\") pod \"f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207\" (UID: \"f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207\") "
Sep 18 19:33:50.894310 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:50.894295    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207-kube-rbac-proxy-sar-config\") pod \"f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207\" (UID: \"f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207\") "
Sep 18 19:33:50.894592 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:50.894323    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4da5ab65-7c8d-41b7-b723-ac3108fc43e2-proxy-tls\") pod \"4da5ab65-7c8d-41b7-b723-ac3108fc43e2\" (UID: \"4da5ab65-7c8d-41b7-b723-ac3108fc43e2\") "
Sep 18 19:33:50.894592 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:50.894348    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4da5ab65-7c8d-41b7-b723-ac3108fc43e2-kserve-provision-location\") pod \"4da5ab65-7c8d-41b7-b723-ac3108fc43e2\" (UID: \"4da5ab65-7c8d-41b7-b723-ac3108fc43e2\") "
Sep 18 19:33:50.894592 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:50.894473    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207-proxy-tls\") pod \"f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207\" (UID: \"f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207\") "
Sep 18 19:33:50.894592 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:50.894517    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4da5ab65-7c8d-41b7-b723-ac3108fc43e2-cabundle-cert\") pod \"4da5ab65-7c8d-41b7-b723-ac3108fc43e2\" (UID: \"4da5ab65-7c8d-41b7-b723-ac3108fc43e2\") "
Sep 18 19:33:50.894592 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:50.894553    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mcjk4\" (UniqueName: \"kubernetes.io/projected/f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207-kube-api-access-mcjk4\") pod \"f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207\" (UID: \"f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207\") "
Sep 18 19:33:50.894592 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:50.894580    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-67fs9\" (UniqueName: \"kubernetes.io/projected/4da5ab65-7c8d-41b7-b723-ac3108fc43e2-kube-api-access-67fs9\") pod \"4da5ab65-7c8d-41b7-b723-ac3108fc43e2\" (UID: \"4da5ab65-7c8d-41b7-b723-ac3108fc43e2\") "
Sep 18 19:33:50.894873 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:50.894607    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4da5ab65-7c8d-41b7-b723-ac3108fc43e2-kube-rbac-proxy-sar-config\") pod \"4da5ab65-7c8d-41b7-b723-ac3108fc43e2\" (UID: \"4da5ab65-7c8d-41b7-b723-ac3108fc43e2\") "
Sep 18 19:33:50.894873 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:50.894626    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4da5ab65-7c8d-41b7-b723-ac3108fc43e2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4da5ab65-7c8d-41b7-b723-ac3108fc43e2" (UID: "4da5ab65-7c8d-41b7-b723-ac3108fc43e2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:33:50.894873 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:50.894639    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207" (UID: "f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:33:50.894873 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:50.894671    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207" (UID: "f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:33:50.895086 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:50.894901    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4da5ab65-7c8d-41b7-b723-ac3108fc43e2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4da5ab65-7c8d-41b7-b723-ac3108fc43e2" (UID: "4da5ab65-7c8d-41b7-b723-ac3108fc43e2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:33:50.895086 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:50.894970    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4da5ab65-7c8d-41b7-b723-ac3108fc43e2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4da5ab65-7c8d-41b7-b723-ac3108fc43e2" (UID: "4da5ab65-7c8d-41b7-b723-ac3108fc43e2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:33:50.895086 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:50.895014    2532 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207-kserve-provision-location\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:33:50.895086 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:50.895037    2532 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:33:50.895086 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:50.895052    2532 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4da5ab65-7c8d-41b7-b723-ac3108fc43e2-kserve-provision-location\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:33:50.895261 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:50.895092    2532 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4da5ab65-7c8d-41b7-b723-ac3108fc43e2-cabundle-cert\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:33:50.895261 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:50.895146    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207" (UID: "f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:33:50.896649 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:50.896631    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207-kube-api-access-mcjk4" (OuterVolumeSpecName: "kube-api-access-mcjk4") pod "f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207" (UID: "f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207"). InnerVolumeSpecName "kube-api-access-mcjk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:33:50.896706 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:50.896653    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207" (UID: "f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:33:50.896882 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:50.896862    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4da5ab65-7c8d-41b7-b723-ac3108fc43e2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4da5ab65-7c8d-41b7-b723-ac3108fc43e2" (UID: "4da5ab65-7c8d-41b7-b723-ac3108fc43e2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:33:50.896933 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:50.896879    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4da5ab65-7c8d-41b7-b723-ac3108fc43e2-kube-api-access-67fs9" (OuterVolumeSpecName: "kube-api-access-67fs9") pod "4da5ab65-7c8d-41b7-b723-ac3108fc43e2" (UID: "4da5ab65-7c8d-41b7-b723-ac3108fc43e2"). InnerVolumeSpecName "kube-api-access-67fs9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:33:50.995667 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:50.995639    2532 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207-cabundle-cert\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:33:50.995667 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:50.995665    2532 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4da5ab65-7c8d-41b7-b723-ac3108fc43e2-proxy-tls\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:33:50.995667 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:50.995673    2532 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207-proxy-tls\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:33:50.995884 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:50.995682    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcjk4\" (UniqueName: \"kubernetes.io/projected/f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207-kube-api-access-mcjk4\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:33:50.995884 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:50.995692    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67fs9\" (UniqueName: \"kubernetes.io/projected/4da5ab65-7c8d-41b7-b723-ac3108fc43e2-kube-api-access-67fs9\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:33:50.995884 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:50.995701    2532 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4da5ab65-7c8d-41b7-b723-ac3108fc43e2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:33:51.008609 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:51.008579    2532 generic.go:356] "Generic (PLEG): container finished" podID="4da5ab65-7c8d-41b7-b723-ac3108fc43e2" containerID="f6e63ec8965c60d077fd17cbe05328f619245d8f4a52d159e527b97853ea29ae" exitCode=0
Sep 18 19:33:51.008741 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:51.008637    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8"
Sep 18 19:33:51.008741 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:51.008668    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8" event={"ID":"4da5ab65-7c8d-41b7-b723-ac3108fc43e2","Type":"ContainerDied","Data":"f6e63ec8965c60d077fd17cbe05328f619245d8f4a52d159e527b97853ea29ae"}
Sep 18 19:33:51.008826 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:51.008744    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8" event={"ID":"4da5ab65-7c8d-41b7-b723-ac3108fc43e2","Type":"ContainerDied","Data":"e18edc40305188ff1aa03ccea2952e082479c078c2159c1b51ceacc3765becf4"}
Sep 18 19:33:51.008826 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:51.008789    2532 scope.go:117] "RemoveContainer" containerID="6eb035007ac764145f50cd934a31d4ec67286ab9ef36c7aca53a0159b93f3596"
Sep 18 19:33:51.010730 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:51.010704    2532 generic.go:356] "Generic (PLEG): container finished" podID="f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207" containerID="dcd704c0066a0f787c6361dab9ecb37c2a1b7d5d0e8263e55ba45f2ea415196f" exitCode=0
Sep 18 19:33:51.010824 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:51.010785    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4"
Sep 18 19:33:51.010824 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:51.010787    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4" event={"ID":"f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207","Type":"ContainerDied","Data":"dcd704c0066a0f787c6361dab9ecb37c2a1b7d5d0e8263e55ba45f2ea415196f"}
Sep 18 19:33:51.010911 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:51.010833    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4" event={"ID":"f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207","Type":"ContainerDied","Data":"aa24c74db10de4eb560c879d3628e31d554c3550c2c630efa7a85f7a0d35a719"}
Sep 18 19:33:51.018046 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:51.018026    2532 scope.go:117] "RemoveContainer" containerID="f6e63ec8965c60d077fd17cbe05328f619245d8f4a52d159e527b97853ea29ae"
Sep 18 19:33:51.025438 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:51.025420    2532 scope.go:117] "RemoveContainer" containerID="f7886435c039d0e397221152b038db52dceb0c7d2ae0cd27e9c695bde577c4e4"
Sep 18 19:33:51.031882 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:51.031866    2532 scope.go:117] "RemoveContainer" containerID="6eb035007ac764145f50cd934a31d4ec67286ab9ef36c7aca53a0159b93f3596"
Sep 18 19:33:51.032132 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:33:51.032112    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6eb035007ac764145f50cd934a31d4ec67286ab9ef36c7aca53a0159b93f3596\": container with ID starting with 6eb035007ac764145f50cd934a31d4ec67286ab9ef36c7aca53a0159b93f3596 not found: ID does not exist" containerID="6eb035007ac764145f50cd934a31d4ec67286ab9ef36c7aca53a0159b93f3596"
Sep 18 19:33:51.032178 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:51.032145    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6eb035007ac764145f50cd934a31d4ec67286ab9ef36c7aca53a0159b93f3596"} err="failed to get container status \"6eb035007ac764145f50cd934a31d4ec67286ab9ef36c7aca53a0159b93f3596\": rpc error: code = NotFound desc = could not find container \"6eb035007ac764145f50cd934a31d4ec67286ab9ef36c7aca53a0159b93f3596\": container with ID starting with 6eb035007ac764145f50cd934a31d4ec67286ab9ef36c7aca53a0159b93f3596 not found: ID does not exist"
Sep 18 19:33:51.032178 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:51.032163    2532 scope.go:117] "RemoveContainer" containerID="f6e63ec8965c60d077fd17cbe05328f619245d8f4a52d159e527b97853ea29ae"
Sep 18 19:33:51.032389 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:33:51.032372    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6e63ec8965c60d077fd17cbe05328f619245d8f4a52d159e527b97853ea29ae\": container with ID starting with f6e63ec8965c60d077fd17cbe05328f619245d8f4a52d159e527b97853ea29ae not found: ID does not exist" containerID="f6e63ec8965c60d077fd17cbe05328f619245d8f4a52d159e527b97853ea29ae"
Sep 18 19:33:51.032454 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:51.032398    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6e63ec8965c60d077fd17cbe05328f619245d8f4a52d159e527b97853ea29ae"} err="failed to get container status \"f6e63ec8965c60d077fd17cbe05328f619245d8f4a52d159e527b97853ea29ae\": rpc error: code = NotFound desc = could not find container \"f6e63ec8965c60d077fd17cbe05328f619245d8f4a52d159e527b97853ea29ae\": container with ID starting with f6e63ec8965c60d077fd17cbe05328f619245d8f4a52d159e527b97853ea29ae not found: ID does not exist"
Sep 18 19:33:51.032454 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:51.032421    2532 scope.go:117] "RemoveContainer" containerID="f7886435c039d0e397221152b038db52dceb0c7d2ae0cd27e9c695bde577c4e4"
Sep 18 19:33:51.032652 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:33:51.032636    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7886435c039d0e397221152b038db52dceb0c7d2ae0cd27e9c695bde577c4e4\": container with ID starting with f7886435c039d0e397221152b038db52dceb0c7d2ae0cd27e9c695bde577c4e4 not found: ID does not exist" containerID="f7886435c039d0e397221152b038db52dceb0c7d2ae0cd27e9c695bde577c4e4"
Sep 18 19:33:51.032690 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:51.032656    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7886435c039d0e397221152b038db52dceb0c7d2ae0cd27e9c695bde577c4e4"} err="failed to get container status \"f7886435c039d0e397221152b038db52dceb0c7d2ae0cd27e9c695bde577c4e4\": rpc error: code = NotFound desc = could not find container \"f7886435c039d0e397221152b038db52dceb0c7d2ae0cd27e9c695bde577c4e4\": container with ID starting with f7886435c039d0e397221152b038db52dceb0c7d2ae0cd27e9c695bde577c4e4 not found: ID does not exist"
Sep 18 19:33:51.032690 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:51.032668    2532 scope.go:117] "RemoveContainer" containerID="3e563d87953aed5c400e667573d9d48a02cabc605198c0d2ce2f19b3bd2c6bf6"
Sep 18 19:33:51.038992 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:51.038972    2532 scope.go:117] "RemoveContainer" containerID="dcd704c0066a0f787c6361dab9ecb37c2a1b7d5d0e8263e55ba45f2ea415196f"
Sep 18 19:33:51.043243 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:51.043221    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8"]
Sep 18 19:33:51.045558 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:51.045543    2532 scope.go:117] "RemoveContainer" containerID="a609864be33d7b4e0caa7a775273e5c5bdc2afdcc6478f2dec3506c54ed139e7"
Sep 18 19:33:51.048619 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:51.048601    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-2r8d8"]
Sep 18 19:33:51.052532 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:51.052517    2532 scope.go:117] "RemoveContainer" containerID="3e563d87953aed5c400e667573d9d48a02cabc605198c0d2ce2f19b3bd2c6bf6"
Sep 18 19:33:51.052752 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:33:51.052734    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e563d87953aed5c400e667573d9d48a02cabc605198c0d2ce2f19b3bd2c6bf6\": container with ID starting with 3e563d87953aed5c400e667573d9d48a02cabc605198c0d2ce2f19b3bd2c6bf6 not found: ID does not exist" containerID="3e563d87953aed5c400e667573d9d48a02cabc605198c0d2ce2f19b3bd2c6bf6"
Sep 18 19:33:51.052797 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:51.052757    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e563d87953aed5c400e667573d9d48a02cabc605198c0d2ce2f19b3bd2c6bf6"} err="failed to get container status \"3e563d87953aed5c400e667573d9d48a02cabc605198c0d2ce2f19b3bd2c6bf6\": rpc error: code = NotFound desc = could not find container \"3e563d87953aed5c400e667573d9d48a02cabc605198c0d2ce2f19b3bd2c6bf6\": container with ID starting with 3e563d87953aed5c400e667573d9d48a02cabc605198c0d2ce2f19b3bd2c6bf6 not found: ID does not exist"
Sep 18 19:33:51.052797 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:51.052772    2532 scope.go:117] "RemoveContainer" containerID="dcd704c0066a0f787c6361dab9ecb37c2a1b7d5d0e8263e55ba45f2ea415196f"
Sep 18 19:33:51.053009 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:33:51.052991    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcd704c0066a0f787c6361dab9ecb37c2a1b7d5d0e8263e55ba45f2ea415196f\": container with ID starting with dcd704c0066a0f787c6361dab9ecb37c2a1b7d5d0e8263e55ba45f2ea415196f not found: ID does not exist" containerID="dcd704c0066a0f787c6361dab9ecb37c2a1b7d5d0e8263e55ba45f2ea415196f"
Sep 18 19:33:51.053047 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:51.053014    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcd704c0066a0f787c6361dab9ecb37c2a1b7d5d0e8263e55ba45f2ea415196f"} err="failed to get container status \"dcd704c0066a0f787c6361dab9ecb37c2a1b7d5d0e8263e55ba45f2ea415196f\": rpc error: code = NotFound desc = could not find container \"dcd704c0066a0f787c6361dab9ecb37c2a1b7d5d0e8263e55ba45f2ea415196f\": container with ID starting with dcd704c0066a0f787c6361dab9ecb37c2a1b7d5d0e8263e55ba45f2ea415196f not found: ID does not exist"
Sep 18 19:33:51.053047 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:51.053027    2532 scope.go:117] "RemoveContainer" containerID="a609864be33d7b4e0caa7a775273e5c5bdc2afdcc6478f2dec3506c54ed139e7"
Sep 18 19:33:51.053278 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:33:51.053259    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a609864be33d7b4e0caa7a775273e5c5bdc2afdcc6478f2dec3506c54ed139e7\": container with ID starting with a609864be33d7b4e0caa7a775273e5c5bdc2afdcc6478f2dec3506c54ed139e7 not found: ID does not exist" containerID="a609864be33d7b4e0caa7a775273e5c5bdc2afdcc6478f2dec3506c54ed139e7"
Sep 18 19:33:51.053319 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:51.053284    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a609864be33d7b4e0caa7a775273e5c5bdc2afdcc6478f2dec3506c54ed139e7"} err="failed to get container status \"a609864be33d7b4e0caa7a775273e5c5bdc2afdcc6478f2dec3506c54ed139e7\": rpc error: code = NotFound desc = could not find container \"a609864be33d7b4e0caa7a775273e5c5bdc2afdcc6478f2dec3506c54ed139e7\": container with ID starting with a609864be33d7b4e0caa7a775273e5c5bdc2afdcc6478f2dec3506c54ed139e7 not found: ID does not exist"
Sep 18 19:33:51.064092 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:51.064070    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4"]
Sep 18 19:33:51.068080 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:51.068047    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-8lmm4"]
Sep 18 19:33:51.620726 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:51.620698    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4da5ab65-7c8d-41b7-b723-ac3108fc43e2" path="/var/lib/kubelet/pods/4da5ab65-7c8d-41b7-b723-ac3108fc43e2/volumes"
Sep 18 19:33:51.621177 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:51.621163    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207" path="/var/lib/kubelet/pods/f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207/volumes"
Sep 18 19:33:53.561521 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:53.561493    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sksw8_2cef0f4e-e44f-4996-8561-da9d261036c7/ovn-acl-logging/0.log"
Sep 18 19:33:53.563508 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:53.563487    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sksw8_2cef0f4e-e44f-4996-8561-da9d261036c7/ovn-acl-logging/0.log"
Sep 18 19:33:56.451393 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:56.451354    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:33:56.484345 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:56.484319    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-2a1456-predictor-5b6757476c-rddt5"]
Sep 18 19:33:56.484556 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:56.484544    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd73555e-e1dd-4509-8607-b23a135b0734" containerName="kserve-container"
Sep 18 19:33:56.484604 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:56.484558    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd73555e-e1dd-4509-8607-b23a135b0734" containerName="kserve-container"
Sep 18 19:33:56.484604 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:56.484568    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207" containerName="kube-rbac-proxy"
Sep 18 19:33:56.484604 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:56.484574    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207" containerName="kube-rbac-proxy"
Sep 18 19:33:56.484604 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:56.484582    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4da5ab65-7c8d-41b7-b723-ac3108fc43e2" containerName="kube-rbac-proxy"
Sep 18 19:33:56.484604 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:56.484587    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="4da5ab65-7c8d-41b7-b723-ac3108fc43e2" containerName="kube-rbac-proxy"
Sep 18 19:33:56.484604 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:56.484599    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd73555e-e1dd-4509-8607-b23a135b0734" containerName="storage-initializer"
Sep 18 19:33:56.484604 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:56.484604    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd73555e-e1dd-4509-8607-b23a135b0734" containerName="storage-initializer"
Sep 18 19:33:56.484826 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:56.484612    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207" containerName="kserve-container"
Sep 18 19:33:56.484826 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:56.484617    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207" containerName="kserve-container"
Sep 18 19:33:56.484826 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:56.484624    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4da5ab65-7c8d-41b7-b723-ac3108fc43e2" containerName="kserve-container"
Sep 18 19:33:56.484826 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:56.484630    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="4da5ab65-7c8d-41b7-b723-ac3108fc43e2" containerName="kserve-container"
Sep 18 19:33:56.484826 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:56.484637    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd73555e-e1dd-4509-8607-b23a135b0734" containerName="kube-rbac-proxy"
Sep 18 19:33:56.484826 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:56.484642    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd73555e-e1dd-4509-8607-b23a135b0734" containerName="kube-rbac-proxy"
Sep 18 19:33:56.484826 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:56.484647    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207" containerName="storage-initializer"
Sep 18 19:33:56.484826 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:56.484652    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207" containerName="storage-initializer"
Sep 18 19:33:56.484826 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:56.484659    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4da5ab65-7c8d-41b7-b723-ac3108fc43e2" containerName="storage-initializer"
Sep 18 19:33:56.484826 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:56.484664    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="4da5ab65-7c8d-41b7-b723-ac3108fc43e2" containerName="storage-initializer"
Sep 18 19:33:56.484826 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:56.484704    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207" containerName="kserve-container"
Sep 18 19:33:56.484826 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:56.484712    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd73555e-e1dd-4509-8607-b23a135b0734" containerName="kube-rbac-proxy"
Sep 18 19:33:56.484826 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:56.484717    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="4da5ab65-7c8d-41b7-b723-ac3108fc43e2" containerName="kube-rbac-proxy"
Sep 18 19:33:56.484826 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:56.484722    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd73555e-e1dd-4509-8607-b23a135b0734" containerName="kserve-container"
Sep 18 19:33:56.484826 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:56.484727    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4c80a9f-e8b4-4df5-ba16-c84bcdf0e207" containerName="kube-rbac-proxy"
Sep 18 19:33:56.484826 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:56.484734    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="4da5ab65-7c8d-41b7-b723-ac3108fc43e2" containerName="kserve-container"
Sep 18 19:33:56.489096 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:56.489081    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-2a1456-predictor-5b6757476c-rddt5"
Sep 18 19:33:56.491564 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:56.491527    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-2a1456-predictor-serving-cert\""
Sep 18 19:33:56.491658 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:56.491618    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 19:33:56.491658 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:56.491618    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 19:33:56.491758 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:56.491665    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 19:33:56.491841 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:56.491779    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-2a1456-kube-rbac-proxy-sar-config\""
Sep 18 19:33:56.492241 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:56.492227    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 19:33:56.492415 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:56.492403    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-ct9wb\""
Sep 18 19:33:56.500073 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:56.500031    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-2a1456-predictor-5b6757476c-rddt5"]
Sep 18 19:33:56.621204 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:56.621175    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:33:56.634276 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:56.634254    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/397350de-2ff0-4a9c-8d78-67b1199faea9-cabundle-cert\") pod \"isvc-primary-2a1456-predictor-5b6757476c-rddt5\" (UID: \"397350de-2ff0-4a9c-8d78-67b1199faea9\") " pod="kserve-ci-e2e-test/isvc-primary-2a1456-predictor-5b6757476c-rddt5"
Sep 18 19:33:56.634363 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:56.634283    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/397350de-2ff0-4a9c-8d78-67b1199faea9-kserve-provision-location\") pod \"isvc-primary-2a1456-predictor-5b6757476c-rddt5\" (UID: \"397350de-2ff0-4a9c-8d78-67b1199faea9\") " pod="kserve-ci-e2e-test/isvc-primary-2a1456-predictor-5b6757476c-rddt5"
Sep 18 19:33:56.634363 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:56.634308    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/397350de-2ff0-4a9c-8d78-67b1199faea9-kube-rbac-proxy-sar-config\") pod \"isvc-primary-2a1456-predictor-5b6757476c-rddt5\" (UID: \"397350de-2ff0-4a9c-8d78-67b1199faea9\") " pod="kserve-ci-e2e-test/isvc-primary-2a1456-predictor-5b6757476c-rddt5"
Sep 18 19:33:56.634457 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:56.634367    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/397350de-2ff0-4a9c-8d78-67b1199faea9-proxy-tls\") pod \"isvc-primary-2a1456-predictor-5b6757476c-rddt5\" (UID: \"397350de-2ff0-4a9c-8d78-67b1199faea9\") " pod="kserve-ci-e2e-test/isvc-primary-2a1456-predictor-5b6757476c-rddt5"
Sep 18 19:33:56.634496 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:56.634467    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xnpr\" (UniqueName: \"kubernetes.io/projected/397350de-2ff0-4a9c-8d78-67b1199faea9-kube-api-access-4xnpr\") pod \"isvc-primary-2a1456-predictor-5b6757476c-rddt5\" (UID: \"397350de-2ff0-4a9c-8d78-67b1199faea9\") " pod="kserve-ci-e2e-test/isvc-primary-2a1456-predictor-5b6757476c-rddt5"
Sep 18 19:33:56.735077 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:56.735013    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xnpr\" (UniqueName: \"kubernetes.io/projected/397350de-2ff0-4a9c-8d78-67b1199faea9-kube-api-access-4xnpr\") pod \"isvc-primary-2a1456-predictor-5b6757476c-rddt5\" (UID: \"397350de-2ff0-4a9c-8d78-67b1199faea9\") " pod="kserve-ci-e2e-test/isvc-primary-2a1456-predictor-5b6757476c-rddt5"
Sep 18 19:33:56.735077 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:56.735046    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/397350de-2ff0-4a9c-8d78-67b1199faea9-cabundle-cert\") pod \"isvc-primary-2a1456-predictor-5b6757476c-rddt5\" (UID: \"397350de-2ff0-4a9c-8d78-67b1199faea9\") " pod="kserve-ci-e2e-test/isvc-primary-2a1456-predictor-5b6757476c-rddt5"
Sep 18 19:33:56.735223 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:56.735081    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/397350de-2ff0-4a9c-8d78-67b1199faea9-kserve-provision-location\") pod \"isvc-primary-2a1456-predictor-5b6757476c-rddt5\" (UID: \"397350de-2ff0-4a9c-8d78-67b1199faea9\") " pod="kserve-ci-e2e-test/isvc-primary-2a1456-predictor-5b6757476c-rddt5"
Sep 18 19:33:56.735223 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:56.735179    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/397350de-2ff0-4a9c-8d78-67b1199faea9-kube-rbac-proxy-sar-config\") pod \"isvc-primary-2a1456-predictor-5b6757476c-rddt5\" (UID: \"397350de-2ff0-4a9c-8d78-67b1199faea9\") " pod="kserve-ci-e2e-test/isvc-primary-2a1456-predictor-5b6757476c-rddt5"
Sep 18 19:33:56.735223 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:56.735212    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/397350de-2ff0-4a9c-8d78-67b1199faea9-proxy-tls\") pod \"isvc-primary-2a1456-predictor-5b6757476c-rddt5\" (UID: \"397350de-2ff0-4a9c-8d78-67b1199faea9\") " pod="kserve-ci-e2e-test/isvc-primary-2a1456-predictor-5b6757476c-rddt5"
Sep 18 19:33:56.735395 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:56.735378    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/397350de-2ff0-4a9c-8d78-67b1199faea9-kserve-provision-location\") pod \"isvc-primary-2a1456-predictor-5b6757476c-rddt5\" (UID: \"397350de-2ff0-4a9c-8d78-67b1199faea9\") " pod="kserve-ci-e2e-test/isvc-primary-2a1456-predictor-5b6757476c-rddt5"
Sep 18 19:33:56.735735 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:56.735714    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/397350de-2ff0-4a9c-8d78-67b1199faea9-cabundle-cert\") pod \"isvc-primary-2a1456-predictor-5b6757476c-rddt5\" (UID: \"397350de-2ff0-4a9c-8d78-67b1199faea9\") " pod="kserve-ci-e2e-test/isvc-primary-2a1456-predictor-5b6757476c-rddt5"
Sep 18 19:33:56.735866 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:56.735849    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/397350de-2ff0-4a9c-8d78-67b1199faea9-kube-rbac-proxy-sar-config\") pod \"isvc-primary-2a1456-predictor-5b6757476c-rddt5\" (UID: \"397350de-2ff0-4a9c-8d78-67b1199faea9\") " pod="kserve-ci-e2e-test/isvc-primary-2a1456-predictor-5b6757476c-rddt5"
Sep 18 19:33:56.737593 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:56.737577    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/397350de-2ff0-4a9c-8d78-67b1199faea9-proxy-tls\") pod \"isvc-primary-2a1456-predictor-5b6757476c-rddt5\" (UID: \"397350de-2ff0-4a9c-8d78-67b1199faea9\") " pod="kserve-ci-e2e-test/isvc-primary-2a1456-predictor-5b6757476c-rddt5"
Sep 18 19:33:56.745154 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:56.745131    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xnpr\" (UniqueName: \"kubernetes.io/projected/397350de-2ff0-4a9c-8d78-67b1199faea9-kube-api-access-4xnpr\") pod \"isvc-primary-2a1456-predictor-5b6757476c-rddt5\" (UID: \"397350de-2ff0-4a9c-8d78-67b1199faea9\") " pod="kserve-ci-e2e-test/isvc-primary-2a1456-predictor-5b6757476c-rddt5"
Sep 18 19:33:56.799146 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:56.799122    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-2a1456-predictor-5b6757476c-rddt5"
Sep 18 19:33:57.073906 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:57.073837    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-2a1456-predictor-5b6757476c-rddt5"]
Sep 18 19:33:57.076878 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:33:57.076850    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod397350de_2ff0_4a9c_8d78_67b1199faea9.slice/crio-a589826af46128c0b60c43cb0952e03f19bb318b1b2e0d41e297a813e9aba7c7 WatchSource:0}: Error finding container a589826af46128c0b60c43cb0952e03f19bb318b1b2e0d41e297a813e9aba7c7: Status 404 returned error can't find the container with id a589826af46128c0b60c43cb0952e03f19bb318b1b2e0d41e297a813e9aba7c7
Sep 18 19:33:58.042794 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:58.042755    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-2a1456-predictor-5b6757476c-rddt5" event={"ID":"397350de-2ff0-4a9c-8d78-67b1199faea9","Type":"ContainerStarted","Data":"a79895ce513da1d746eaf4d16cf09803e5b5114baf4c0784a8cc4f8ed19accfd"}
Sep 18 19:33:58.043183 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:33:58.042822    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-2a1456-predictor-5b6757476c-rddt5" event={"ID":"397350de-2ff0-4a9c-8d78-67b1199faea9","Type":"ContainerStarted","Data":"a589826af46128c0b60c43cb0952e03f19bb318b1b2e0d41e297a813e9aba7c7"}
Sep 18 19:34:01.055373 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:01.055339    2532 generic.go:356] "Generic (PLEG): container finished" podID="397350de-2ff0-4a9c-8d78-67b1199faea9" containerID="a79895ce513da1d746eaf4d16cf09803e5b5114baf4c0784a8cc4f8ed19accfd" exitCode=0
Sep 18 19:34:01.055700 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:01.055392    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-2a1456-predictor-5b6757476c-rddt5" event={"ID":"397350de-2ff0-4a9c-8d78-67b1199faea9","Type":"ContainerDied","Data":"a79895ce513da1d746eaf4d16cf09803e5b5114baf4c0784a8cc4f8ed19accfd"}
Sep 18 19:34:02.060954 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:02.060917    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-2a1456-predictor-5b6757476c-rddt5" event={"ID":"397350de-2ff0-4a9c-8d78-67b1199faea9","Type":"ContainerStarted","Data":"8c6886b6c5c0a5f8f90a8b23cfe688bf8954d0abb607aed9b8b6d71fea6bb855"}
Sep 18 19:34:02.061359 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:02.060982    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-2a1456-predictor-5b6757476c-rddt5" event={"ID":"397350de-2ff0-4a9c-8d78-67b1199faea9","Type":"ContainerStarted","Data":"e70d6746db73d2c531e73f389eab061cbaefd29fec37bcad1ff3a4f847f7b2ac"}
Sep 18 19:34:02.061359 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:02.061313    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-2a1456-predictor-5b6757476c-rddt5"
Sep 18 19:34:02.061359 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:02.061361    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-2a1456-predictor-5b6757476c-rddt5"
Sep 18 19:34:02.062787 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:02.062761    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-2a1456-predictor-5b6757476c-rddt5" podUID="397350de-2ff0-4a9c-8d78-67b1199faea9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 18 19:34:03.064847 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:03.064814    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-2a1456-predictor-5b6757476c-rddt5" podUID="397350de-2ff0-4a9c-8d78-67b1199faea9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 18 19:34:08.069749 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:08.069723    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-2a1456-predictor-5b6757476c-rddt5"
Sep 18 19:34:08.070213 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:08.070186    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-2a1456-predictor-5b6757476c-rddt5" podUID="397350de-2ff0-4a9c-8d78-67b1199faea9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 18 19:34:08.113544 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:08.113503    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-2a1456-predictor-5b6757476c-rddt5" podStartSLOduration=8.25404858 podStartE2EDuration="12.113489738s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.859441158s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:33:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:34:02.089313193 +0000 UTC m=+1209.009817063" watchObservedRunningTime="2026-09-18 19:34:08.113489738 +0000 UTC m=+1215.033993595"
Sep 18 19:34:14.219283 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:14.219252    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:34:16.370655 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:16.370624    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:34:16.617323 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:16.617291    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:34:18.070941 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:18.070906    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-2a1456-predictor-5b6757476c-rddt5" podUID="397350de-2ff0-4a9c-8d78-67b1199faea9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 18 19:34:26.438212 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:26.438179    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:34:26.597200 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:26.597165    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:34:26.708910 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:26.708831    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:34:26.834165 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:26.834124    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:34:26.835604 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:26.835570    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-b9657-predictor-8bd5d747d-9pl8t"]
Sep 18 19:34:26.838729 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:26.838712    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-b9657-predictor-8bd5d747d-9pl8t"
Sep 18 19:34:26.842699 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:26.842684    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-b9657-predictor-serving-cert\""
Sep 18 19:34:26.843378 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:26.843362    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-b9657-kube-rbac-proxy-sar-config\""
Sep 18 19:34:26.863531 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:26.863511    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-b9657-predictor-8bd5d747d-9pl8t"]
Sep 18 19:34:26.915589 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:26.915555    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:34:26.959556 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:26.959492    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f63bf0ea-f130-4d38-aa1d-7ae6aa48858d-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-b9657-predictor-8bd5d747d-9pl8t\" (UID: \"f63bf0ea-f130-4d38-aa1d-7ae6aa48858d\") " pod="kserve-ci-e2e-test/message-dumper-raw-b9657-predictor-8bd5d747d-9pl8t"
Sep 18 19:34:26.959556 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:26.959521    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cdgv\" (UniqueName: \"kubernetes.io/projected/f63bf0ea-f130-4d38-aa1d-7ae6aa48858d-kube-api-access-2cdgv\") pod \"message-dumper-raw-b9657-predictor-8bd5d747d-9pl8t\" (UID: \"f63bf0ea-f130-4d38-aa1d-7ae6aa48858d\") " pod="kserve-ci-e2e-test/message-dumper-raw-b9657-predictor-8bd5d747d-9pl8t"
Sep 18 19:34:26.959556 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:26.959544    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f63bf0ea-f130-4d38-aa1d-7ae6aa48858d-proxy-tls\") pod \"message-dumper-raw-b9657-predictor-8bd5d747d-9pl8t\" (UID: \"f63bf0ea-f130-4d38-aa1d-7ae6aa48858d\") " pod="kserve-ci-e2e-test/message-dumper-raw-b9657-predictor-8bd5d747d-9pl8t"
Sep 18 19:34:27.060895 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:27.060861    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cdgv\" (UniqueName: \"kubernetes.io/projected/f63bf0ea-f130-4d38-aa1d-7ae6aa48858d-kube-api-access-2cdgv\") pod \"message-dumper-raw-b9657-predictor-8bd5d747d-9pl8t\" (UID: \"f63bf0ea-f130-4d38-aa1d-7ae6aa48858d\") " pod="kserve-ci-e2e-test/message-dumper-raw-b9657-predictor-8bd5d747d-9pl8t"
Sep 18 19:34:27.061032 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:27.060905    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f63bf0ea-f130-4d38-aa1d-7ae6aa48858d-proxy-tls\") pod \"message-dumper-raw-b9657-predictor-8bd5d747d-9pl8t\" (UID: \"f63bf0ea-f130-4d38-aa1d-7ae6aa48858d\") " pod="kserve-ci-e2e-test/message-dumper-raw-b9657-predictor-8bd5d747d-9pl8t"
Sep 18 19:34:27.061032 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:27.060989    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f63bf0ea-f130-4d38-aa1d-7ae6aa48858d-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-b9657-predictor-8bd5d747d-9pl8t\" (UID: \"f63bf0ea-f130-4d38-aa1d-7ae6aa48858d\") " pod="kserve-ci-e2e-test/message-dumper-raw-b9657-predictor-8bd5d747d-9pl8t"
Sep 18 19:34:27.061622 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:27.061599    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f63bf0ea-f130-4d38-aa1d-7ae6aa48858d-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-b9657-predictor-8bd5d747d-9pl8t\" (UID: \"f63bf0ea-f130-4d38-aa1d-7ae6aa48858d\") " pod="kserve-ci-e2e-test/message-dumper-raw-b9657-predictor-8bd5d747d-9pl8t"
Sep 18 19:34:27.063263 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:27.063247    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f63bf0ea-f130-4d38-aa1d-7ae6aa48858d-proxy-tls\") pod \"message-dumper-raw-b9657-predictor-8bd5d747d-9pl8t\" (UID: \"f63bf0ea-f130-4d38-aa1d-7ae6aa48858d\") " pod="kserve-ci-e2e-test/message-dumper-raw-b9657-predictor-8bd5d747d-9pl8t"
Sep 18 19:34:27.075611 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:27.075593    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cdgv\" (UniqueName: \"kubernetes.io/projected/f63bf0ea-f130-4d38-aa1d-7ae6aa48858d-kube-api-access-2cdgv\") pod \"message-dumper-raw-b9657-predictor-8bd5d747d-9pl8t\" (UID: \"f63bf0ea-f130-4d38-aa1d-7ae6aa48858d\") " pod="kserve-ci-e2e-test/message-dumper-raw-b9657-predictor-8bd5d747d-9pl8t"
Sep 18 19:34:27.148482 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:27.148457    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-b9657-predictor-8bd5d747d-9pl8t"
Sep 18 19:34:27.428733 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:27.428711    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-b9657-predictor-8bd5d747d-9pl8t"]
Sep 18 19:34:27.430498 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:34:27.430473    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf63bf0ea_f130_4d38_aa1d_7ae6aa48858d.slice/crio-b541df774b9bd9ae840cd2f4e927c06c63affe715a5fc3175dd9ce48cc3f2b97 WatchSource:0}: Error finding container b541df774b9bd9ae840cd2f4e927c06c63affe715a5fc3175dd9ce48cc3f2b97: Status 404 returned error can't find the container with id b541df774b9bd9ae840cd2f4e927c06c63affe715a5fc3175dd9ce48cc3f2b97
Sep 18 19:34:27.432916 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:27.432897    2532 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:34:28.070471 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:28.070429    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-2a1456-predictor-5b6757476c-rddt5" podUID="397350de-2ff0-4a9c-8d78-67b1199faea9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 18 19:34:28.156214 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:28.156183    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-b9657-predictor-8bd5d747d-9pl8t" event={"ID":"f63bf0ea-f130-4d38-aa1d-7ae6aa48858d","Type":"ContainerStarted","Data":"b541df774b9bd9ae840cd2f4e927c06c63affe715a5fc3175dd9ce48cc3f2b97"}
Sep 18 19:34:29.161129 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:29.161097    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-b9657-predictor-8bd5d747d-9pl8t" event={"ID":"f63bf0ea-f130-4d38-aa1d-7ae6aa48858d","Type":"ContainerStarted","Data":"c30ba5b1bde034f226361d0110cc4129d234d81c4cb0982f73e69dcd0301ac6e"}
Sep 18 19:34:29.161498 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:29.161158    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-b9657-predictor-8bd5d747d-9pl8t" event={"ID":"f63bf0ea-f130-4d38-aa1d-7ae6aa48858d","Type":"ContainerStarted","Data":"55b8a85f7900d3f973c33b08dcab23059342fa811729743498ff6cea6d068fa4"}
Sep 18 19:34:29.161498 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:29.161298    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-b9657-predictor-8bd5d747d-9pl8t"
Sep 18 19:34:29.161498 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:29.161433    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-b9657-predictor-8bd5d747d-9pl8t"
Sep 18 19:34:29.163082 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:29.163051    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-b9657-predictor-8bd5d747d-9pl8t"
Sep 18 19:34:29.203239 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:29.203200    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-b9657-predictor-8bd5d747d-9pl8t" podStartSLOduration=2.052568136 podStartE2EDuration="3.203186637s" totalImagesPullingTime="1.150618501s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:34:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:34:29.202440254 +0000 UTC m=+1236.122944224" watchObservedRunningTime="2026-09-18 19:34:29.203186637 +0000 UTC m=+1236.123690494"
Sep 18 19:34:36.171781 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:36.171752    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-b9657-predictor-8bd5d747d-9pl8t"
Sep 18 19:34:36.749840 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:36.749808    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:34:36.804405 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:36.804376    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784"]
Sep 18 19:34:36.807889 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:36.807873    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784"
Sep 18 19:34:36.810806 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:36.810776    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-b9657-kube-rbac-proxy-sar-config\""
Sep 18 19:34:36.810806 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:36.810775    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-b9657-predictor-serving-cert\""
Sep 18 19:34:36.821392 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:36.821369    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784"]
Sep 18 19:34:36.831536 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:36.831509    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09665ef2-2574-4057-b347-b5884c1f9664-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-b9657-predictor-67b57f7854-8l784\" (UID: \"09665ef2-2574-4057-b347-b5884c1f9664\") " pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784"
Sep 18 19:34:36.831628 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:36.831548    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09665ef2-2574-4057-b347-b5884c1f9664-cabundle-cert\") pod \"isvc-logger-raw-b9657-predictor-67b57f7854-8l784\" (UID: \"09665ef2-2574-4057-b347-b5884c1f9664\") " pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784"
Sep 18 19:34:36.831628 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:36.831587    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09665ef2-2574-4057-b347-b5884c1f9664-kserve-provision-location\") pod \"isvc-logger-raw-b9657-predictor-67b57f7854-8l784\" (UID: \"09665ef2-2574-4057-b347-b5884c1f9664\") " pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784"
Sep 18 19:34:36.831628 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:36.831604    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09665ef2-2574-4057-b347-b5884c1f9664-proxy-tls\") pod \"isvc-logger-raw-b9657-predictor-67b57f7854-8l784\" (UID: \"09665ef2-2574-4057-b347-b5884c1f9664\") " pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784"
Sep 18 19:34:36.831734 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:36.831666    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2mqt\" (UniqueName: \"kubernetes.io/projected/09665ef2-2574-4057-b347-b5884c1f9664-kube-api-access-f2mqt\") pod \"isvc-logger-raw-b9657-predictor-67b57f7854-8l784\" (UID: \"09665ef2-2574-4057-b347-b5884c1f9664\") " pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784"
Sep 18 19:34:36.906190 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:36.906163    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:34:36.932394 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:36.932367    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2mqt\" (UniqueName: \"kubernetes.io/projected/09665ef2-2574-4057-b347-b5884c1f9664-kube-api-access-f2mqt\") pod \"isvc-logger-raw-b9657-predictor-67b57f7854-8l784\" (UID: \"09665ef2-2574-4057-b347-b5884c1f9664\") " pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784"
Sep 18 19:34:36.932550 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:36.932416    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09665ef2-2574-4057-b347-b5884c1f9664-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-b9657-predictor-67b57f7854-8l784\" (UID: \"09665ef2-2574-4057-b347-b5884c1f9664\") " pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784"
Sep 18 19:34:36.932550 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:36.932453    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09665ef2-2574-4057-b347-b5884c1f9664-cabundle-cert\") pod \"isvc-logger-raw-b9657-predictor-67b57f7854-8l784\" (UID: \"09665ef2-2574-4057-b347-b5884c1f9664\") " pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784"
Sep 18 19:34:36.932550 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:36.932503    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09665ef2-2574-4057-b347-b5884c1f9664-kserve-provision-location\") pod \"isvc-logger-raw-b9657-predictor-67b57f7854-8l784\" (UID: \"09665ef2-2574-4057-b347-b5884c1f9664\") " pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784"
Sep 18 19:34:36.932550 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:36.932528    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09665ef2-2574-4057-b347-b5884c1f9664-proxy-tls\") pod \"isvc-logger-raw-b9657-predictor-67b57f7854-8l784\" (UID: \"09665ef2-2574-4057-b347-b5884c1f9664\") " pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784"
Sep 18 19:34:36.932983 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:36.932961    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09665ef2-2574-4057-b347-b5884c1f9664-kserve-provision-location\") pod \"isvc-logger-raw-b9657-predictor-67b57f7854-8l784\" (UID: \"09665ef2-2574-4057-b347-b5884c1f9664\") " pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784"
Sep 18 19:34:36.933236 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:36.933220    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09665ef2-2574-4057-b347-b5884c1f9664-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-b9657-predictor-67b57f7854-8l784\" (UID: \"09665ef2-2574-4057-b347-b5884c1f9664\") " pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784"
Sep 18 19:34:36.933409 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:36.933388    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09665ef2-2574-4057-b347-b5884c1f9664-cabundle-cert\") pod \"isvc-logger-raw-b9657-predictor-67b57f7854-8l784\" (UID: \"09665ef2-2574-4057-b347-b5884c1f9664\") " pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784"
Sep 18 19:34:36.934865 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:36.934848    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09665ef2-2574-4057-b347-b5884c1f9664-proxy-tls\") pod \"isvc-logger-raw-b9657-predictor-67b57f7854-8l784\" (UID: \"09665ef2-2574-4057-b347-b5884c1f9664\") " pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784"
Sep 18 19:34:36.944218 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:36.944199    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2mqt\" (UniqueName: \"kubernetes.io/projected/09665ef2-2574-4057-b347-b5884c1f9664-kube-api-access-f2mqt\") pod \"isvc-logger-raw-b9657-predictor-67b57f7854-8l784\" (UID: \"09665ef2-2574-4057-b347-b5884c1f9664\") " pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784"
Sep 18 19:34:37.117485 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:37.117408    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784"
Sep 18 19:34:37.400640 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:37.400616    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784"]
Sep 18 19:34:37.402275 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:34:37.402244    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09665ef2_2574_4057_b347_b5884c1f9664.slice/crio-531df2d72bc0d8470ced85ac4778a7305f98a0d9975896c59ab17939d309fb9e WatchSource:0}: Error finding container 531df2d72bc0d8470ced85ac4778a7305f98a0d9975896c59ab17939d309fb9e: Status 404 returned error can't find the container with id 531df2d72bc0d8470ced85ac4778a7305f98a0d9975896c59ab17939d309fb9e
Sep 18 19:34:38.070310 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:38.070276    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-2a1456-predictor-5b6757476c-rddt5" podUID="397350de-2ff0-4a9c-8d78-67b1199faea9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 18 19:34:38.195606 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:38.195574    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784" event={"ID":"09665ef2-2574-4057-b347-b5884c1f9664","Type":"ContainerStarted","Data":"c71f01751ac017df21598986ef5c8a0b5087ade98c8c45dee1cc4be8aa037bb8"}
Sep 18 19:34:38.195757 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:38.195636    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784" event={"ID":"09665ef2-2574-4057-b347-b5884c1f9664","Type":"ContainerStarted","Data":"531df2d72bc0d8470ced85ac4778a7305f98a0d9975896c59ab17939d309fb9e"}
Sep 18 19:34:42.211109 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:42.211074    2532 generic.go:356] "Generic (PLEG): container finished" podID="09665ef2-2574-4057-b347-b5884c1f9664" containerID="c71f01751ac017df21598986ef5c8a0b5087ade98c8c45dee1cc4be8aa037bb8" exitCode=0
Sep 18 19:34:42.211583 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:42.211146    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784" event={"ID":"09665ef2-2574-4057-b347-b5884c1f9664","Type":"ContainerDied","Data":"c71f01751ac017df21598986ef5c8a0b5087ade98c8c45dee1cc4be8aa037bb8"}
Sep 18 19:34:43.217687 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:43.217655    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784" event={"ID":"09665ef2-2574-4057-b347-b5884c1f9664","Type":"ContainerStarted","Data":"63e3aaa18758f67bd0fa4691f478e66a977dba5df835cbc54aa5f6d8e759b9d1"}
Sep 18 19:34:43.218092 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:43.217745    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784" event={"ID":"09665ef2-2574-4057-b347-b5884c1f9664","Type":"ContainerStarted","Data":"5b65352a6aa331e4b57b2bcb9fbe4881536f7e3ac45a7e089209bc87978ffc27"}
Sep 18 19:34:43.218092 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:43.217770    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784" event={"ID":"09665ef2-2574-4057-b347-b5884c1f9664","Type":"ContainerStarted","Data":"2def92bfa09444b011984b6a95e7b162efabfdb9e2c68b787478031fe3458bb9"}
Sep 18 19:34:43.218092 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:43.218081    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784"
Sep 18 19:34:43.218216 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:43.218198    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784"
Sep 18 19:34:43.219421 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:43.219398    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784" podUID="09665ef2-2574-4057-b347-b5884c1f9664" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 18 19:34:43.242610 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:43.242565    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784" podStartSLOduration=3.363059302 podStartE2EDuration="7.242553336s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.879494034s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:34:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:34:43.241761 +0000 UTC m=+1250.162264861" watchObservedRunningTime="2026-09-18 19:34:43.242553336 +0000 UTC m=+1250.163057192"
Sep 18 19:34:44.221829 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:44.221797    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784"
Sep 18 19:34:44.222306 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:44.221876    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784" podUID="09665ef2-2574-4057-b347-b5884c1f9664" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 18 19:34:44.222930 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:44.222910    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784" podUID="09665ef2-2574-4057-b347-b5884c1f9664" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:34:45.225578 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:45.225528    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784" podUID="09665ef2-2574-4057-b347-b5884c1f9664" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 18 19:34:45.226109 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:45.225895    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784" podUID="09665ef2-2574-4057-b347-b5884c1f9664" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:34:48.070348 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:48.070313    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-2a1456-predictor-5b6757476c-rddt5" podUID="397350de-2ff0-4a9c-8d78-67b1199faea9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 18 19:34:50.229720 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:50.229690    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784"
Sep 18 19:34:50.230311 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:50.230276    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784" podUID="09665ef2-2574-4057-b347-b5884c1f9664" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 18 19:34:50.230680 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:50.230657    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784" podUID="09665ef2-2574-4057-b347-b5884c1f9664" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:34:58.071096 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:34:58.071043    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-2a1456-predictor-5b6757476c-rddt5" podUID="397350de-2ff0-4a9c-8d78-67b1199faea9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 18 19:35:00.230658 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:35:00.230606    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784" podUID="09665ef2-2574-4057-b347-b5884c1f9664" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 18 19:35:00.231144 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:35:00.231026    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784" podUID="09665ef2-2574-4057-b347-b5884c1f9664" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:35:08.070220 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:35:08.070182    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-2a1456-predictor-5b6757476c-rddt5" podUID="397350de-2ff0-4a9c-8d78-67b1199faea9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 18 19:35:10.230608 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:35:10.230565    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784" podUID="09665ef2-2574-4057-b347-b5884c1f9664" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 18 19:35:10.231071 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:35:10.231034    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784" podUID="09665ef2-2574-4057-b347-b5884c1f9664" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:35:18.070922 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:35:18.070887    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-2a1456-predictor-5b6757476c-rddt5"
Sep 18 19:35:20.230661 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:35:20.230624    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784" podUID="09665ef2-2574-4057-b347-b5884c1f9664" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 18 19:35:20.231245 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:35:20.231219    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784" podUID="09665ef2-2574-4057-b347-b5884c1f9664" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:35:30.230219 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:35:30.230167    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784" podUID="09665ef2-2574-4057-b347-b5884c1f9664" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 18 19:35:30.230682 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:35:30.230542    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784" podUID="09665ef2-2574-4057-b347-b5884c1f9664" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:35:40.231156 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:35:40.231100    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784" podUID="09665ef2-2574-4057-b347-b5884c1f9664" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 18 19:35:40.231558 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:35:40.231534    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784" podUID="09665ef2-2574-4057-b347-b5884c1f9664" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:35:41.621155 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:35:41.621118    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:35:41.820036 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:35:41.820004    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:35:46.409290 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:35:46.409252    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:35:50.230679 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:35:50.230638    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784" podUID="09665ef2-2574-4057-b347-b5884c1f9664" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 18 19:35:50.231191 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:35:50.231152    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784" podUID="09665ef2-2574-4057-b347-b5884c1f9664" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:35:52.617018 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:35:52.616983    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784" podUID="09665ef2-2574-4057-b347-b5884c1f9664" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 18 19:35:52.617409 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:35:52.617251    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784" podUID="09665ef2-2574-4057-b347-b5884c1f9664" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:35:57.766040 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:35:57.766002    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:35:57.766426 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:35:57.766288    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-2a1456-predictor-5b6757476c-rddt5"]
Sep 18 19:35:57.766596 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:35:57.766568    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-2a1456-predictor-5b6757476c-rddt5" podUID="397350de-2ff0-4a9c-8d78-67b1199faea9" containerName="kserve-container" containerID="cri-o://e70d6746db73d2c531e73f389eab061cbaefd29fec37bcad1ff3a4f847f7b2ac" gracePeriod=30
Sep 18 19:35:57.766713 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:35:57.766668    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-2a1456-predictor-5b6757476c-rddt5" podUID="397350de-2ff0-4a9c-8d78-67b1199faea9" containerName="kube-rbac-proxy" containerID="cri-o://8c6886b6c5c0a5f8f90a8b23cfe688bf8954d0abb607aed9b8b6d71fea6bb855" gracePeriod=30
Sep 18 19:35:57.847979 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:35:57.847946    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:35:58.065521 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:35:58.065460    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-2a1456-predictor-5b6757476c-rddt5" podUID="397350de-2ff0-4a9c-8d78-67b1199faea9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Sep 18 19:35:58.070803 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:35:58.070783    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-2a1456-predictor-5b6757476c-rddt5" podUID="397350de-2ff0-4a9c-8d78-67b1199faea9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 18 19:35:58.503315 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:35:58.503285    2532 generic.go:356] "Generic (PLEG): container finished" podID="397350de-2ff0-4a9c-8d78-67b1199faea9" containerID="8c6886b6c5c0a5f8f90a8b23cfe688bf8954d0abb607aed9b8b6d71fea6bb855" exitCode=2
Sep 18 19:35:58.503479 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:35:58.503360    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-2a1456-predictor-5b6757476c-rddt5" event={"ID":"397350de-2ff0-4a9c-8d78-67b1199faea9","Type":"ContainerDied","Data":"8c6886b6c5c0a5f8f90a8b23cfe688bf8954d0abb607aed9b8b6d71fea6bb855"}
Sep 18 19:36:02.618094 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:02.618033    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784"
Sep 18 19:36:02.618540 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:02.618242    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784"
Sep 18 19:36:03.015693 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:03.015656    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:36:03.065931 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:03.065893    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-2a1456-predictor-5b6757476c-rddt5" podUID="397350de-2ff0-4a9c-8d78-67b1199faea9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Sep 18 19:36:03.286701 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:03.286615    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:36:06.304870 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:06.304849    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-2a1456-predictor-5b6757476c-rddt5"
Sep 18 19:36:06.392109 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:06.392030    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/397350de-2ff0-4a9c-8d78-67b1199faea9-proxy-tls\") pod \"397350de-2ff0-4a9c-8d78-67b1199faea9\" (UID: \"397350de-2ff0-4a9c-8d78-67b1199faea9\") "
Sep 18 19:36:06.392109 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:06.392075    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/397350de-2ff0-4a9c-8d78-67b1199faea9-kserve-provision-location\") pod \"397350de-2ff0-4a9c-8d78-67b1199faea9\" (UID: \"397350de-2ff0-4a9c-8d78-67b1199faea9\") "
Sep 18 19:36:06.392109 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:06.392093    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/397350de-2ff0-4a9c-8d78-67b1199faea9-kube-rbac-proxy-sar-config\") pod \"397350de-2ff0-4a9c-8d78-67b1199faea9\" (UID: \"397350de-2ff0-4a9c-8d78-67b1199faea9\") "
Sep 18 19:36:06.392328 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:06.392113    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/397350de-2ff0-4a9c-8d78-67b1199faea9-cabundle-cert\") pod \"397350de-2ff0-4a9c-8d78-67b1199faea9\" (UID: \"397350de-2ff0-4a9c-8d78-67b1199faea9\") "
Sep 18 19:36:06.392328 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:06.392158    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4xnpr\" (UniqueName: \"kubernetes.io/projected/397350de-2ff0-4a9c-8d78-67b1199faea9-kube-api-access-4xnpr\") pod \"397350de-2ff0-4a9c-8d78-67b1199faea9\" (UID: \"397350de-2ff0-4a9c-8d78-67b1199faea9\") "
Sep 18 19:36:06.392466 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:06.392443    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/397350de-2ff0-4a9c-8d78-67b1199faea9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "397350de-2ff0-4a9c-8d78-67b1199faea9" (UID: "397350de-2ff0-4a9c-8d78-67b1199faea9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:36:06.392545 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:06.392461    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/397350de-2ff0-4a9c-8d78-67b1199faea9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "397350de-2ff0-4a9c-8d78-67b1199faea9" (UID: "397350de-2ff0-4a9c-8d78-67b1199faea9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:36:06.392545 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:06.392479    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/397350de-2ff0-4a9c-8d78-67b1199faea9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "397350de-2ff0-4a9c-8d78-67b1199faea9" (UID: "397350de-2ff0-4a9c-8d78-67b1199faea9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:36:06.394105 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:06.394076    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/397350de-2ff0-4a9c-8d78-67b1199faea9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "397350de-2ff0-4a9c-8d78-67b1199faea9" (UID: "397350de-2ff0-4a9c-8d78-67b1199faea9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:36:06.394277 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:06.394258    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/397350de-2ff0-4a9c-8d78-67b1199faea9-kube-api-access-4xnpr" (OuterVolumeSpecName: "kube-api-access-4xnpr") pod "397350de-2ff0-4a9c-8d78-67b1199faea9" (UID: "397350de-2ff0-4a9c-8d78-67b1199faea9"). InnerVolumeSpecName "kube-api-access-4xnpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:36:06.493021 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:06.492997    2532 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/397350de-2ff0-4a9c-8d78-67b1199faea9-proxy-tls\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:36:06.493021 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:06.493018    2532 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/397350de-2ff0-4a9c-8d78-67b1199faea9-kserve-provision-location\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:36:06.493180 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:06.493027    2532 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/397350de-2ff0-4a9c-8d78-67b1199faea9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:36:06.493180 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:06.493037    2532 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/397350de-2ff0-4a9c-8d78-67b1199faea9-cabundle-cert\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:36:06.493180 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:06.493045    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xnpr\" (UniqueName: \"kubernetes.io/projected/397350de-2ff0-4a9c-8d78-67b1199faea9-kube-api-access-4xnpr\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:36:06.533616 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:06.533592    2532 generic.go:356] "Generic (PLEG): container finished" podID="397350de-2ff0-4a9c-8d78-67b1199faea9" containerID="e70d6746db73d2c531e73f389eab061cbaefd29fec37bcad1ff3a4f847f7b2ac" exitCode=0
Sep 18 19:36:06.533728 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:06.533645    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-2a1456-predictor-5b6757476c-rddt5" event={"ID":"397350de-2ff0-4a9c-8d78-67b1199faea9","Type":"ContainerDied","Data":"e70d6746db73d2c531e73f389eab061cbaefd29fec37bcad1ff3a4f847f7b2ac"}
Sep 18 19:36:06.533728 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:06.533653    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-2a1456-predictor-5b6757476c-rddt5"
Sep 18 19:36:06.533728 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:06.533692    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-2a1456-predictor-5b6757476c-rddt5" event={"ID":"397350de-2ff0-4a9c-8d78-67b1199faea9","Type":"ContainerDied","Data":"a589826af46128c0b60c43cb0952e03f19bb318b1b2e0d41e297a813e9aba7c7"}
Sep 18 19:36:06.533728 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:06.533720    2532 scope.go:117] "RemoveContainer" containerID="8c6886b6c5c0a5f8f90a8b23cfe688bf8954d0abb607aed9b8b6d71fea6bb855"
Sep 18 19:36:06.547419 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:06.547399    2532 scope.go:117] "RemoveContainer" containerID="e70d6746db73d2c531e73f389eab061cbaefd29fec37bcad1ff3a4f847f7b2ac"
Sep 18 19:36:06.554447 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:06.554429    2532 scope.go:117] "RemoveContainer" containerID="a79895ce513da1d746eaf4d16cf09803e5b5114baf4c0784a8cc4f8ed19accfd"
Sep 18 19:36:06.561188 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:06.561170    2532 scope.go:117] "RemoveContainer" containerID="8c6886b6c5c0a5f8f90a8b23cfe688bf8954d0abb607aed9b8b6d71fea6bb855"
Sep 18 19:36:06.561486 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:36:06.561469    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c6886b6c5c0a5f8f90a8b23cfe688bf8954d0abb607aed9b8b6d71fea6bb855\": container with ID starting with 8c6886b6c5c0a5f8f90a8b23cfe688bf8954d0abb607aed9b8b6d71fea6bb855 not found: ID does not exist" containerID="8c6886b6c5c0a5f8f90a8b23cfe688bf8954d0abb607aed9b8b6d71fea6bb855"
Sep 18 19:36:06.561536 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:06.561492    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c6886b6c5c0a5f8f90a8b23cfe688bf8954d0abb607aed9b8b6d71fea6bb855"} err="failed to get container status \"8c6886b6c5c0a5f8f90a8b23cfe688bf8954d0abb607aed9b8b6d71fea6bb855\": rpc error: code = NotFound desc = could not find container \"8c6886b6c5c0a5f8f90a8b23cfe688bf8954d0abb607aed9b8b6d71fea6bb855\": container with ID starting with 8c6886b6c5c0a5f8f90a8b23cfe688bf8954d0abb607aed9b8b6d71fea6bb855 not found: ID does not exist"
Sep 18 19:36:06.561536 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:06.561507    2532 scope.go:117] "RemoveContainer" containerID="e70d6746db73d2c531e73f389eab061cbaefd29fec37bcad1ff3a4f847f7b2ac"
Sep 18 19:36:06.561747 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:36:06.561731    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e70d6746db73d2c531e73f389eab061cbaefd29fec37bcad1ff3a4f847f7b2ac\": container with ID starting with e70d6746db73d2c531e73f389eab061cbaefd29fec37bcad1ff3a4f847f7b2ac not found: ID does not exist" containerID="e70d6746db73d2c531e73f389eab061cbaefd29fec37bcad1ff3a4f847f7b2ac"
Sep 18 19:36:06.561796 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:06.561750    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e70d6746db73d2c531e73f389eab061cbaefd29fec37bcad1ff3a4f847f7b2ac"} err="failed to get container status \"e70d6746db73d2c531e73f389eab061cbaefd29fec37bcad1ff3a4f847f7b2ac\": rpc error: code = NotFound desc = could not find container \"e70d6746db73d2c531e73f389eab061cbaefd29fec37bcad1ff3a4f847f7b2ac\": container with ID starting with e70d6746db73d2c531e73f389eab061cbaefd29fec37bcad1ff3a4f847f7b2ac not found: ID does not exist"
Sep 18 19:36:06.561796 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:06.561761    2532 scope.go:117] "RemoveContainer" containerID="a79895ce513da1d746eaf4d16cf09803e5b5114baf4c0784a8cc4f8ed19accfd"
Sep 18 19:36:06.561964 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:36:06.561950    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a79895ce513da1d746eaf4d16cf09803e5b5114baf4c0784a8cc4f8ed19accfd\": container with ID starting with a79895ce513da1d746eaf4d16cf09803e5b5114baf4c0784a8cc4f8ed19accfd not found: ID does not exist" containerID="a79895ce513da1d746eaf4d16cf09803e5b5114baf4c0784a8cc4f8ed19accfd"
Sep 18 19:36:06.562003 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:06.561966    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a79895ce513da1d746eaf4d16cf09803e5b5114baf4c0784a8cc4f8ed19accfd"} err="failed to get container status \"a79895ce513da1d746eaf4d16cf09803e5b5114baf4c0784a8cc4f8ed19accfd\": rpc error: code = NotFound desc = could not find container \"a79895ce513da1d746eaf4d16cf09803e5b5114baf4c0784a8cc4f8ed19accfd\": container with ID starting with a79895ce513da1d746eaf4d16cf09803e5b5114baf4c0784a8cc4f8ed19accfd not found: ID does not exist"
Sep 18 19:36:06.572092 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:06.572050    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-2a1456-predictor-5b6757476c-rddt5"]
Sep 18 19:36:06.577593 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:06.577573    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-2a1456-predictor-5b6757476c-rddt5"]
Sep 18 19:36:07.620539 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:07.620507    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="397350de-2ff0-4a9c-8d78-67b1199faea9" path="/var/lib/kubelet/pods/397350de-2ff0-4a9c-8d78-67b1199faea9/volumes"
Sep 18 19:36:12.081035 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:12.081008    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-b9657-predictor-8bd5d747d-9pl8t_f63bf0ea-f130-4d38-aa1d-7ae6aa48858d/kserve-container/0.log"
Sep 18 19:36:12.286047 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:12.285033    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784"]
Sep 18 19:36:12.286047 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:12.285516    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784" podUID="09665ef2-2574-4057-b347-b5884c1f9664" containerName="kserve-container" containerID="cri-o://2def92bfa09444b011984b6a95e7b162efabfdb9e2c68b787478031fe3458bb9" gracePeriod=30
Sep 18 19:36:12.286047 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:12.285785    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784" podUID="09665ef2-2574-4057-b347-b5884c1f9664" containerName="agent" containerID="cri-o://63e3aaa18758f67bd0fa4691f478e66a977dba5df835cbc54aa5f6d8e759b9d1" gracePeriod=30
Sep 18 19:36:12.286047 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:12.285890    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784" podUID="09665ef2-2574-4057-b347-b5884c1f9664" containerName="kube-rbac-proxy" containerID="cri-o://5b65352a6aa331e4b57b2bcb9fbe4881536f7e3ac45a7e089209bc87978ffc27" gracePeriod=30
Sep 18 19:36:12.289853 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:12.289806    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:36:12.315356 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:12.315192    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-b9657-predictor-8bd5d747d-9pl8t"]
Sep 18 19:36:12.315505 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:12.315453    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-b9657-predictor-8bd5d747d-9pl8t" podUID="f63bf0ea-f130-4d38-aa1d-7ae6aa48858d" containerName="kserve-container" containerID="cri-o://55b8a85f7900d3f973c33b08dcab23059342fa811729743498ff6cea6d068fa4" gracePeriod=30
Sep 18 19:36:12.315627 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:12.315607    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-b9657-predictor-8bd5d747d-9pl8t" podUID="f63bf0ea-f130-4d38-aa1d-7ae6aa48858d" containerName="kube-rbac-proxy" containerID="cri-o://c30ba5b1bde034f226361d0110cc4129d234d81c4cb0982f73e69dcd0301ac6e" gracePeriod=30
Sep 18 19:36:12.359740 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:12.359683    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:36:12.444975 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:12.444952    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:36:12.491492 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:12.491467    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c"]
Sep 18 19:36:12.491790 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:12.491777    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="397350de-2ff0-4a9c-8d78-67b1199faea9" containerName="kube-rbac-proxy"
Sep 18 19:36:12.491841 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:12.491793    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="397350de-2ff0-4a9c-8d78-67b1199faea9" containerName="kube-rbac-proxy"
Sep 18 19:36:12.491841 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:12.491816    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="397350de-2ff0-4a9c-8d78-67b1199faea9" containerName="kserve-container"
Sep 18 19:36:12.491841 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:12.491821    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="397350de-2ff0-4a9c-8d78-67b1199faea9" containerName="kserve-container"
Sep 18 19:36:12.491841 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:12.491830    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="397350de-2ff0-4a9c-8d78-67b1199faea9" containerName="storage-initializer"
Sep 18 19:36:12.491841 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:12.491836    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="397350de-2ff0-4a9c-8d78-67b1199faea9" containerName="storage-initializer"
Sep 18 19:36:12.491990 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:12.491877    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="397350de-2ff0-4a9c-8d78-67b1199faea9" containerName="kserve-container"
Sep 18 19:36:12.491990 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:12.491888    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="397350de-2ff0-4a9c-8d78-67b1199faea9" containerName="kube-rbac-proxy"
Sep 18 19:36:12.496319 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:12.496294    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c"
Sep 18 19:36:12.498755 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:12.498734    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-5b49b-predictor-serving-cert\""
Sep 18 19:36:12.498902 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:12.498816    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-5b49b-kube-rbac-proxy-sar-config\""
Sep 18 19:36:12.506158 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:12.506136    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c"]
Sep 18 19:36:12.536737 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:12.536711    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/95d33a63-27ff-440a-a6ed-72f24ca79498-cabundle-cert\") pod \"isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c\" (UID: \"95d33a63-27ff-440a-a6ed-72f24ca79498\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c"
Sep 18 19:36:12.536859 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:12.536770    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/95d33a63-27ff-440a-a6ed-72f24ca79498-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c\" (UID: \"95d33a63-27ff-440a-a6ed-72f24ca79498\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c"
Sep 18 19:36:12.536859 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:12.536804    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqwf9\" (UniqueName: \"kubernetes.io/projected/95d33a63-27ff-440a-a6ed-72f24ca79498-kube-api-access-dqwf9\") pod \"isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c\" (UID: \"95d33a63-27ff-440a-a6ed-72f24ca79498\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c"
Sep 18 19:36:12.536859 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:12.536841    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/95d33a63-27ff-440a-a6ed-72f24ca79498-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c\" (UID: \"95d33a63-27ff-440a-a6ed-72f24ca79498\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c"
Sep 18 19:36:12.536967 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:12.536883    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/95d33a63-27ff-440a-a6ed-72f24ca79498-proxy-tls\") pod \"isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c\" (UID: \"95d33a63-27ff-440a-a6ed-72f24ca79498\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c"
Sep 18 19:36:12.559259 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:12.559237    2532 generic.go:356] "Generic (PLEG): container finished" podID="f63bf0ea-f130-4d38-aa1d-7ae6aa48858d" containerID="c30ba5b1bde034f226361d0110cc4129d234d81c4cb0982f73e69dcd0301ac6e" exitCode=2
Sep 18 19:36:12.559259 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:12.559257    2532 generic.go:356] "Generic (PLEG): container finished" podID="f63bf0ea-f130-4d38-aa1d-7ae6aa48858d" containerID="55b8a85f7900d3f973c33b08dcab23059342fa811729743498ff6cea6d068fa4" exitCode=2
Sep 18 19:36:12.559393 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:12.559306    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-b9657-predictor-8bd5d747d-9pl8t" event={"ID":"f63bf0ea-f130-4d38-aa1d-7ae6aa48858d","Type":"ContainerDied","Data":"c30ba5b1bde034f226361d0110cc4129d234d81c4cb0982f73e69dcd0301ac6e"}
Sep 18 19:36:12.559393 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:12.559358    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-b9657-predictor-8bd5d747d-9pl8t" event={"ID":"f63bf0ea-f130-4d38-aa1d-7ae6aa48858d","Type":"ContainerDied","Data":"55b8a85f7900d3f973c33b08dcab23059342fa811729743498ff6cea6d068fa4"}
Sep 18 19:36:12.559393 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:12.559382    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-b9657-predictor-8bd5d747d-9pl8t" event={"ID":"f63bf0ea-f130-4d38-aa1d-7ae6aa48858d","Type":"ContainerDied","Data":"b541df774b9bd9ae840cd2f4e927c06c63affe715a5fc3175dd9ce48cc3f2b97"}
Sep 18 19:36:12.559568 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:12.559409    2532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b541df774b9bd9ae840cd2f4e927c06c63affe715a5fc3175dd9ce48cc3f2b97"
Sep 18 19:36:12.560356 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:12.560329    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-b9657-predictor-8bd5d747d-9pl8t"
Sep 18 19:36:12.561639 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:12.561621    2532 generic.go:356] "Generic (PLEG): container finished" podID="09665ef2-2574-4057-b347-b5884c1f9664" containerID="5b65352a6aa331e4b57b2bcb9fbe4881536f7e3ac45a7e089209bc87978ffc27" exitCode=2
Sep 18 19:36:12.561706 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:12.561684    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784" event={"ID":"09665ef2-2574-4057-b347-b5884c1f9664","Type":"ContainerDied","Data":"5b65352a6aa331e4b57b2bcb9fbe4881536f7e3ac45a7e089209bc87978ffc27"}
Sep 18 19:36:12.586904 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:12.586884    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:36:12.617638 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:12.617583    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784" podUID="09665ef2-2574-4057-b347-b5884c1f9664" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 18 19:36:12.617893 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:12.617872    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784" podUID="09665ef2-2574-4057-b347-b5884c1f9664" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:36:12.638169 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:12.638144    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f63bf0ea-f130-4d38-aa1d-7ae6aa48858d-kube-rbac-proxy-sar-config\") pod \"f63bf0ea-f130-4d38-aa1d-7ae6aa48858d\" (UID: \"f63bf0ea-f130-4d38-aa1d-7ae6aa48858d\") "
Sep 18 19:36:12.638254 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:12.638179    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2cdgv\" (UniqueName: \"kubernetes.io/projected/f63bf0ea-f130-4d38-aa1d-7ae6aa48858d-kube-api-access-2cdgv\") pod \"f63bf0ea-f130-4d38-aa1d-7ae6aa48858d\" (UID: \"f63bf0ea-f130-4d38-aa1d-7ae6aa48858d\") "
Sep 18 19:36:12.638254 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:12.638227    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f63bf0ea-f130-4d38-aa1d-7ae6aa48858d-proxy-tls\") pod \"f63bf0ea-f130-4d38-aa1d-7ae6aa48858d\" (UID: \"f63bf0ea-f130-4d38-aa1d-7ae6aa48858d\") "
Sep 18 19:36:12.638465 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:12.638364    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/95d33a63-27ff-440a-a6ed-72f24ca79498-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c\" (UID: \"95d33a63-27ff-440a-a6ed-72f24ca79498\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c"
Sep 18 19:36:12.638465 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:12.638403    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqwf9\" (UniqueName: \"kubernetes.io/projected/95d33a63-27ff-440a-a6ed-72f24ca79498-kube-api-access-dqwf9\") pod \"isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c\" (UID: \"95d33a63-27ff-440a-a6ed-72f24ca79498\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c"
Sep 18 19:36:12.638465 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:12.638428    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/95d33a63-27ff-440a-a6ed-72f24ca79498-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c\" (UID: \"95d33a63-27ff-440a-a6ed-72f24ca79498\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c"
Sep 18 19:36:12.638650 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:12.638490    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/95d33a63-27ff-440a-a6ed-72f24ca79498-proxy-tls\") pod \"isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c\" (UID: \"95d33a63-27ff-440a-a6ed-72f24ca79498\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c"
Sep 18 19:36:12.638650 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:12.638507    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f63bf0ea-f130-4d38-aa1d-7ae6aa48858d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f63bf0ea-f130-4d38-aa1d-7ae6aa48858d" (UID: "f63bf0ea-f130-4d38-aa1d-7ae6aa48858d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:36:12.638650 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:12.638529    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/95d33a63-27ff-440a-a6ed-72f24ca79498-cabundle-cert\") pod \"isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c\" (UID: \"95d33a63-27ff-440a-a6ed-72f24ca79498\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c"
Sep 18 19:36:12.639003 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:12.638861    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/95d33a63-27ff-440a-a6ed-72f24ca79498-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c\" (UID: \"95d33a63-27ff-440a-a6ed-72f24ca79498\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c"
Sep 18 19:36:12.639003 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:12.638881    2532 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f63bf0ea-f130-4d38-aa1d-7ae6aa48858d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:36:12.639368 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:12.639337    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/95d33a63-27ff-440a-a6ed-72f24ca79498-cabundle-cert\") pod \"isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c\" (UID: \"95d33a63-27ff-440a-a6ed-72f24ca79498\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c"
Sep 18 19:36:12.639456 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:12.639390    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/95d33a63-27ff-440a-a6ed-72f24ca79498-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c\" (UID: \"95d33a63-27ff-440a-a6ed-72f24ca79498\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c"
Sep 18 19:36:12.640871 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:12.640852    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f63bf0ea-f130-4d38-aa1d-7ae6aa48858d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f63bf0ea-f130-4d38-aa1d-7ae6aa48858d" (UID: "f63bf0ea-f130-4d38-aa1d-7ae6aa48858d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:36:12.640871 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:12.640861    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f63bf0ea-f130-4d38-aa1d-7ae6aa48858d-kube-api-access-2cdgv" (OuterVolumeSpecName: "kube-api-access-2cdgv") pod "f63bf0ea-f130-4d38-aa1d-7ae6aa48858d" (UID: "f63bf0ea-f130-4d38-aa1d-7ae6aa48858d"). InnerVolumeSpecName "kube-api-access-2cdgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:36:12.641019 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:12.641004    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/95d33a63-27ff-440a-a6ed-72f24ca79498-proxy-tls\") pod \"isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c\" (UID: \"95d33a63-27ff-440a-a6ed-72f24ca79498\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c"
Sep 18 19:36:12.648078 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:12.648040    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqwf9\" (UniqueName: \"kubernetes.io/projected/95d33a63-27ff-440a-a6ed-72f24ca79498-kube-api-access-dqwf9\") pod \"isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c\" (UID: \"95d33a63-27ff-440a-a6ed-72f24ca79498\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c"
Sep 18 19:36:12.739839 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:12.739819    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cdgv\" (UniqueName: \"kubernetes.io/projected/f63bf0ea-f130-4d38-aa1d-7ae6aa48858d-kube-api-access-2cdgv\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:36:12.739839 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:12.739840    2532 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f63bf0ea-f130-4d38-aa1d-7ae6aa48858d-proxy-tls\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:36:12.808867 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:12.808836    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c"
Sep 18 19:36:12.968565 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:12.968532    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:36:13.090998 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:13.090974    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c"]
Sep 18 19:36:13.094070 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:36:13.094022    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95d33a63_27ff_440a_a6ed_72f24ca79498.slice/crio-3913d52771f5569df5781a055f7575826e4baca45a2186f2c0f769cb289e33e9 WatchSource:0}: Error finding container 3913d52771f5569df5781a055f7575826e4baca45a2186f2c0f769cb289e33e9: Status 404 returned error can't find the container with id 3913d52771f5569df5781a055f7575826e4baca45a2186f2c0f769cb289e33e9
Sep 18 19:36:13.567155 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:13.567073    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c" event={"ID":"95d33a63-27ff-440a-a6ed-72f24ca79498","Type":"ContainerStarted","Data":"fef69983e9bd8b3eac44d343ae4ab8a79fd442a997eb48a215e84183b5a617a9"}
Sep 18 19:36:13.567155 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:13.567125    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c" event={"ID":"95d33a63-27ff-440a-a6ed-72f24ca79498","Type":"ContainerStarted","Data":"3913d52771f5569df5781a055f7575826e4baca45a2186f2c0f769cb289e33e9"}
Sep 18 19:36:13.567398 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:13.567173    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-b9657-predictor-8bd5d747d-9pl8t"
Sep 18 19:36:13.602048 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:13.602021    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-b9657-predictor-8bd5d747d-9pl8t"]
Sep 18 19:36:13.606834 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:13.606810    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-b9657-predictor-8bd5d747d-9pl8t"]
Sep 18 19:36:13.622842 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:13.622819    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f63bf0ea-f130-4d38-aa1d-7ae6aa48858d" path="/var/lib/kubelet/pods/f63bf0ea-f130-4d38-aa1d-7ae6aa48858d/volumes"
Sep 18 19:36:15.225917 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:15.225878    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784" podUID="09665ef2-2574-4057-b347-b5884c1f9664" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Sep 18 19:36:16.580474 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:16.580446    2532 generic.go:356] "Generic (PLEG): container finished" podID="09665ef2-2574-4057-b347-b5884c1f9664" containerID="2def92bfa09444b011984b6a95e7b162efabfdb9e2c68b787478031fe3458bb9" exitCode=0
Sep 18 19:36:16.580762 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:16.580522    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784" event={"ID":"09665ef2-2574-4057-b347-b5884c1f9664","Type":"ContainerDied","Data":"2def92bfa09444b011984b6a95e7b162efabfdb9e2c68b787478031fe3458bb9"}
Sep 18 19:36:17.586269 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:17.586236    2532 generic.go:356] "Generic (PLEG): container finished" podID="95d33a63-27ff-440a-a6ed-72f24ca79498" containerID="fef69983e9bd8b3eac44d343ae4ab8a79fd442a997eb48a215e84183b5a617a9" exitCode=0
Sep 18 19:36:17.586627 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:17.586309    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c" event={"ID":"95d33a63-27ff-440a-a6ed-72f24ca79498","Type":"ContainerDied","Data":"fef69983e9bd8b3eac44d343ae4ab8a79fd442a997eb48a215e84183b5a617a9"}
Sep 18 19:36:18.156264 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:18.156230    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:36:18.328762 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:18.328722    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:36:18.593115 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:18.593081    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c" event={"ID":"95d33a63-27ff-440a-a6ed-72f24ca79498","Type":"ContainerStarted","Data":"95347162b5de295e0494163bb829bb6d59d67c20ecdc2fb31fae9aa975c3c960"}
Sep 18 19:36:18.593566 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:18.593150    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c" event={"ID":"95d33a63-27ff-440a-a6ed-72f24ca79498","Type":"ContainerStarted","Data":"6df41818b60623fdebe6b4dc45a602271a201b5e18b719e6d17aff25d0aa9552"}
Sep 18 19:36:18.593566 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:18.593370    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c"
Sep 18 19:36:18.615390 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:18.615345    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c" podStartSLOduration=2.824052202 podStartE2EDuration="6.615330703s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.791278501s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:36:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:36:18.613895371 +0000 UTC m=+1345.534399229" watchObservedRunningTime="2026-09-18 19:36:18.615330703 +0000 UTC m=+1345.535834559"
Sep 18 19:36:19.597301 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:19.597270    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c"
Sep 18 19:36:19.598582 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:19.598554    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c" podUID="95d33a63-27ff-440a-a6ed-72f24ca79498" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 18 19:36:20.226717 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:20.226671    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784" podUID="09665ef2-2574-4057-b347-b5884c1f9664" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Sep 18 19:36:20.601516 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:20.601427    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c" podUID="95d33a63-27ff-440a-a6ed-72f24ca79498" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 18 19:36:22.617729 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:22.617679    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784" podUID="09665ef2-2574-4057-b347-b5884c1f9664" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 18 19:36:22.618255 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:22.618038    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784" podUID="09665ef2-2574-4057-b347-b5884c1f9664" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:36:25.226330 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:25.226287    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784" podUID="09665ef2-2574-4057-b347-b5884c1f9664" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Sep 18 19:36:25.226696 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:25.226402    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784"
Sep 18 19:36:25.606444 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:25.606370    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c"
Sep 18 19:36:25.606837 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:25.606814    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c" podUID="95d33a63-27ff-440a-a6ed-72f24ca79498" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 18 19:36:30.226547 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:30.226503    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784" podUID="09665ef2-2574-4057-b347-b5884c1f9664" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Sep 18 19:36:32.617187 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:32.617144    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784" podUID="09665ef2-2574-4057-b347-b5884c1f9664" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 18 19:36:32.617613 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:32.617279    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784"
Sep 18 19:36:32.617613 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:32.617504    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784" podUID="09665ef2-2574-4057-b347-b5884c1f9664" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:36:32.617613 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:32.617576    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784"
Sep 18 19:36:35.226185 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:35.226144    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784" podUID="09665ef2-2574-4057-b347-b5884c1f9664" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Sep 18 19:36:35.606873 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:35.606787    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c" podUID="95d33a63-27ff-440a-a6ed-72f24ca79498" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 18 19:36:40.226038 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:40.225990    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784" podUID="09665ef2-2574-4057-b347-b5884c1f9664" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Sep 18 19:36:42.438314 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:42.438294    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784"
Sep 18 19:36:42.541911 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:42.541852    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09665ef2-2574-4057-b347-b5884c1f9664-kube-rbac-proxy-sar-config\") pod \"09665ef2-2574-4057-b347-b5884c1f9664\" (UID: \"09665ef2-2574-4057-b347-b5884c1f9664\") "
Sep 18 19:36:42.541911 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:42.541885    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09665ef2-2574-4057-b347-b5884c1f9664-cabundle-cert\") pod \"09665ef2-2574-4057-b347-b5884c1f9664\" (UID: \"09665ef2-2574-4057-b347-b5884c1f9664\") "
Sep 18 19:36:42.541911 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:42.541912    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09665ef2-2574-4057-b347-b5884c1f9664-kserve-provision-location\") pod \"09665ef2-2574-4057-b347-b5884c1f9664\" (UID: \"09665ef2-2574-4057-b347-b5884c1f9664\") "
Sep 18 19:36:42.542134 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:42.541956    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09665ef2-2574-4057-b347-b5884c1f9664-proxy-tls\") pod \"09665ef2-2574-4057-b347-b5884c1f9664\" (UID: \"09665ef2-2574-4057-b347-b5884c1f9664\") "
Sep 18 19:36:42.542134 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:42.541975    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f2mqt\" (UniqueName: \"kubernetes.io/projected/09665ef2-2574-4057-b347-b5884c1f9664-kube-api-access-f2mqt\") pod \"09665ef2-2574-4057-b347-b5884c1f9664\" (UID: \"09665ef2-2574-4057-b347-b5884c1f9664\") "
Sep 18 19:36:42.542252 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:42.542223    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09665ef2-2574-4057-b347-b5884c1f9664-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "09665ef2-2574-4057-b347-b5884c1f9664" (UID: "09665ef2-2574-4057-b347-b5884c1f9664"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:36:42.542298 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:42.542262    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09665ef2-2574-4057-b347-b5884c1f9664-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "09665ef2-2574-4057-b347-b5884c1f9664" (UID: "09665ef2-2574-4057-b347-b5884c1f9664"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:36:42.542335 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:42.542319    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09665ef2-2574-4057-b347-b5884c1f9664-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "09665ef2-2574-4057-b347-b5884c1f9664" (UID: "09665ef2-2574-4057-b347-b5884c1f9664"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:36:42.543995 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:42.543974    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09665ef2-2574-4057-b347-b5884c1f9664-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "09665ef2-2574-4057-b347-b5884c1f9664" (UID: "09665ef2-2574-4057-b347-b5884c1f9664"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:36:42.544124 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:42.544091    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09665ef2-2574-4057-b347-b5884c1f9664-kube-api-access-f2mqt" (OuterVolumeSpecName: "kube-api-access-f2mqt") pod "09665ef2-2574-4057-b347-b5884c1f9664" (UID: "09665ef2-2574-4057-b347-b5884c1f9664"). InnerVolumeSpecName "kube-api-access-f2mqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:36:42.642517 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:42.642494    2532 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09665ef2-2574-4057-b347-b5884c1f9664-kserve-provision-location\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:36:42.642517 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:42.642513    2532 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09665ef2-2574-4057-b347-b5884c1f9664-proxy-tls\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:36:42.642646 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:42.642523    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2mqt\" (UniqueName: \"kubernetes.io/projected/09665ef2-2574-4057-b347-b5884c1f9664-kube-api-access-f2mqt\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:36:42.642646 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:42.642532    2532 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09665ef2-2574-4057-b347-b5884c1f9664-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:36:42.642646 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:42.642541    2532 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09665ef2-2574-4057-b347-b5884c1f9664-cabundle-cert\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:36:42.682493 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:42.682470    2532 generic.go:356] "Generic (PLEG): container finished" podID="09665ef2-2574-4057-b347-b5884c1f9664" containerID="63e3aaa18758f67bd0fa4691f478e66a977dba5df835cbc54aa5f6d8e759b9d1" exitCode=0
Sep 18 19:36:42.682608 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:42.682527    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784"
Sep 18 19:36:42.682608 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:42.682557    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784" event={"ID":"09665ef2-2574-4057-b347-b5884c1f9664","Type":"ContainerDied","Data":"63e3aaa18758f67bd0fa4691f478e66a977dba5df835cbc54aa5f6d8e759b9d1"}
Sep 18 19:36:42.682757 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:42.682639    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784" event={"ID":"09665ef2-2574-4057-b347-b5884c1f9664","Type":"ContainerDied","Data":"531df2d72bc0d8470ced85ac4778a7305f98a0d9975896c59ab17939d309fb9e"}
Sep 18 19:36:42.682757 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:42.682687    2532 scope.go:117] "RemoveContainer" containerID="63e3aaa18758f67bd0fa4691f478e66a977dba5df835cbc54aa5f6d8e759b9d1"
Sep 18 19:36:42.691130 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:42.691116    2532 scope.go:117] "RemoveContainer" containerID="5b65352a6aa331e4b57b2bcb9fbe4881536f7e3ac45a7e089209bc87978ffc27"
Sep 18 19:36:42.697860 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:42.697846    2532 scope.go:117] "RemoveContainer" containerID="2def92bfa09444b011984b6a95e7b162efabfdb9e2c68b787478031fe3458bb9"
Sep 18 19:36:42.704529 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:42.704513    2532 scope.go:117] "RemoveContainer" containerID="c71f01751ac017df21598986ef5c8a0b5087ade98c8c45dee1cc4be8aa037bb8"
Sep 18 19:36:42.705405 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:42.705384    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784"]
Sep 18 19:36:42.710500 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:42.710479    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-b9657-predictor-67b57f7854-8l784"]
Sep 18 19:36:42.711729 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:42.711715    2532 scope.go:117] "RemoveContainer" containerID="63e3aaa18758f67bd0fa4691f478e66a977dba5df835cbc54aa5f6d8e759b9d1"
Sep 18 19:36:42.711968 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:36:42.711949    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63e3aaa18758f67bd0fa4691f478e66a977dba5df835cbc54aa5f6d8e759b9d1\": container with ID starting with 63e3aaa18758f67bd0fa4691f478e66a977dba5df835cbc54aa5f6d8e759b9d1 not found: ID does not exist" containerID="63e3aaa18758f67bd0fa4691f478e66a977dba5df835cbc54aa5f6d8e759b9d1"
Sep 18 19:36:42.712026 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:42.711974    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63e3aaa18758f67bd0fa4691f478e66a977dba5df835cbc54aa5f6d8e759b9d1"} err="failed to get container status \"63e3aaa18758f67bd0fa4691f478e66a977dba5df835cbc54aa5f6d8e759b9d1\": rpc error: code = NotFound desc = could not find container \"63e3aaa18758f67bd0fa4691f478e66a977dba5df835cbc54aa5f6d8e759b9d1\": container with ID starting with 63e3aaa18758f67bd0fa4691f478e66a977dba5df835cbc54aa5f6d8e759b9d1 not found: ID does not exist"
Sep 18 19:36:42.712026 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:42.711990    2532 scope.go:117] "RemoveContainer" containerID="5b65352a6aa331e4b57b2bcb9fbe4881536f7e3ac45a7e089209bc87978ffc27"
Sep 18 19:36:42.712270 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:36:42.712255    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b65352a6aa331e4b57b2bcb9fbe4881536f7e3ac45a7e089209bc87978ffc27\": container with ID starting with 5b65352a6aa331e4b57b2bcb9fbe4881536f7e3ac45a7e089209bc87978ffc27 not found: ID does not exist" containerID="5b65352a6aa331e4b57b2bcb9fbe4881536f7e3ac45a7e089209bc87978ffc27"
Sep 18 19:36:42.712312 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:42.712275    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b65352a6aa331e4b57b2bcb9fbe4881536f7e3ac45a7e089209bc87978ffc27"} err="failed to get container status \"5b65352a6aa331e4b57b2bcb9fbe4881536f7e3ac45a7e089209bc87978ffc27\": rpc error: code = NotFound desc = could not find container \"5b65352a6aa331e4b57b2bcb9fbe4881536f7e3ac45a7e089209bc87978ffc27\": container with ID starting with 5b65352a6aa331e4b57b2bcb9fbe4881536f7e3ac45a7e089209bc87978ffc27 not found: ID does not exist"
Sep 18 19:36:42.712312 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:42.712287    2532 scope.go:117] "RemoveContainer" containerID="2def92bfa09444b011984b6a95e7b162efabfdb9e2c68b787478031fe3458bb9"
Sep 18 19:36:42.712548 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:36:42.712524    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2def92bfa09444b011984b6a95e7b162efabfdb9e2c68b787478031fe3458bb9\": container with ID starting with 2def92bfa09444b011984b6a95e7b162efabfdb9e2c68b787478031fe3458bb9 not found: ID does not exist" containerID="2def92bfa09444b011984b6a95e7b162efabfdb9e2c68b787478031fe3458bb9"
Sep 18 19:36:42.712635 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:42.712545    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2def92bfa09444b011984b6a95e7b162efabfdb9e2c68b787478031fe3458bb9"} err="failed to get container status \"2def92bfa09444b011984b6a95e7b162efabfdb9e2c68b787478031fe3458bb9\": rpc error: code = NotFound desc = could not find container \"2def92bfa09444b011984b6a95e7b162efabfdb9e2c68b787478031fe3458bb9\": container with ID starting with 2def92bfa09444b011984b6a95e7b162efabfdb9e2c68b787478031fe3458bb9 not found: ID does not exist"
Sep 18 19:36:42.712635 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:42.712560    2532 scope.go:117] "RemoveContainer" containerID="c71f01751ac017df21598986ef5c8a0b5087ade98c8c45dee1cc4be8aa037bb8"
Sep 18 19:36:42.712816 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:36:42.712801    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c71f01751ac017df21598986ef5c8a0b5087ade98c8c45dee1cc4be8aa037bb8\": container with ID starting with c71f01751ac017df21598986ef5c8a0b5087ade98c8c45dee1cc4be8aa037bb8 not found: ID does not exist" containerID="c71f01751ac017df21598986ef5c8a0b5087ade98c8c45dee1cc4be8aa037bb8"
Sep 18 19:36:42.712854 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:42.712820    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c71f01751ac017df21598986ef5c8a0b5087ade98c8c45dee1cc4be8aa037bb8"} err="failed to get container status \"c71f01751ac017df21598986ef5c8a0b5087ade98c8c45dee1cc4be8aa037bb8\": rpc error: code = NotFound desc = could not find container \"c71f01751ac017df21598986ef5c8a0b5087ade98c8c45dee1cc4be8aa037bb8\": container with ID starting with c71f01751ac017df21598986ef5c8a0b5087ade98c8c45dee1cc4be8aa037bb8 not found: ID does not exist"
Sep 18 19:36:43.620391 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:43.620358    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09665ef2-2574-4057-b347-b5884c1f9664" path="/var/lib/kubelet/pods/09665ef2-2574-4057-b347-b5884c1f9664/volumes"
Sep 18 19:36:45.607377 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:45.607339    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c" podUID="95d33a63-27ff-440a-a6ed-72f24ca79498" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 18 19:36:55.607011 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:36:55.606974    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c" podUID="95d33a63-27ff-440a-a6ed-72f24ca79498" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 18 19:37:05.607832 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:37:05.607795    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c" podUID="95d33a63-27ff-440a-a6ed-72f24ca79498" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 18 19:37:15.606791 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:37:15.606754    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c" podUID="95d33a63-27ff-440a-a6ed-72f24ca79498" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 18 19:37:25.607233 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:37:25.607194    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c" podUID="95d33a63-27ff-440a-a6ed-72f24ca79498" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 18 19:37:35.607319 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:37:35.607283    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c" podUID="95d33a63-27ff-440a-a6ed-72f24ca79498" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 18 19:37:35.616297 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:37:35.616259    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c" podUID="95d33a63-27ff-440a-a6ed-72f24ca79498" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 18 19:37:45.616231 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:37:45.616188    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c" podUID="95d33a63-27ff-440a-a6ed-72f24ca79498" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 18 19:37:48.764608 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:37:48.764525    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:37:53.871388 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:37:53.871352    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:37:53.963729 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:37:53.963697    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r"]
Sep 18 19:37:53.963972 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:37:53.963960    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f63bf0ea-f130-4d38-aa1d-7ae6aa48858d" containerName="kserve-container"
Sep 18 19:37:53.964016 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:37:53.963976    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="f63bf0ea-f130-4d38-aa1d-7ae6aa48858d" containerName="kserve-container"
Sep 18 19:37:53.964016 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:37:53.963987    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09665ef2-2574-4057-b347-b5884c1f9664" containerName="kube-rbac-proxy"
Sep 18 19:37:53.964016 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:37:53.963992    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="09665ef2-2574-4057-b347-b5884c1f9664" containerName="kube-rbac-proxy"
Sep 18 19:37:53.964016 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:37:53.963998    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f63bf0ea-f130-4d38-aa1d-7ae6aa48858d" containerName="kube-rbac-proxy"
Sep 18 19:37:53.964016 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:37:53.964004    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="f63bf0ea-f130-4d38-aa1d-7ae6aa48858d" containerName="kube-rbac-proxy"
Sep 18 19:37:53.964016 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:37:53.964010    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09665ef2-2574-4057-b347-b5884c1f9664" containerName="storage-initializer"
Sep 18 19:37:53.964016 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:37:53.964016    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="09665ef2-2574-4057-b347-b5884c1f9664" containerName="storage-initializer"
Sep 18 19:37:53.964252 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:37:53.964029    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09665ef2-2574-4057-b347-b5884c1f9664" containerName="agent"
Sep 18 19:37:53.964252 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:37:53.964034    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="09665ef2-2574-4057-b347-b5884c1f9664" containerName="agent"
Sep 18 19:37:53.964252 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:37:53.964044    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09665ef2-2574-4057-b347-b5884c1f9664" containerName="kserve-container"
Sep 18 19:37:53.964252 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:37:53.964050    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="09665ef2-2574-4057-b347-b5884c1f9664" containerName="kserve-container"
Sep 18 19:37:53.964252 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:37:53.964107    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="f63bf0ea-f130-4d38-aa1d-7ae6aa48858d" containerName="kserve-container"
Sep 18 19:37:53.964252 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:37:53.964114    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="09665ef2-2574-4057-b347-b5884c1f9664" containerName="kube-rbac-proxy"
Sep 18 19:37:53.964252 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:37:53.964120    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="f63bf0ea-f130-4d38-aa1d-7ae6aa48858d" containerName="kube-rbac-proxy"
Sep 18 19:37:53.964252 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:37:53.964125    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="09665ef2-2574-4057-b347-b5884c1f9664" containerName="kserve-container"
Sep 18 19:37:53.964252 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:37:53.964131    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="09665ef2-2574-4057-b347-b5884c1f9664" containerName="agent"
Sep 18 19:37:53.966956 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:37:53.966941    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r"
Sep 18 19:37:53.969666 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:37:53.969648    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-82555-predictor-serving-cert\""
Sep 18 19:37:53.969951 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:37:53.969936    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-82555-kube-rbac-proxy-sar-config\""
Sep 18 19:37:53.980815 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:37:53.980797    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r"]
Sep 18 19:37:54.039769 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:37:54.039747    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r\" (UID: \"d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r"
Sep 18 19:37:54.039875 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:37:54.039776    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3-cabundle-cert\") pod \"raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r\" (UID: \"d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r"
Sep 18 19:37:54.039875 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:37:54.039804    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkfl7\" (UniqueName: \"kubernetes.io/projected/d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3-kube-api-access-lkfl7\") pod \"raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r\" (UID: \"d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r"
Sep 18 19:37:54.039875 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:37:54.039836    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3-proxy-tls\") pod \"raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r\" (UID: \"d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r"
Sep 18 19:37:54.039875 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:37:54.039869    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3-kserve-provision-location\") pod \"raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r\" (UID: \"d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r"
Sep 18 19:37:54.050188 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:37:54.050167    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:37:54.140771 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:37:54.140720    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r\" (UID: \"d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r"
Sep 18 19:37:54.140771 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:37:54.140748    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3-cabundle-cert\") pod \"raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r\" (UID: \"d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r"
Sep 18 19:37:54.140883 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:37:54.140776    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkfl7\" (UniqueName: \"kubernetes.io/projected/d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3-kube-api-access-lkfl7\") pod \"raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r\" (UID: \"d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r"
Sep 18 19:37:54.140883 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:37:54.140838    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3-proxy-tls\") pod \"raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r\" (UID: \"d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r"
Sep 18 19:37:54.140883 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:37:54.140868    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3-kserve-provision-location\") pod \"raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r\" (UID: \"d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r"
Sep 18 19:37:54.141369 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:37:54.141342    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3-kserve-provision-location\") pod \"raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r\" (UID: \"d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r"
Sep 18 19:37:54.141483 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:37:54.141463    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r\" (UID: \"d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r"
Sep 18 19:37:54.141545 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:37:54.141515    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3-cabundle-cert\") pod \"raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r\" (UID: \"d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r"
Sep 18 19:37:54.143218 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:37:54.143203    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3-proxy-tls\") pod \"raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r\" (UID: \"d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r"
Sep 18 19:37:54.150295 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:37:54.150274    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkfl7\" (UniqueName: \"kubernetes.io/projected/d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3-kube-api-access-lkfl7\") pod \"raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r\" (UID: \"d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r"
Sep 18 19:37:54.276518 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:37:54.276491    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r"
Sep 18 19:37:54.554831 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:37:54.554791    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r"]
Sep 18 19:37:54.557460 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:37:54.557433    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1243a4e_c7fd_4be3_b73f_3c4ec714f5f3.slice/crio-0ddf988846d03fd869d306f912fd6dde59636dce95ca6572ee2525a91b3e2d4c WatchSource:0}: Error finding container 0ddf988846d03fd869d306f912fd6dde59636dce95ca6572ee2525a91b3e2d4c: Status 404 returned error can't find the container with id 0ddf988846d03fd869d306f912fd6dde59636dce95ca6572ee2525a91b3e2d4c
Sep 18 19:37:54.939594 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:37:54.939555    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r" event={"ID":"d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3","Type":"ContainerStarted","Data":"9139fde173093de15470274541648631f54abc3af768f3e45a595d6972b2eb2f"}
Sep 18 19:37:54.940036 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:37:54.939620    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r" event={"ID":"d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3","Type":"ContainerStarted","Data":"0ddf988846d03fd869d306f912fd6dde59636dce95ca6572ee2525a91b3e2d4c"}
Sep 18 19:37:55.616768 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:37:55.616727    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c" podUID="95d33a63-27ff-440a-a6ed-72f24ca79498" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 18 19:37:58.958035 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:37:58.958000    2532 generic.go:356] "Generic (PLEG): container finished" podID="d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3" containerID="9139fde173093de15470274541648631f54abc3af768f3e45a595d6972b2eb2f" exitCode=0
Sep 18 19:37:58.958422 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:37:58.958045    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r" event={"ID":"d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3","Type":"ContainerDied","Data":"9139fde173093de15470274541648631f54abc3af768f3e45a595d6972b2eb2f"}
Sep 18 19:37:59.964308 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:37:59.964274    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r" event={"ID":"d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3","Type":"ContainerStarted","Data":"7b7e71ba35dfa111cc693d76554e51022726c01f3ff47bb6051597ba0c7ee4cd"}
Sep 18 19:37:59.964666 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:37:59.964334    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r" event={"ID":"d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3","Type":"ContainerStarted","Data":"e8c322af5eb75f72c85971f5660f8bc5c01a61fde6a737419c8347b1314c2db3"}
Sep 18 19:37:59.964666 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:37:59.964648    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r"
Sep 18 19:37:59.964805 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:37:59.964786    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r"
Sep 18 19:37:59.965987 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:37:59.965965    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r" podUID="d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 18 19:37:59.990606 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:37:59.990564    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r" podStartSLOduration=3.188587686 podStartE2EDuration="6.99055138s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.801963694s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:37:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:37:59.988892858 +0000 UTC m=+1446.909396715" watchObservedRunningTime="2026-09-18 19:37:59.99055138 +0000 UTC m=+1446.911055238"
Sep 18 19:38:00.968128 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:38:00.968092    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r" podUID="d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 18 19:38:05.616383 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:38:05.616340    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c" podUID="95d33a63-27ff-440a-a6ed-72f24ca79498" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 18 19:38:05.973125 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:38:05.973096    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r"
Sep 18 19:38:05.973504 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:38:05.973479    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r" podUID="d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 18 19:38:15.616755 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:38:15.616692    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c" podUID="95d33a63-27ff-440a-a6ed-72f24ca79498" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 18 19:38:15.973784 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:38:15.973747    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r" podUID="d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 18 19:38:25.617118 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:38:25.617078    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c" podUID="95d33a63-27ff-440a-a6ed-72f24ca79498" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 18 19:38:25.974300 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:38:25.974265    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r" podUID="d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 18 19:38:35.616478 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:38:35.616436    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c" podUID="95d33a63-27ff-440a-a6ed-72f24ca79498" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 18 19:38:35.973927 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:38:35.973892    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r" podUID="d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 18 19:38:40.868311 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:38:40.868277    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:41.620947 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:38:41.620911    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c" podUID="95d33a63-27ff-440a-a6ed-72f24ca79498" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 18 19:38:45.973973 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:38:45.973938    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r" podUID="d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 18 19:38:51.621641 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:38:51.621608    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c"
Sep 18 19:38:53.490705 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:38:53.490670    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c"]
Sep 18 19:38:53.491172 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:38:53.490877    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c" podUID="95d33a63-27ff-440a-a6ed-72f24ca79498" containerName="kserve-container" containerID="cri-o://6df41818b60623fdebe6b4dc45a602271a201b5e18b719e6d17aff25d0aa9552" gracePeriod=30
Sep 18 19:38:53.491172 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:38:53.490935    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c" podUID="95d33a63-27ff-440a-a6ed-72f24ca79498" containerName="kube-rbac-proxy" containerID="cri-o://95347162b5de295e0494163bb829bb6d59d67c20ecdc2fb31fae9aa975c3c960" gracePeriod=30
Sep 18 19:38:53.515195 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:38:53.515165    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:53.586859 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:38:53.586832    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sksw8_2cef0f4e-e44f-4996-8561-da9d261036c7/ovn-acl-logging/0.log"
Sep 18 19:38:53.587103 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:38:53.587087    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sksw8_2cef0f4e-e44f-4996-8561-da9d261036c7/ovn-acl-logging/0.log"
Sep 18 19:38:54.171820 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:38:54.171785    2532 generic.go:356] "Generic (PLEG): container finished" podID="95d33a63-27ff-440a-a6ed-72f24ca79498" containerID="95347162b5de295e0494163bb829bb6d59d67c20ecdc2fb31fae9aa975c3c960" exitCode=2
Sep 18 19:38:54.171988 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:38:54.171857    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c" event={"ID":"95d33a63-27ff-440a-a6ed-72f24ca79498","Type":"ContainerDied","Data":"95347162b5de295e0494163bb829bb6d59d67c20ecdc2fb31fae9aa975c3c960"}
Sep 18 19:38:55.602633 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:38:55.602582    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c" podUID="95d33a63-27ff-440a-a6ed-72f24ca79498" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused"
Sep 18 19:38:55.973779 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:38:55.973739    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r" podUID="d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 18 19:38:58.627022 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:38:58.626991    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:58.732651 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:38:58.732614    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-s26zm"]
Sep 18 19:38:58.734828 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:38:58.734803    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:58.736020 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:38:58.736005    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-s26zm"
Sep 18 19:38:58.738577 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:38:58.738558    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 18 19:38:58.739251 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:38:58.739234    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 18 19:38:58.753478 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:38:58.753460    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-s26zm"]
Sep 18 19:38:58.919915 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:38:58.919843    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/410b89b0-faf6-40a1-86ef-d7eafa392524-kserve-provision-location\") pod \"isvc-canary-create-predictor-7b568f75-s26zm\" (UID: \"410b89b0-faf6-40a1-86ef-d7eafa392524\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-s26zm"
Sep 18 19:38:58.919915 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:38:58.919877    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrs8r\" (UniqueName: \"kubernetes.io/projected/410b89b0-faf6-40a1-86ef-d7eafa392524-kube-api-access-lrs8r\") pod \"isvc-canary-create-predictor-7b568f75-s26zm\" (UID: \"410b89b0-faf6-40a1-86ef-d7eafa392524\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-s26zm"
Sep 18 19:38:58.920101 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:38:58.919918    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/410b89b0-faf6-40a1-86ef-d7eafa392524-cabundle-cert\") pod \"isvc-canary-create-predictor-7b568f75-s26zm\" (UID: \"410b89b0-faf6-40a1-86ef-d7eafa392524\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-s26zm"
Sep 18 19:38:58.920101 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:38:58.919974    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/410b89b0-faf6-40a1-86ef-d7eafa392524-proxy-tls\") pod \"isvc-canary-create-predictor-7b568f75-s26zm\" (UID: \"410b89b0-faf6-40a1-86ef-d7eafa392524\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-s26zm"
Sep 18 19:38:58.920101 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:38:58.920015    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/410b89b0-faf6-40a1-86ef-d7eafa392524-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-7b568f75-s26zm\" (UID: \"410b89b0-faf6-40a1-86ef-d7eafa392524\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-s26zm"
Sep 18 19:38:58.937978 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:38:58.937946    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:59.020882 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:38:59.020852    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/410b89b0-faf6-40a1-86ef-d7eafa392524-cabundle-cert\") pod \"isvc-canary-create-predictor-7b568f75-s26zm\" (UID: \"410b89b0-faf6-40a1-86ef-d7eafa392524\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-s26zm"
Sep 18 19:38:59.021027 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:38:59.020906    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/410b89b0-faf6-40a1-86ef-d7eafa392524-proxy-tls\") pod \"isvc-canary-create-predictor-7b568f75-s26zm\" (UID: \"410b89b0-faf6-40a1-86ef-d7eafa392524\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-s26zm"
Sep 18 19:38:59.021027 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:38:59.020935    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/410b89b0-faf6-40a1-86ef-d7eafa392524-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-7b568f75-s26zm\" (UID: \"410b89b0-faf6-40a1-86ef-d7eafa392524\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-s26zm"
Sep 18 19:38:59.021027 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:38:59.020971    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/410b89b0-faf6-40a1-86ef-d7eafa392524-kserve-provision-location\") pod \"isvc-canary-create-predictor-7b568f75-s26zm\" (UID: \"410b89b0-faf6-40a1-86ef-d7eafa392524\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-s26zm"
Sep 18 19:38:59.021027 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:38:59.021004    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrs8r\" (UniqueName: \"kubernetes.io/projected/410b89b0-faf6-40a1-86ef-d7eafa392524-kube-api-access-lrs8r\") pod \"isvc-canary-create-predictor-7b568f75-s26zm\" (UID: \"410b89b0-faf6-40a1-86ef-d7eafa392524\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-s26zm"
Sep 18 19:38:59.021402 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:38:59.021378    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/410b89b0-faf6-40a1-86ef-d7eafa392524-kserve-provision-location\") pod \"isvc-canary-create-predictor-7b568f75-s26zm\" (UID: \"410b89b0-faf6-40a1-86ef-d7eafa392524\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-s26zm"
Sep 18 19:38:59.021568 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:38:59.021550    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/410b89b0-faf6-40a1-86ef-d7eafa392524-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-7b568f75-s26zm\" (UID: \"410b89b0-faf6-40a1-86ef-d7eafa392524\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-s26zm"
Sep 18 19:38:59.021626 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:38:59.021552    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/410b89b0-faf6-40a1-86ef-d7eafa392524-cabundle-cert\") pod \"isvc-canary-create-predictor-7b568f75-s26zm\" (UID: \"410b89b0-faf6-40a1-86ef-d7eafa392524\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-s26zm"
Sep 18 19:38:59.023623 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:38:59.023602    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/410b89b0-faf6-40a1-86ef-d7eafa392524-proxy-tls\") pod \"isvc-canary-create-predictor-7b568f75-s26zm\" (UID: \"410b89b0-faf6-40a1-86ef-d7eafa392524\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-s26zm"
Sep 18 19:38:59.030455 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:38:59.030422    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrs8r\" (UniqueName: \"kubernetes.io/projected/410b89b0-faf6-40a1-86ef-d7eafa392524-kube-api-access-lrs8r\") pod \"isvc-canary-create-predictor-7b568f75-s26zm\" (UID: \"410b89b0-faf6-40a1-86ef-d7eafa392524\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-s26zm"
Sep 18 19:38:59.046178 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:38:59.046161    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-s26zm"
Sep 18 19:38:59.280587 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:38:59.280542    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:59.338697 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:38:59.338665    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-s26zm"]
Sep 18 19:38:59.342135 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:38:59.342112    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod410b89b0_faf6_40a1_86ef_d7eafa392524.slice/crio-889e5329831fb042a17cb530fc434b0a1058b78ab533ba9330af570a11e21752 WatchSource:0}: Error finding container 889e5329831fb042a17cb530fc434b0a1058b78ab533ba9330af570a11e21752: Status 404 returned error can't find the container with id 889e5329831fb042a17cb530fc434b0a1058b78ab533ba9330af570a11e21752
Sep 18 19:39:00.196403 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:00.196371    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-s26zm" event={"ID":"410b89b0-faf6-40a1-86ef-d7eafa392524","Type":"ContainerStarted","Data":"df0b041e08b32812020a057d99ec77b5e0054228bca7c40598254e282d8f62a2"}
Sep 18 19:39:00.196832 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:00.196430    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-s26zm" event={"ID":"410b89b0-faf6-40a1-86ef-d7eafa392524","Type":"ContainerStarted","Data":"889e5329831fb042a17cb530fc434b0a1058b78ab533ba9330af570a11e21752"}
Sep 18 19:39:00.601806 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:00.601719    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c" podUID="95d33a63-27ff-440a-a6ed-72f24ca79498" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused"
Sep 18 19:39:01.624228 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:01.623182    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c" podUID="95d33a63-27ff-440a-a6ed-72f24ca79498" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 18 19:39:03.208728 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:03.208695    2532 generic.go:356] "Generic (PLEG): container finished" podID="410b89b0-faf6-40a1-86ef-d7eafa392524" containerID="df0b041e08b32812020a057d99ec77b5e0054228bca7c40598254e282d8f62a2" exitCode=0
Sep 18 19:39:03.208728 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:03.208731    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-s26zm" event={"ID":"410b89b0-faf6-40a1-86ef-d7eafa392524","Type":"ContainerDied","Data":"df0b041e08b32812020a057d99ec77b5e0054228bca7c40598254e282d8f62a2"}
Sep 18 19:39:03.837404 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:03.837378    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c"
Sep 18 19:39:03.959586 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:03.959557    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dqwf9\" (UniqueName: \"kubernetes.io/projected/95d33a63-27ff-440a-a6ed-72f24ca79498-kube-api-access-dqwf9\") pod \"95d33a63-27ff-440a-a6ed-72f24ca79498\" (UID: \"95d33a63-27ff-440a-a6ed-72f24ca79498\") "
Sep 18 19:39:03.959766 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:03.959637    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/95d33a63-27ff-440a-a6ed-72f24ca79498-kube-rbac-proxy-sar-config\") pod \"95d33a63-27ff-440a-a6ed-72f24ca79498\" (UID: \"95d33a63-27ff-440a-a6ed-72f24ca79498\") "
Sep 18 19:39:03.959766 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:03.959658    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/95d33a63-27ff-440a-a6ed-72f24ca79498-cabundle-cert\") pod \"95d33a63-27ff-440a-a6ed-72f24ca79498\" (UID: \"95d33a63-27ff-440a-a6ed-72f24ca79498\") "
Sep 18 19:39:03.959766 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:03.959699    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/95d33a63-27ff-440a-a6ed-72f24ca79498-proxy-tls\") pod \"95d33a63-27ff-440a-a6ed-72f24ca79498\" (UID: \"95d33a63-27ff-440a-a6ed-72f24ca79498\") "
Sep 18 19:39:03.959766 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:03.959737    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/95d33a63-27ff-440a-a6ed-72f24ca79498-kserve-provision-location\") pod \"95d33a63-27ff-440a-a6ed-72f24ca79498\" (UID: \"95d33a63-27ff-440a-a6ed-72f24ca79498\") "
Sep 18 19:39:03.960040 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:03.960014    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95d33a63-27ff-440a-a6ed-72f24ca79498-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "95d33a63-27ff-440a-a6ed-72f24ca79498" (UID: "95d33a63-27ff-440a-a6ed-72f24ca79498"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:39:03.960126 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:03.960088    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95d33a63-27ff-440a-a6ed-72f24ca79498-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "95d33a63-27ff-440a-a6ed-72f24ca79498" (UID: "95d33a63-27ff-440a-a6ed-72f24ca79498"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:39:03.960164 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:03.960117    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95d33a63-27ff-440a-a6ed-72f24ca79498-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "95d33a63-27ff-440a-a6ed-72f24ca79498" (UID: "95d33a63-27ff-440a-a6ed-72f24ca79498"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:39:03.961798 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:03.961779    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95d33a63-27ff-440a-a6ed-72f24ca79498-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "95d33a63-27ff-440a-a6ed-72f24ca79498" (UID: "95d33a63-27ff-440a-a6ed-72f24ca79498"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:39:03.961893 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:03.961778    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95d33a63-27ff-440a-a6ed-72f24ca79498-kube-api-access-dqwf9" (OuterVolumeSpecName: "kube-api-access-dqwf9") pod "95d33a63-27ff-440a-a6ed-72f24ca79498" (UID: "95d33a63-27ff-440a-a6ed-72f24ca79498"). InnerVolumeSpecName "kube-api-access-dqwf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:39:04.061158 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:04.061123    2532 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/95d33a63-27ff-440a-a6ed-72f24ca79498-proxy-tls\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:39:04.061158 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:04.061153    2532 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/95d33a63-27ff-440a-a6ed-72f24ca79498-kserve-provision-location\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:39:04.061158 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:04.061162    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqwf9\" (UniqueName: \"kubernetes.io/projected/95d33a63-27ff-440a-a6ed-72f24ca79498-kube-api-access-dqwf9\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:39:04.061397 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:04.061173    2532 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/95d33a63-27ff-440a-a6ed-72f24ca79498-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:39:04.061397 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:04.061182    2532 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/95d33a63-27ff-440a-a6ed-72f24ca79498-cabundle-cert\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:39:04.214306 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:04.214224    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-s26zm" event={"ID":"410b89b0-faf6-40a1-86ef-d7eafa392524","Type":"ContainerStarted","Data":"a77dceece0ff236cf32042428ae167f23d45152825fbe0a03e856a4f67532db4"}
Sep 18 19:39:04.214306 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:04.214289    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-s26zm" event={"ID":"410b89b0-faf6-40a1-86ef-d7eafa392524","Type":"ContainerStarted","Data":"389c409cb86829996fc1873214ea243b290386160a1e5e939c7a3f3c9ae298ba"}
Sep 18 19:39:04.214780 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:04.214552    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-s26zm"
Sep 18 19:39:04.214780 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:04.214689    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-s26zm"
Sep 18 19:39:04.215843 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:04.215812    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-s26zm" podUID="410b89b0-faf6-40a1-86ef-d7eafa392524" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 18 19:39:04.216256 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:04.216238    2532 generic.go:356] "Generic (PLEG): container finished" podID="95d33a63-27ff-440a-a6ed-72f24ca79498" containerID="6df41818b60623fdebe6b4dc45a602271a201b5e18b719e6d17aff25d0aa9552" exitCode=0
Sep 18 19:39:04.216372 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:04.216281    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c"
Sep 18 19:39:04.216372 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:04.216299    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c" event={"ID":"95d33a63-27ff-440a-a6ed-72f24ca79498","Type":"ContainerDied","Data":"6df41818b60623fdebe6b4dc45a602271a201b5e18b719e6d17aff25d0aa9552"}
Sep 18 19:39:04.216372 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:04.216351    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c" event={"ID":"95d33a63-27ff-440a-a6ed-72f24ca79498","Type":"ContainerDied","Data":"3913d52771f5569df5781a055f7575826e4baca45a2186f2c0f769cb289e33e9"}
Sep 18 19:39:04.216527 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:04.216390    2532 scope.go:117] "RemoveContainer" containerID="95347162b5de295e0494163bb829bb6d59d67c20ecdc2fb31fae9aa975c3c960"
Sep 18 19:39:04.224888 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:04.224873    2532 scope.go:117] "RemoveContainer" containerID="6df41818b60623fdebe6b4dc45a602271a201b5e18b719e6d17aff25d0aa9552"
Sep 18 19:39:04.232296 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:04.232280    2532 scope.go:117] "RemoveContainer" containerID="fef69983e9bd8b3eac44d343ae4ab8a79fd442a997eb48a215e84183b5a617a9"
Sep 18 19:39:04.239221 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:04.239199    2532 scope.go:117] "RemoveContainer" containerID="95347162b5de295e0494163bb829bb6d59d67c20ecdc2fb31fae9aa975c3c960"
Sep 18 19:39:04.239502 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:39:04.239484    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95347162b5de295e0494163bb829bb6d59d67c20ecdc2fb31fae9aa975c3c960\": container with ID starting with 95347162b5de295e0494163bb829bb6d59d67c20ecdc2fb31fae9aa975c3c960 not found: ID does not exist" containerID="95347162b5de295e0494163bb829bb6d59d67c20ecdc2fb31fae9aa975c3c960"
Sep 18 19:39:04.239587 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:04.239512    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95347162b5de295e0494163bb829bb6d59d67c20ecdc2fb31fae9aa975c3c960"} err="failed to get container status \"95347162b5de295e0494163bb829bb6d59d67c20ecdc2fb31fae9aa975c3c960\": rpc error: code = NotFound desc = could not find container \"95347162b5de295e0494163bb829bb6d59d67c20ecdc2fb31fae9aa975c3c960\": container with ID starting with 95347162b5de295e0494163bb829bb6d59d67c20ecdc2fb31fae9aa975c3c960 not found: ID does not exist"
Sep 18 19:39:04.239587 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:04.239534    2532 scope.go:117] "RemoveContainer" containerID="6df41818b60623fdebe6b4dc45a602271a201b5e18b719e6d17aff25d0aa9552"
Sep 18 19:39:04.239786 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:39:04.239767    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6df41818b60623fdebe6b4dc45a602271a201b5e18b719e6d17aff25d0aa9552\": container with ID starting with 6df41818b60623fdebe6b4dc45a602271a201b5e18b719e6d17aff25d0aa9552 not found: ID does not exist" containerID="6df41818b60623fdebe6b4dc45a602271a201b5e18b719e6d17aff25d0aa9552"
Sep 18 19:39:04.239825 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:04.239793    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6df41818b60623fdebe6b4dc45a602271a201b5e18b719e6d17aff25d0aa9552"} err="failed to get container status \"6df41818b60623fdebe6b4dc45a602271a201b5e18b719e6d17aff25d0aa9552\": rpc error: code = NotFound desc = could not find container \"6df41818b60623fdebe6b4dc45a602271a201b5e18b719e6d17aff25d0aa9552\": container with ID starting with 6df41818b60623fdebe6b4dc45a602271a201b5e18b719e6d17aff25d0aa9552 not found: ID does not exist"
Sep 18 19:39:04.239825 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:04.239807    2532 scope.go:117] "RemoveContainer" containerID="fef69983e9bd8b3eac44d343ae4ab8a79fd442a997eb48a215e84183b5a617a9"
Sep 18 19:39:04.240026 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:39:04.240008    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fef69983e9bd8b3eac44d343ae4ab8a79fd442a997eb48a215e84183b5a617a9\": container with ID starting with fef69983e9bd8b3eac44d343ae4ab8a79fd442a997eb48a215e84183b5a617a9 not found: ID does not exist" containerID="fef69983e9bd8b3eac44d343ae4ab8a79fd442a997eb48a215e84183b5a617a9"
Sep 18 19:39:04.240105 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:04.240034    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fef69983e9bd8b3eac44d343ae4ab8a79fd442a997eb48a215e84183b5a617a9"} err="failed to get container status \"fef69983e9bd8b3eac44d343ae4ab8a79fd442a997eb48a215e84183b5a617a9\": rpc error: code = NotFound desc = could not find container \"fef69983e9bd8b3eac44d343ae4ab8a79fd442a997eb48a215e84183b5a617a9\": container with ID starting with fef69983e9bd8b3eac44d343ae4ab8a79fd442a997eb48a215e84183b5a617a9 not found: ID does not exist"
Sep 18 19:39:04.269398 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:04.269355    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-s26zm" podStartSLOduration=2.73681034 podStartE2EDuration="6.26934041s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.53253007s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:38:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:04.267658767 +0000 UTC m=+1511.188162624" watchObservedRunningTime="2026-09-18 19:39:04.26934041 +0000 UTC m=+1511.189844266"
Sep 18 19:39:04.285154 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:04.285132    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c"]
Sep 18 19:39:04.296911 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:04.296887    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-5b49b-predictor-7c7c598584-lvc2c"]
Sep 18 19:39:05.222301 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:05.222262    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-s26zm" podUID="410b89b0-faf6-40a1-86ef-d7eafa392524" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 18 19:39:05.620667 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:05.620590    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95d33a63-27ff-440a-a6ed-72f24ca79498" path="/var/lib/kubelet/pods/95d33a63-27ff-440a-a6ed-72f24ca79498/volumes"
Sep 18 19:39:05.974180 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:05.974144    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r" podUID="d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 18 19:39:10.227345 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:10.227312    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-s26zm"
Sep 18 19:39:10.227913 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:10.227884    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-s26zm" podUID="410b89b0-faf6-40a1-86ef-d7eafa392524" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 18 19:39:15.974250 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:15.974170    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r"
Sep 18 19:39:20.228366 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:20.228329    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-s26zm" podUID="410b89b0-faf6-40a1-86ef-d7eafa392524" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 18 19:39:24.441316 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:24.441275    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r"]
Sep 18 19:39:24.441869 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:24.441588    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r" podUID="d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3" containerName="kserve-container" containerID="cri-o://e8c322af5eb75f72c85971f5660f8bc5c01a61fde6a737419c8347b1314c2db3" gracePeriod=30
Sep 18 19:39:24.441869 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:24.441597    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r" podUID="d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3" containerName="kube-rbac-proxy" containerID="cri-o://7b7e71ba35dfa111cc693d76554e51022726c01f3ff47bb6051597ba0c7ee4cd" gracePeriod=30
Sep 18 19:39:24.473358 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:24.473336    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:25.302144 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:25.302112    2532 generic.go:356] "Generic (PLEG): container finished" podID="d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3" containerID="7b7e71ba35dfa111cc693d76554e51022726c01f3ff47bb6051597ba0c7ee4cd" exitCode=2
Sep 18 19:39:25.302320 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:25.302188    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r" event={"ID":"d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3","Type":"ContainerDied","Data":"7b7e71ba35dfa111cc693d76554e51022726c01f3ff47bb6051597ba0c7ee4cd"}
Sep 18 19:39:25.968510 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:25.968470    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r" podUID="d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Sep 18 19:39:25.974395 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:25.974365    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r" podUID="d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 18 19:39:29.076122 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:29.076098    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r"
Sep 18 19:39:29.129856 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:29.129832    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3-cabundle-cert\") pod \"d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3\" (UID: \"d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3\") "
Sep 18 19:39:29.129982 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:29.129889    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3-kserve-provision-location\") pod \"d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3\" (UID: \"d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3\") "
Sep 18 19:39:29.129982 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:29.129911    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3-proxy-tls\") pod \"d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3\" (UID: \"d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3\") "
Sep 18 19:39:29.129982 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:29.129948    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3-kube-rbac-proxy-sar-config\") pod \"d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3\" (UID: \"d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3\") "
Sep 18 19:39:29.130191 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:29.129983    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lkfl7\" (UniqueName: \"kubernetes.io/projected/d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3-kube-api-access-lkfl7\") pod \"d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3\" (UID: \"d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3\") "
Sep 18 19:39:29.130250 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:29.130181    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3" (UID: "d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:39:29.130310 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:29.130281    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3" (UID: "d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:39:29.130368 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:29.130336    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3" (UID: "d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:39:29.132093 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:29.132071    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3" (UID: "d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:39:29.132187 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:29.132146    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3-kube-api-access-lkfl7" (OuterVolumeSpecName: "kube-api-access-lkfl7") pod "d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3" (UID: "d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3"). InnerVolumeSpecName "kube-api-access-lkfl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:39:29.231117 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:29.231050    2532 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3-proxy-tls\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:39:29.231117 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:29.231087    2532 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:39:29.231117 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:29.231096    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkfl7\" (UniqueName: \"kubernetes.io/projected/d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3-kube-api-access-lkfl7\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:39:29.231117 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:29.231105    2532 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3-cabundle-cert\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:39:29.231117 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:29.231113    2532 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3-kserve-provision-location\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:39:29.319257 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:29.319223    2532 generic.go:356] "Generic (PLEG): container finished" podID="d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3" containerID="e8c322af5eb75f72c85971f5660f8bc5c01a61fde6a737419c8347b1314c2db3" exitCode=0
Sep 18 19:39:29.319371 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:29.319283    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r"
Sep 18 19:39:29.319371 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:29.319308    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r" event={"ID":"d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3","Type":"ContainerDied","Data":"e8c322af5eb75f72c85971f5660f8bc5c01a61fde6a737419c8347b1314c2db3"}
Sep 18 19:39:29.319453 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:29.319393    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r" event={"ID":"d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3","Type":"ContainerDied","Data":"0ddf988846d03fd869d306f912fd6dde59636dce95ca6572ee2525a91b3e2d4c"}
Sep 18 19:39:29.319453 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:29.319426    2532 scope.go:117] "RemoveContainer" containerID="7b7e71ba35dfa111cc693d76554e51022726c01f3ff47bb6051597ba0c7ee4cd"
Sep 18 19:39:29.327844 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:29.327828    2532 scope.go:117] "RemoveContainer" containerID="e8c322af5eb75f72c85971f5660f8bc5c01a61fde6a737419c8347b1314c2db3"
Sep 18 19:39:29.336230 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:29.336210    2532 scope.go:117] "RemoveContainer" containerID="9139fde173093de15470274541648631f54abc3af768f3e45a595d6972b2eb2f"
Sep 18 19:39:29.340446 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:29.340427    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r"]
Sep 18 19:39:29.343274 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:29.343256    2532 scope.go:117] "RemoveContainer" containerID="7b7e71ba35dfa111cc693d76554e51022726c01f3ff47bb6051597ba0c7ee4cd"
Sep 18 19:39:29.343530 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:39:29.343509    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b7e71ba35dfa111cc693d76554e51022726c01f3ff47bb6051597ba0c7ee4cd\": container with ID starting with 7b7e71ba35dfa111cc693d76554e51022726c01f3ff47bb6051597ba0c7ee4cd not found: ID does not exist" containerID="7b7e71ba35dfa111cc693d76554e51022726c01f3ff47bb6051597ba0c7ee4cd"
Sep 18 19:39:29.343616 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:29.343536    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b7e71ba35dfa111cc693d76554e51022726c01f3ff47bb6051597ba0c7ee4cd"} err="failed to get container status \"7b7e71ba35dfa111cc693d76554e51022726c01f3ff47bb6051597ba0c7ee4cd\": rpc error: code = NotFound desc = could not find container \"7b7e71ba35dfa111cc693d76554e51022726c01f3ff47bb6051597ba0c7ee4cd\": container with ID starting with 7b7e71ba35dfa111cc693d76554e51022726c01f3ff47bb6051597ba0c7ee4cd not found: ID does not exist"
Sep 18 19:39:29.343616 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:29.343553    2532 scope.go:117] "RemoveContainer" containerID="e8c322af5eb75f72c85971f5660f8bc5c01a61fde6a737419c8347b1314c2db3"
Sep 18 19:39:29.343774 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:39:29.343760    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8c322af5eb75f72c85971f5660f8bc5c01a61fde6a737419c8347b1314c2db3\": container with ID starting with e8c322af5eb75f72c85971f5660f8bc5c01a61fde6a737419c8347b1314c2db3 not found: ID does not exist" containerID="e8c322af5eb75f72c85971f5660f8bc5c01a61fde6a737419c8347b1314c2db3"
Sep 18 19:39:29.343815 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:29.343776    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8c322af5eb75f72c85971f5660f8bc5c01a61fde6a737419c8347b1314c2db3"} err="failed to get container status \"e8c322af5eb75f72c85971f5660f8bc5c01a61fde6a737419c8347b1314c2db3\": rpc error: code = NotFound desc = could not find container \"e8c322af5eb75f72c85971f5660f8bc5c01a61fde6a737419c8347b1314c2db3\": container with ID starting with e8c322af5eb75f72c85971f5660f8bc5c01a61fde6a737419c8347b1314c2db3 not found: ID does not exist"
Sep 18 19:39:29.343815 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:29.343788    2532 scope.go:117] "RemoveContainer" containerID="9139fde173093de15470274541648631f54abc3af768f3e45a595d6972b2eb2f"
Sep 18 19:39:29.344026 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:39:29.344011    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9139fde173093de15470274541648631f54abc3af768f3e45a595d6972b2eb2f\": container with ID starting with 9139fde173093de15470274541648631f54abc3af768f3e45a595d6972b2eb2f not found: ID does not exist" containerID="9139fde173093de15470274541648631f54abc3af768f3e45a595d6972b2eb2f"
Sep 18 19:39:29.344077 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:29.344027    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9139fde173093de15470274541648631f54abc3af768f3e45a595d6972b2eb2f"} err="failed to get container status \"9139fde173093de15470274541648631f54abc3af768f3e45a595d6972b2eb2f\": rpc error: code = NotFound desc = could not find container \"9139fde173093de15470274541648631f54abc3af768f3e45a595d6972b2eb2f\": container with ID starting with 9139fde173093de15470274541648631f54abc3af768f3e45a595d6972b2eb2f not found: ID does not exist"
Sep 18 19:39:29.346164 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:29.346145    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-82555-predictor-6687b46c9d-k226r"]
Sep 18 19:39:29.620531 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:29.620467    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3" path="/var/lib/kubelet/pods/d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3/volumes"
Sep 18 19:39:30.228442 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:30.228410    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-s26zm" podUID="410b89b0-faf6-40a1-86ef-d7eafa392524" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 18 19:39:40.228524 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:40.228482    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-s26zm" podUID="410b89b0-faf6-40a1-86ef-d7eafa392524" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 18 19:39:50.227996 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:39:50.227960    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-s26zm" podUID="410b89b0-faf6-40a1-86ef-d7eafa392524" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 18 19:40:00.228275 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:00.228232    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-s26zm" podUID="410b89b0-faf6-40a1-86ef-d7eafa392524" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 18 19:40:04.851653 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:04.851620    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:10.228564 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:10.228522    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-s26zm" podUID="410b89b0-faf6-40a1-86ef-d7eafa392524" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 18 19:40:20.229036 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:20.229000    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-s26zm"
Sep 18 19:40:38.788819 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:38.788787    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-s26zm"]
Sep 18 19:40:38.789379 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:38.789022    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-s26zm" podUID="410b89b0-faf6-40a1-86ef-d7eafa392524" containerName="kserve-container" containerID="cri-o://389c409cb86829996fc1873214ea243b290386160a1e5e939c7a3f3c9ae298ba" gracePeriod=30
Sep 18 19:40:38.789379 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:38.789111    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-s26zm" podUID="410b89b0-faf6-40a1-86ef-d7eafa392524" containerName="kube-rbac-proxy" containerID="cri-o://a77dceece0ff236cf32042428ae167f23d45152825fbe0a03e856a4f67532db4" gracePeriod=30
Sep 18 19:40:38.835614 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:38.835587    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:39.000696 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:39.000664    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:39.581948 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:39.581917    2532 generic.go:356] "Generic (PLEG): container finished" podID="410b89b0-faf6-40a1-86ef-d7eafa392524" containerID="a77dceece0ff236cf32042428ae167f23d45152825fbe0a03e856a4f67532db4" exitCode=2
Sep 18 19:40:39.582140 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:39.581990    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-s26zm" event={"ID":"410b89b0-faf6-40a1-86ef-d7eafa392524","Type":"ContainerDied","Data":"a77dceece0ff236cf32042428ae167f23d45152825fbe0a03e856a4f67532db4"}
Sep 18 19:40:40.223190 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:40.223153    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-s26zm" podUID="410b89b0-faf6-40a1-86ef-d7eafa392524" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Sep 18 19:40:40.228548 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:40.228524    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-s26zm" podUID="410b89b0-faf6-40a1-86ef-d7eafa392524" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 18 19:40:43.223571 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:43.223546    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-s26zm"
Sep 18 19:40:43.353646 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:43.353568    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/410b89b0-faf6-40a1-86ef-d7eafa392524-kube-rbac-proxy-sar-config\") pod \"410b89b0-faf6-40a1-86ef-d7eafa392524\" (UID: \"410b89b0-faf6-40a1-86ef-d7eafa392524\") "
Sep 18 19:40:43.353646 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:43.353606    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/410b89b0-faf6-40a1-86ef-d7eafa392524-kserve-provision-location\") pod \"410b89b0-faf6-40a1-86ef-d7eafa392524\" (UID: \"410b89b0-faf6-40a1-86ef-d7eafa392524\") "
Sep 18 19:40:43.353646 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:43.353622    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/410b89b0-faf6-40a1-86ef-d7eafa392524-cabundle-cert\") pod \"410b89b0-faf6-40a1-86ef-d7eafa392524\" (UID: \"410b89b0-faf6-40a1-86ef-d7eafa392524\") "
Sep 18 19:40:43.353907 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:43.353651    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrs8r\" (UniqueName: \"kubernetes.io/projected/410b89b0-faf6-40a1-86ef-d7eafa392524-kube-api-access-lrs8r\") pod \"410b89b0-faf6-40a1-86ef-d7eafa392524\" (UID: \"410b89b0-faf6-40a1-86ef-d7eafa392524\") "
Sep 18 19:40:43.353907 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:43.353756    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/410b89b0-faf6-40a1-86ef-d7eafa392524-proxy-tls\") pod \"410b89b0-faf6-40a1-86ef-d7eafa392524\" (UID: \"410b89b0-faf6-40a1-86ef-d7eafa392524\") "
Sep 18 19:40:43.354010 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:43.353947    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/410b89b0-faf6-40a1-86ef-d7eafa392524-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "410b89b0-faf6-40a1-86ef-d7eafa392524" (UID: "410b89b0-faf6-40a1-86ef-d7eafa392524"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:40:43.354010 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:43.353967    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/410b89b0-faf6-40a1-86ef-d7eafa392524-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "410b89b0-faf6-40a1-86ef-d7eafa392524" (UID: "410b89b0-faf6-40a1-86ef-d7eafa392524"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:40:43.354010 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:43.353978    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/410b89b0-faf6-40a1-86ef-d7eafa392524-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "410b89b0-faf6-40a1-86ef-d7eafa392524" (UID: "410b89b0-faf6-40a1-86ef-d7eafa392524"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:40:43.354152 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:43.354024    2532 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/410b89b0-faf6-40a1-86ef-d7eafa392524-kserve-provision-location\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:40:43.355658 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:43.355638    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/410b89b0-faf6-40a1-86ef-d7eafa392524-kube-api-access-lrs8r" (OuterVolumeSpecName: "kube-api-access-lrs8r") pod "410b89b0-faf6-40a1-86ef-d7eafa392524" (UID: "410b89b0-faf6-40a1-86ef-d7eafa392524"). InnerVolumeSpecName "kube-api-access-lrs8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:40:43.355825 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:43.355807    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/410b89b0-faf6-40a1-86ef-d7eafa392524-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "410b89b0-faf6-40a1-86ef-d7eafa392524" (UID: "410b89b0-faf6-40a1-86ef-d7eafa392524"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:40:43.455139 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:43.455117    2532 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/410b89b0-faf6-40a1-86ef-d7eafa392524-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:40:43.455139 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:43.455136    2532 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/410b89b0-faf6-40a1-86ef-d7eafa392524-cabundle-cert\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:40:43.455278 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:43.455145    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrs8r\" (UniqueName: \"kubernetes.io/projected/410b89b0-faf6-40a1-86ef-d7eafa392524-kube-api-access-lrs8r\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:40:43.455278 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:43.455154    2532 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/410b89b0-faf6-40a1-86ef-d7eafa392524-proxy-tls\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:40:43.597842 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:43.597817    2532 generic.go:356] "Generic (PLEG): container finished" podID="410b89b0-faf6-40a1-86ef-d7eafa392524" containerID="389c409cb86829996fc1873214ea243b290386160a1e5e939c7a3f3c9ae298ba" exitCode=0
Sep 18 19:40:43.597952 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:43.597871    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-s26zm"
Sep 18 19:40:43.597952 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:43.597913    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-s26zm" event={"ID":"410b89b0-faf6-40a1-86ef-d7eafa392524","Type":"ContainerDied","Data":"389c409cb86829996fc1873214ea243b290386160a1e5e939c7a3f3c9ae298ba"}
Sep 18 19:40:43.598030 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:43.597973    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-s26zm" event={"ID":"410b89b0-faf6-40a1-86ef-d7eafa392524","Type":"ContainerDied","Data":"889e5329831fb042a17cb530fc434b0a1058b78ab533ba9330af570a11e21752"}
Sep 18 19:40:43.598030 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:43.598000    2532 scope.go:117] "RemoveContainer" containerID="a77dceece0ff236cf32042428ae167f23d45152825fbe0a03e856a4f67532db4"
Sep 18 19:40:43.606563 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:43.606547    2532 scope.go:117] "RemoveContainer" containerID="389c409cb86829996fc1873214ea243b290386160a1e5e939c7a3f3c9ae298ba"
Sep 18 19:40:43.613464 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:43.613447    2532 scope.go:117] "RemoveContainer" containerID="df0b041e08b32812020a057d99ec77b5e0054228bca7c40598254e282d8f62a2"
Sep 18 19:40:43.620623 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:43.620604    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-s26zm"]
Sep 18 19:40:43.621101 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:43.621087    2532 scope.go:117] "RemoveContainer" containerID="a77dceece0ff236cf32042428ae167f23d45152825fbe0a03e856a4f67532db4"
Sep 18 19:40:43.621372 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:40:43.621356    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a77dceece0ff236cf32042428ae167f23d45152825fbe0a03e856a4f67532db4\": container with ID starting with a77dceece0ff236cf32042428ae167f23d45152825fbe0a03e856a4f67532db4 not found: ID does not exist" containerID="a77dceece0ff236cf32042428ae167f23d45152825fbe0a03e856a4f67532db4"
Sep 18 19:40:43.621459 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:43.621379    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a77dceece0ff236cf32042428ae167f23d45152825fbe0a03e856a4f67532db4"} err="failed to get container status \"a77dceece0ff236cf32042428ae167f23d45152825fbe0a03e856a4f67532db4\": rpc error: code = NotFound desc = could not find container \"a77dceece0ff236cf32042428ae167f23d45152825fbe0a03e856a4f67532db4\": container with ID starting with a77dceece0ff236cf32042428ae167f23d45152825fbe0a03e856a4f67532db4 not found: ID does not exist"
Sep 18 19:40:43.621459 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:43.621395    2532 scope.go:117] "RemoveContainer" containerID="389c409cb86829996fc1873214ea243b290386160a1e5e939c7a3f3c9ae298ba"
Sep 18 19:40:43.621634 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:40:43.621616    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"389c409cb86829996fc1873214ea243b290386160a1e5e939c7a3f3c9ae298ba\": container with ID starting with 389c409cb86829996fc1873214ea243b290386160a1e5e939c7a3f3c9ae298ba not found: ID does not exist" containerID="389c409cb86829996fc1873214ea243b290386160a1e5e939c7a3f3c9ae298ba"
Sep 18 19:40:43.621673 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:43.621640    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"389c409cb86829996fc1873214ea243b290386160a1e5e939c7a3f3c9ae298ba"} err="failed to get container status \"389c409cb86829996fc1873214ea243b290386160a1e5e939c7a3f3c9ae298ba\": rpc error: code = NotFound desc = could not find container \"389c409cb86829996fc1873214ea243b290386160a1e5e939c7a3f3c9ae298ba\": container with ID starting with 389c409cb86829996fc1873214ea243b290386160a1e5e939c7a3f3c9ae298ba not found: ID does not exist"
Sep 18 19:40:43.621673 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:43.621656    2532 scope.go:117] "RemoveContainer" containerID="df0b041e08b32812020a057d99ec77b5e0054228bca7c40598254e282d8f62a2"
Sep 18 19:40:43.621896 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:40:43.621877    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df0b041e08b32812020a057d99ec77b5e0054228bca7c40598254e282d8f62a2\": container with ID starting with df0b041e08b32812020a057d99ec77b5e0054228bca7c40598254e282d8f62a2 not found: ID does not exist" containerID="df0b041e08b32812020a057d99ec77b5e0054228bca7c40598254e282d8f62a2"
Sep 18 19:40:43.621949 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:43.621900    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df0b041e08b32812020a057d99ec77b5e0054228bca7c40598254e282d8f62a2"} err="failed to get container status \"df0b041e08b32812020a057d99ec77b5e0054228bca7c40598254e282d8f62a2\": rpc error: code = NotFound desc = could not find container \"df0b041e08b32812020a057d99ec77b5e0054228bca7c40598254e282d8f62a2\": container with ID starting with df0b041e08b32812020a057d99ec77b5e0054228bca7c40598254e282d8f62a2 not found: ID does not exist"
Sep 18 19:40:43.624353 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:43.624329    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-s26zm"]
Sep 18 19:40:43.940540 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:43.940513    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:44.017735 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.017708    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt"]
Sep 18 19:40:44.017947 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.017936    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3" containerName="storage-initializer"
Sep 18 19:40:44.017996 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.017949    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3" containerName="storage-initializer"
Sep 18 19:40:44.017996 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.017960    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="410b89b0-faf6-40a1-86ef-d7eafa392524" containerName="storage-initializer"
Sep 18 19:40:44.017996 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.017965    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="410b89b0-faf6-40a1-86ef-d7eafa392524" containerName="storage-initializer"
Sep 18 19:40:44.017996 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.017974    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95d33a63-27ff-440a-a6ed-72f24ca79498" containerName="kube-rbac-proxy"
Sep 18 19:40:44.017996 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.017979    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="95d33a63-27ff-440a-a6ed-72f24ca79498" containerName="kube-rbac-proxy"
Sep 18 19:40:44.017996 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.017986    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95d33a63-27ff-440a-a6ed-72f24ca79498" containerName="kserve-container"
Sep 18 19:40:44.017996 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.017991    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="95d33a63-27ff-440a-a6ed-72f24ca79498" containerName="kserve-container"
Sep 18 19:40:44.018244 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.018003    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="410b89b0-faf6-40a1-86ef-d7eafa392524" containerName="kube-rbac-proxy"
Sep 18 19:40:44.018244 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.018008    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="410b89b0-faf6-40a1-86ef-d7eafa392524" containerName="kube-rbac-proxy"
Sep 18 19:40:44.018244 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.018017    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95d33a63-27ff-440a-a6ed-72f24ca79498" containerName="storage-initializer"
Sep 18 19:40:44.018244 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.018022    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="95d33a63-27ff-440a-a6ed-72f24ca79498" containerName="storage-initializer"
Sep 18 19:40:44.018244 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.018029    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3" containerName="kserve-container"
Sep 18 19:40:44.018244 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.018034    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3" containerName="kserve-container"
Sep 18 19:40:44.018244 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.018039    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3" containerName="kube-rbac-proxy"
Sep 18 19:40:44.018244 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.018044    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3" containerName="kube-rbac-proxy"
Sep 18 19:40:44.018244 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.018050    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="410b89b0-faf6-40a1-86ef-d7eafa392524" containerName="kserve-container"
Sep 18 19:40:44.018244 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.018067    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="410b89b0-faf6-40a1-86ef-d7eafa392524" containerName="kserve-container"
Sep 18 19:40:44.018244 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.018104    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3" containerName="kube-rbac-proxy"
Sep 18 19:40:44.018244 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.018111    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="410b89b0-faf6-40a1-86ef-d7eafa392524" containerName="kserve-container"
Sep 18 19:40:44.018244 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.018116    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="410b89b0-faf6-40a1-86ef-d7eafa392524" containerName="kube-rbac-proxy"
Sep 18 19:40:44.018244 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.018123    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1243a4e-c7fd-4be3-b73f-3c4ec714f5f3" containerName="kserve-container"
Sep 18 19:40:44.018244 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.018129    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="95d33a63-27ff-440a-a6ed-72f24ca79498" containerName="kserve-container"
Sep 18 19:40:44.018244 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.018134    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="95d33a63-27ff-440a-a6ed-72f24ca79498" containerName="kube-rbac-proxy"
Sep 18 19:40:44.022357 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.022337    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-x8hjf"]
Sep 18 19:40:44.022465 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.022448    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt"
Sep 18 19:40:44.025301 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.025283    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 18 19:40:44.025402 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.025288    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 18 19:40:44.025402 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.025322    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-x8hjf"
Sep 18 19:40:44.025618 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.025602    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 19:40:44.025809 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.025792    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 19:40:44.025886 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.025796    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 19:40:44.025886 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.025856    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-ct9wb\""
Sep 18 19:40:44.026002 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.025978    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 19:40:44.027646 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.027629    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 18 19:40:44.037806 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.037787    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt"]
Sep 18 19:40:44.040594 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.040563    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-x8hjf"]
Sep 18 19:40:44.053188 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.053149    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:44.159006 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.158984    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl47s\" (UniqueName: \"kubernetes.io/projected/1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc-kube-api-access-wl47s\") pod \"isvc-canary-promote-predictor-75f64ff6c9-x8hjf\" (UID: \"1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-x8hjf"
Sep 18 19:40:44.159203 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.159011    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/abaa8cdf-c42d-439a-824b-55139c1b8fb6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt\" (UID: \"abaa8cdf-c42d-439a-824b-55139c1b8fb6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt"
Sep 18 19:40:44.159203 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.159032    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc-cabundle-cert\") pod \"isvc-canary-promote-predictor-75f64ff6c9-x8hjf\" (UID: \"1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-x8hjf"
Sep 18 19:40:44.159203 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.159070    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-75f64ff6c9-x8hjf\" (UID: \"1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-x8hjf"
Sep 18 19:40:44.159203 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.159107    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc-kserve-provision-location\") pod \"isvc-canary-promote-predictor-75f64ff6c9-x8hjf\" (UID: \"1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-x8hjf"
Sep 18 19:40:44.159203 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.159134    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/abaa8cdf-c42d-439a-824b-55139c1b8fb6-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt\" (UID: \"abaa8cdf-c42d-439a-824b-55139c1b8fb6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt"
Sep 18 19:40:44.159203 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.159157    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/abaa8cdf-c42d-439a-824b-55139c1b8fb6-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt\" (UID: \"abaa8cdf-c42d-439a-824b-55139c1b8fb6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt"
Sep 18 19:40:44.159203 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.159174    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abaa8cdf-c42d-439a-824b-55139c1b8fb6-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt\" (UID: \"abaa8cdf-c42d-439a-824b-55139c1b8fb6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt"
Sep 18 19:40:44.159203 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.159193    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc-proxy-tls\") pod \"isvc-canary-promote-predictor-75f64ff6c9-x8hjf\" (UID: \"1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-x8hjf"
Sep 18 19:40:44.159475 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.159213    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cph7n\" (UniqueName: \"kubernetes.io/projected/abaa8cdf-c42d-439a-824b-55139c1b8fb6-kube-api-access-cph7n\") pod \"isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt\" (UID: \"abaa8cdf-c42d-439a-824b-55139c1b8fb6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt"
Sep 18 19:40:44.259917 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.259859    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc-proxy-tls\") pod \"isvc-canary-promote-predictor-75f64ff6c9-x8hjf\" (UID: \"1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-x8hjf"
Sep 18 19:40:44.259917 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.259887    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cph7n\" (UniqueName: \"kubernetes.io/projected/abaa8cdf-c42d-439a-824b-55139c1b8fb6-kube-api-access-cph7n\") pod \"isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt\" (UID: \"abaa8cdf-c42d-439a-824b-55139c1b8fb6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt"
Sep 18 19:40:44.259917 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.259913    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl47s\" (UniqueName: \"kubernetes.io/projected/1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc-kube-api-access-wl47s\") pod \"isvc-canary-promote-predictor-75f64ff6c9-x8hjf\" (UID: \"1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-x8hjf"
Sep 18 19:40:44.260415 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.260017    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/abaa8cdf-c42d-439a-824b-55139c1b8fb6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt\" (UID: \"abaa8cdf-c42d-439a-824b-55139c1b8fb6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt"
Sep 18 19:40:44.260415 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.260050    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc-cabundle-cert\") pod \"isvc-canary-promote-predictor-75f64ff6c9-x8hjf\" (UID: \"1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-x8hjf"
Sep 18 19:40:44.260415 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.260107    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-75f64ff6c9-x8hjf\" (UID: \"1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-x8hjf"
Sep 18 19:40:44.260415 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.260149    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc-kserve-provision-location\") pod \"isvc-canary-promote-predictor-75f64ff6c9-x8hjf\" (UID: \"1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-x8hjf"
Sep 18 19:40:44.260415 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.260195    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/abaa8cdf-c42d-439a-824b-55139c1b8fb6-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt\" (UID: \"abaa8cdf-c42d-439a-824b-55139c1b8fb6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt"
Sep 18 19:40:44.260415 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.260237    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/abaa8cdf-c42d-439a-824b-55139c1b8fb6-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt\" (UID: \"abaa8cdf-c42d-439a-824b-55139c1b8fb6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt"
Sep 18 19:40:44.260415 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.260271    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abaa8cdf-c42d-439a-824b-55139c1b8fb6-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt\" (UID: \"abaa8cdf-c42d-439a-824b-55139c1b8fb6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt"
Sep 18 19:40:44.260779 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.260592    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/abaa8cdf-c42d-439a-824b-55139c1b8fb6-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt\" (UID: \"abaa8cdf-c42d-439a-824b-55139c1b8fb6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt"
Sep 18 19:40:44.260911 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.260891    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-75f64ff6c9-x8hjf\" (UID: \"1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-x8hjf"
Sep 18 19:40:44.260911 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.260905    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc-cabundle-cert\") pod \"isvc-canary-promote-predictor-75f64ff6c9-x8hjf\" (UID: \"1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-x8hjf"
Sep 18 19:40:44.261027 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.260916    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/abaa8cdf-c42d-439a-824b-55139c1b8fb6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt\" (UID: \"abaa8cdf-c42d-439a-824b-55139c1b8fb6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt"
Sep 18 19:40:44.261027 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.260894    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc-kserve-provision-location\") pod \"isvc-canary-promote-predictor-75f64ff6c9-x8hjf\" (UID: \"1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-x8hjf"
Sep 18 19:40:44.261120 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.261083    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/abaa8cdf-c42d-439a-824b-55139c1b8fb6-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt\" (UID: \"abaa8cdf-c42d-439a-824b-55139c1b8fb6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt"
Sep 18 19:40:44.262506 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.262486    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc-proxy-tls\") pod \"isvc-canary-promote-predictor-75f64ff6c9-x8hjf\" (UID: \"1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-x8hjf"
Sep 18 19:40:44.262598 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.262582    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abaa8cdf-c42d-439a-824b-55139c1b8fb6-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt\" (UID: \"abaa8cdf-c42d-439a-824b-55139c1b8fb6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt"
Sep 18 19:40:44.267584 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.267563    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:44.268586 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.268562    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl47s\" (UniqueName: \"kubernetes.io/projected/1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc-kube-api-access-wl47s\") pod \"isvc-canary-promote-predictor-75f64ff6c9-x8hjf\" (UID: \"1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-x8hjf"
Sep 18 19:40:44.268807 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.268787    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cph7n\" (UniqueName: \"kubernetes.io/projected/abaa8cdf-c42d-439a-824b-55139c1b8fb6-kube-api-access-cph7n\") pod \"isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt\" (UID: \"abaa8cdf-c42d-439a-824b-55139c1b8fb6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt"
Sep 18 19:40:44.333205 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.333179    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt"
Sep 18 19:40:44.341011 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.340982    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-x8hjf"
Sep 18 19:40:44.422621 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.422596    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:44.644774 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.644749    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-x8hjf"]
Sep 18 19:40:44.645920 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:40:44.645893    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ebc87e5_aef7_44dc_9d8e_0ad3410a41cc.slice/crio-07be89ca2abfb724078c592fe70bd6118c82189e7eb9c2fb4b64deacff79738f WatchSource:0}: Error finding container 07be89ca2abfb724078c592fe70bd6118c82189e7eb9c2fb4b64deacff79738f: Status 404 returned error can't find the container with id 07be89ca2abfb724078c592fe70bd6118c82189e7eb9c2fb4b64deacff79738f
Sep 18 19:40:44.650484 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:44.650464    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt"]
Sep 18 19:40:44.653322 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:40:44.653248    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabaa8cdf_c42d_439a_824b_55139c1b8fb6.slice/crio-386f0d32fb97a8cc809b629db0c8a2a1e689986613e5355c109524b6109e0678 WatchSource:0}: Error finding container 386f0d32fb97a8cc809b629db0c8a2a1e689986613e5355c109524b6109e0678: Status 404 returned error can't find the container with id 386f0d32fb97a8cc809b629db0c8a2a1e689986613e5355c109524b6109e0678
Sep 18 19:40:45.611878 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:45.611844    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt" event={"ID":"abaa8cdf-c42d-439a-824b-55139c1b8fb6","Type":"ContainerStarted","Data":"416b413ea6eb8530a0312ed9a08a842cfab041c966bccb35b633896a8ddd3863"}
Sep 18 19:40:45.612298 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:45.611943    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt" event={"ID":"abaa8cdf-c42d-439a-824b-55139c1b8fb6","Type":"ContainerStarted","Data":"386f0d32fb97a8cc809b629db0c8a2a1e689986613e5355c109524b6109e0678"}
Sep 18 19:40:45.613490 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:45.613463    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-x8hjf" event={"ID":"1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc","Type":"ContainerStarted","Data":"568d704aaa4cd1454285d1d44db33c0384947d415257f9037120d3609259b600"}
Sep 18 19:40:45.613599 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:45.613528    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-x8hjf" event={"ID":"1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc","Type":"ContainerStarted","Data":"07be89ca2abfb724078c592fe70bd6118c82189e7eb9c2fb4b64deacff79738f"}
Sep 18 19:40:45.619982 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:45.619960    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="410b89b0-faf6-40a1-86ef-d7eafa392524" path="/var/lib/kubelet/pods/410b89b0-faf6-40a1-86ef-d7eafa392524/volumes"
Sep 18 19:40:48.625968 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:48.625941    2532 generic.go:356] "Generic (PLEG): container finished" podID="abaa8cdf-c42d-439a-824b-55139c1b8fb6" containerID="416b413ea6eb8530a0312ed9a08a842cfab041c966bccb35b633896a8ddd3863" exitCode=0
Sep 18 19:40:48.626316 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:48.626021    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt" event={"ID":"abaa8cdf-c42d-439a-824b-55139c1b8fb6","Type":"ContainerDied","Data":"416b413ea6eb8530a0312ed9a08a842cfab041c966bccb35b633896a8ddd3863"}
Sep 18 19:40:48.627649 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:48.627632    2532 generic.go:356] "Generic (PLEG): container finished" podID="1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc" containerID="568d704aaa4cd1454285d1d44db33c0384947d415257f9037120d3609259b600" exitCode=0
Sep 18 19:40:48.627719 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:48.627690    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-x8hjf" event={"ID":"1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc","Type":"ContainerDied","Data":"568d704aaa4cd1454285d1d44db33c0384947d415257f9037120d3609259b600"}
Sep 18 19:40:49.634291 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:49.634253    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt" event={"ID":"abaa8cdf-c42d-439a-824b-55139c1b8fb6","Type":"ContainerStarted","Data":"52f90640fcafe7688e74a1f6e7d9eb726800fdd794568b37cbd92a807f97ecb1"}
Sep 18 19:40:49.634734 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:49.634316    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt" event={"ID":"abaa8cdf-c42d-439a-824b-55139c1b8fb6","Type":"ContainerStarted","Data":"7e75097adb413f3f70313096758e00ac485a7738b9622bde009ebc87626c4cdf"}
Sep 18 19:40:49.634854 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:49.634832    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt"
Sep 18 19:40:49.634927 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:49.634920    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt"
Sep 18 19:40:49.636341 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:49.636315    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt" podUID="abaa8cdf-c42d-439a-824b-55139c1b8fb6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 18 19:40:49.636767 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:49.636749    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-x8hjf" event={"ID":"1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc","Type":"ContainerStarted","Data":"715b13b01cb190fb8c8e8915fcfae82914f6d54335a2c724dcba97bb853c105a"}
Sep 18 19:40:49.636870 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:49.636791    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-x8hjf" event={"ID":"1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc","Type":"ContainerStarted","Data":"c5c68f7e213f85d2303a569c75f1ff2b15d38b368c7de600742577c33f4919f5"}
Sep 18 19:40:49.636971 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:49.636955    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-x8hjf"
Sep 18 19:40:49.637031 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:49.636997    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-x8hjf"
Sep 18 19:40:49.638080 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:49.638040    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-x8hjf" podUID="1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 18 19:40:49.660908 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:49.660863    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt" podStartSLOduration=2.793804165 podStartE2EDuration="6.660847347s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.867043182s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:40:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:40:49.659975403 +0000 UTC m=+1616.580479260" watchObservedRunningTime="2026-09-18 19:40:49.660847347 +0000 UTC m=+1616.581351206"
Sep 18 19:40:49.684558 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:49.684519    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-x8hjf" podStartSLOduration=2.886145404 podStartE2EDuration="6.684506038s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.798360634s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:40:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:40:49.682853385 +0000 UTC m=+1616.603357255" watchObservedRunningTime="2026-09-18 19:40:49.684506038 +0000 UTC m=+1616.605009894"
Sep 18 19:40:50.640586 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:50.640550    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-x8hjf" podUID="1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 18 19:40:50.641033 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:50.640731    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt" podUID="abaa8cdf-c42d-439a-824b-55139c1b8fb6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 18 19:40:53.652665 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:53.652638    2532 scope.go:117] "RemoveContainer" containerID="c30ba5b1bde034f226361d0110cc4129d234d81c4cb0982f73e69dcd0301ac6e"
Sep 18 19:40:53.660477 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:53.660451    2532 scope.go:117] "RemoveContainer" containerID="55b8a85f7900d3f973c33b08dcab23059342fa811729743498ff6cea6d068fa4"
Sep 18 19:40:55.645556 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:55.645518    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt"
Sep 18 19:40:55.645966 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:55.645801    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-x8hjf"
Sep 18 19:40:55.646191 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:55.646162    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt" podUID="abaa8cdf-c42d-439a-824b-55139c1b8fb6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 18 19:40:55.646489 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:40:55.646466    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-x8hjf" podUID="1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 18 19:41:05.646509 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:41:05.646476    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt" podUID="abaa8cdf-c42d-439a-824b-55139c1b8fb6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 18 19:41:05.646898 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:41:05.646487    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-x8hjf" podUID="1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 18 19:41:15.647101 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:41:15.647047    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-x8hjf" podUID="1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 18 19:41:15.647562 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:41:15.647047    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt" podUID="abaa8cdf-c42d-439a-824b-55139c1b8fb6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 18 19:41:15.890242 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:41:15.890208    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:41:25.646621 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:41:25.646581    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt" podUID="abaa8cdf-c42d-439a-824b-55139c1b8fb6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 18 19:41:25.647009 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:41:25.646581    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-x8hjf" podUID="1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 18 19:41:35.646921 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:41:35.646885    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt" podUID="abaa8cdf-c42d-439a-824b-55139c1b8fb6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 18 19:41:35.647384 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:41:35.646892    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-x8hjf" podUID="1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 18 19:41:45.646261 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:41:45.646221    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt" podUID="abaa8cdf-c42d-439a-824b-55139c1b8fb6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 18 19:41:45.646707 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:41:45.646291    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-x8hjf" podUID="1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 18 19:41:55.646258 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:41:55.646218    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt" podUID="abaa8cdf-c42d-439a-824b-55139c1b8fb6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 18 19:41:55.646812 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:41:55.646638    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-x8hjf" podUID="1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 18 19:42:05.646911 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:42:05.646881    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-x8hjf"
Sep 18 19:42:05.647346 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:42:05.646979    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt"
Sep 18 19:42:14.101162 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:42:14.101128    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-x8hjf"]
Sep 18 19:42:14.101550 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:42:14.101388    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-x8hjf" podUID="1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc" containerName="kserve-container" containerID="cri-o://c5c68f7e213f85d2303a569c75f1ff2b15d38b368c7de600742577c33f4919f5" gracePeriod=30
Sep 18 19:42:14.101550 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:42:14.101396    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-x8hjf" podUID="1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc" containerName="kube-rbac-proxy" containerID="cri-o://715b13b01cb190fb8c8e8915fcfae82914f6d54335a2c724dcba97bb853c105a" gracePeriod=30
Sep 18 19:42:14.117082 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:42:14.117041    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:14.956299 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:42:14.956267    2532 generic.go:356] "Generic (PLEG): container finished" podID="1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc" containerID="715b13b01cb190fb8c8e8915fcfae82914f6d54335a2c724dcba97bb853c105a" exitCode=2
Sep 18 19:42:14.956452 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:42:14.956342    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-x8hjf" event={"ID":"1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc","Type":"ContainerDied","Data":"715b13b01cb190fb8c8e8915fcfae82914f6d54335a2c724dcba97bb853c105a"}
Sep 18 19:42:15.641656 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:42:15.641616    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-x8hjf" podUID="1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Sep 18 19:42:15.646931 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:42:15.646891    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-x8hjf" podUID="1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 18 19:42:18.539015 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:42:18.538992    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-x8hjf"
Sep 18 19:42:18.641296 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:42:18.641230    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wl47s\" (UniqueName: \"kubernetes.io/projected/1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc-kube-api-access-wl47s\") pod \"1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc\" (UID: \"1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc\") "
Sep 18 19:42:18.641296 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:42:18.641273    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc-kube-rbac-proxy-sar-config\") pod \"1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc\" (UID: \"1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc\") "
Sep 18 19:42:18.641493 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:42:18.641326    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc-cabundle-cert\") pod \"1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc\" (UID: \"1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc\") "
Sep 18 19:42:18.641493 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:42:18.641353    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc-kserve-provision-location\") pod \"1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc\" (UID: \"1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc\") "
Sep 18 19:42:18.641493 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:42:18.641375    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc-proxy-tls\") pod \"1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc\" (UID: \"1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc\") "
Sep 18 19:42:18.641692 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:42:18.641655    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc" (UID: "1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:42:18.641692 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:42:18.641673    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc" (UID: "1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:42:18.641840 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:42:18.641670    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc" (UID: "1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:42:18.643355 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:42:18.643333    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc-kube-api-access-wl47s" (OuterVolumeSpecName: "kube-api-access-wl47s") pod "1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc" (UID: "1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc"). InnerVolumeSpecName "kube-api-access-wl47s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:42:18.643433 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:42:18.643405    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc" (UID: "1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:42:18.741853 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:42:18.741829    2532 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc-cabundle-cert\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:42:18.741853 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:42:18.741850    2532 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc-kserve-provision-location\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:42:18.741978 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:42:18.741860    2532 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc-proxy-tls\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:42:18.741978 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:42:18.741870    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl47s\" (UniqueName: \"kubernetes.io/projected/1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc-kube-api-access-wl47s\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:42:18.741978 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:42:18.741878    2532 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:42:18.972205 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:42:18.972177    2532 generic.go:356] "Generic (PLEG): container finished" podID="1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc" containerID="c5c68f7e213f85d2303a569c75f1ff2b15d38b368c7de600742577c33f4919f5" exitCode=0
Sep 18 19:42:18.972356 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:42:18.972232    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-x8hjf"
Sep 18 19:42:18.972356 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:42:18.972260    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-x8hjf" event={"ID":"1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc","Type":"ContainerDied","Data":"c5c68f7e213f85d2303a569c75f1ff2b15d38b368c7de600742577c33f4919f5"}
Sep 18 19:42:18.972356 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:42:18.972320    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-x8hjf" event={"ID":"1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc","Type":"ContainerDied","Data":"07be89ca2abfb724078c592fe70bd6118c82189e7eb9c2fb4b64deacff79738f"}
Sep 18 19:42:18.972356 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:42:18.972348    2532 scope.go:117] "RemoveContainer" containerID="715b13b01cb190fb8c8e8915fcfae82914f6d54335a2c724dcba97bb853c105a"
Sep 18 19:42:18.980251 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:42:18.980228    2532 scope.go:117] "RemoveContainer" containerID="c5c68f7e213f85d2303a569c75f1ff2b15d38b368c7de600742577c33f4919f5"
Sep 18 19:42:18.987025 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:42:18.987007    2532 scope.go:117] "RemoveContainer" containerID="568d704aaa4cd1454285d1d44db33c0384947d415257f9037120d3609259b600"
Sep 18 19:42:18.993742 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:42:18.993726    2532 scope.go:117] "RemoveContainer" containerID="715b13b01cb190fb8c8e8915fcfae82914f6d54335a2c724dcba97bb853c105a"
Sep 18 19:42:18.993999 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:42:18.993981    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"715b13b01cb190fb8c8e8915fcfae82914f6d54335a2c724dcba97bb853c105a\": container with ID starting with 715b13b01cb190fb8c8e8915fcfae82914f6d54335a2c724dcba97bb853c105a not found: ID does not exist" containerID="715b13b01cb190fb8c8e8915fcfae82914f6d54335a2c724dcba97bb853c105a"
Sep 18 19:42:18.994092 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:42:18.994010    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"715b13b01cb190fb8c8e8915fcfae82914f6d54335a2c724dcba97bb853c105a"} err="failed to get container status \"715b13b01cb190fb8c8e8915fcfae82914f6d54335a2c724dcba97bb853c105a\": rpc error: code = NotFound desc = could not find container \"715b13b01cb190fb8c8e8915fcfae82914f6d54335a2c724dcba97bb853c105a\": container with ID starting with 715b13b01cb190fb8c8e8915fcfae82914f6d54335a2c724dcba97bb853c105a not found: ID does not exist"
Sep 18 19:42:18.994092 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:42:18.994034    2532 scope.go:117] "RemoveContainer" containerID="c5c68f7e213f85d2303a569c75f1ff2b15d38b368c7de600742577c33f4919f5"
Sep 18 19:42:18.994295 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:42:18.994278    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5c68f7e213f85d2303a569c75f1ff2b15d38b368c7de600742577c33f4919f5\": container with ID starting with c5c68f7e213f85d2303a569c75f1ff2b15d38b368c7de600742577c33f4919f5 not found: ID does not exist" containerID="c5c68f7e213f85d2303a569c75f1ff2b15d38b368c7de600742577c33f4919f5"
Sep 18 19:42:18.994343 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:42:18.994302    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5c68f7e213f85d2303a569c75f1ff2b15d38b368c7de600742577c33f4919f5"} err="failed to get container status \"c5c68f7e213f85d2303a569c75f1ff2b15d38b368c7de600742577c33f4919f5\": rpc error: code = NotFound desc = could not find container \"c5c68f7e213f85d2303a569c75f1ff2b15d38b368c7de600742577c33f4919f5\": container with ID starting with c5c68f7e213f85d2303a569c75f1ff2b15d38b368c7de600742577c33f4919f5 not found: ID does not exist"
Sep 18 19:42:18.994343 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:42:18.994317    2532 scope.go:117] "RemoveContainer" containerID="568d704aaa4cd1454285d1d44db33c0384947d415257f9037120d3609259b600"
Sep 18 19:42:18.994526 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:42:18.994510    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"568d704aaa4cd1454285d1d44db33c0384947d415257f9037120d3609259b600\": container with ID starting with 568d704aaa4cd1454285d1d44db33c0384947d415257f9037120d3609259b600 not found: ID does not exist" containerID="568d704aaa4cd1454285d1d44db33c0384947d415257f9037120d3609259b600"
Sep 18 19:42:18.994576 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:42:18.994535    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"568d704aaa4cd1454285d1d44db33c0384947d415257f9037120d3609259b600"} err="failed to get container status \"568d704aaa4cd1454285d1d44db33c0384947d415257f9037120d3609259b600\": rpc error: code = NotFound desc = could not find container \"568d704aaa4cd1454285d1d44db33c0384947d415257f9037120d3609259b600\": container with ID starting with 568d704aaa4cd1454285d1d44db33c0384947d415257f9037120d3609259b600 not found: ID does not exist"
Sep 18 19:42:19.005141 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:42:19.005122    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-x8hjf"]
Sep 18 19:42:19.010386 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:42:19.010368    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-x8hjf"]
Sep 18 19:42:19.621103 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:42:19.621050    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc" path="/var/lib/kubelet/pods/1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc/volumes"
Sep 18 19:43:44.847143 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:44.847109    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt"]
Sep 18 19:43:44.847652 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:44.847333    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt" podUID="abaa8cdf-c42d-439a-824b-55139c1b8fb6" containerName="kserve-container" containerID="cri-o://7e75097adb413f3f70313096758e00ac485a7738b9622bde009ebc87626c4cdf" gracePeriod=30
Sep 18 19:43:44.847652 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:44.847390    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt" podUID="abaa8cdf-c42d-439a-824b-55139c1b8fb6" containerName="kube-rbac-proxy" containerID="cri-o://52f90640fcafe7688e74a1f6e7d9eb726800fdd794568b37cbd92a807f97ecb1" gracePeriod=30
Sep 18 19:43:44.941368 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:44.941339    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:45.279094 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:45.279047    2532 generic.go:356] "Generic (PLEG): container finished" podID="abaa8cdf-c42d-439a-824b-55139c1b8fb6" containerID="52f90640fcafe7688e74a1f6e7d9eb726800fdd794568b37cbd92a807f97ecb1" exitCode=2
Sep 18 19:43:45.279271 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:45.279127    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt" event={"ID":"abaa8cdf-c42d-439a-824b-55139c1b8fb6","Type":"ContainerDied","Data":"52f90640fcafe7688e74a1f6e7d9eb726800fdd794568b37cbd92a807f97ecb1"}
Sep 18 19:43:45.640975 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:45.640890    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt" podUID="abaa8cdf-c42d-439a-824b-55139c1b8fb6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Sep 18 19:43:45.646349 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:45.646328    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt" podUID="abaa8cdf-c42d-439a-824b-55139c1b8fb6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 18 19:43:49.082299 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:49.082279    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt"
Sep 18 19:43:49.164156 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:49.164135    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cph7n\" (UniqueName: \"kubernetes.io/projected/abaa8cdf-c42d-439a-824b-55139c1b8fb6-kube-api-access-cph7n\") pod \"abaa8cdf-c42d-439a-824b-55139c1b8fb6\" (UID: \"abaa8cdf-c42d-439a-824b-55139c1b8fb6\") "
Sep 18 19:43:49.164302 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:49.164173    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/abaa8cdf-c42d-439a-824b-55139c1b8fb6-kserve-provision-location\") pod \"abaa8cdf-c42d-439a-824b-55139c1b8fb6\" (UID: \"abaa8cdf-c42d-439a-824b-55139c1b8fb6\") "
Sep 18 19:43:49.164302 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:49.164210    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abaa8cdf-c42d-439a-824b-55139c1b8fb6-proxy-tls\") pod \"abaa8cdf-c42d-439a-824b-55139c1b8fb6\" (UID: \"abaa8cdf-c42d-439a-824b-55139c1b8fb6\") "
Sep 18 19:43:49.164429 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:49.164373    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/abaa8cdf-c42d-439a-824b-55139c1b8fb6-kube-rbac-proxy-sar-config\") pod \"abaa8cdf-c42d-439a-824b-55139c1b8fb6\" (UID: \"abaa8cdf-c42d-439a-824b-55139c1b8fb6\") "
Sep 18 19:43:49.164429 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:49.164426    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/abaa8cdf-c42d-439a-824b-55139c1b8fb6-cabundle-cert\") pod \"abaa8cdf-c42d-439a-824b-55139c1b8fb6\" (UID: \"abaa8cdf-c42d-439a-824b-55139c1b8fb6\") "
Sep 18 19:43:49.164547 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:49.164460    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abaa8cdf-c42d-439a-824b-55139c1b8fb6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "abaa8cdf-c42d-439a-824b-55139c1b8fb6" (UID: "abaa8cdf-c42d-439a-824b-55139c1b8fb6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:43:49.164671 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:49.164593    2532 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/abaa8cdf-c42d-439a-824b-55139c1b8fb6-kserve-provision-location\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:43:49.164782 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:49.164726    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abaa8cdf-c42d-439a-824b-55139c1b8fb6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "abaa8cdf-c42d-439a-824b-55139c1b8fb6" (UID: "abaa8cdf-c42d-439a-824b-55139c1b8fb6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:43:49.164782 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:49.164751    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abaa8cdf-c42d-439a-824b-55139c1b8fb6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "abaa8cdf-c42d-439a-824b-55139c1b8fb6" (UID: "abaa8cdf-c42d-439a-824b-55139c1b8fb6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:43:49.166192 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:49.166172    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abaa8cdf-c42d-439a-824b-55139c1b8fb6-kube-api-access-cph7n" (OuterVolumeSpecName: "kube-api-access-cph7n") pod "abaa8cdf-c42d-439a-824b-55139c1b8fb6" (UID: "abaa8cdf-c42d-439a-824b-55139c1b8fb6"). InnerVolumeSpecName "kube-api-access-cph7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:43:49.166350 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:49.166328    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abaa8cdf-c42d-439a-824b-55139c1b8fb6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "abaa8cdf-c42d-439a-824b-55139c1b8fb6" (UID: "abaa8cdf-c42d-439a-824b-55139c1b8fb6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:43:49.265807 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:49.265743    2532 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/abaa8cdf-c42d-439a-824b-55139c1b8fb6-cabundle-cert\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:43:49.265807 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:49.265773    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cph7n\" (UniqueName: \"kubernetes.io/projected/abaa8cdf-c42d-439a-824b-55139c1b8fb6-kube-api-access-cph7n\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:43:49.265807 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:49.265783    2532 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abaa8cdf-c42d-439a-824b-55139c1b8fb6-proxy-tls\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:43:49.265807 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:49.265791    2532 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/abaa8cdf-c42d-439a-824b-55139c1b8fb6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:43:49.296290 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:49.296262    2532 generic.go:356] "Generic (PLEG): container finished" podID="abaa8cdf-c42d-439a-824b-55139c1b8fb6" containerID="7e75097adb413f3f70313096758e00ac485a7738b9622bde009ebc87626c4cdf" exitCode=0
Sep 18 19:43:49.296404 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:49.296323    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt"
Sep 18 19:43:49.296404 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:49.296350    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt" event={"ID":"abaa8cdf-c42d-439a-824b-55139c1b8fb6","Type":"ContainerDied","Data":"7e75097adb413f3f70313096758e00ac485a7738b9622bde009ebc87626c4cdf"}
Sep 18 19:43:49.296526 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:49.296429    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt" event={"ID":"abaa8cdf-c42d-439a-824b-55139c1b8fb6","Type":"ContainerDied","Data":"386f0d32fb97a8cc809b629db0c8a2a1e689986613e5355c109524b6109e0678"}
Sep 18 19:43:49.296526 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:49.296501    2532 scope.go:117] "RemoveContainer" containerID="52f90640fcafe7688e74a1f6e7d9eb726800fdd794568b37cbd92a807f97ecb1"
Sep 18 19:43:49.307956 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:49.305335    2532 scope.go:117] "RemoveContainer" containerID="7e75097adb413f3f70313096758e00ac485a7738b9622bde009ebc87626c4cdf"
Sep 18 19:43:49.315235 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:49.315216    2532 scope.go:117] "RemoveContainer" containerID="416b413ea6eb8530a0312ed9a08a842cfab041c966bccb35b633896a8ddd3863"
Sep 18 19:43:49.321710 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:49.321692    2532 scope.go:117] "RemoveContainer" containerID="52f90640fcafe7688e74a1f6e7d9eb726800fdd794568b37cbd92a807f97ecb1"
Sep 18 19:43:49.321964 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:43:49.321947    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52f90640fcafe7688e74a1f6e7d9eb726800fdd794568b37cbd92a807f97ecb1\": container with ID starting with 52f90640fcafe7688e74a1f6e7d9eb726800fdd794568b37cbd92a807f97ecb1 not found: ID does not exist" containerID="52f90640fcafe7688e74a1f6e7d9eb726800fdd794568b37cbd92a807f97ecb1"
Sep 18 19:43:49.322023 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:49.321970    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52f90640fcafe7688e74a1f6e7d9eb726800fdd794568b37cbd92a807f97ecb1"} err="failed to get container status \"52f90640fcafe7688e74a1f6e7d9eb726800fdd794568b37cbd92a807f97ecb1\": rpc error: code = NotFound desc = could not find container \"52f90640fcafe7688e74a1f6e7d9eb726800fdd794568b37cbd92a807f97ecb1\": container with ID starting with 52f90640fcafe7688e74a1f6e7d9eb726800fdd794568b37cbd92a807f97ecb1 not found: ID does not exist"
Sep 18 19:43:49.322023 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:49.321984    2532 scope.go:117] "RemoveContainer" containerID="7e75097adb413f3f70313096758e00ac485a7738b9622bde009ebc87626c4cdf"
Sep 18 19:43:49.322234 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:43:49.322217    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e75097adb413f3f70313096758e00ac485a7738b9622bde009ebc87626c4cdf\": container with ID starting with 7e75097adb413f3f70313096758e00ac485a7738b9622bde009ebc87626c4cdf not found: ID does not exist" containerID="7e75097adb413f3f70313096758e00ac485a7738b9622bde009ebc87626c4cdf"
Sep 18 19:43:49.322275 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:49.322239    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e75097adb413f3f70313096758e00ac485a7738b9622bde009ebc87626c4cdf"} err="failed to get container status \"7e75097adb413f3f70313096758e00ac485a7738b9622bde009ebc87626c4cdf\": rpc error: code = NotFound desc = could not find container \"7e75097adb413f3f70313096758e00ac485a7738b9622bde009ebc87626c4cdf\": container with ID starting with 7e75097adb413f3f70313096758e00ac485a7738b9622bde009ebc87626c4cdf not found: ID does not exist"
Sep 18 19:43:49.322275 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:49.322252    2532 scope.go:117] "RemoveContainer" containerID="416b413ea6eb8530a0312ed9a08a842cfab041c966bccb35b633896a8ddd3863"
Sep 18 19:43:49.322476 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:43:49.322460    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"416b413ea6eb8530a0312ed9a08a842cfab041c966bccb35b633896a8ddd3863\": container with ID starting with 416b413ea6eb8530a0312ed9a08a842cfab041c966bccb35b633896a8ddd3863 not found: ID does not exist" containerID="416b413ea6eb8530a0312ed9a08a842cfab041c966bccb35b633896a8ddd3863"
Sep 18 19:43:49.322524 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:49.322478    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"416b413ea6eb8530a0312ed9a08a842cfab041c966bccb35b633896a8ddd3863"} err="failed to get container status \"416b413ea6eb8530a0312ed9a08a842cfab041c966bccb35b633896a8ddd3863\": rpc error: code = NotFound desc = could not find container \"416b413ea6eb8530a0312ed9a08a842cfab041c966bccb35b633896a8ddd3863\": container with ID starting with 416b413ea6eb8530a0312ed9a08a842cfab041c966bccb35b633896a8ddd3863 not found: ID does not exist"
Sep 18 19:43:49.333614 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:49.333590    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt"]
Sep 18 19:43:49.341780 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:49.341758    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-4gqxt"]
Sep 18 19:43:49.621297 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:49.621228    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abaa8cdf-c42d-439a-824b-55139c1b8fb6" path="/var/lib/kubelet/pods/abaa8cdf-c42d-439a-824b-55139c1b8fb6/volumes"
Sep 18 19:43:53.608620 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:53.608589    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sksw8_2cef0f4e-e44f-4996-8561-da9d261036c7/ovn-acl-logging/0.log"
Sep 18 19:43:53.609434 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:53.609410    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sksw8_2cef0f4e-e44f-4996-8561-da9d261036c7/ovn-acl-logging/0.log"
Sep 18 19:43:59.085083 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:59.085023    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:59.132361 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:59.132330    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-8f8f5f7df-gdh57"]
Sep 18 19:43:59.132561 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:59.132550    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc" containerName="kube-rbac-proxy"
Sep 18 19:43:59.132615 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:59.132563    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc" containerName="kube-rbac-proxy"
Sep 18 19:43:59.132615 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:59.132572    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc" containerName="kserve-container"
Sep 18 19:43:59.132615 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:59.132578    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc" containerName="kserve-container"
Sep 18 19:43:59.132615 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:59.132585    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abaa8cdf-c42d-439a-824b-55139c1b8fb6" containerName="kserve-container"
Sep 18 19:43:59.132615 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:59.132590    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="abaa8cdf-c42d-439a-824b-55139c1b8fb6" containerName="kserve-container"
Sep 18 19:43:59.132615 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:59.132599    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abaa8cdf-c42d-439a-824b-55139c1b8fb6" containerName="kube-rbac-proxy"
Sep 18 19:43:59.132615 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:59.132604    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="abaa8cdf-c42d-439a-824b-55139c1b8fb6" containerName="kube-rbac-proxy"
Sep 18 19:43:59.132615 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:59.132610    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abaa8cdf-c42d-439a-824b-55139c1b8fb6" containerName="storage-initializer"
Sep 18 19:43:59.132615 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:59.132616    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="abaa8cdf-c42d-439a-824b-55139c1b8fb6" containerName="storage-initializer"
Sep 18 19:43:59.132905 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:59.132623    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc" containerName="storage-initializer"
Sep 18 19:43:59.132905 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:59.132628    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc" containerName="storage-initializer"
Sep 18 19:43:59.132905 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:59.132675    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="abaa8cdf-c42d-439a-824b-55139c1b8fb6" containerName="kserve-container"
Sep 18 19:43:59.132905 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:59.132683    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="abaa8cdf-c42d-439a-824b-55139c1b8fb6" containerName="kube-rbac-proxy"
Sep 18 19:43:59.132905 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:59.132689    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc" containerName="kube-rbac-proxy"
Sep 18 19:43:59.132905 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:59.132695    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ebc87e5-aef7-44dc-9d8e-0ad3410a41cc" containerName="kserve-container"
Sep 18 19:43:59.135480 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:59.135464    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-8f8f5f7df-gdh57"
Sep 18 19:43:59.138293 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:59.138265    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 19:43:59.138420 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:59.138362    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-cipn-kube-rbac-proxy-sar-config\""
Sep 18 19:43:59.138629 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:59.138613    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 19:43:59.138733 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:59.138719    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-cipn-predictor-serving-cert\""
Sep 18 19:43:59.139271 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:59.139254    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-ct9wb\""
Sep 18 19:43:59.147467 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:59.147444    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-8f8f5f7df-gdh57"]
Sep 18 19:43:59.221023 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:59.220987    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:59.230126 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:59.230106    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws8wp\" (UniqueName: \"kubernetes.io/projected/f45128db-6729-4ede-856c-debf4f3a3531-kube-api-access-ws8wp\") pod \"message-dumper-raw-cipn-predictor-8f8f5f7df-gdh57\" (UID: \"f45128db-6729-4ede-856c-debf4f3a3531\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-8f8f5f7df-gdh57"
Sep 18 19:43:59.230202 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:59.230148    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f45128db-6729-4ede-856c-debf4f3a3531-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-8f8f5f7df-gdh57\" (UID: \"f45128db-6729-4ede-856c-debf4f3a3531\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-8f8f5f7df-gdh57"
Sep 18 19:43:59.230202 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:59.230180    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f45128db-6729-4ede-856c-debf4f3a3531-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-8f8f5f7df-gdh57\" (UID: \"f45128db-6729-4ede-856c-debf4f3a3531\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-8f8f5f7df-gdh57"
Sep 18 19:43:59.330673 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:59.330647    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws8wp\" (UniqueName: \"kubernetes.io/projected/f45128db-6729-4ede-856c-debf4f3a3531-kube-api-access-ws8wp\") pod \"message-dumper-raw-cipn-predictor-8f8f5f7df-gdh57\" (UID: \"f45128db-6729-4ede-856c-debf4f3a3531\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-8f8f5f7df-gdh57"
Sep 18 19:43:59.330844 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:59.330690    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f45128db-6729-4ede-856c-debf4f3a3531-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-8f8f5f7df-gdh57\" (UID: \"f45128db-6729-4ede-856c-debf4f3a3531\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-8f8f5f7df-gdh57"
Sep 18 19:43:59.330844 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:59.330723    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f45128db-6729-4ede-856c-debf4f3a3531-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-8f8f5f7df-gdh57\" (UID: \"f45128db-6729-4ede-856c-debf4f3a3531\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-8f8f5f7df-gdh57"
Sep 18 19:43:59.331470 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:59.331444    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f45128db-6729-4ede-856c-debf4f3a3531-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-8f8f5f7df-gdh57\" (UID: \"f45128db-6729-4ede-856c-debf4f3a3531\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-8f8f5f7df-gdh57"
Sep 18 19:43:59.333255 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:59.333235    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f45128db-6729-4ede-856c-debf4f3a3531-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-8f8f5f7df-gdh57\" (UID: \"f45128db-6729-4ede-856c-debf4f3a3531\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-8f8f5f7df-gdh57"
Sep 18 19:43:59.339768 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:59.339716    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws8wp\" (UniqueName: \"kubernetes.io/projected/f45128db-6729-4ede-856c-debf4f3a3531-kube-api-access-ws8wp\") pod \"message-dumper-raw-cipn-predictor-8f8f5f7df-gdh57\" (UID: \"f45128db-6729-4ede-856c-debf4f3a3531\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-8f8f5f7df-gdh57"
Sep 18 19:43:59.444950 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:59.444925    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-8f8f5f7df-gdh57"
Sep 18 19:43:59.725518 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:43:59.725496    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-8f8f5f7df-gdh57"]
Sep 18 19:43:59.727697 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:43:59.727667    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf45128db_6729_4ede_856c_debf4f3a3531.slice/crio-2baedd8e1db980e49330b7c7b0949ff1812b8b48c955cd530181fb8c9bf1fcbd WatchSource:0}: Error finding container 2baedd8e1db980e49330b7c7b0949ff1812b8b48c955cd530181fb8c9bf1fcbd: Status 404 returned error can't find the container with id 2baedd8e1db980e49330b7c7b0949ff1812b8b48c955cd530181fb8c9bf1fcbd
Sep 18 19:44:00.337367 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:44:00.337327    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-8f8f5f7df-gdh57" event={"ID":"f45128db-6729-4ede-856c-debf4f3a3531","Type":"ContainerStarted","Data":"425803cddf2677cd6d9b18c029bdae9e7c95fc7d7605a0a473d9fc354d4522a5"}
Sep 18 19:44:00.337739 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:44:00.337400    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-8f8f5f7df-gdh57" event={"ID":"f45128db-6729-4ede-856c-debf4f3a3531","Type":"ContainerStarted","Data":"bcd39fb092379a55952500f53fb4e2a5de4077165f6210abbe063aa9685af346"}
Sep 18 19:44:00.337739 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:44:00.337433    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-8f8f5f7df-gdh57" event={"ID":"f45128db-6729-4ede-856c-debf4f3a3531","Type":"ContainerStarted","Data":"2baedd8e1db980e49330b7c7b0949ff1812b8b48c955cd530181fb8c9bf1fcbd"}
Sep 18 19:44:00.337739 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:44:00.337551    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-8f8f5f7df-gdh57"
Sep 18 19:44:00.358313 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:44:00.358256    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-8f8f5f7df-gdh57" podStartSLOduration=1.358242907 podStartE2EDuration="1.358242907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:43:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:44:00.358014218 +0000 UTC m=+1807.278518077" watchObservedRunningTime="2026-09-18 19:44:00.358242907 +0000 UTC m=+1807.278746763"
Sep 18 19:44:01.340733 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:44:01.340700    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-8f8f5f7df-gdh57"
Sep 18 19:44:01.342476 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:44:01.342453    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-8f8f5f7df-gdh57"
Sep 18 19:44:08.352544 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:44:08.352518    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-8f8f5f7df-gdh57"
Sep 18 19:44:09.160093 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:44:09.160043    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:44:09.294246 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:44:09.294219    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq"]
Sep 18 19:44:09.297361 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:44:09.297344    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq"
Sep 18 19:44:09.300111 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:44:09.300093    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 19:44:09.300577 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:44:09.300547    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 19:44:09.300945 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:44:09.300925    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-cipn-kube-rbac-proxy-sar-config\""
Sep 18 19:44:09.301399 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:44:09.301383    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-cipn-predictor-serving-cert\""
Sep 18 19:44:09.330397 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:44:09.330378    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq"]
Sep 18 19:44:09.400046 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:44:09.400017    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:44:09.402113 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:44:09.402096    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c9d6bfb1-cbb4-4471-b278-7828d76c8cde-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq\" (UID: \"c9d6bfb1-cbb4-4471-b278-7828d76c8cde\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq"
Sep 18 19:44:09.402177 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:44:09.402122    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c9d6bfb1-cbb4-4471-b278-7828d76c8cde-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq\" (UID: \"c9d6bfb1-cbb4-4471-b278-7828d76c8cde\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq"
Sep 18 19:44:09.402177 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:44:09.402146    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpbg7\" (UniqueName: \"kubernetes.io/projected/c9d6bfb1-cbb4-4471-b278-7828d76c8cde-kube-api-access-bpbg7\") pod \"isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq\" (UID: \"c9d6bfb1-cbb4-4471-b278-7828d76c8cde\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq"
Sep 18 19:44:09.402177 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:44:09.402161    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c9d6bfb1-cbb4-4471-b278-7828d76c8cde-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq\" (UID: \"c9d6bfb1-cbb4-4471-b278-7828d76c8cde\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq"
Sep 18 19:44:09.402273 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:44:09.402223    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c9d6bfb1-cbb4-4471-b278-7828d76c8cde-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq\" (UID: \"c9d6bfb1-cbb4-4471-b278-7828d76c8cde\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq"
Sep 18 19:44:09.503619 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:44:09.503574    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c9d6bfb1-cbb4-4471-b278-7828d76c8cde-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq\" (UID: \"c9d6bfb1-cbb4-4471-b278-7828d76c8cde\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq"
Sep 18 19:44:09.503619 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:44:09.503622    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c9d6bfb1-cbb4-4471-b278-7828d76c8cde-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq\" (UID: \"c9d6bfb1-cbb4-4471-b278-7828d76c8cde\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq"
Sep 18 19:44:09.503841 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:44:09.503741    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpbg7\" (UniqueName: \"kubernetes.io/projected/c9d6bfb1-cbb4-4471-b278-7828d76c8cde-kube-api-access-bpbg7\") pod \"isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq\" (UID: \"c9d6bfb1-cbb4-4471-b278-7828d76c8cde\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq"
Sep 18 19:44:09.503841 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:44:09.503769    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c9d6bfb1-cbb4-4471-b278-7828d76c8cde-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq\" (UID: \"c9d6bfb1-cbb4-4471-b278-7828d76c8cde\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq"
Sep 18 19:44:09.503841 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:44:09.503808    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c9d6bfb1-cbb4-4471-b278-7828d76c8cde-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq\" (UID: \"c9d6bfb1-cbb4-4471-b278-7828d76c8cde\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq"
Sep 18 19:44:09.503975 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:44:09.503952    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c9d6bfb1-cbb4-4471-b278-7828d76c8cde-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq\" (UID: \"c9d6bfb1-cbb4-4471-b278-7828d76c8cde\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq"
Sep 18 19:44:09.504328 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:44:09.504308    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c9d6bfb1-cbb4-4471-b278-7828d76c8cde-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq\" (UID: \"c9d6bfb1-cbb4-4471-b278-7828d76c8cde\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq"
Sep 18 19:44:09.504435 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:44:09.504415    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c9d6bfb1-cbb4-4471-b278-7828d76c8cde-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq\" (UID: \"c9d6bfb1-cbb4-4471-b278-7828d76c8cde\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq"
Sep 18 19:44:09.506259 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:44:09.506238    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c9d6bfb1-cbb4-4471-b278-7828d76c8cde-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq\" (UID: \"c9d6bfb1-cbb4-4471-b278-7828d76c8cde\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq"
Sep 18 19:44:09.524734 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:44:09.524707    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpbg7\" (UniqueName: \"kubernetes.io/projected/c9d6bfb1-cbb4-4471-b278-7828d76c8cde-kube-api-access-bpbg7\") pod \"isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq\" (UID: \"c9d6bfb1-cbb4-4471-b278-7828d76c8cde\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq"
Sep 18 19:44:09.607349 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:44:09.607325    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq"
Sep 18 19:44:09.884223 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:44:09.884200    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq"]
Sep 18 19:44:09.886808 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:44:09.886775    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9d6bfb1_cbb4_4471_b278_7828d76c8cde.slice/crio-184030ccfebbf7e7e443f921225dc16ead4f80f962a1f2b77782a2c5fa80f5b4 WatchSource:0}: Error finding container 184030ccfebbf7e7e443f921225dc16ead4f80f962a1f2b77782a2c5fa80f5b4: Status 404 returned error can't find the container with id 184030ccfebbf7e7e443f921225dc16ead4f80f962a1f2b77782a2c5fa80f5b4
Sep 18 19:44:10.374808 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:44:10.374774    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq" event={"ID":"c9d6bfb1-cbb4-4471-b278-7828d76c8cde","Type":"ContainerStarted","Data":"1dca856a379a1d1375509d2a744797d368df023fa4c40e6b618c670b9e1cc8e9"}
Sep 18 19:44:10.375002 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:44:10.374855    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq" event={"ID":"c9d6bfb1-cbb4-4471-b278-7828d76c8cde","Type":"ContainerStarted","Data":"184030ccfebbf7e7e443f921225dc16ead4f80f962a1f2b77782a2c5fa80f5b4"}
Sep 18 19:44:14.390676 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:44:14.390643    2532 generic.go:356] "Generic (PLEG): container finished" podID="c9d6bfb1-cbb4-4471-b278-7828d76c8cde" containerID="1dca856a379a1d1375509d2a744797d368df023fa4c40e6b618c670b9e1cc8e9" exitCode=0
Sep 18 19:44:14.391084 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:44:14.390696    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq" event={"ID":"c9d6bfb1-cbb4-4471-b278-7828d76c8cde","Type":"ContainerDied","Data":"1dca856a379a1d1375509d2a744797d368df023fa4c40e6b618c670b9e1cc8e9"}
Sep 18 19:44:15.396712 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:44:15.396681    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq" event={"ID":"c9d6bfb1-cbb4-4471-b278-7828d76c8cde","Type":"ContainerStarted","Data":"0305ce30466237768789d47b644f90e64781a5babf36bac205ba36940fbb11f7"}
Sep 18 19:44:15.397111 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:44:15.396747    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq" event={"ID":"c9d6bfb1-cbb4-4471-b278-7828d76c8cde","Type":"ContainerStarted","Data":"97891fb3c3db48def4ece2a942507d1b9ca9f506b304a08c0afee5d9c402cc1f"}
Sep 18 19:44:15.397111 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:44:15.396770    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq" event={"ID":"c9d6bfb1-cbb4-4471-b278-7828d76c8cde","Type":"ContainerStarted","Data":"54d33b42e6c01d5d52abf2873936a60c4bc1a361dfe1f44d6f90bbec9acc1b8b"}
Sep 18 19:44:15.397111 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:44:15.396963    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq"
Sep 18 19:44:15.429926 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:44:15.429883    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq" podStartSLOduration=2.653063952 podStartE2EDuration="6.429868132s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.77680418s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:44:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:44:15.428874418 +0000 UTC m=+1822.349378275" watchObservedRunningTime="2026-09-18 19:44:15.429868132 +0000 UTC m=+1822.350371990"
Sep 18 19:44:16.400094 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:44:16.400045    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq"
Sep 18 19:44:16.400437 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:44:16.400136    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq"
Sep 18 19:44:16.401388 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:44:16.401360    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq" podUID="c9d6bfb1-cbb4-4471-b278-7828d76c8cde" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 18 19:44:16.401997 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:44:16.401973    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq" podUID="c9d6bfb1-cbb4-4471-b278-7828d76c8cde" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:44:17.403560 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:44:17.403517    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq" podUID="c9d6bfb1-cbb4-4471-b278-7828d76c8cde" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 18 19:44:17.404001 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:44:17.403981    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq" podUID="c9d6bfb1-cbb4-4471-b278-7828d76c8cde" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:44:22.408194 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:44:22.408166    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq"
Sep 18 19:44:22.410653 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:44:22.408768    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq" podUID="c9d6bfb1-cbb4-4471-b278-7828d76c8cde" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 18 19:44:22.410653 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:44:22.409022    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq" podUID="c9d6bfb1-cbb4-4471-b278-7828d76c8cde" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:44:32.409263 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:44:32.409224    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq" podUID="c9d6bfb1-cbb4-4471-b278-7828d76c8cde" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 18 19:44:32.409704 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:44:32.409666    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq" podUID="c9d6bfb1-cbb4-4471-b278-7828d76c8cde" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:44:42.409322 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:44:42.409282    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq" podUID="c9d6bfb1-cbb4-4471-b278-7828d76c8cde" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 18 19:44:42.409742 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:44:42.409713    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq" podUID="c9d6bfb1-cbb4-4471-b278-7828d76c8cde" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:44:52.409546 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:44:52.409497    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq" podUID="c9d6bfb1-cbb4-4471-b278-7828d76c8cde" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 18 19:44:52.410123 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:44:52.409871    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq" podUID="c9d6bfb1-cbb4-4471-b278-7828d76c8cde" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:45:02.408887 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:02.408849    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq" podUID="c9d6bfb1-cbb4-4471-b278-7828d76c8cde" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 18 19:45:02.409358 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:02.409293    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq" podUID="c9d6bfb1-cbb4-4471-b278-7828d76c8cde" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:45:12.408844 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:12.408801    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq" podUID="c9d6bfb1-cbb4-4471-b278-7828d76c8cde" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 18 19:45:12.409308 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:12.409286    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq" podUID="c9d6bfb1-cbb4-4471-b278-7828d76c8cde" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:45:22.408938 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:22.408849    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq" podUID="c9d6bfb1-cbb4-4471-b278-7828d76c8cde" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 18 19:45:22.409329 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:22.409246    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq" podUID="c9d6bfb1-cbb4-4471-b278-7828d76c8cde" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:45:28.617247 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:28.617218    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq"
Sep 18 19:45:28.617708 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:28.617296    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq"
Sep 18 19:45:34.573052 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:34.573019    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-cipn-predictor-8f8f5f7df-gdh57_f45128db-6729-4ede-856c-debf4f3a3531/kserve-container/0.log"
Sep 18 19:45:34.740351 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:34.740321    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq"]
Sep 18 19:45:34.740638 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:34.740598    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq" podUID="c9d6bfb1-cbb4-4471-b278-7828d76c8cde" containerName="kserve-container" containerID="cri-o://54d33b42e6c01d5d52abf2873936a60c4bc1a361dfe1f44d6f90bbec9acc1b8b" gracePeriod=30
Sep 18 19:45:34.740787 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:34.740625    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq" podUID="c9d6bfb1-cbb4-4471-b278-7828d76c8cde" containerName="kube-rbac-proxy" containerID="cri-o://97891fb3c3db48def4ece2a942507d1b9ca9f506b304a08c0afee5d9c402cc1f" gracePeriod=30
Sep 18 19:45:34.740787 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:34.740597    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq" podUID="c9d6bfb1-cbb4-4471-b278-7828d76c8cde" containerName="agent" containerID="cri-o://0305ce30466237768789d47b644f90e64781a5babf36bac205ba36940fbb11f7" gracePeriod=30
Sep 18 19:45:34.765323 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:34.765297    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:34.774300 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:34.774274    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-8f8f5f7df-gdh57"]
Sep 18 19:45:34.774479 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:34.774456    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-8f8f5f7df-gdh57" podUID="f45128db-6729-4ede-856c-debf4f3a3531" containerName="kserve-container" containerID="cri-o://bcd39fb092379a55952500f53fb4e2a5de4077165f6210abbe063aa9685af346" gracePeriod=30
Sep 18 19:45:34.774594 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:34.774505    2532 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-8f8f5f7df-gdh57" podUID="f45128db-6729-4ede-856c-debf4f3a3531" containerName="kube-rbac-proxy" containerID="cri-o://425803cddf2677cd6d9b18c029bdae9e7c95fc7d7605a0a473d9fc354d4522a5" gracePeriod=30
Sep 18 19:45:34.832132 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:34.832074    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:35.011126 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:35.011104    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-8f8f5f7df-gdh57"
Sep 18 19:45:35.101036 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:35.100953    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f45128db-6729-4ede-856c-debf4f3a3531-proxy-tls\") pod \"f45128db-6729-4ede-856c-debf4f3a3531\" (UID: \"f45128db-6729-4ede-856c-debf4f3a3531\") "
Sep 18 19:45:35.101036 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:35.101000    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f45128db-6729-4ede-856c-debf4f3a3531-kube-rbac-proxy-sar-config\") pod \"f45128db-6729-4ede-856c-debf4f3a3531\" (UID: \"f45128db-6729-4ede-856c-debf4f3a3531\") "
Sep 18 19:45:35.101036 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:35.101018    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ws8wp\" (UniqueName: \"kubernetes.io/projected/f45128db-6729-4ede-856c-debf4f3a3531-kube-api-access-ws8wp\") pod \"f45128db-6729-4ede-856c-debf4f3a3531\" (UID: \"f45128db-6729-4ede-856c-debf4f3a3531\") "
Sep 18 19:45:35.101435 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:35.101408    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f45128db-6729-4ede-856c-debf4f3a3531-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f45128db-6729-4ede-856c-debf4f3a3531" (UID: "f45128db-6729-4ede-856c-debf4f3a3531"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:45:35.103076 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:35.103043    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f45128db-6729-4ede-856c-debf4f3a3531-kube-api-access-ws8wp" (OuterVolumeSpecName: "kube-api-access-ws8wp") pod "f45128db-6729-4ede-856c-debf4f3a3531" (UID: "f45128db-6729-4ede-856c-debf4f3a3531"). InnerVolumeSpecName "kube-api-access-ws8wp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:45:35.103181 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:35.103076    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f45128db-6729-4ede-856c-debf4f3a3531-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f45128db-6729-4ede-856c-debf4f3a3531" (UID: "f45128db-6729-4ede-856c-debf4f3a3531"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:45:35.201484 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:35.201438    2532 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f45128db-6729-4ede-856c-debf4f3a3531-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:45:35.201484 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:35.201478    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ws8wp\" (UniqueName: \"kubernetes.io/projected/f45128db-6729-4ede-856c-debf4f3a3531-kube-api-access-ws8wp\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:45:35.201484 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:35.201487    2532 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f45128db-6729-4ede-856c-debf4f3a3531-proxy-tls\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:45:35.688705 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:35.688672    2532 generic.go:356] "Generic (PLEG): container finished" podID="f45128db-6729-4ede-856c-debf4f3a3531" containerID="425803cddf2677cd6d9b18c029bdae9e7c95fc7d7605a0a473d9fc354d4522a5" exitCode=2
Sep 18 19:45:35.688705 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:35.688694    2532 generic.go:356] "Generic (PLEG): container finished" podID="f45128db-6729-4ede-856c-debf4f3a3531" containerID="bcd39fb092379a55952500f53fb4e2a5de4077165f6210abbe063aa9685af346" exitCode=2
Sep 18 19:45:35.689206 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:35.688719    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-8f8f5f7df-gdh57"
Sep 18 19:45:35.689206 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:35.688759    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-8f8f5f7df-gdh57" event={"ID":"f45128db-6729-4ede-856c-debf4f3a3531","Type":"ContainerDied","Data":"425803cddf2677cd6d9b18c029bdae9e7c95fc7d7605a0a473d9fc354d4522a5"}
Sep 18 19:45:35.689206 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:35.688832    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-8f8f5f7df-gdh57" event={"ID":"f45128db-6729-4ede-856c-debf4f3a3531","Type":"ContainerDied","Data":"bcd39fb092379a55952500f53fb4e2a5de4077165f6210abbe063aa9685af346"}
Sep 18 19:45:35.689206 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:35.688867    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-8f8f5f7df-gdh57" event={"ID":"f45128db-6729-4ede-856c-debf4f3a3531","Type":"ContainerDied","Data":"2baedd8e1db980e49330b7c7b0949ff1812b8b48c955cd530181fb8c9bf1fcbd"}
Sep 18 19:45:35.689206 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:35.688902    2532 scope.go:117] "RemoveContainer" containerID="425803cddf2677cd6d9b18c029bdae9e7c95fc7d7605a0a473d9fc354d4522a5"
Sep 18 19:45:35.691622 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:35.691603    2532 generic.go:356] "Generic (PLEG): container finished" podID="c9d6bfb1-cbb4-4471-b278-7828d76c8cde" containerID="97891fb3c3db48def4ece2a942507d1b9ca9f506b304a08c0afee5d9c402cc1f" exitCode=2
Sep 18 19:45:35.691713 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:35.691646    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq" event={"ID":"c9d6bfb1-cbb4-4471-b278-7828d76c8cde","Type":"ContainerDied","Data":"97891fb3c3db48def4ece2a942507d1b9ca9f506b304a08c0afee5d9c402cc1f"}
Sep 18 19:45:35.696968 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:35.696957    2532 scope.go:117] "RemoveContainer" containerID="bcd39fb092379a55952500f53fb4e2a5de4077165f6210abbe063aa9685af346"
Sep 18 19:45:35.703771 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:35.703756    2532 scope.go:117] "RemoveContainer" containerID="425803cddf2677cd6d9b18c029bdae9e7c95fc7d7605a0a473d9fc354d4522a5"
Sep 18 19:45:35.704050 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:45:35.704030    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"425803cddf2677cd6d9b18c029bdae9e7c95fc7d7605a0a473d9fc354d4522a5\": container with ID starting with 425803cddf2677cd6d9b18c029bdae9e7c95fc7d7605a0a473d9fc354d4522a5 not found: ID does not exist" containerID="425803cddf2677cd6d9b18c029bdae9e7c95fc7d7605a0a473d9fc354d4522a5"
Sep 18 19:45:35.704186 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:35.704068    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"425803cddf2677cd6d9b18c029bdae9e7c95fc7d7605a0a473d9fc354d4522a5"} err="failed to get container status \"425803cddf2677cd6d9b18c029bdae9e7c95fc7d7605a0a473d9fc354d4522a5\": rpc error: code = NotFound desc = could not find container \"425803cddf2677cd6d9b18c029bdae9e7c95fc7d7605a0a473d9fc354d4522a5\": container with ID starting with 425803cddf2677cd6d9b18c029bdae9e7c95fc7d7605a0a473d9fc354d4522a5 not found: ID does not exist"
Sep 18 19:45:35.704186 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:35.704102    2532 scope.go:117] "RemoveContainer" containerID="bcd39fb092379a55952500f53fb4e2a5de4077165f6210abbe063aa9685af346"
Sep 18 19:45:35.704409 ip-10-0-138-252 kubenswrapper[2532]: E0918 19:45:35.704388    2532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcd39fb092379a55952500f53fb4e2a5de4077165f6210abbe063aa9685af346\": container with ID starting with bcd39fb092379a55952500f53fb4e2a5de4077165f6210abbe063aa9685af346 not found: ID does not exist" containerID="bcd39fb092379a55952500f53fb4e2a5de4077165f6210abbe063aa9685af346"
Sep 18 19:45:35.704493 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:35.704431    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcd39fb092379a55952500f53fb4e2a5de4077165f6210abbe063aa9685af346"} err="failed to get container status \"bcd39fb092379a55952500f53fb4e2a5de4077165f6210abbe063aa9685af346\": rpc error: code = NotFound desc = could not find container \"bcd39fb092379a55952500f53fb4e2a5de4077165f6210abbe063aa9685af346\": container with ID starting with bcd39fb092379a55952500f53fb4e2a5de4077165f6210abbe063aa9685af346 not found: ID does not exist"
Sep 18 19:45:35.704493 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:35.704456    2532 scope.go:117] "RemoveContainer" containerID="425803cddf2677cd6d9b18c029bdae9e7c95fc7d7605a0a473d9fc354d4522a5"
Sep 18 19:45:35.704752 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:35.704733    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"425803cddf2677cd6d9b18c029bdae9e7c95fc7d7605a0a473d9fc354d4522a5"} err="failed to get container status \"425803cddf2677cd6d9b18c029bdae9e7c95fc7d7605a0a473d9fc354d4522a5\": rpc error: code = NotFound desc = could not find container \"425803cddf2677cd6d9b18c029bdae9e7c95fc7d7605a0a473d9fc354d4522a5\": container with ID starting with 425803cddf2677cd6d9b18c029bdae9e7c95fc7d7605a0a473d9fc354d4522a5 not found: ID does not exist"
Sep 18 19:45:35.704807 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:35.704754    2532 scope.go:117] "RemoveContainer" containerID="bcd39fb092379a55952500f53fb4e2a5de4077165f6210abbe063aa9685af346"
Sep 18 19:45:35.704999 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:35.704977    2532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcd39fb092379a55952500f53fb4e2a5de4077165f6210abbe063aa9685af346"} err="failed to get container status \"bcd39fb092379a55952500f53fb4e2a5de4077165f6210abbe063aa9685af346\": rpc error: code = NotFound desc = could not find container \"bcd39fb092379a55952500f53fb4e2a5de4077165f6210abbe063aa9685af346\": container with ID starting with bcd39fb092379a55952500f53fb4e2a5de4077165f6210abbe063aa9685af346 not found: ID does not exist"
Sep 18 19:45:35.709392 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:35.709372    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-8f8f5f7df-gdh57"]
Sep 18 19:45:35.721692 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:35.721674    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-8f8f5f7df-gdh57"]
Sep 18 19:45:35.759360 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:35.759339    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:35.829807 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:35.829779    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:37.404426 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:37.404388    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq" podUID="c9d6bfb1-cbb4-4471-b278-7828d76c8cde" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Sep 18 19:45:37.620687 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:37.620652    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f45128db-6729-4ede-856c-debf4f3a3531" path="/var/lib/kubelet/pods/f45128db-6729-4ede-856c-debf4f3a3531/volumes"
Sep 18 19:45:38.617460 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:38.617415    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq" podUID="c9d6bfb1-cbb4-4471-b278-7828d76c8cde" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 18 19:45:38.617860 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:38.617786    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq" podUID="c9d6bfb1-cbb4-4471-b278-7828d76c8cde" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:45:39.709931 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:39.709900    2532 generic.go:356] "Generic (PLEG): container finished" podID="c9d6bfb1-cbb4-4471-b278-7828d76c8cde" containerID="54d33b42e6c01d5d52abf2873936a60c4bc1a361dfe1f44d6f90bbec9acc1b8b" exitCode=0
Sep 18 19:45:39.710315 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:39.709962    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq" event={"ID":"c9d6bfb1-cbb4-4471-b278-7828d76c8cde","Type":"ContainerDied","Data":"54d33b42e6c01d5d52abf2873936a60c4bc1a361dfe1f44d6f90bbec9acc1b8b"}
Sep 18 19:45:42.403885 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:42.403838    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq" podUID="c9d6bfb1-cbb4-4471-b278-7828d76c8cde" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Sep 18 19:45:47.404012 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:47.403970    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq" podUID="c9d6bfb1-cbb4-4471-b278-7828d76c8cde" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Sep 18 19:45:47.404414 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:47.404138    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq"
Sep 18 19:45:48.617417 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:48.617376    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq" podUID="c9d6bfb1-cbb4-4471-b278-7828d76c8cde" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 18 19:45:48.617840 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:48.617767    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq" podUID="c9d6bfb1-cbb4-4471-b278-7828d76c8cde" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:45:51.759268 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:51.759237    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq_c9d6bfb1-cbb4-4471-b278-7828d76c8cde/kserve-container/0.log"
Sep 18 19:45:51.787474 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:51.787453    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq_c9d6bfb1-cbb4-4471-b278-7828d76c8cde/kube-rbac-proxy/0.log"
Sep 18 19:45:51.799419 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:51.799390    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq_c9d6bfb1-cbb4-4471-b278-7828d76c8cde/agent/0.log"
Sep 18 19:45:51.812182 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:51.812159    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq_c9d6bfb1-cbb4-4471-b278-7828d76c8cde/storage-initializer/0.log"
Sep 18 19:45:52.404607 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:52.404567    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq" podUID="c9d6bfb1-cbb4-4471-b278-7828d76c8cde" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Sep 18 19:45:52.656298 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:52.656228    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq_c9d6bfb1-cbb4-4471-b278-7828d76c8cde/kserve-container/0.log"
Sep 18 19:45:52.670200 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:52.670176    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq_c9d6bfb1-cbb4-4471-b278-7828d76c8cde/kube-rbac-proxy/0.log"
Sep 18 19:45:52.682980 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:52.682957    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq_c9d6bfb1-cbb4-4471-b278-7828d76c8cde/agent/0.log"
Sep 18 19:45:52.694485 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:52.694467    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq_c9d6bfb1-cbb4-4471-b278-7828d76c8cde/storage-initializer/0.log"
Sep 18 19:45:53.616932 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:53.616902    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq_c9d6bfb1-cbb4-4471-b278-7828d76c8cde/kserve-container/0.log"
Sep 18 19:45:53.630466 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:53.630443    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq_c9d6bfb1-cbb4-4471-b278-7828d76c8cde/kube-rbac-proxy/0.log"
Sep 18 19:45:53.642936 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:53.642916    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq_c9d6bfb1-cbb4-4471-b278-7828d76c8cde/agent/0.log"
Sep 18 19:45:53.654763 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:53.654747    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq_c9d6bfb1-cbb4-4471-b278-7828d76c8cde/storage-initializer/0.log"
Sep 18 19:45:54.485382 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:54.485358    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq_c9d6bfb1-cbb4-4471-b278-7828d76c8cde/kserve-container/0.log"
Sep 18 19:45:54.499547 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:54.499522    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq_c9d6bfb1-cbb4-4471-b278-7828d76c8cde/kube-rbac-proxy/0.log"
Sep 18 19:45:54.513804 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:54.513785    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq_c9d6bfb1-cbb4-4471-b278-7828d76c8cde/agent/0.log"
Sep 18 19:45:54.529004 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:54.528984    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq_c9d6bfb1-cbb4-4471-b278-7828d76c8cde/storage-initializer/0.log"
Sep 18 19:45:55.396859 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:55.396827    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq_c9d6bfb1-cbb4-4471-b278-7828d76c8cde/kserve-container/0.log"
Sep 18 19:45:55.413154 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:55.413130    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq_c9d6bfb1-cbb4-4471-b278-7828d76c8cde/kube-rbac-proxy/0.log"
Sep 18 19:45:55.427096 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:55.427076    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq_c9d6bfb1-cbb4-4471-b278-7828d76c8cde/agent/0.log"
Sep 18 19:45:55.440773 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:55.440752    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq_c9d6bfb1-cbb4-4471-b278-7828d76c8cde/storage-initializer/0.log"
Sep 18 19:45:56.299199 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:56.299171    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq_c9d6bfb1-cbb4-4471-b278-7828d76c8cde/kserve-container/0.log"
Sep 18 19:45:56.319670 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:56.319645    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq_c9d6bfb1-cbb4-4471-b278-7828d76c8cde/kube-rbac-proxy/0.log"
Sep 18 19:45:56.333015 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:56.332994    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq_c9d6bfb1-cbb4-4471-b278-7828d76c8cde/agent/0.log"
Sep 18 19:45:56.346820 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:56.346805    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq_c9d6bfb1-cbb4-4471-b278-7828d76c8cde/storage-initializer/0.log"
Sep 18 19:45:57.336568 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:57.336544    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq_c9d6bfb1-cbb4-4471-b278-7828d76c8cde/kserve-container/0.log"
Sep 18 19:45:57.373579 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:57.373557    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq_c9d6bfb1-cbb4-4471-b278-7828d76c8cde/kube-rbac-proxy/0.log"
Sep 18 19:45:57.395944 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:57.395927    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq_c9d6bfb1-cbb4-4471-b278-7828d76c8cde/agent/0.log"
Sep 18 19:45:57.404342 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:57.404320    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq" podUID="c9d6bfb1-cbb4-4471-b278-7828d76c8cde" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Sep 18 19:45:57.416946 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:57.416930    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq_c9d6bfb1-cbb4-4471-b278-7828d76c8cde/storage-initializer/0.log"
Sep 18 19:45:58.340295 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:58.340254    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq_c9d6bfb1-cbb4-4471-b278-7828d76c8cde/kserve-container/0.log"
Sep 18 19:45:58.357668 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:58.357647    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq_c9d6bfb1-cbb4-4471-b278-7828d76c8cde/kube-rbac-proxy/0.log"
Sep 18 19:45:58.370936 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:58.370914    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq_c9d6bfb1-cbb4-4471-b278-7828d76c8cde/agent/0.log"
Sep 18 19:45:58.384502 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:58.384486    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq_c9d6bfb1-cbb4-4471-b278-7828d76c8cde/storage-initializer/0.log"
Sep 18 19:45:58.617492 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:58.617430    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq" podUID="c9d6bfb1-cbb4-4471-b278-7828d76c8cde" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 18 19:45:58.617627 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:58.617556    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq"
Sep 18 19:45:58.617805 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:58.617787    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq" podUID="c9d6bfb1-cbb4-4471-b278-7828d76c8cde" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:45:58.617882 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:58.617871    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq"
Sep 18 19:45:59.250628 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:59.250602    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq_c9d6bfb1-cbb4-4471-b278-7828d76c8cde/kserve-container/0.log"
Sep 18 19:45:59.265435 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:59.265412    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq_c9d6bfb1-cbb4-4471-b278-7828d76c8cde/kube-rbac-proxy/0.log"
Sep 18 19:45:59.282096 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:59.282072    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq_c9d6bfb1-cbb4-4471-b278-7828d76c8cde/agent/0.log"
Sep 18 19:45:59.296702 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:45:59.296684    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq_c9d6bfb1-cbb4-4471-b278-7828d76c8cde/storage-initializer/0.log"
Sep 18 19:46:00.139550 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:00.139522    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq_c9d6bfb1-cbb4-4471-b278-7828d76c8cde/kserve-container/0.log"
Sep 18 19:46:00.154743 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:00.154720    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq_c9d6bfb1-cbb4-4471-b278-7828d76c8cde/kube-rbac-proxy/0.log"
Sep 18 19:46:00.168119 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:00.168098    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq_c9d6bfb1-cbb4-4471-b278-7828d76c8cde/agent/0.log"
Sep 18 19:46:00.181940 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:00.181923    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq_c9d6bfb1-cbb4-4471-b278-7828d76c8cde/storage-initializer/0.log"
Sep 18 19:46:01.027160 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:01.027131    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq_c9d6bfb1-cbb4-4471-b278-7828d76c8cde/kserve-container/0.log"
Sep 18 19:46:01.049044 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:01.049017    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq_c9d6bfb1-cbb4-4471-b278-7828d76c8cde/kube-rbac-proxy/0.log"
Sep 18 19:46:01.074962 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:01.074942    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq_c9d6bfb1-cbb4-4471-b278-7828d76c8cde/agent/0.log"
Sep 18 19:46:01.090834 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:01.090811    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq_c9d6bfb1-cbb4-4471-b278-7828d76c8cde/storage-initializer/0.log"
Sep 18 19:46:01.938419 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:01.938389    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq_c9d6bfb1-cbb4-4471-b278-7828d76c8cde/kserve-container/0.log"
Sep 18 19:46:01.955666 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:01.955641    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq_c9d6bfb1-cbb4-4471-b278-7828d76c8cde/kube-rbac-proxy/0.log"
Sep 18 19:46:01.969556 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:01.969532    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq_c9d6bfb1-cbb4-4471-b278-7828d76c8cde/agent/0.log"
Sep 18 19:46:01.983475 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:01.983452    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq_c9d6bfb1-cbb4-4471-b278-7828d76c8cde/storage-initializer/0.log"
Sep 18 19:46:02.404588 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:02.404485    2532 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq" podUID="c9d6bfb1-cbb4-4471-b278-7828d76c8cde" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Sep 18 19:46:02.947597 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:02.947564    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq_c9d6bfb1-cbb4-4471-b278-7828d76c8cde/kserve-container/0.log"
Sep 18 19:46:02.963047 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:02.963021    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq_c9d6bfb1-cbb4-4471-b278-7828d76c8cde/kube-rbac-proxy/0.log"
Sep 18 19:46:02.985880 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:02.985855    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq_c9d6bfb1-cbb4-4471-b278-7828d76c8cde/agent/0.log"
Sep 18 19:46:03.021355 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:03.021331    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq_c9d6bfb1-cbb4-4471-b278-7828d76c8cde/storage-initializer/0.log"
Sep 18 19:46:04.025178 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:04.025150    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq_c9d6bfb1-cbb4-4471-b278-7828d76c8cde/kserve-container/0.log"
Sep 18 19:46:04.051644 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:04.051614    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq_c9d6bfb1-cbb4-4471-b278-7828d76c8cde/kube-rbac-proxy/0.log"
Sep 18 19:46:04.089918 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:04.089892    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq_c9d6bfb1-cbb4-4471-b278-7828d76c8cde/agent/0.log"
Sep 18 19:46:04.112998 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:04.112974    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq_c9d6bfb1-cbb4-4471-b278-7828d76c8cde/storage-initializer/0.log"
Sep 18 19:46:04.800517 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:04.800485    2532 generic.go:356] "Generic (PLEG): container finished" podID="c9d6bfb1-cbb4-4471-b278-7828d76c8cde" containerID="0305ce30466237768789d47b644f90e64781a5babf36bac205ba36940fbb11f7" exitCode=0
Sep 18 19:46:04.800647 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:04.800533    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq" event={"ID":"c9d6bfb1-cbb4-4471-b278-7828d76c8cde","Type":"ContainerDied","Data":"0305ce30466237768789d47b644f90e64781a5babf36bac205ba36940fbb11f7"}
Sep 18 19:46:04.882544 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:04.882522    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq"
Sep 18 19:46:04.996970 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:04.996909    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c9d6bfb1-cbb4-4471-b278-7828d76c8cde-proxy-tls\") pod \"c9d6bfb1-cbb4-4471-b278-7828d76c8cde\" (UID: \"c9d6bfb1-cbb4-4471-b278-7828d76c8cde\") "
Sep 18 19:46:04.996970 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:04.996957    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c9d6bfb1-cbb4-4471-b278-7828d76c8cde-kserve-provision-location\") pod \"c9d6bfb1-cbb4-4471-b278-7828d76c8cde\" (UID: \"c9d6bfb1-cbb4-4471-b278-7828d76c8cde\") "
Sep 18 19:46:04.997157 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:04.996983    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bpbg7\" (UniqueName: \"kubernetes.io/projected/c9d6bfb1-cbb4-4471-b278-7828d76c8cde-kube-api-access-bpbg7\") pod \"c9d6bfb1-cbb4-4471-b278-7828d76c8cde\" (UID: \"c9d6bfb1-cbb4-4471-b278-7828d76c8cde\") "
Sep 18 19:46:04.997157 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:04.997016    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c9d6bfb1-cbb4-4471-b278-7828d76c8cde-cabundle-cert\") pod \"c9d6bfb1-cbb4-4471-b278-7828d76c8cde\" (UID: \"c9d6bfb1-cbb4-4471-b278-7828d76c8cde\") "
Sep 18 19:46:04.997157 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:04.997042    2532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c9d6bfb1-cbb4-4471-b278-7828d76c8cde-kube-rbac-proxy-sar-config\") pod \"c9d6bfb1-cbb4-4471-b278-7828d76c8cde\" (UID: \"c9d6bfb1-cbb4-4471-b278-7828d76c8cde\") "
Sep 18 19:46:04.997315 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:04.997264    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9d6bfb1-cbb4-4471-b278-7828d76c8cde-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c9d6bfb1-cbb4-4471-b278-7828d76c8cde" (UID: "c9d6bfb1-cbb4-4471-b278-7828d76c8cde"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:46:04.997438 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:04.997354    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9d6bfb1-cbb4-4471-b278-7828d76c8cde-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c9d6bfb1-cbb4-4471-b278-7828d76c8cde" (UID: "c9d6bfb1-cbb4-4471-b278-7828d76c8cde"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:46:04.997557 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:04.997490    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9d6bfb1-cbb4-4471-b278-7828d76c8cde-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c9d6bfb1-cbb4-4471-b278-7828d76c8cde" (UID: "c9d6bfb1-cbb4-4471-b278-7828d76c8cde"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:46:04.999016 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:04.998996    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9d6bfb1-cbb4-4471-b278-7828d76c8cde-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c9d6bfb1-cbb4-4471-b278-7828d76c8cde" (UID: "c9d6bfb1-cbb4-4471-b278-7828d76c8cde"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:46:04.999165 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:04.999067    2532 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9d6bfb1-cbb4-4471-b278-7828d76c8cde-kube-api-access-bpbg7" (OuterVolumeSpecName: "kube-api-access-bpbg7") pod "c9d6bfb1-cbb4-4471-b278-7828d76c8cde" (UID: "c9d6bfb1-cbb4-4471-b278-7828d76c8cde"). InnerVolumeSpecName "kube-api-access-bpbg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:46:05.092394 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:05.092371    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq_c9d6bfb1-cbb4-4471-b278-7828d76c8cde/kserve-container/0.log"
Sep 18 19:46:05.097787 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:05.097771    2532 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c9d6bfb1-cbb4-4471-b278-7828d76c8cde-kserve-provision-location\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:46:05.097861 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:05.097791    2532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpbg7\" (UniqueName: \"kubernetes.io/projected/c9d6bfb1-cbb4-4471-b278-7828d76c8cde-kube-api-access-bpbg7\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:46:05.097861 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:05.097800    2532 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c9d6bfb1-cbb4-4471-b278-7828d76c8cde-cabundle-cert\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:46:05.097861 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:05.097809    2532 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c9d6bfb1-cbb4-4471-b278-7828d76c8cde-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:46:05.097861 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:05.097819    2532 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c9d6bfb1-cbb4-4471-b278-7828d76c8cde-proxy-tls\") on node \"ip-10-0-138-252.ec2.internal\" DevicePath \"\""
Sep 18 19:46:05.106220 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:05.106203    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq_c9d6bfb1-cbb4-4471-b278-7828d76c8cde/kube-rbac-proxy/0.log"
Sep 18 19:46:05.120794 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:05.120773    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq_c9d6bfb1-cbb4-4471-b278-7828d76c8cde/agent/0.log"
Sep 18 19:46:05.134942 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:05.134923    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq_c9d6bfb1-cbb4-4471-b278-7828d76c8cde/storage-initializer/0.log"
Sep 18 19:46:05.806069 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:05.806027    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq" event={"ID":"c9d6bfb1-cbb4-4471-b278-7828d76c8cde","Type":"ContainerDied","Data":"184030ccfebbf7e7e443f921225dc16ead4f80f962a1f2b77782a2c5fa80f5b4"}
Sep 18 19:46:05.806218 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:05.806119    2532 scope.go:117] "RemoveContainer" containerID="0305ce30466237768789d47b644f90e64781a5babf36bac205ba36940fbb11f7"
Sep 18 19:46:05.806218 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:05.806049    2532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq"
Sep 18 19:46:05.813506 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:05.813488    2532 scope.go:117] "RemoveContainer" containerID="97891fb3c3db48def4ece2a942507d1b9ca9f506b304a08c0afee5d9c402cc1f"
Sep 18 19:46:05.820324 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:05.820307    2532 scope.go:117] "RemoveContainer" containerID="54d33b42e6c01d5d52abf2873936a60c4bc1a361dfe1f44d6f90bbec9acc1b8b"
Sep 18 19:46:05.825250 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:05.825231    2532 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq"]
Sep 18 19:46:05.827458 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:05.827442    2532 scope.go:117] "RemoveContainer" containerID="1dca856a379a1d1375509d2a744797d368df023fa4c40e6b618c670b9e1cc8e9"
Sep 18 19:46:05.829455 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:05.829437    2532 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-66ff446ff-k6tnq"]
Sep 18 19:46:07.553218 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:07.553187    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:46:07.620835 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:07.620804    2532 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9d6bfb1-cbb4-4471-b278-7828d76c8cde" path="/var/lib/kubelet/pods/c9d6bfb1-cbb4-4471-b278-7828d76c8cde/volumes"
Sep 18 19:46:07.739864 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:07.739838    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hzj7l/must-gather-xljkd"]
Sep 18 19:46:07.740085 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:07.740073    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f45128db-6729-4ede-856c-debf4f3a3531" containerName="kserve-container"
Sep 18 19:46:07.740133 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:07.740086    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="f45128db-6729-4ede-856c-debf4f3a3531" containerName="kserve-container"
Sep 18 19:46:07.740133 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:07.740095    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9d6bfb1-cbb4-4471-b278-7828d76c8cde" containerName="storage-initializer"
Sep 18 19:46:07.740133 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:07.740101    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9d6bfb1-cbb4-4471-b278-7828d76c8cde" containerName="storage-initializer"
Sep 18 19:46:07.740133 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:07.740110    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9d6bfb1-cbb4-4471-b278-7828d76c8cde" containerName="agent"
Sep 18 19:46:07.740133 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:07.740116    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9d6bfb1-cbb4-4471-b278-7828d76c8cde" containerName="agent"
Sep 18 19:46:07.740133 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:07.740122    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f45128db-6729-4ede-856c-debf4f3a3531" containerName="kube-rbac-proxy"
Sep 18 19:46:07.740133 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:07.740127    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="f45128db-6729-4ede-856c-debf4f3a3531" containerName="kube-rbac-proxy"
Sep 18 19:46:07.740349 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:07.740138    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9d6bfb1-cbb4-4471-b278-7828d76c8cde" containerName="kube-rbac-proxy"
Sep 18 19:46:07.740349 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:07.740143    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9d6bfb1-cbb4-4471-b278-7828d76c8cde" containerName="kube-rbac-proxy"
Sep 18 19:46:07.740349 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:07.740149    2532 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9d6bfb1-cbb4-4471-b278-7828d76c8cde" containerName="kserve-container"
Sep 18 19:46:07.740349 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:07.740154    2532 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9d6bfb1-cbb4-4471-b278-7828d76c8cde" containerName="kserve-container"
Sep 18 19:46:07.740349 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:07.740191    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="f45128db-6729-4ede-856c-debf4f3a3531" containerName="kube-rbac-proxy"
Sep 18 19:46:07.740349 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:07.740197    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9d6bfb1-cbb4-4471-b278-7828d76c8cde" containerName="kserve-container"
Sep 18 19:46:07.740349 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:07.740204    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9d6bfb1-cbb4-4471-b278-7828d76c8cde" containerName="agent"
Sep 18 19:46:07.740349 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:07.740209    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="f45128db-6729-4ede-856c-debf4f3a3531" containerName="kserve-container"
Sep 18 19:46:07.740349 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:07.740215    2532 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9d6bfb1-cbb4-4471-b278-7828d76c8cde" containerName="kube-rbac-proxy"
Sep 18 19:46:07.744659 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:07.744641    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hzj7l/must-gather-xljkd"
Sep 18 19:46:07.747161 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:07.747136    2532 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-hzj7l\"/\"default-dockercfg-4b7ng\""
Sep 18 19:46:07.747161 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:07.747139    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hzj7l\"/\"openshift-service-ca.crt\""
Sep 18 19:46:07.747805 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:07.747787    2532 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hzj7l\"/\"kube-root-ca.crt\""
Sep 18 19:46:07.753641 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:07.753619    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hzj7l/must-gather-xljkd"]
Sep 18 19:46:07.766941 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:07.766920    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:46:07.818235 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:07.818186    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcnhj\" (UniqueName: \"kubernetes.io/projected/590feaec-b78d-4d94-9737-ee34685df117-kube-api-access-lcnhj\") pod \"must-gather-xljkd\" (UID: \"590feaec-b78d-4d94-9737-ee34685df117\") " pod="openshift-must-gather-hzj7l/must-gather-xljkd"
Sep 18 19:46:07.818235 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:07.818220    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/590feaec-b78d-4d94-9737-ee34685df117-must-gather-output\") pod \"must-gather-xljkd\" (UID: \"590feaec-b78d-4d94-9737-ee34685df117\") " pod="openshift-must-gather-hzj7l/must-gather-xljkd"
Sep 18 19:46:07.841433 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:07.841409    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:46:07.918960 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:07.918940    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/590feaec-b78d-4d94-9737-ee34685df117-must-gather-output\") pod \"must-gather-xljkd\" (UID: \"590feaec-b78d-4d94-9737-ee34685df117\") " pod="openshift-must-gather-hzj7l/must-gather-xljkd"
Sep 18 19:46:07.919079 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:07.918987    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcnhj\" (UniqueName: \"kubernetes.io/projected/590feaec-b78d-4d94-9737-ee34685df117-kube-api-access-lcnhj\") pod \"must-gather-xljkd\" (UID: \"590feaec-b78d-4d94-9737-ee34685df117\") " pod="openshift-must-gather-hzj7l/must-gather-xljkd"
Sep 18 19:46:07.919294 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:07.919276    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/590feaec-b78d-4d94-9737-ee34685df117-must-gather-output\") pod \"must-gather-xljkd\" (UID: \"590feaec-b78d-4d94-9737-ee34685df117\") " pod="openshift-must-gather-hzj7l/must-gather-xljkd"
Sep 18 19:46:07.928244 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:07.928223    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcnhj\" (UniqueName: \"kubernetes.io/projected/590feaec-b78d-4d94-9737-ee34685df117-kube-api-access-lcnhj\") pod \"must-gather-xljkd\" (UID: \"590feaec-b78d-4d94-9737-ee34685df117\") " pod="openshift-must-gather-hzj7l/must-gather-xljkd"
Sep 18 19:46:08.054183 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:08.054147    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hzj7l/must-gather-xljkd"
Sep 18 19:46:08.329193 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:08.329157    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hzj7l/must-gather-xljkd"]
Sep 18 19:46:08.332405 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:46:08.332375    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod590feaec_b78d_4d94_9737_ee34685df117.slice/crio-940357b4ef1c820cd02c820ef5277952a4d0863014938f180f1b3b62a069d8fc WatchSource:0}: Error finding container 940357b4ef1c820cd02c820ef5277952a4d0863014938f180f1b3b62a069d8fc: Status 404 returned error can't find the container with id 940357b4ef1c820cd02c820ef5277952a4d0863014938f180f1b3b62a069d8fc
Sep 18 19:46:08.334212 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:08.334195    2532 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:46:08.819927 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:08.819896    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hzj7l/must-gather-xljkd" event={"ID":"590feaec-b78d-4d94-9737-ee34685df117","Type":"ContainerStarted","Data":"940357b4ef1c820cd02c820ef5277952a4d0863014938f180f1b3b62a069d8fc"}
Sep 18 19:46:09.826711 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:09.826673    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hzj7l/must-gather-xljkd" event={"ID":"590feaec-b78d-4d94-9737-ee34685df117","Type":"ContainerStarted","Data":"1bce0ed44d68730e6fe940cd38f25311e896c6d4863443eedd6f2915c38d96d8"}
Sep 18 19:46:09.826711 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:09.826711    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hzj7l/must-gather-xljkd" event={"ID":"590feaec-b78d-4d94-9737-ee34685df117","Type":"ContainerStarted","Data":"2eafa741035169e0341b621fdbe6253d4c64f0d64d4b49b0747a47552cdddac1"}
Sep 18 19:46:09.844524 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:09.844473    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-hzj7l/must-gather-xljkd" podStartSLOduration=2.113056046 podStartE2EDuration="2.844458541s" totalImagesPullingTime="731.402495ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:46:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:46:09.84400837 +0000 UTC m=+1936.764512227" watchObservedRunningTime="2026-09-18 19:46:09.844458541 +0000 UTC m=+1936.764962397"
Sep 18 19:46:10.582980 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:10.582954    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-d9rlj_7cae3b8f-84ca-4cb2-9003-4f38a71ab6ed/global-pull-secret-syncer/0.log"
Sep 18 19:46:10.672366 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:10.672321    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log"
Sep 18 19:46:10.760729 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:10.760694    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-252.ec2.internal_2d5d27fed1ef1d219785d3f7c796a73d/haproxy/0.log"
Sep 18 19:46:13.513098 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:13.513039    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:46:13.586264 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:13.586233    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:46:15.209436 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:15.209397    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5dfbb88bb-tpbp8_4eba974d-5929-4fe6-8dec-4e3d7fab0e0e/metrics-server/0.log"
Sep 18 19:46:15.406438 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:15.406407    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fcd5d_a661c12a-c025-4e31-9c98-ba3d6a66089e/node-exporter/0.log"
Sep 18 19:46:15.437556 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:15.437524    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fcd5d_a661c12a-c025-4e31-9c98-ba3d6a66089e/kube-rbac-proxy/0.log"
Sep 18 19:46:15.469980 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:15.469899    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fcd5d_a661c12a-c025-4e31-9c98-ba3d6a66089e/init-textfile/0.log"
Sep 18 19:46:18.194516 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:18.194473    2532 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hzj7l/perf-node-gather-daemonset-pf77h"]
Sep 18 19:46:18.199697 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:18.199674    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hzj7l/perf-node-gather-daemonset-pf77h"
Sep 18 19:46:18.208504 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:18.208475    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hzj7l/perf-node-gather-daemonset-pf77h"]
Sep 18 19:46:18.303448 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:18.303413    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30137774-32e1-42f9-a216-da027d9248eb-sys\") pod \"perf-node-gather-daemonset-pf77h\" (UID: \"30137774-32e1-42f9-a216-da027d9248eb\") " pod="openshift-must-gather-hzj7l/perf-node-gather-daemonset-pf77h"
Sep 18 19:46:18.303620 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:18.303458    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/30137774-32e1-42f9-a216-da027d9248eb-podres\") pod \"perf-node-gather-daemonset-pf77h\" (UID: \"30137774-32e1-42f9-a216-da027d9248eb\") " pod="openshift-must-gather-hzj7l/perf-node-gather-daemonset-pf77h"
Sep 18 19:46:18.303620 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:18.303538    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qkzl\" (UniqueName: \"kubernetes.io/projected/30137774-32e1-42f9-a216-da027d9248eb-kube-api-access-7qkzl\") pod \"perf-node-gather-daemonset-pf77h\" (UID: \"30137774-32e1-42f9-a216-da027d9248eb\") " pod="openshift-must-gather-hzj7l/perf-node-gather-daemonset-pf77h"
Sep 18 19:46:18.303620 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:18.303596    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/30137774-32e1-42f9-a216-da027d9248eb-proc\") pod \"perf-node-gather-daemonset-pf77h\" (UID: \"30137774-32e1-42f9-a216-da027d9248eb\") " pod="openshift-must-gather-hzj7l/perf-node-gather-daemonset-pf77h"
Sep 18 19:46:18.303745 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:18.303626    2532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/30137774-32e1-42f9-a216-da027d9248eb-lib-modules\") pod \"perf-node-gather-daemonset-pf77h\" (UID: \"30137774-32e1-42f9-a216-da027d9248eb\") " pod="openshift-must-gather-hzj7l/perf-node-gather-daemonset-pf77h"
Sep 18 19:46:18.404560 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:18.404516    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qkzl\" (UniqueName: \"kubernetes.io/projected/30137774-32e1-42f9-a216-da027d9248eb-kube-api-access-7qkzl\") pod \"perf-node-gather-daemonset-pf77h\" (UID: \"30137774-32e1-42f9-a216-da027d9248eb\") " pod="openshift-must-gather-hzj7l/perf-node-gather-daemonset-pf77h"
Sep 18 19:46:18.404742 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:18.404576    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/30137774-32e1-42f9-a216-da027d9248eb-proc\") pod \"perf-node-gather-daemonset-pf77h\" (UID: \"30137774-32e1-42f9-a216-da027d9248eb\") " pod="openshift-must-gather-hzj7l/perf-node-gather-daemonset-pf77h"
Sep 18 19:46:18.404742 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:18.404632    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/30137774-32e1-42f9-a216-da027d9248eb-lib-modules\") pod \"perf-node-gather-daemonset-pf77h\" (UID: \"30137774-32e1-42f9-a216-da027d9248eb\") " pod="openshift-must-gather-hzj7l/perf-node-gather-daemonset-pf77h"
Sep 18 19:46:18.404742 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:18.404652    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/30137774-32e1-42f9-a216-da027d9248eb-proc\") pod \"perf-node-gather-daemonset-pf77h\" (UID: \"30137774-32e1-42f9-a216-da027d9248eb\") " pod="openshift-must-gather-hzj7l/perf-node-gather-daemonset-pf77h"
Sep 18 19:46:18.404927 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:18.404754    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30137774-32e1-42f9-a216-da027d9248eb-sys\") pod \"perf-node-gather-daemonset-pf77h\" (UID: \"30137774-32e1-42f9-a216-da027d9248eb\") " pod="openshift-must-gather-hzj7l/perf-node-gather-daemonset-pf77h"
Sep 18 19:46:18.404927 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:18.404779    2532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/30137774-32e1-42f9-a216-da027d9248eb-podres\") pod \"perf-node-gather-daemonset-pf77h\" (UID: \"30137774-32e1-42f9-a216-da027d9248eb\") " pod="openshift-must-gather-hzj7l/perf-node-gather-daemonset-pf77h"
Sep 18 19:46:18.404927 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:18.404848    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/30137774-32e1-42f9-a216-da027d9248eb-lib-modules\") pod \"perf-node-gather-daemonset-pf77h\" (UID: \"30137774-32e1-42f9-a216-da027d9248eb\") " pod="openshift-must-gather-hzj7l/perf-node-gather-daemonset-pf77h"
Sep 18 19:46:18.404927 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:18.404861    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30137774-32e1-42f9-a216-da027d9248eb-sys\") pod \"perf-node-gather-daemonset-pf77h\" (UID: \"30137774-32e1-42f9-a216-da027d9248eb\") " pod="openshift-must-gather-hzj7l/perf-node-gather-daemonset-pf77h"
Sep 18 19:46:18.405140 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:18.404936    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/30137774-32e1-42f9-a216-da027d9248eb-podres\") pod \"perf-node-gather-daemonset-pf77h\" (UID: \"30137774-32e1-42f9-a216-da027d9248eb\") " pod="openshift-must-gather-hzj7l/perf-node-gather-daemonset-pf77h"
Sep 18 19:46:18.413324 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:18.413302    2532 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qkzl\" (UniqueName: \"kubernetes.io/projected/30137774-32e1-42f9-a216-da027d9248eb-kube-api-access-7qkzl\") pod \"perf-node-gather-daemonset-pf77h\" (UID: \"30137774-32e1-42f9-a216-da027d9248eb\") " pod="openshift-must-gather-hzj7l/perf-node-gather-daemonset-pf77h"
Sep 18 19:46:18.511710 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:18.511621    2532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hzj7l/perf-node-gather-daemonset-pf77h"
Sep 18 19:46:18.797898 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:18.797769    2532 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hzj7l/perf-node-gather-daemonset-pf77h"]
Sep 18 19:46:18.801897 ip-10-0-138-252 kubenswrapper[2532]: W0918 19:46:18.801696    2532 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod30137774_32e1_42f9_a216_da027d9248eb.slice/crio-1bdf4ee19b99e577dc2005bba113222525edfcf591fa04a25893f439345312b2 WatchSource:0}: Error finding container 1bdf4ee19b99e577dc2005bba113222525edfcf591fa04a25893f439345312b2: Status 404 returned error can't find the container with id 1bdf4ee19b99e577dc2005bba113222525edfcf591fa04a25893f439345312b2
Sep 18 19:46:18.869416 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:18.869379    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hzj7l/perf-node-gather-daemonset-pf77h" event={"ID":"30137774-32e1-42f9-a216-da027d9248eb","Type":"ContainerStarted","Data":"1bdf4ee19b99e577dc2005bba113222525edfcf591fa04a25893f439345312b2"}
Sep 18 19:46:19.875881 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:19.875846    2532 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hzj7l/perf-node-gather-daemonset-pf77h" event={"ID":"30137774-32e1-42f9-a216-da027d9248eb","Type":"ContainerStarted","Data":"1e40a32c14577d5c3514199d4fd7e2c01e061c604d83d73072e2e9952355f8b6"}
Sep 18 19:46:19.876384 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:19.875938    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-hzj7l/perf-node-gather-daemonset-pf77h"
Sep 18 19:46:19.894847 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:19.894609    2532 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-hzj7l/perf-node-gather-daemonset-pf77h" podStartSLOduration=1.894589775 podStartE2EDuration="1.894589775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:46:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:46:19.892647035 +0000 UTC m=+1946.813150928" watchObservedRunningTime="2026-09-18 19:46:19.894589775 +0000 UTC m=+1946.815093633"
Sep 18 19:46:20.231290 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:20.231265    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-54cns_ad80f379-b5dc-4d02-a61d-b8b2a956a551/dns/0.log"
Sep 18 19:46:20.274261 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:20.274236    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-54cns_ad80f379-b5dc-4d02-a61d-b8b2a956a551/kube-rbac-proxy/0.log"
Sep 18 19:46:20.502162 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:20.502090    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qbs4d_851825cd-f968-471c-87c0-b7d2423c9c99/dns-node-resolver/0.log"
Sep 18 19:46:21.121523 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:21.121498    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7s97w_812cd3c0-1c61-4d21-8dd7-77d98f0edd2d/node-ca/0.log"
Sep 18 19:46:22.282009 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:22.281981    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wgxds_35de2d74-739d-411b-ad9b-dc99018ad770/serve-healthcheck-canary/0.log"
Sep 18 19:46:22.815149 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:22.815121    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tz4cf_3cd69cee-5d0e-41d9-8dc9-a0fb2898085d/kube-rbac-proxy/0.log"
Sep 18 19:46:22.838536 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:22.838507    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tz4cf_3cd69cee-5d0e-41d9-8dc9-a0fb2898085d/exporter/0.log"
Sep 18 19:46:22.864112 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:22.864092    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tz4cf_3cd69cee-5d0e-41d9-8dc9-a0fb2898085d/extractor/0.log"
Sep 18 19:46:25.444836 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:25.444801    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-cg6kz_6632b05b-6137-43b0-90d6-c2faf0a9c31f/manager/0.log"
Sep 18 19:46:25.480891 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:25.480864    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-bwxwl_12b53e18-63de-43fc-85e6-2e3508ea739b/s3-init/0.log"
Sep 18 19:46:25.500847 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:25.500823    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-bwxwl_12b53e18-63de-43fc-85e6-2e3508ea739b/download-hf-model/0.log"
Sep 18 19:46:25.538522 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:25.538490    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-2kcjk_e9d74016-32fe-4df7-9403-062ad106dd6d/seaweedfs/0.log"
Sep 18 19:46:25.890498 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:25.890422    2532 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-hzj7l/perf-node-gather-daemonset-pf77h"
Sep 18 19:46:29.897844 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:29.897808    2532 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-226rm_d45a88d4-e763-4a5e-87a3-91c0bb3bbf68/konnectivity-agent/0.log" limit=1024
Sep 18 19:46:32.252883 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:32.252849    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rc579_7959688f-be33-494c-a8e9-08d33f2c2d3b/kube-multus-additional-cni-plugins/0.log"
Sep 18 19:46:32.314979 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:32.314952    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rc579_7959688f-be33-494c-a8e9-08d33f2c2d3b/egress-router-binary-copy/0.log"
Sep 18 19:46:32.343875 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:32.343834    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rc579_7959688f-be33-494c-a8e9-08d33f2c2d3b/cni-plugins/0.log"
Sep 18 19:46:32.383550 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:32.383525    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rc579_7959688f-be33-494c-a8e9-08d33f2c2d3b/bond-cni-plugin/0.log"
Sep 18 19:46:32.413918 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:32.413894    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rc579_7959688f-be33-494c-a8e9-08d33f2c2d3b/routeoverride-cni/0.log"
Sep 18 19:46:32.451923 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:32.451892    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rc579_7959688f-be33-494c-a8e9-08d33f2c2d3b/whereabouts-cni-bincopy/0.log"
Sep 18 19:46:32.483707 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:32.483680    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rc579_7959688f-be33-494c-a8e9-08d33f2c2d3b/whereabouts-cni/0.log"
Sep 18 19:46:32.530749 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:32.530680    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-mmlqn_210a5221-5e8c-4bf5-9bf1-abe8413e3897/kube-multus/0.log"
Sep 18 19:46:32.683051 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:32.683021    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ps7w2_05c957d5-da77-47c2-8b9c-e7cdc087120c/network-metrics-daemon/0.log"
Sep 18 19:46:32.709545 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:32.709520    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ps7w2_05c957d5-da77-47c2-8b9c-e7cdc087120c/kube-rbac-proxy/0.log"
Sep 18 19:46:34.467218 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:34.467144    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sksw8_2cef0f4e-e44f-4996-8561-da9d261036c7/ovn-controller/0.log"
Sep 18 19:46:34.521683 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:34.521658    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sksw8_2cef0f4e-e44f-4996-8561-da9d261036c7/ovn-acl-logging/0.log"
Sep 18 19:46:34.521840 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:34.521810    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sksw8_2cef0f4e-e44f-4996-8561-da9d261036c7/ovn-acl-logging/1.log"
Sep 18 19:46:34.571195 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:34.571167    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sksw8_2cef0f4e-e44f-4996-8561-da9d261036c7/kube-rbac-proxy-node/0.log"
Sep 18 19:46:34.617171 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:34.617139    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sksw8_2cef0f4e-e44f-4996-8561-da9d261036c7/kube-rbac-proxy-ovn-metrics/0.log"
Sep 18 19:46:34.656469 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:34.656436    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sksw8_2cef0f4e-e44f-4996-8561-da9d261036c7/northd/0.log"
Sep 18 19:46:34.699246 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:34.699219    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sksw8_2cef0f4e-e44f-4996-8561-da9d261036c7/nbdb/0.log"
Sep 18 19:46:34.770790 ip-10-0-138-252 kubenswrapper[2532]: I0918 19:46:34.770718    2532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sksw8_2cef0f4e-e44f-4996-8561-da9d261036c7/sbdb/0.log"