Jul 28 12:53:06.354189 ip-10-0-138-158 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 12:53:06.794191 ip-10-0-138-158 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 12:53:06.794191 ip-10-0-138-158 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 12:53:06.794191 ip-10-0-138-158 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 12:53:06.794191 ip-10-0-138-158 kubenswrapper[2574]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 12:53:06.794191 ip-10-0-138-158 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 12:53:06.796402 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.796317    2574 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 12:53:06.800389 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800368    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 12:53:06.800389 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800384    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 12:53:06.800389 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800389    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 12:53:06.800389 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800393    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 12:53:06.800605 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800398    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 12:53:06.800605 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800402    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 12:53:06.800605 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800406    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 12:53:06.800605 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800410    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 12:53:06.800605 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800414    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 12:53:06.800605 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800418    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 12:53:06.800605 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800424    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 12:53:06.800605 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800429    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 12:53:06.800605 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800433    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 12:53:06.800605 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800437    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 12:53:06.800605 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800441    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 12:53:06.800605 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800447    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 12:53:06.800605 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800451    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 12:53:06.800605 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800455    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 12:53:06.800605 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800459    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 12:53:06.800605 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800463    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 12:53:06.800605 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800474    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 12:53:06.800605 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800478    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 12:53:06.800605 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800482    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 12:53:06.801409 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800486    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 12:53:06.801409 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800491    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 12:53:06.801409 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800495    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 12:53:06.801409 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800499    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 12:53:06.801409 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800504    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 12:53:06.801409 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800508    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 12:53:06.801409 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800512    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 12:53:06.801409 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800516    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 12:53:06.801409 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800520    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 12:53:06.801409 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800528    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 12:53:06.801409 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800534    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 12:53:06.801409 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800539    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 12:53:06.801409 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800544    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 12:53:06.801409 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800549    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 12:53:06.801409 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800554    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 12:53:06.801409 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800559    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 12:53:06.801409 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800563    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 12:53:06.801409 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800569    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 12:53:06.801409 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800574    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 12:53:06.801409 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800578    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 12:53:06.802306 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800582    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 12:53:06.802306 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800586    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 12:53:06.802306 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800591    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 12:53:06.802306 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800597    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 12:53:06.802306 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800602    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 12:53:06.802306 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800606    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 12:53:06.802306 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800610    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 12:53:06.802306 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800614    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 12:53:06.802306 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800618    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 12:53:06.802306 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800623    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 12:53:06.802306 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800627    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 12:53:06.802306 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800631    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 12:53:06.802306 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800635    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 12:53:06.802306 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800639    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 12:53:06.802306 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800644    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 12:53:06.802306 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800648    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 12:53:06.802306 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800652    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 12:53:06.802306 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800656    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 12:53:06.802306 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800661    2574 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 12:53:06.802306 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800665    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 12:53:06.803159 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800669    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 12:53:06.803159 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800675    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 12:53:06.803159 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800680    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 12:53:06.803159 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800684    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 12:53:06.803159 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800688    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 12:53:06.803159 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800693    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 12:53:06.803159 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800698    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 12:53:06.803159 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800702    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 12:53:06.803159 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800706    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 12:53:06.803159 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800710    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 12:53:06.803159 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800715    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 12:53:06.803159 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800719    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 12:53:06.803159 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800724    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 12:53:06.803159 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800728    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 12:53:06.803159 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800732    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 12:53:06.803159 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800736    2574 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 12:53:06.803159 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800740    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 12:53:06.803159 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800745    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 12:53:06.803159 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800750    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 12:53:06.803159 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800753    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 12:53:06.804041 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800757    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 12:53:06.804041 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800762    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 12:53:06.804041 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800766    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 12:53:06.804041 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800770    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 12:53:06.804041 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800775    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 12:53:06.804041 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800779    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 12:53:06.804041 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800785    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 12:53:06.804041 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800791    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 12:53:06.804041 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800796    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 12:53:06.804041 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800800    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 12:53:06.804041 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800804    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 12:53:06.804041 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800809    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 12:53:06.804041 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800813    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 12:53:06.804041 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800817    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 12:53:06.804041 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800822    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 12:53:06.804041 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.800827    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 12:53:06.804041 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802022    2574 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 12:53:06.804041 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802032    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 12:53:06.804041 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802036    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 12:53:06.804899 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802041    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 12:53:06.804899 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802045    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 12:53:06.804899 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802049    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 12:53:06.804899 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802055    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 12:53:06.804899 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802060    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 12:53:06.804899 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802065    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 12:53:06.804899 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802069    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 12:53:06.804899 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802073    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 12:53:06.804899 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802077    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 12:53:06.804899 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802081    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 12:53:06.804899 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802085    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 12:53:06.804899 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802090    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 12:53:06.804899 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802094    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 12:53:06.804899 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802098    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 12:53:06.804899 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802103    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 12:53:06.804899 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802107    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 12:53:06.804899 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802111    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 12:53:06.804899 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802116    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 12:53:06.804899 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802120    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 12:53:06.804899 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802125    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 12:53:06.805610 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802129    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 12:53:06.805610 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802133    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 12:53:06.805610 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802137    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 12:53:06.805610 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802141    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 12:53:06.805610 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802146    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 12:53:06.805610 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802150    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 12:53:06.805610 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802154    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 12:53:06.805610 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802159    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 12:53:06.805610 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802163    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 12:53:06.805610 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802168    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 12:53:06.805610 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802173    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 12:53:06.805610 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802177    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 12:53:06.805610 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802181    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 12:53:06.805610 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802187    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 12:53:06.805610 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802191    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 12:53:06.805610 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802195    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 12:53:06.805610 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802199    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 12:53:06.805610 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802204    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 12:53:06.805610 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802211    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 12:53:06.805610 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802217    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 12:53:06.806253 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802222    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 12:53:06.806253 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802226    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 12:53:06.806253 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802230    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 12:53:06.806253 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802235    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 12:53:06.806253 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802240    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 12:53:06.806253 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802244    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 12:53:06.806253 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802249    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 12:53:06.806253 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802253    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 12:53:06.806253 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802257    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 12:53:06.806253 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802261    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 12:53:06.806253 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802265    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 12:53:06.806253 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802270    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 12:53:06.806253 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802277    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 12:53:06.806253 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802283    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 12:53:06.806253 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802289    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 12:53:06.806253 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802294    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 12:53:06.806253 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802299    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 12:53:06.806253 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802305    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 12:53:06.806761 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802310    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 12:53:06.806761 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802316    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 12:53:06.806761 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802321    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 12:53:06.806761 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802325    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 12:53:06.806761 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802329    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 12:53:06.806761 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802333    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 12:53:06.806761 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802337    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 12:53:06.806761 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802341    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 12:53:06.806761 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802346    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 12:53:06.806761 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802350    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 12:53:06.806761 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802354    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 12:53:06.806761 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802358    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 12:53:06.806761 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802364    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 12:53:06.806761 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802369    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 12:53:06.806761 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802373    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 12:53:06.806761 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802377    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 12:53:06.806761 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802381    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 12:53:06.806761 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802386    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 12:53:06.806761 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802390    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 12:53:06.807248 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802394    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 12:53:06.807248 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802399    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 12:53:06.807248 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802403    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 12:53:06.807248 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802407    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 12:53:06.807248 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802411    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 12:53:06.807248 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802416    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 12:53:06.807248 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802420    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 12:53:06.807248 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802424    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 12:53:06.807248 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802428    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 12:53:06.807248 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802432    2574 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 12:53:06.807248 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802437    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 12:53:06.807248 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802441    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 12:53:06.807248 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802445    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 12:53:06.807248 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802449    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 12:53:06.807248 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802455    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 12:53:06.807248 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802459    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 12:53:06.807248 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802463    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 12:53:06.807248 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802467    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 12:53:06.807248 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.802472    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 12:53:06.807248 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803096    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 12:53:06.807811 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803106    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 12:53:06.807811 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803111    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 12:53:06.807811 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803116    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 12:53:06.807811 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803121    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 12:53:06.807811 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803125    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 12:53:06.807811 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803130    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 12:53:06.807811 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803135    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 12:53:06.807811 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803139    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 12:53:06.807811 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803144    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 12:53:06.807811 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803148    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 12:53:06.807811 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803152    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 12:53:06.807811 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803156    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 12:53:06.807811 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803160    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 12:53:06.807811 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803164    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 12:53:06.807811 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803171    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 12:53:06.807811 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803177    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 12:53:06.807811 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803181    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 12:53:06.807811 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803186    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 12:53:06.807811 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803191    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 12:53:06.808358 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803195    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 12:53:06.808358 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803200    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 12:53:06.808358 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803204    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 12:53:06.808358 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803208    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 12:53:06.808358 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803212    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 12:53:06.808358 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803216    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 12:53:06.808358 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803221    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 12:53:06.808358 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803225    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 12:53:06.808358 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803229    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 12:53:06.808358 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803233    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 12:53:06.808358 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803238    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 12:53:06.808358 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803242    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 12:53:06.808358 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803246    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 12:53:06.808358 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803251    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 12:53:06.808358 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803255    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 12:53:06.808358 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803259    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 12:53:06.808358 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803263    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 12:53:06.808358 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803267    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 12:53:06.808358 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803271    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 12:53:06.808358 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803280    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 12:53:06.808949 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803287    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 12:53:06.808949 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803291    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 12:53:06.808949 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803295    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 12:53:06.808949 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803299    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 12:53:06.808949 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803303    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 12:53:06.808949 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803310    2574 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 12:53:06.808949 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803314    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 12:53:06.808949 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803318    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 12:53:06.808949 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803322    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 12:53:06.808949 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803327    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 12:53:06.808949 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803331    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 12:53:06.808949 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803336    2574 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 12:53:06.808949 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803340    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 12:53:06.808949 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803344    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 12:53:06.808949 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803348    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 12:53:06.808949 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803352    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 12:53:06.808949 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803356    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 12:53:06.808949 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803360    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 12:53:06.808949 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803365    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 12:53:06.808949 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803369    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 12:53:06.809430 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803373    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 12:53:06.809430 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803377    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 12:53:06.809430 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803382    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 12:53:06.809430 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803386    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 12:53:06.809430 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803390    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 12:53:06.809430 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803394    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 12:53:06.809430 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803398    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 12:53:06.809430 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803402    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 12:53:06.809430 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803406    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 12:53:06.809430 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803410    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 12:53:06.809430 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803414    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 12:53:06.809430 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803418    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 12:53:06.809430 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803423    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 12:53:06.809430 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803427    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 12:53:06.809430 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803431    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 12:53:06.809430 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803435    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 12:53:06.809430 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803439    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 12:53:06.809430 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803445    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 12:53:06.809430 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803449    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 12:53:06.809430 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803453    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 12:53:06.809912 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803458    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 12:53:06.809912 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803464    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 12:53:06.809912 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803468    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 12:53:06.809912 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803473    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 12:53:06.809912 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803477    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 12:53:06.809912 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803482    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 12:53:06.809912 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803485    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 12:53:06.809912 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803489    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 12:53:06.809912 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803493    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 12:53:06.809912 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803498    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 12:53:06.809912 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803504    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 12:53:06.809912 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803509    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 12:53:06.809912 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803514    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 12:53:06.809912 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803519    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 12:53:06.809912 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803523    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 12:53:06.809912 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803527    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 12:53:06.809912 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803532    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 12:53:06.809912 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803537    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 12:53:06.809912 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.803542    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 12:53:06.810429 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.803631    2574 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 12:53:06.810429 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.803649    2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 12:53:06.810429 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.803658    2574 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 12:53:06.810429 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.803664    2574 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 12:53:06.810429 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.803670    2574 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 12:53:06.810429 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.803675    2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 12:53:06.810429 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.803682    2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 12:53:06.810429 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.803688    2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 12:53:06.810429 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.803693    2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 12:53:06.810429 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.803698    2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 12:53:06.810429 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.803705    2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 12:53:06.810429 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.803711    2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 12:53:06.810429 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.803715    2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 12:53:06.810429 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.803721    2574 flags.go:64] FLAG: --cgroup-root=""
Jul 28 12:53:06.810429 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.803725    2574 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 12:53:06.810429 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.803730    2574 flags.go:64] FLAG: --client-ca-file=""
Jul 28 12:53:06.810429 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.803738    2574 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 12:53:06.810429 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.803744    2574 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 12:53:06.810429 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.803751    2574 flags.go:64] FLAG: --cluster-domain=""
Jul 28 12:53:06.810429 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.803755    2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 12:53:06.810429 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.803761    2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 12:53:06.810429 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.803766    2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 12:53:06.810429 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.803771    2574 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 12:53:06.810429 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.803777    2574 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 12:53:06.811104 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.803782    2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 12:53:06.811104 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.803787    2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 12:53:06.811104 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.803792    2574 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 12:53:06.811104 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.803797    2574 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 12:53:06.811104 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.803802    2574 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 12:53:06.811104 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.803806    2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 12:53:06.811104 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.803811    2574 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 12:53:06.811104 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.803816    2574 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 12:53:06.811104 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.803822    2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 12:53:06.811104 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.803827    2574 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 12:53:06.811104 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.803832    2574 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 12:53:06.811104 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.803837    2574 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 12:53:06.811104 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.803841    2574 flags.go:64] FLAG: --enable-server="true"
Jul 28 12:53:06.811104 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.803846    2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 12:53:06.811104 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.803852    2574 flags.go:64] FLAG: --event-burst="100"
Jul 28 12:53:06.811104 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.803858    2574 flags.go:64] FLAG: --event-qps="50"
Jul 28 12:53:06.811104 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.803863    2574 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 12:53:06.811104 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.803868    2574 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 12:53:06.811104 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.803874    2574 flags.go:64] FLAG: --eviction-hard=""
Jul 28 12:53:06.811104 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.803880    2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 12:53:06.811104 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.803885    2574 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 12:53:06.811104 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.803890    2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 12:53:06.811104 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.803894    2574 flags.go:64] FLAG: --eviction-soft=""
Jul 28 12:53:06.811104 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.803899    2574 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 12:53:06.811104 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.803905    2574 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 12:53:06.812008 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.803910    2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 12:53:06.812008 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.803915    2574 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 12:53:06.812008 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.803940    2574 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 12:53:06.812008 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.803945    2574 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 12:53:06.812008 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.803950    2574 flags.go:64] FLAG: --feature-gates=""
Jul 28 12:53:06.812008 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.803956    2574 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 12:53:06.812008 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.803961    2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 12:53:06.812008 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.803965    2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 12:53:06.812008 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.803971    2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 12:53:06.812008 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.803976    2574 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 12:53:06.812008 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.803981    2574 flags.go:64] FLAG: --help="false"
Jul 28 12:53:06.812008 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.803985    2574 flags.go:64] FLAG: --hostname-override="ip-10-0-138-158.ec2.internal"
Jul 28 12:53:06.812008 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.803991    2574 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 12:53:06.812008 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.803995    2574 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 12:53:06.812008 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804000    2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 12:53:06.812008 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804005    2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 12:53:06.812008 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804011    2574 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 12:53:06.812008 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804016    2574 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 12:53:06.812008 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804021    2574 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 12:53:06.812008 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804025    2574 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 12:53:06.812008 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804030    2574 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 12:53:06.812008 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804034    2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 12:53:06.812008 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804039    2574 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 12:53:06.812008 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804050    2574 flags.go:64] FLAG: --kube-reserved=""
Jul 28 12:53:06.812789 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804055    2574 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 12:53:06.812789 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804060    2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 12:53:06.812789 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804065    2574 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 12:53:06.812789 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804069    2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 12:53:06.812789 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804074    2574 flags.go:64] FLAG: --lock-file=""
Jul 28 12:53:06.812789 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804079    2574 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 12:53:06.812789 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804083    2574 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 12:53:06.812789 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804090    2574 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 12:53:06.812789 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804105    2574 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 12:53:06.812789 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804110    2574 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 12:53:06.812789 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804115    2574 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 12:53:06.812789 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804119    2574 flags.go:64] FLAG: --logging-format="text"
Jul 28 12:53:06.812789 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804124    2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 12:53:06.812789 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804129    2574 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 12:53:06.812789 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804134    2574 flags.go:64] FLAG: --manifest-url=""
Jul 28 12:53:06.812789 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804138    2574 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 12:53:06.812789 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804146    2574 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 12:53:06.812789 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804151    2574 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 12:53:06.812789 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804157    2574 flags.go:64] FLAG: --max-pods="110"
Jul 28 12:53:06.812789 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804162    2574 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 12:53:06.812789 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804167    2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 12:53:06.812789 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804172    2574 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 12:53:06.812789 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804176    2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 12:53:06.812789 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804181    2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 12:53:06.812789 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804186    2574 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 12:53:06.813598 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804191    2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 12:53:06.813598 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804202    2574 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 12:53:06.813598 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804207    2574 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 12:53:06.813598 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804212    2574 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 12:53:06.813598 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804217    2574 flags.go:64] FLAG: --pod-cidr=""
Jul 28 12:53:06.813598 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804222    2574 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 12:53:06.813598 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804230    2574 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 12:53:06.813598 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804235    2574 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 12:53:06.813598 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804241    2574 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 12:53:06.813598 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804246    2574 flags.go:64] FLAG: --port="10250"
Jul 28 12:53:06.813598 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804251    2574 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 12:53:06.813598 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804256    2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c7f9d41b49da1556"
Jul 28 12:53:06.813598 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804261    2574 flags.go:64] FLAG: --qos-reserved=""
Jul 28 12:53:06.813598 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804266    2574 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 12:53:06.813598 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804270    2574 flags.go:64] FLAG: --register-node="true"
Jul 28 12:53:06.813598 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804277    2574 flags.go:64] FLAG: --register-with-taints=""
Jul 28 12:53:06.813598 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804283    2574 flags.go:64] FLAG: --registry-burst="10"
Jul 28 12:53:06.813598 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804288    2574 flags.go:64] FLAG: --registry-qps="5"
Jul 28 12:53:06.813598 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804293    2574 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 12:53:06.813598 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804298    2574 flags.go:64] FLAG: --reserved-memory=""
Jul 28 12:53:06.813598 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804303    2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 12:53:06.813598 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804308    2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 12:53:06.813598 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804313    2574 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 12:53:06.813598 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804317    2574 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 12:53:06.813598 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804322    2574 flags.go:64] FLAG: --runonce="false"
Jul 28 12:53:06.814220 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804327    2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 12:53:06.814220 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804332    2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 12:53:06.814220 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804337    2574 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 12:53:06.814220 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804342    2574 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 12:53:06.814220 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804346    2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 12:53:06.814220 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804351    2574 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 12:53:06.814220 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804356    2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 12:53:06.814220 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804361    2574 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 12:53:06.814220 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804365    2574 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 12:53:06.814220 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804370    2574 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 12:53:06.814220 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804375    2574 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 12:53:06.814220 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804380    2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 12:53:06.814220 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804384    2574 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 12:53:06.814220 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804390    2574 flags.go:64] FLAG: --system-cgroups=""
Jul 28 12:53:06.814220 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804395    2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 12:53:06.814220 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804404    2574 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 12:53:06.814220 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804409    2574 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 12:53:06.814220 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804414    2574 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 12:53:06.814220 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804421    2574 flags.go:64] FLAG: --tls-min-version=""
Jul 28 12:53:06.814220 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804425    2574 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 12:53:06.814220 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804430    2574 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 12:53:06.814220 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804435    2574 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 12:53:06.814220 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804441    2574 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 12:53:06.814220 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804446    2574 flags.go:64] FLAG: --v="2"
Jul 28 12:53:06.814220 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804452    2574 flags.go:64] FLAG: --version="false"
Jul 28 12:53:06.814825 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804458    2574 flags.go:64] FLAG: --vmodule=""
Jul 28 12:53:06.814825 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804464    2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 12:53:06.814825 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.804470    2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 12:53:06.814825 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804668    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 12:53:06.814825 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804677    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 12:53:06.814825 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804682    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 12:53:06.814825 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804687    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 12:53:06.814825 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804691    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 12:53:06.814825 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804696    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 12:53:06.814825 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804700    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 12:53:06.814825 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804704    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 12:53:06.814825 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804709    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 12:53:06.814825 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804713    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 12:53:06.814825 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804717    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 12:53:06.814825 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804721    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 12:53:06.814825 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804725    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 12:53:06.814825 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804730    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 12:53:06.814825 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804734    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 12:53:06.814825 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804739    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 12:53:06.814825 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804743    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 12:53:06.815332 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804748    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 12:53:06.815332 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804752    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 12:53:06.815332 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804757    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 12:53:06.815332 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804766    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 12:53:06.815332 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804770    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 12:53:06.815332 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804774    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 12:53:06.815332 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804779    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 12:53:06.815332 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804783    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 12:53:06.815332 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804787    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 12:53:06.815332 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804796    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 12:53:06.815332 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804801    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 12:53:06.815332 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804805    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 12:53:06.815332 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804809    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 12:53:06.815332 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804813    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 12:53:06.815332 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804817    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 12:53:06.815332 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804821    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 12:53:06.815332 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804825    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 12:53:06.815332 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804830    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 12:53:06.815332 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804834    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 12:53:06.815332 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804838    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 12:53:06.815847 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804842    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 12:53:06.815847 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804846    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 12:53:06.815847 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804851    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 12:53:06.815847 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804855    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 12:53:06.815847 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804860    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 12:53:06.815847 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804864    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 12:53:06.815847 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804871    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 12:53:06.815847 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804877    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 12:53:06.815847 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804881    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 12:53:06.815847 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804886    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 12:53:06.815847 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804891    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 12:53:06.815847 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804895    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 12:53:06.815847 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804901    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 12:53:06.815847 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804906    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 12:53:06.815847 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804910    2574 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 12:53:06.815847 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804913    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 12:53:06.815847 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804937    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 12:53:06.815847 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804943    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 12:53:06.815847 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804947    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 12:53:06.816327 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804951    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 12:53:06.816327 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804955    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 12:53:06.816327 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804962    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 12:53:06.816327 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804966    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 12:53:06.816327 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804970    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 12:53:06.816327 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804975    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 12:53:06.816327 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804979    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 12:53:06.816327 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804983    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 12:53:06.816327 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804988    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 12:53:06.816327 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804992    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 12:53:06.816327 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.804998    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 12:53:06.816327 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.805003    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 12:53:06.816327 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.805008    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 12:53:06.816327 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.805012    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 12:53:06.816327 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.805016    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 12:53:06.816327 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.805021    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 12:53:06.816327 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.805026    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 12:53:06.816327 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.805030    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 12:53:06.816327 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.805035    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 12:53:06.816792 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.805041    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 12:53:06.816792 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.805045    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 12:53:06.816792 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.805050    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 12:53:06.816792 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.805054    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 12:53:06.816792 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.805058    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 12:53:06.816792 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.805062    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 12:53:06.816792 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.805068    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 12:53:06.816792 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.805073    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 12:53:06.816792 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.805076    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 12:53:06.816792 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.805080    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 12:53:06.816792 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.805085    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 12:53:06.816792 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.805090    2574 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 12:53:06.816792 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.805094    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 12:53:06.816792 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.805098    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 12:53:06.816792 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.805103    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 12:53:06.816792 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.805109    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 12:53:06.816792 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.805114    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 12:53:06.816792 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.805118    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 12:53:06.816792 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.805122    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 12:53:06.816792 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.805126    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 12:53:06.817346 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.805130    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 12:53:06.817346 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.805134    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 12:53:06.817346 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.805139    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 12:53:06.817346 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.805143    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 12:53:06.817346 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.805151    2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 12:53:06.817346 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.813867    2574 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 12:53:06.817346 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.813880    2574 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 12:53:06.817346 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.814789    2574 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 12:53:06.817346 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.814797    2574 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 12:53:06.817346 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.814852    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 12:53:06.817346 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.814857    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 12:53:06.817346 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.814861    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 12:53:06.817346 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.814864    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 12:53:06.817346 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.814868    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 12:53:06.817695 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.814871    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 12:53:06.817695 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.814874    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 12:53:06.817695 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.814877    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 12:53:06.817695 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.814879    2574 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 12:53:06.817695 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.814882    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 12:53:06.817695 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.814885    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 12:53:06.817695 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.814888    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 12:53:06.817695 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.814891    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 12:53:06.817695 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.814893    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 12:53:06.817695 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.814896    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 12:53:06.817695 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.814899    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 12:53:06.817695 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.814902    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 12:53:06.817695 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.814904    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 12:53:06.817695 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.814907    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 12:53:06.817695 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.814909    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 12:53:06.817695 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.814912    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 12:53:06.817695 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.814914    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 12:53:06.817695 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.814917    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 12:53:06.817695 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.814934    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 12:53:06.817695 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.814937    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 12:53:06.818186 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.814940    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 12:53:06.818186 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.814944    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 12:53:06.818186 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.814949    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 12:53:06.818186 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.814952    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 12:53:06.818186 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.814956    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 12:53:06.818186 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.814959    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 12:53:06.818186 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.814962    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 12:53:06.818186 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.814965    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 12:53:06.818186 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.814968    2574 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 12:53:06.818186 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.814970    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 12:53:06.818186 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.814973    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 12:53:06.818186 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.814975    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 12:53:06.818186 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.814978    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 12:53:06.818186 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.814980    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 12:53:06.818186 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.814983    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 12:53:06.818186 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.814985    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 12:53:06.818186 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.814988    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 12:53:06.818186 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.814990    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 12:53:06.818186 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.814993    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 12:53:06.818631 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.814995    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 12:53:06.818631 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.814998    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 12:53:06.818631 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815001    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 12:53:06.818631 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815003    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 12:53:06.818631 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815006    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 12:53:06.818631 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815008    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 12:53:06.818631 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815011    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 12:53:06.818631 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815013    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 12:53:06.818631 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815016    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 12:53:06.818631 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815018    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 12:53:06.818631 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815021    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 12:53:06.818631 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815023    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 12:53:06.818631 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815026    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 12:53:06.818631 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815028    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 12:53:06.818631 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815032    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 12:53:06.818631 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815036    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 12:53:06.818631 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815040    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 12:53:06.818631 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815043    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 12:53:06.818631 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815046    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 12:53:06.819110 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815049    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 12:53:06.819110 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815051    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 12:53:06.819110 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815054    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 12:53:06.819110 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815057    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 12:53:06.819110 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815059    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 12:53:06.819110 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815062    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 12:53:06.819110 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815064    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 12:53:06.819110 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815067    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 12:53:06.819110 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815069    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 12:53:06.819110 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815072    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 12:53:06.819110 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815074    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 12:53:06.819110 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815077    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 12:53:06.819110 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815080    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 12:53:06.819110 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815082    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 12:53:06.819110 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815085    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 12:53:06.819110 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815087    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 12:53:06.819110 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815091    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 12:53:06.819110 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815093    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 12:53:06.819110 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815096    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 12:53:06.819110 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815098    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 12:53:06.819589 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815101    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 12:53:06.819589 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815103    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 12:53:06.819589 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815106    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 12:53:06.819589 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815108    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 12:53:06.819589 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815111    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 12:53:06.819589 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815113    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 12:53:06.819589 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815116    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 12:53:06.819589 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815119    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 12:53:06.819589 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815121    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 12:53:06.819589 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815124    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 12:53:06.819589 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815126    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 12:53:06.819589 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815129    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 12:53:06.819589 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815132    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 12:53:06.819589 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815134    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 12:53:06.819589 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815137    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 12:53:06.819589 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815140    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 12:53:06.820023 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.815145    2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 12:53:06.820023 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815265    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 12:53:06.820023 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815270    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 12:53:06.820023 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815274    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 12:53:06.820023 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815277    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 12:53:06.820023 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815279    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 12:53:06.820023 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815282    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 12:53:06.820023 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815284    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 12:53:06.820023 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815288    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 12:53:06.820023 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815292    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 12:53:06.820023 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815294    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 12:53:06.820023 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815297    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 12:53:06.820023 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815300    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 12:53:06.820023 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815303    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 12:53:06.820023 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815305    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 12:53:06.820393 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815309    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 12:53:06.820393 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815313    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 12:53:06.820393 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815315    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 12:53:06.820393 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815318    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 12:53:06.820393 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815321    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 12:53:06.820393 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815324    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 12:53:06.820393 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815326    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 12:53:06.820393 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815329    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 12:53:06.820393 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815331    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 12:53:06.820393 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815334    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 12:53:06.820393 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815337    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 12:53:06.820393 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815339    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 12:53:06.820393 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815342    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 12:53:06.820393 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815344    2574 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 12:53:06.820393 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815347    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 12:53:06.820393 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815350    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 12:53:06.820393 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815353    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 12:53:06.820393 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815355    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 12:53:06.820393 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815358    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 12:53:06.820393 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815361    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 12:53:06.820891 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815363    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 12:53:06.820891 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815366    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 12:53:06.820891 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815368    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 12:53:06.820891 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815371    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 12:53:06.820891 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815373    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 12:53:06.820891 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815376    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 12:53:06.820891 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815378    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 12:53:06.820891 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815381    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 12:53:06.820891 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815383    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 12:53:06.820891 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815386    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 12:53:06.820891 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815388    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 12:53:06.820891 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815392    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 12:53:06.820891 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815394    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 12:53:06.820891 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815397    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 12:53:06.820891 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815400    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 12:53:06.820891 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815402    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 12:53:06.820891 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815405    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 12:53:06.820891 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815407    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 12:53:06.820891 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815409    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 12:53:06.820891 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815412    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 12:53:06.821397 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815415    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 12:53:06.821397 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815417    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 12:53:06.821397 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815420    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 12:53:06.821397 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815422    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 12:53:06.821397 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815425    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 12:53:06.821397 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815427    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 12:53:06.821397 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815429    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 12:53:06.821397 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815432    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 12:53:06.821397 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815434    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 12:53:06.821397 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815437    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 12:53:06.821397 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815441    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 12:53:06.821397 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815444    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 12:53:06.821397 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815447    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 12:53:06.821397 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815450    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 12:53:06.821397 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815453    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 12:53:06.821397 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815455    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 12:53:06.821397 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815458    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 12:53:06.821397 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815460    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 12:53:06.821397 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815463    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 12:53:06.821875 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815465    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 12:53:06.821875 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815468    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 12:53:06.821875 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815471    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 12:53:06.821875 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815473    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 12:53:06.821875 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815476    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 12:53:06.821875 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815479    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 12:53:06.821875 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815481    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 12:53:06.821875 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815484    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 12:53:06.821875 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815487    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 12:53:06.821875 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815489    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 12:53:06.821875 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815492    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 12:53:06.821875 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815494    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 12:53:06.821875 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815497    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 12:53:06.821875 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815500    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 12:53:06.821875 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815502    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 12:53:06.821875 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815505    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 12:53:06.821875 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815507    2574 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 12:53:06.821875 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815510    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 12:53:06.821875 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815512    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 12:53:06.821875 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815515    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 12:53:06.822416 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815517    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 12:53:06.822416 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815520    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 12:53:06.822416 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815522    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 12:53:06.822416 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815525    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 12:53:06.822416 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815528    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 12:53:06.822416 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:06.815530    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 12:53:06.822416 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.815534    2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 12:53:06.822416 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.815636    2574 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 12:53:06.822416 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.819830    2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 12:53:06.822416 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.821480    2574 server.go:1019] "Starting client certificate rotation"
Jul 28 12:53:06.822416 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.821577    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 12:53:06.822416 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.821612    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 12:53:06.824292 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.824280    2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 12:53:06.826966 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.826952    2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 12:53:06.837549 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.837527    2574 log.go:25] "Validated CRI v1 runtime API"
Jul 28 12:53:06.840177 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.840160    2574 log.go:25] "Validated CRI v1 image API"
Jul 28 12:53:06.840962 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.840943    2574 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 12:53:06.843542 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.843523    2574 fs.go:135] Filesystem UUIDs: map[1e457516-c1ed-4cda-a8e0-9b3a3a34b5ae:/dev/nvme0n1p3 39e73af6-a863-434d-9e60-38a3c792f525:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Jul 28 12:53:06.843601 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.843539    2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 12:53:06.848735 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.848634    2574 manager.go:216] Machine: {Timestamp:2026-07-28 12:53:06.847226192 +0000 UTC m=+0.367375050 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3126180 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec280ac297728ca38630486fd098dff3 SystemUUID:ec280ac2-9772-8ca3-8630-486fd098dff3 BootID:772d2f76-1e25-4bb4-8198-4c629d7ce9b1 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:c8:8e:05:91:d9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:c8:8e:05:91:d9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2a:42:56:0c:64:3b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 12:53:06.848735 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.848734    2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 12:53:06.848850 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.848817    2574 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 12:53:06.852182 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.852161    2574 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 12:53:06.852316 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.852183    2574 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-158.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 12:53:06.852665 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.852647    2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 12:53:06.853203 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.853186    2574 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 12:53:06.853240 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.853209    2574 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 12:53:06.853240 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.853220    2574 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 12:53:06.853616 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.853538    2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 12:53:06.854977 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.854956    2574 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 12:53:06.857828 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.857817    2574 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 12:53:06.857948 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.857938    2574 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 12:53:06.857980 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.857966    2574 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 12:53:06.858012 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.857982    2574 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 12:53:06.858012 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.857999    2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 12:53:06.858012 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.858008    2574 kubelet.go:398] "Adding apiserver pod source"
Jul 28 12:53:06.858101 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.858017    2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 12:53:06.858314 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.858303    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 12:53:06.858349 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.858321    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 12:53:06.860654 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.860640    2574 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 12:53:06.863087 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.863075    2574 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 12:53:06.863146 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.863104    2574 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 12:53:06.865302 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.865289    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 12:53:06.865366 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.865306    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 12:53:06.865366 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.865312    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 12:53:06.865366 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.865318    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 12:53:06.865366 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.865327    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 12:53:06.865366 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.865335    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 12:53:06.865366 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.865348    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 12:53:06.865366 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.865355    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 12:53:06.865366 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.865361    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 12:53:06.865366 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.865367    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 12:53:06.865604 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.865375    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 12:53:06.865604 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.865380    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 12:53:06.866304 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.866294    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 12:53:06.866336 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.866305    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 12:53:06.868827 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.868810    2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-158.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 12:53:06.868883 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:06.868832    2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-158.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 12:53:06.868915 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:06.868883    2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 12:53:06.869792 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.869770    2574 server.go:1268] "Started kubelet"
Jul 28 12:53:06.869845 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.869803    2574 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 12:53:06.869989 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.869941    2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 12:53:06.870061 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.870010    2574 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 12:53:06.871346 ip-10-0-138-158 systemd[1]: Started Kubernetes Kubelet.
Jul 28 12:53:06.871984 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.871965    2574 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 12:53:06.872452 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.872438    2574 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 12:53:06.875152 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:06.875130    2574 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 12:53:06.877182 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:06.876036    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-158.ec2.internal.18c67538470e79e9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-158.ec2.internal,UID:ip-10-0-138-158.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-158.ec2.internal,},FirstTimestamp:2026-07-28 12:53:06.869729769 +0000 UTC m=+0.389878610,LastTimestamp:2026-07-28 12:53:06.869729769 +0000 UTC m=+0.389878610,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-158.ec2.internal,}"
Jul 28 12:53:06.881630 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.881602    2574 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 12:53:06.881719 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.881680    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 12:53:06.881769 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:06.881732    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-158.ec2.internal\" not found"
Jul 28 12:53:06.882468 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.882448    2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 12:53:06.882468 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.882451    2574 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 12:53:06.882623 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.882479    2574 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 12:53:06.882623 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.882578    2574 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 12:53:06.882623 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.882590    2574 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 12:53:06.882910 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.882891    2574 factory.go:153] Registering CRI-O factory
Jul 28 12:53:06.882910 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.882913    2574 factory.go:223] Registration of the crio container factory successfully
Jul 28 12:53:06.883053 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.882988    2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 12:53:06.883053 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.883006    2574 factory.go:55] Registering systemd factory
Jul 28 12:53:06.883053 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.883015    2574 factory.go:223] Registration of the systemd container factory successfully
Jul 28 12:53:06.883053 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.883033    2574 factory.go:103] Registering Raw factory
Jul 28 12:53:06.883053 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.883057    2574 manager.go:1196] Started watching for new ooms in manager
Jul 28 12:53:06.883691 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.883677    2574 manager.go:349] Starting recovery of all containers
Jul 28 12:53:06.883962 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:06.883936    2574 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 28 12:53:06.884071 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:06.884034    2574 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-158.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 28 12:53:06.892258 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.892225    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 12:53:06.897286 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.897265    2574 manager.go:354] Recovery completed
Jul 28 12:53:06.898705 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:06.898689    2574 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Jul 28 12:53:06.902312 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.902295    2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lkkmp"
Jul 28 12:53:06.902716 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.902705    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:53:06.905446 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.905425    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-158.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:53:06.905531 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.905451    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-158.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:53:06.905531 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.905461    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-158.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:53:06.906250 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.906227    2574 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 12:53:06.906250 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.906240    2574 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 12:53:06.906408 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.906255    2574 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 12:53:06.906408 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.906292    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:53:06.907075 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.907059    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-158.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:53:06.907147 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.907088    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-158.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:53:06.907147 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.907101    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-158.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:53:06.908912 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.908899    2574 policy_none.go:49] "None policy: Start"
Jul 28 12:53:06.908965 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.908916    2574 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 12:53:06.908965 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.908947    2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 12:53:06.910164 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.910148    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lkkmp"
Jul 28 12:53:06.912097 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.912084    2574 policy_none.go:47] "Start"
Jul 28 12:53:06.916176 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:06.916098    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-158.ec2.internal.18c67538492f5e28  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-158.ec2.internal,UID:ip-10-0-138-158.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-158.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-158.ec2.internal,},FirstTimestamp:2026-07-28 12:53:06.905439784 +0000 UTC m=+0.425588630,LastTimestamp:2026-07-28 12:53:06.905439784 +0000 UTC m=+0.425588630,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-158.ec2.internal,}"
Jul 28 12:53:06.917956 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:06.917887    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-158.ec2.internal.18c67538492f9d4c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-158.ec2.internal,UID:ip-10-0-138-158.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-158.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-158.ec2.internal,},FirstTimestamp:2026-07-28 12:53:06.905455948 +0000 UTC m=+0.425604787,LastTimestamp:2026-07-28 12:53:06.905455948 +0000 UTC m=+0.425604787,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-158.ec2.internal,}"
Jul 28 12:53:06.966221 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.945745    2574 manager.go:337] "Starting Device Plugin manager"
Jul 28 12:53:06.966221 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:06.945768    2574 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 12:53:06.966221 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.945777    2574 server.go:87] "Starting device plugin registration server"
Jul 28 12:53:06.966221 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.945984    2574 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 12:53:06.966221 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.945998    2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 12:53:06.966221 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.946135    2574 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 12:53:06.966221 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.946200    2574 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 12:53:06.966221 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.946211    2574 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 12:53:06.966221 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:06.946716    2574 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 12:53:06.966221 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:06.946785    2574 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-158.ec2.internal\" not found"
Jul 28 12:53:06.981234 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.981217    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 12:53:06.981329 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.981262    2574 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 12:53:06.981329 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.981284    2574 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 12:53:06.981411 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:06.981349    2574 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 12:53:06.983991 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:06.983975    2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 12:53:07.046247 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:07.046187    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:53:07.046962 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:07.046939    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-158.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:53:07.047051 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:07.046971    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-158.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:53:07.047051 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:07.046981    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-158.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:53:07.047051 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:07.047012    2574 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-158.ec2.internal"
Jul 28 12:53:07.059365 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:07.059343    2574 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-158.ec2.internal"
Jul 28 12:53:07.059365 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:07.059362    2574 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-158.ec2.internal\": node \"ip-10-0-138-158.ec2.internal\" not found"
Jul 28 12:53:07.080570 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:07.080548    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-158.ec2.internal\" not found"
Jul 28 12:53:07.081543 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:07.081525    2574 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-158.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-158.ec2.internal"]
Jul 28 12:53:07.081590 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:07.081579    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:53:07.082806 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:07.082789    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ead970784cc741b3c8d768dce86822b5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-158.ec2.internal\" (UID: \"ead970784cc741b3c8d768dce86822b5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-158.ec2.internal"
Jul 28 12:53:07.082893 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:07.082821    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ead970784cc741b3c8d768dce86822b5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-158.ec2.internal\" (UID: \"ead970784cc741b3c8d768dce86822b5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-158.ec2.internal"
Jul 28 12:53:07.082893 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:07.082844    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-158.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:53:07.082893 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:07.082863    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-158.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:53:07.082893 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:07.082872    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-158.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:53:07.085093 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:07.085079    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:53:07.085181 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:07.085165    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-158.ec2.internal"
Jul 28 12:53:07.085219 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:07.085199    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:53:07.085688 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:07.085672    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-158.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:53:07.085688 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:07.085680    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-158.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:53:07.085789 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:07.085696    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-158.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:53:07.085789 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:07.085700    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-158.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:53:07.085789 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:07.085708    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-158.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:53:07.085789 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:07.085710    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-158.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:53:07.087929 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:07.087906    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-158.ec2.internal"
Jul 28 12:53:07.088017 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:07.087945    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:53:07.088693 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:07.088679    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-158.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:53:07.088750 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:07.088700    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-158.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:53:07.088750 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:07.088714    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-158.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:53:07.113467 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:07.113451    2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-158.ec2.internal\" not found" node="ip-10-0-138-158.ec2.internal"
Jul 28 12:53:07.117719 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:07.117703    2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-158.ec2.internal\" not found" node="ip-10-0-138-158.ec2.internal"
Jul 28 12:53:07.181430 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:07.181412    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-158.ec2.internal\" not found"
Jul 28 12:53:07.183641 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:07.183626    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ead970784cc741b3c8d768dce86822b5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-158.ec2.internal\" (UID: \"ead970784cc741b3c8d768dce86822b5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-158.ec2.internal"
Jul 28 12:53:07.183727 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:07.183654    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ead970784cc741b3c8d768dce86822b5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-158.ec2.internal\" (UID: \"ead970784cc741b3c8d768dce86822b5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-158.ec2.internal"
Jul 28 12:53:07.183727 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:07.183690    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6f72deeae9606046a5729c2df486eb15-config\") pod \"kube-apiserver-proxy-ip-10-0-138-158.ec2.internal\" (UID: \"6f72deeae9606046a5729c2df486eb15\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-158.ec2.internal"
Jul 28 12:53:07.183727 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:07.183716    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ead970784cc741b3c8d768dce86822b5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-158.ec2.internal\" (UID: \"ead970784cc741b3c8d768dce86822b5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-158.ec2.internal"
Jul 28 12:53:07.183727 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:07.183721    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ead970784cc741b3c8d768dce86822b5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-158.ec2.internal\" (UID: \"ead970784cc741b3c8d768dce86822b5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-158.ec2.internal"
Jul 28 12:53:07.281970 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:07.281944    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-158.ec2.internal\" not found"
Jul 28 12:53:07.284125 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:07.284113    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6f72deeae9606046a5729c2df486eb15-config\") pod \"kube-apiserver-proxy-ip-10-0-138-158.ec2.internal\" (UID: \"6f72deeae9606046a5729c2df486eb15\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-158.ec2.internal"
Jul 28 12:53:07.284176 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:07.284138    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6f72deeae9606046a5729c2df486eb15-config\") pod \"kube-apiserver-proxy-ip-10-0-138-158.ec2.internal\" (UID: \"6f72deeae9606046a5729c2df486eb15\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-158.ec2.internal"
Jul 28 12:53:07.382560 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:07.382514    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-158.ec2.internal\" not found"
Jul 28 12:53:07.415859 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:07.415842    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-158.ec2.internal"
Jul 28 12:53:07.420699 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:07.420674    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-158.ec2.internal"
Jul 28 12:53:07.482693 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:07.482671    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-158.ec2.internal\" not found"
Jul 28 12:53:07.583139 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:07.583111    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-158.ec2.internal\" not found"
Jul 28 12:53:08.998397 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:08.998340    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-158.ec2.internal\" not found"
Jul 28 12:53:09.009383 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.009365    2574 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 12:53:09.096563 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.096531    2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-158.ec2.internal"
Jul 28 12:53:09.110372 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.110349    2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 12:53:09.111919 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.111904    2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-158.ec2.internal"
Jul 28 12:53:09.121031 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.121019    2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 12:53:09.135145 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.135126    2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 12:53:09.135268 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.135250    2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 12:53:09.135319 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.135267    2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 12:53:09.173434 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.173419    2574 apiserver.go:52] "Watching apiserver"
Jul 28 12:53:09.196607 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.196593    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 12:53:09.208357 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.208339    2574 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 12:53:09.209266 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.209246    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-158.ec2.internal","openshift-multus/multus-additional-cni-plugins-zf5gt","openshift-multus/network-metrics-daemon-ps58v","openshift-network-operator/iptables-alerter-bc4gl","kube-system/global-pull-secret-syncer-bdgzk","kube-system/konnectivity-agent-cxz2b","openshift-dns/node-resolver-66b94","openshift-multus/multus-xtlx4","openshift-network-diagnostics/network-check-target-wk6vn","openshift-ovn-kubernetes/ovnkube-node-nd4vq","kube-system/kube-apiserver-proxy-ip-10-0-138-158.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csb84","openshift-cluster-node-tuning-operator/tuned-xnfmf","openshift-image-registry/node-ca-7n245"]
Jul 28 12:53:09.212265 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.212249    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-66b94"
Jul 28 12:53:09.214732 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.214715    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xnfmf"
Jul 28 12:53:09.215523 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.215507    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2nw72\""
Jul 28 12:53:09.215757 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.215743    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 12:53:09.216198 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.216183    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 12:53:09.216735 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.216721    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ps58v"
Jul 28 12:53:09.216817 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:09.216789    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ps58v" podUID="a418b22c-fa2d-4d69-8a21-7e2209a892de"
Jul 28 12:53:09.219195 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.219178    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bc4gl"
Jul 28 12:53:09.221542 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.221524    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sqwtd\""
Jul 28 12:53:09.221792 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.221775    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 12:53:09.221883 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.221801    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 12:53:09.221883 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.221834    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 12:53:09.224548 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.224532    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cxz2b"
Jul 28 12:53:09.233046 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.232844    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 12:48:08 +0000 UTC" deadline="2028-02-04 01:22:49.471327226 +0000 UTC"
Jul 28 12:53:09.233046 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.232874    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13332h29m40.23845707s"
Jul 28 12:53:09.233046 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.232880    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdgzk"
Jul 28 12:53:09.233046 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.232887    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7n245"
Jul 28 12:53:09.233046 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.232948    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wk6vn"
Jul 28 12:53:09.233046 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:09.232944    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdgzk" podUID="da8b58f9-2528-4906-b7d2-aadea159b4f6"
Jul 28 12:53:09.233046 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.233023    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 12:53:09.233539 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.233059    2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 12:53:09.233539 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.233123    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 12:53:09.233539 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:09.233127    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wk6vn" podUID="4807337c-67b7-4b80-bf38-58ecdd8d972c"
Jul 28 12:53:09.233539 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.233255    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 12:53:09.233539 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.233394    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-dgmgt\""
Jul 28 12:53:09.233539 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.233481    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 12:53:09.235294 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.235280    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zf5gt"
Jul 28 12:53:09.235696 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.235663    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 12:53:09.235856 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.235841    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 12:53:09.235856 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.235854    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 12:53:09.235989 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.235901    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-j2lz2\""
Jul 28 12:53:09.235989 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.235961    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4tvpp\""
Jul 28 12:53:09.237586 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.237570    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-s7sd4\""
Jul 28 12:53:09.237778 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.237760    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 12:53:09.237778 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.237770    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 12:53:09.237916 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.237810    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 12:53:09.237916 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.237759    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 12:53:09.238205 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.238183    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 12:53:09.238364 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.238351    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:09.240468 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.240448    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xtlx4"
Jul 28 12:53:09.240797 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.240775    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 12:53:09.240993 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.240979    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cgp79\""
Jul 28 12:53:09.241058 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.241011    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 12:53:09.242026 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.242013    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 12:53:09.242026 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.242022    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 12:53:09.242138 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.242042    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 12:53:09.242138 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.242058    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 12:53:09.242651 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.242636    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-7nxlq\""
Jul 28 12:53:09.242895 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.242883    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 12:53:09.242947 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.242908    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csb84"
Jul 28 12:53:09.245341 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.245324    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 12:53:09.245341 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.245334    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vwcrj\""
Jul 28 12:53:09.245451 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.245325    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 12:53:09.245451 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.245358    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 12:53:09.246378 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.246360    2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-l7qpz"
Jul 28 12:53:09.256114 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.253784    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-l7qpz"
Jul 28 12:53:09.298383 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.298367    2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 12:53:09.301060 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.301039    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1-lib-modules\") pod \"tuned-xnfmf\" (UID: \"6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1\") " pod="openshift-cluster-node-tuning-operator/tuned-xnfmf"
Jul 28 12:53:09.301145 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.301070    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e03e76bd-a14d-4bab-8fea-35b382ecabd7-multus-socket-dir-parent\") pod \"multus-xtlx4\" (UID: \"e03e76bd-a14d-4bab-8fea-35b382ecabd7\") " pod="openshift-multus/multus-xtlx4"
Jul 28 12:53:09.301145 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.301091    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e03e76bd-a14d-4bab-8fea-35b382ecabd7-etc-kubernetes\") pod \"multus-xtlx4\" (UID: \"e03e76bd-a14d-4bab-8fea-35b382ecabd7\") " pod="openshift-multus/multus-xtlx4"
Jul 28 12:53:09.301145 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.301128    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a874f52e-baf3-4227-b2ef-f66115db2ffb-run-ovn\") pod \"ovnkube-node-nd4vq\" (UID: \"a874f52e-baf3-4227-b2ef-f66115db2ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:09.301273 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.301151    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a874f52e-baf3-4227-b2ef-f66115db2ffb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nd4vq\" (UID: \"a874f52e-baf3-4227-b2ef-f66115db2ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:09.301273 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.301170    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a874f52e-baf3-4227-b2ef-f66115db2ffb-ovn-node-metrics-cert\") pod \"ovnkube-node-nd4vq\" (UID: \"a874f52e-baf3-4227-b2ef-f66115db2ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:09.301273 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.301185    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a874f52e-baf3-4227-b2ef-f66115db2ffb-ovnkube-script-lib\") pod \"ovnkube-node-nd4vq\" (UID: \"a874f52e-baf3-4227-b2ef-f66115db2ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:09.301273 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.301203    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e03e76bd-a14d-4bab-8fea-35b382ecabd7-cnibin\") pod \"multus-xtlx4\" (UID: \"e03e76bd-a14d-4bab-8fea-35b382ecabd7\") " pod="openshift-multus/multus-xtlx4"
Jul 28 12:53:09.301273 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.301230    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e03e76bd-a14d-4bab-8fea-35b382ecabd7-multus-daemon-config\") pod \"multus-xtlx4\" (UID: \"e03e76bd-a14d-4bab-8fea-35b382ecabd7\") " pod="openshift-multus/multus-xtlx4"
Jul 28 12:53:09.301273 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.301244    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5842m\" (UniqueName: \"kubernetes.io/projected/e03e76bd-a14d-4bab-8fea-35b382ecabd7-kube-api-access-5842m\") pod \"multus-xtlx4\" (UID: \"e03e76bd-a14d-4bab-8fea-35b382ecabd7\") " pod="openshift-multus/multus-xtlx4"
Jul 28 12:53:09.301273 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.301258    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1-tmp\") pod \"tuned-xnfmf\" (UID: \"6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1\") " pod="openshift-cluster-node-tuning-operator/tuned-xnfmf"
Jul 28 12:53:09.301638 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.301277    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a874f52e-baf3-4227-b2ef-f66115db2ffb-etc-openvswitch\") pod \"ovnkube-node-nd4vq\" (UID: \"a874f52e-baf3-4227-b2ef-f66115db2ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:09.301638 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.301300    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a874f52e-baf3-4227-b2ef-f66115db2ffb-host-run-ovn-kubernetes\") pod \"ovnkube-node-nd4vq\" (UID: \"a874f52e-baf3-4227-b2ef-f66115db2ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:09.301638 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.301334    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e03e76bd-a14d-4bab-8fea-35b382ecabd7-system-cni-dir\") pod \"multus-xtlx4\" (UID: \"e03e76bd-a14d-4bab-8fea-35b382ecabd7\") " pod="openshift-multus/multus-xtlx4"
Jul 28 12:53:09.301638 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.301365    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e03e76bd-a14d-4bab-8fea-35b382ecabd7-multus-conf-dir\") pod \"multus-xtlx4\" (UID: \"e03e76bd-a14d-4bab-8fea-35b382ecabd7\") " pod="openshift-multus/multus-xtlx4"
Jul 28 12:53:09.301638 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.301388    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a418b22c-fa2d-4d69-8a21-7e2209a892de-metrics-certs\") pod \"network-metrics-daemon-ps58v\" (UID: \"a418b22c-fa2d-4d69-8a21-7e2209a892de\") " pod="openshift-multus/network-metrics-daemon-ps58v"
Jul 28 12:53:09.301638 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.301414    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f562a438-3615-401c-9c89-a8e553a96877-sys-fs\") pod \"aws-ebs-csi-driver-node-csb84\" (UID: \"f562a438-3615-401c-9c89-a8e553a96877\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csb84"
Jul 28 12:53:09.301638 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.301435    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1-etc-sysctl-conf\") pod \"tuned-xnfmf\" (UID: \"6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1\") " pod="openshift-cluster-node-tuning-operator/tuned-xnfmf"
Jul 28 12:53:09.301638 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.301457    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1-etc-systemd\") pod \"tuned-xnfmf\" (UID: \"6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1\") " pod="openshift-cluster-node-tuning-operator/tuned-xnfmf"
Jul 28 12:53:09.301638 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.301500    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1-etc-tuned\") pod \"tuned-xnfmf\" (UID: \"6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1\") " pod="openshift-cluster-node-tuning-operator/tuned-xnfmf"
Jul 28 12:53:09.301638 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.301536    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e03e76bd-a14d-4bab-8fea-35b382ecabd7-host-run-netns\") pod \"multus-xtlx4\" (UID: \"e03e76bd-a14d-4bab-8fea-35b382ecabd7\") " pod="openshift-multus/multus-xtlx4"
Jul 28 12:53:09.301638 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.301560    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a874f52e-baf3-4227-b2ef-f66115db2ffb-log-socket\") pod \"ovnkube-node-nd4vq\" (UID: \"a874f52e-baf3-4227-b2ef-f66115db2ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:09.301638 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.301583    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/03193c25-6f01-4389-b250-6eae144b3298-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zf5gt\" (UID: \"03193c25-6f01-4389-b250-6eae144b3298\") " pod="openshift-multus/multus-additional-cni-plugins-zf5gt"
Jul 28 12:53:09.301638 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.301610    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1-etc-sysctl-d\") pod \"tuned-xnfmf\" (UID: \"6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1\") " pod="openshift-cluster-node-tuning-operator/tuned-xnfmf"
Jul 28 12:53:09.301638 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.301642    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1-run\") pod \"tuned-xnfmf\" (UID: \"6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1\") " pod="openshift-cluster-node-tuning-operator/tuned-xnfmf"
Jul 28 12:53:09.302318 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.301664    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3943de15-60a3-4e53-af9f-125cb8bae2ab-konnectivity-ca\") pod \"konnectivity-agent-cxz2b\" (UID: \"3943de15-60a3-4e53-af9f-125cb8bae2ab\") " pod="kube-system/konnectivity-agent-cxz2b"
Jul 28 12:53:09.302318 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.301705    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a874f52e-baf3-4227-b2ef-f66115db2ffb-host-cni-bin\") pod \"ovnkube-node-nd4vq\" (UID: \"a874f52e-baf3-4227-b2ef-f66115db2ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:09.302318 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.301739    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a874f52e-baf3-4227-b2ef-f66115db2ffb-env-overrides\") pod \"ovnkube-node-nd4vq\" (UID: \"a874f52e-baf3-4227-b2ef-f66115db2ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:09.302318 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.301777    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e03e76bd-a14d-4bab-8fea-35b382ecabd7-os-release\") pod \"multus-xtlx4\" (UID: \"e03e76bd-a14d-4bab-8fea-35b382ecabd7\") " pod="openshift-multus/multus-xtlx4"
Jul 28 12:53:09.302318 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.301808    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f562a438-3615-401c-9c89-a8e553a96877-device-dir\") pod \"aws-ebs-csi-driver-node-csb84\" (UID: \"f562a438-3615-401c-9c89-a8e553a96877\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csb84"
Jul 28 12:53:09.302318 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.301825    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1-etc-sysconfig\") pod \"tuned-xnfmf\" (UID: \"6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1\") " pod="openshift-cluster-node-tuning-operator/tuned-xnfmf"
Jul 28 12:53:09.302318 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.301839    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a89e422d-76c2-4e62-b9fb-4283d5fa3de9-hosts-file\") pod \"node-resolver-66b94\" (UID: \"a89e422d-76c2-4e62-b9fb-4283d5fa3de9\") " pod="openshift-dns/node-resolver-66b94"
Jul 28 12:53:09.302318 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.301853    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtqfq\" (UniqueName: \"kubernetes.io/projected/a89e422d-76c2-4e62-b9fb-4283d5fa3de9-kube-api-access-jtqfq\") pod \"node-resolver-66b94\" (UID: \"a89e422d-76c2-4e62-b9fb-4283d5fa3de9\") " pod="openshift-dns/node-resolver-66b94"
Jul 28 12:53:09.302318 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.301909    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/03193c25-6f01-4389-b250-6eae144b3298-system-cni-dir\") pod \"multus-additional-cni-plugins-zf5gt\" (UID: \"03193c25-6f01-4389-b250-6eae144b3298\") " pod="openshift-multus/multus-additional-cni-plugins-zf5gt"
Jul 28 12:53:09.302318 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.301931    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9jb5\" (UniqueName: \"kubernetes.io/projected/03193c25-6f01-4389-b250-6eae144b3298-kube-api-access-c9jb5\") pod \"multus-additional-cni-plugins-zf5gt\" (UID: \"03193c25-6f01-4389-b250-6eae144b3298\") " pod="openshift-multus/multus-additional-cni-plugins-zf5gt"
Jul 28 12:53:09.302318 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.301949    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f562a438-3615-401c-9c89-a8e553a96877-kubelet-dir\") pod \"aws-ebs-csi-driver-node-csb84\" (UID: \"f562a438-3615-401c-9c89-a8e553a96877\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csb84"
Jul 28 12:53:09.302318 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.301973    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/46ca2ea9-2097-4729-8b59-52882a7ea6bf-serviceca\") pod \"node-ca-7n245\" (UID: \"46ca2ea9-2097-4729-8b59-52882a7ea6bf\") " pod="openshift-image-registry/node-ca-7n245"
Jul 28 12:53:09.302318 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.301996    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a874f52e-baf3-4227-b2ef-f66115db2ffb-host-cni-netd\") pod \"ovnkube-node-nd4vq\" (UID: \"a874f52e-baf3-4227-b2ef-f66115db2ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:09.302318 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.302013    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/03193c25-6f01-4389-b250-6eae144b3298-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zf5gt\" (UID: \"03193c25-6f01-4389-b250-6eae144b3298\") " pod="openshift-multus/multus-additional-cni-plugins-zf5gt"
Jul 28 12:53:09.302318 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.302033    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/03193c25-6f01-4389-b250-6eae144b3298-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zf5gt\" (UID: \"03193c25-6f01-4389-b250-6eae144b3298\") " pod="openshift-multus/multus-additional-cni-plugins-zf5gt"
Jul 28 12:53:09.302318 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.302059    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f562a438-3615-401c-9c89-a8e553a96877-etc-selinux\") pod \"aws-ebs-csi-driver-node-csb84\" (UID: \"f562a438-3615-401c-9c89-a8e553a96877\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csb84"
Jul 28 12:53:09.302917 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.302098    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e03e76bd-a14d-4bab-8fea-35b382ecabd7-host-run-k8s-cni-cncf-io\") pod \"multus-xtlx4\" (UID: \"e03e76bd-a14d-4bab-8fea-35b382ecabd7\") " pod="openshift-multus/multus-xtlx4"
Jul 28 12:53:09.302917 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.302127    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e03e76bd-a14d-4bab-8fea-35b382ecabd7-host-run-multus-certs\") pod \"multus-xtlx4\" (UID: \"e03e76bd-a14d-4bab-8fea-35b382ecabd7\") " pod="openshift-multus/multus-xtlx4"
Jul 28 12:53:09.302917 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.302142    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2e5d8f8f-d54f-4126-97a1-4b91dc7d40f8-iptables-alerter-script\") pod \"iptables-alerter-bc4gl\" (UID: \"2e5d8f8f-d54f-4126-97a1-4b91dc7d40f8\") " pod="openshift-network-operator/iptables-alerter-bc4gl"
Jul 28 12:53:09.302917 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.302173    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1-var-lib-kubelet\") pod \"tuned-xnfmf\" (UID: \"6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1\") " pod="openshift-cluster-node-tuning-operator/tuned-xnfmf"
Jul 28 12:53:09.302917 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.302197    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/46ca2ea9-2097-4729-8b59-52882a7ea6bf-host\") pod \"node-ca-7n245\" (UID: \"46ca2ea9-2097-4729-8b59-52882a7ea6bf\") " pod="openshift-image-registry/node-ca-7n245"
Jul 28 12:53:09.302917 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.302219    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a874f52e-baf3-4227-b2ef-f66115db2ffb-host-kubelet\") pod \"ovnkube-node-nd4vq\" (UID: \"a874f52e-baf3-4227-b2ef-f66115db2ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:09.302917 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.302253    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e03e76bd-a14d-4bab-8fea-35b382ecabd7-host-var-lib-cni-bin\") pod \"multus-xtlx4\" (UID: \"e03e76bd-a14d-4bab-8fea-35b382ecabd7\") " pod="openshift-multus/multus-xtlx4"
Jul 28 12:53:09.302917 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.302327    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e03e76bd-a14d-4bab-8fea-35b382ecabd7-host-var-lib-kubelet\") pod \"multus-xtlx4\" (UID: \"e03e76bd-a14d-4bab-8fea-35b382ecabd7\") " pod="openshift-multus/multus-xtlx4"
Jul 28 12:53:09.302917 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.302359    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da8b58f9-2528-4906-b7d2-aadea159b4f6-original-pull-secret\") pod \"global-pull-secret-syncer-bdgzk\" (UID: \"da8b58f9-2528-4906-b7d2-aadea159b4f6\") " pod="kube-system/global-pull-secret-syncer-bdgzk"
Jul 28 12:53:09.302917 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.302386    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f562a438-3615-401c-9c89-a8e553a96877-registration-dir\") pod \"aws-ebs-csi-driver-node-csb84\" (UID: \"f562a438-3615-401c-9c89-a8e553a96877\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csb84"
Jul 28 12:53:09.302917 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.302419    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3943de15-60a3-4e53-af9f-125cb8bae2ab-agent-certs\") pod \"konnectivity-agent-cxz2b\" (UID: \"3943de15-60a3-4e53-af9f-125cb8bae2ab\") " pod="kube-system/konnectivity-agent-cxz2b"
Jul 28 12:53:09.302917 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.302438    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwtpx\" (UniqueName: \"kubernetes.io/projected/46ca2ea9-2097-4729-8b59-52882a7ea6bf-kube-api-access-jwtpx\") pod \"node-ca-7n245\" (UID: \"46ca2ea9-2097-4729-8b59-52882a7ea6bf\") " pod="openshift-image-registry/node-ca-7n245"
Jul 28 12:53:09.302917 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.302454    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/03193c25-6f01-4389-b250-6eae144b3298-cni-binary-copy\") pod \"multus-additional-cni-plugins-zf5gt\" (UID: \"03193c25-6f01-4389-b250-6eae144b3298\") " pod="openshift-multus/multus-additional-cni-plugins-zf5gt"
Jul 28 12:53:09.302917 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.302480    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1-etc-modprobe-d\") pod \"tuned-xnfmf\" (UID: \"6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1\") " pod="openshift-cluster-node-tuning-operator/tuned-xnfmf"
Jul 28 12:53:09.302917 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.302496    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9krd\" (UniqueName: \"kubernetes.io/projected/4807337c-67b7-4b80-bf38-58ecdd8d972c-kube-api-access-n9krd\") pod \"network-check-target-wk6vn\" (UID: \"4807337c-67b7-4b80-bf38-58ecdd8d972c\") " pod="openshift-network-diagnostics/network-check-target-wk6vn"
Jul 28 12:53:09.302917 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.302509    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a874f52e-baf3-4227-b2ef-f66115db2ffb-ovnkube-config\") pod \"ovnkube-node-nd4vq\" (UID: \"a874f52e-baf3-4227-b2ef-f66115db2ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:09.303452 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.302525    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e03e76bd-a14d-4bab-8fea-35b382ecabd7-multus-cni-dir\") pod \"multus-xtlx4\" (UID: \"e03e76bd-a14d-4bab-8fea-35b382ecabd7\") " pod="openshift-multus/multus-xtlx4"
Jul 28 12:53:09.303452 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.302549    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/da8b58f9-2528-4906-b7d2-aadea159b4f6-kubelet-config\") pod \"global-pull-secret-syncer-bdgzk\" (UID: \"da8b58f9-2528-4906-b7d2-aadea159b4f6\") " pod="kube-system/global-pull-secret-syncer-bdgzk"
Jul 28 12:53:09.303452 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.302612    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/03193c25-6f01-4389-b250-6eae144b3298-cnibin\") pod \"multus-additional-cni-plugins-zf5gt\" (UID: \"03193c25-6f01-4389-b250-6eae144b3298\") " pod="openshift-multus/multus-additional-cni-plugins-zf5gt"
Jul 28 12:53:09.303452 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.302630    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x288h\" (UniqueName: \"kubernetes.io/projected/f562a438-3615-401c-9c89-a8e553a96877-kube-api-access-x288h\") pod \"aws-ebs-csi-driver-node-csb84\" (UID: \"f562a438-3615-401c-9c89-a8e553a96877\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csb84"
Jul 28 12:53:09.303452 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.302644    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a874f52e-baf3-4227-b2ef-f66115db2ffb-host-slash\") pod \"ovnkube-node-nd4vq\" (UID: \"a874f52e-baf3-4227-b2ef-f66115db2ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:09.303452 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.302661    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/da8b58f9-2528-4906-b7d2-aadea159b4f6-dbus\") pod \"global-pull-secret-syncer-bdgzk\" (UID: \"da8b58f9-2528-4906-b7d2-aadea159b4f6\") " pod="kube-system/global-pull-secret-syncer-bdgzk"
Jul 28 12:53:09.303452 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.302697    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1-etc-kubernetes\") pod \"tuned-xnfmf\" (UID: \"6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1\") " pod="openshift-cluster-node-tuning-operator/tuned-xnfmf"
Jul 28 12:53:09.303452 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.302717    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1-host\") pod \"tuned-xnfmf\" (UID: \"6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1\") " pod="openshift-cluster-node-tuning-operator/tuned-xnfmf"
Jul 28 12:53:09.303452 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.302730    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a874f52e-baf3-4227-b2ef-f66115db2ffb-systemd-units\") pod \"ovnkube-node-nd4vq\" (UID: \"a874f52e-baf3-4227-b2ef-f66115db2ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:09.303452 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.302744    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e03e76bd-a14d-4bab-8fea-35b382ecabd7-host-var-lib-cni-multus\") pod \"multus-xtlx4\" (UID: \"e03e76bd-a14d-4bab-8fea-35b382ecabd7\") " pod="openshift-multus/multus-xtlx4"
Jul 28 12:53:09.303452 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.302761    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a89e422d-76c2-4e62-b9fb-4283d5fa3de9-tmp-dir\") pod \"node-resolver-66b94\" (UID: \"a89e422d-76c2-4e62-b9fb-4283d5fa3de9\") " pod="openshift-dns/node-resolver-66b94"
Jul 28 12:53:09.303452 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.302796    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmhqb\" (UniqueName: \"kubernetes.io/projected/a418b22c-fa2d-4d69-8a21-7e2209a892de-kube-api-access-jmhqb\") pod \"network-metrics-daemon-ps58v\" (UID: \"a418b22c-fa2d-4d69-8a21-7e2209a892de\") " pod="openshift-multus/network-metrics-daemon-ps58v"
Jul 28 12:53:09.303452 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.302815    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2e5d8f8f-d54f-4126-97a1-4b91dc7d40f8-host-slash\") pod \"iptables-alerter-bc4gl\" (UID: \"2e5d8f8f-d54f-4126-97a1-4b91dc7d40f8\") " pod="openshift-network-operator/iptables-alerter-bc4gl"
Jul 28 12:53:09.303452 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.302833    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-666cc\" (UniqueName: \"kubernetes.io/projected/2e5d8f8f-d54f-4126-97a1-4b91dc7d40f8-kube-api-access-666cc\") pod \"iptables-alerter-bc4gl\" (UID: \"2e5d8f8f-d54f-4126-97a1-4b91dc7d40f8\") " pod="openshift-network-operator/iptables-alerter-bc4gl"
Jul 28 12:53:09.303452 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.302866    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f562a438-3615-401c-9c89-a8e553a96877-socket-dir\") pod \"aws-ebs-csi-driver-node-csb84\" (UID: \"f562a438-3615-401c-9c89-a8e553a96877\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csb84"
Jul 28 12:53:09.303452 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.302887    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxmqr\" (UniqueName: \"kubernetes.io/projected/6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1-kube-api-access-lxmqr\") pod \"tuned-xnfmf\" (UID: \"6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1\") " pod="openshift-cluster-node-tuning-operator/tuned-xnfmf"
Jul 28 12:53:09.303974 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.302902    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a874f52e-baf3-4227-b2ef-f66115db2ffb-run-systemd\") pod \"ovnkube-node-nd4vq\" (UID: \"a874f52e-baf3-4227-b2ef-f66115db2ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:09.303974 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.302918    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a874f52e-baf3-4227-b2ef-f66115db2ffb-var-lib-openvswitch\") pod \"ovnkube-node-nd4vq\" (UID: \"a874f52e-baf3-4227-b2ef-f66115db2ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:09.303974 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.302942    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a874f52e-baf3-4227-b2ef-f66115db2ffb-node-log\") pod \"ovnkube-node-nd4vq\" (UID: \"a874f52e-baf3-4227-b2ef-f66115db2ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:09.303974 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.302964    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlh6d\" (UniqueName: \"kubernetes.io/projected/a874f52e-baf3-4227-b2ef-f66115db2ffb-kube-api-access-rlh6d\") pod \"ovnkube-node-nd4vq\" (UID: \"a874f52e-baf3-4227-b2ef-f66115db2ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:09.303974 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.302982    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1-sys\") pod \"tuned-xnfmf\" (UID: \"6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1\") " pod="openshift-cluster-node-tuning-operator/tuned-xnfmf"
Jul 28 12:53:09.303974 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.303003    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a874f52e-baf3-4227-b2ef-f66115db2ffb-host-run-netns\") pod \"ovnkube-node-nd4vq\" (UID: \"a874f52e-baf3-4227-b2ef-f66115db2ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:09.303974 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.303016    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a874f52e-baf3-4227-b2ef-f66115db2ffb-run-openvswitch\") pod \"ovnkube-node-nd4vq\" (UID: \"a874f52e-baf3-4227-b2ef-f66115db2ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:09.303974 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.303030    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e03e76bd-a14d-4bab-8fea-35b382ecabd7-cni-binary-copy\") pod \"multus-xtlx4\" (UID: \"e03e76bd-a14d-4bab-8fea-35b382ecabd7\") " pod="openshift-multus/multus-xtlx4"
Jul 28 12:53:09.303974 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.303049    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e03e76bd-a14d-4bab-8fea-35b382ecabd7-hostroot\") pod \"multus-xtlx4\" (UID: \"e03e76bd-a14d-4bab-8fea-35b382ecabd7\") " pod="openshift-multus/multus-xtlx4"
Jul 28 12:53:09.303974 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.303109    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/03193c25-6f01-4389-b250-6eae144b3298-os-release\") pod \"multus-additional-cni-plugins-zf5gt\" (UID: \"03193c25-6f01-4389-b250-6eae144b3298\") " pod="openshift-multus/multus-additional-cni-plugins-zf5gt"
Jul 28 12:53:09.373665 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:09.373638    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f72deeae9606046a5729c2df486eb15.slice/crio-80ed1b564500c13b6cdf4020b29c2bef46903a992ae59903a097a8a71214c2e0 WatchSource:0}: Error finding container 80ed1b564500c13b6cdf4020b29c2bef46903a992ae59903a097a8a71214c2e0: Status 404 returned error can't find the container with id 80ed1b564500c13b6cdf4020b29c2bef46903a992ae59903a097a8a71214c2e0
Jul 28 12:53:09.379181 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.379168    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 12:53:09.403602 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.403272    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a89e422d-76c2-4e62-b9fb-4283d5fa3de9-tmp-dir\") pod \"node-resolver-66b94\" (UID: \"a89e422d-76c2-4e62-b9fb-4283d5fa3de9\") " pod="openshift-dns/node-resolver-66b94"
Jul 28 12:53:09.403701 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.403636    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmhqb\" (UniqueName: \"kubernetes.io/projected/a418b22c-fa2d-4d69-8a21-7e2209a892de-kube-api-access-jmhqb\") pod \"network-metrics-daemon-ps58v\" (UID: \"a418b22c-fa2d-4d69-8a21-7e2209a892de\") " pod="openshift-multus/network-metrics-daemon-ps58v"
Jul 28 12:53:09.403701 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.403675    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2e5d8f8f-d54f-4126-97a1-4b91dc7d40f8-host-slash\") pod \"iptables-alerter-bc4gl\" (UID: \"2e5d8f8f-d54f-4126-97a1-4b91dc7d40f8\") " pod="openshift-network-operator/iptables-alerter-bc4gl"
Jul 28 12:53:09.403816 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.403722    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-666cc\" (UniqueName: \"kubernetes.io/projected/2e5d8f8f-d54f-4126-97a1-4b91dc7d40f8-kube-api-access-666cc\") pod \"iptables-alerter-bc4gl\" (UID: \"2e5d8f8f-d54f-4126-97a1-4b91dc7d40f8\") " pod="openshift-network-operator/iptables-alerter-bc4gl"
Jul 28 12:53:09.403816 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.403758    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a89e422d-76c2-4e62-b9fb-4283d5fa3de9-tmp-dir\") pod \"node-resolver-66b94\" (UID: \"a89e422d-76c2-4e62-b9fb-4283d5fa3de9\") " pod="openshift-dns/node-resolver-66b94"
Jul 28 12:53:09.403919 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.403875    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2e5d8f8f-d54f-4126-97a1-4b91dc7d40f8-host-slash\") pod \"iptables-alerter-bc4gl\" (UID: \"2e5d8f8f-d54f-4126-97a1-4b91dc7d40f8\") " pod="openshift-network-operator/iptables-alerter-bc4gl"
Jul 28 12:53:09.403919 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.403862    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f562a438-3615-401c-9c89-a8e553a96877-socket-dir\") pod \"aws-ebs-csi-driver-node-csb84\" (UID: \"f562a438-3615-401c-9c89-a8e553a96877\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csb84"
Jul 28 12:53:09.404030 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.404007    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f562a438-3615-401c-9c89-a8e553a96877-socket-dir\") pod \"aws-ebs-csi-driver-node-csb84\" (UID: \"f562a438-3615-401c-9c89-a8e553a96877\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csb84"
Jul 28 12:53:09.404136 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.404071    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxmqr\" (UniqueName: \"kubernetes.io/projected/6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1-kube-api-access-lxmqr\") pod \"tuned-xnfmf\" (UID: \"6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1\") " pod="openshift-cluster-node-tuning-operator/tuned-xnfmf"
Jul 28 12:53:09.404136 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.404099    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a874f52e-baf3-4227-b2ef-f66115db2ffb-run-systemd\") pod \"ovnkube-node-nd4vq\" (UID: \"a874f52e-baf3-4227-b2ef-f66115db2ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:09.404136 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.404127    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a874f52e-baf3-4227-b2ef-f66115db2ffb-var-lib-openvswitch\") pod \"ovnkube-node-nd4vq\" (UID: \"a874f52e-baf3-4227-b2ef-f66115db2ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:09.404290 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.404155    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a874f52e-baf3-4227-b2ef-f66115db2ffb-node-log\") pod \"ovnkube-node-nd4vq\" (UID: \"a874f52e-baf3-4227-b2ef-f66115db2ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:09.404290 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.404184    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlh6d\" (UniqueName: \"kubernetes.io/projected/a874f52e-baf3-4227-b2ef-f66115db2ffb-kube-api-access-rlh6d\") pod \"ovnkube-node-nd4vq\" (UID: \"a874f52e-baf3-4227-b2ef-f66115db2ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:09.404290 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.404210    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1-sys\") pod \"tuned-xnfmf\" (UID: \"6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1\") " pod="openshift-cluster-node-tuning-operator/tuned-xnfmf"
Jul 28 12:53:09.404290 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.404229    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a874f52e-baf3-4227-b2ef-f66115db2ffb-run-systemd\") pod \"ovnkube-node-nd4vq\" (UID: \"a874f52e-baf3-4227-b2ef-f66115db2ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:09.404290 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.404234    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a874f52e-baf3-4227-b2ef-f66115db2ffb-host-run-netns\") pod \"ovnkube-node-nd4vq\" (UID: \"a874f52e-baf3-4227-b2ef-f66115db2ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:09.404290 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.404274    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a874f52e-baf3-4227-b2ef-f66115db2ffb-host-run-netns\") pod \"ovnkube-node-nd4vq\" (UID: \"a874f52e-baf3-4227-b2ef-f66115db2ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:09.404290 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.404277    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a874f52e-baf3-4227-b2ef-f66115db2ffb-run-openvswitch\") pod \"ovnkube-node-nd4vq\" (UID: \"a874f52e-baf3-4227-b2ef-f66115db2ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:09.404580 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.404313    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e03e76bd-a14d-4bab-8fea-35b382ecabd7-cni-binary-copy\") pod \"multus-xtlx4\" (UID: \"e03e76bd-a14d-4bab-8fea-35b382ecabd7\") " pod="openshift-multus/multus-xtlx4"
Jul 28 12:53:09.404580 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.404319    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a874f52e-baf3-4227-b2ef-f66115db2ffb-run-openvswitch\") pod \"ovnkube-node-nd4vq\" (UID: \"a874f52e-baf3-4227-b2ef-f66115db2ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:09.404580 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.404356    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e03e76bd-a14d-4bab-8fea-35b382ecabd7-hostroot\") pod \"multus-xtlx4\" (UID: \"e03e76bd-a14d-4bab-8fea-35b382ecabd7\") " pod="openshift-multus/multus-xtlx4"
Jul 28 12:53:09.404580 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.404367    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a874f52e-baf3-4227-b2ef-f66115db2ffb-var-lib-openvswitch\") pod \"ovnkube-node-nd4vq\" (UID: \"a874f52e-baf3-4227-b2ef-f66115db2ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:09.404580 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.404388    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/03193c25-6f01-4389-b250-6eae144b3298-os-release\") pod \"multus-additional-cni-plugins-zf5gt\" (UID: \"03193c25-6f01-4389-b250-6eae144b3298\") " pod="openshift-multus/multus-additional-cni-plugins-zf5gt"
Jul 28 12:53:09.404580 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.404401    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a874f52e-baf3-4227-b2ef-f66115db2ffb-node-log\") pod \"ovnkube-node-nd4vq\" (UID: \"a874f52e-baf3-4227-b2ef-f66115db2ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:09.404580 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.404420    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1-lib-modules\") pod \"tuned-xnfmf\" (UID: \"6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1\") " pod="openshift-cluster-node-tuning-operator/tuned-xnfmf"
Jul 28 12:53:09.404580 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.404450    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e03e76bd-a14d-4bab-8fea-35b382ecabd7-multus-socket-dir-parent\") pod \"multus-xtlx4\" (UID: \"e03e76bd-a14d-4bab-8fea-35b382ecabd7\") " pod="openshift-multus/multus-xtlx4"
Jul 28 12:53:09.404580 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.404479    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e03e76bd-a14d-4bab-8fea-35b382ecabd7-etc-kubernetes\") pod \"multus-xtlx4\" (UID: \"e03e76bd-a14d-4bab-8fea-35b382ecabd7\") " pod="openshift-multus/multus-xtlx4"
Jul 28 12:53:09.404580 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.404503    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a874f52e-baf3-4227-b2ef-f66115db2ffb-run-ovn\") pod \"ovnkube-node-nd4vq\" (UID: \"a874f52e-baf3-4227-b2ef-f66115db2ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:09.404580 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.404533    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a874f52e-baf3-4227-b2ef-f66115db2ffb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nd4vq\" (UID: \"a874f52e-baf3-4227-b2ef-f66115db2ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:09.404580 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.404564    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a874f52e-baf3-4227-b2ef-f66115db2ffb-ovn-node-metrics-cert\") pod \"ovnkube-node-nd4vq\" (UID: \"a874f52e-baf3-4227-b2ef-f66115db2ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:09.405107 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.404593    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a874f52e-baf3-4227-b2ef-f66115db2ffb-ovnkube-script-lib\") pod \"ovnkube-node-nd4vq\" (UID: \"a874f52e-baf3-4227-b2ef-f66115db2ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:09.405107 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.404625    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e03e76bd-a14d-4bab-8fea-35b382ecabd7-cnibin\") pod \"multus-xtlx4\" (UID: \"e03e76bd-a14d-4bab-8fea-35b382ecabd7\") " pod="openshift-multus/multus-xtlx4"
Jul 28 12:53:09.405107 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.404650    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e03e76bd-a14d-4bab-8fea-35b382ecabd7-multus-daemon-config\") pod \"multus-xtlx4\" (UID: \"e03e76bd-a14d-4bab-8fea-35b382ecabd7\") " pod="openshift-multus/multus-xtlx4"
Jul 28 12:53:09.405107 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.404668    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1-sys\") pod \"tuned-xnfmf\" (UID: \"6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1\") " pod="openshift-cluster-node-tuning-operator/tuned-xnfmf"
Jul 28 12:53:09.405107 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.404697    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5842m\" (UniqueName: \"kubernetes.io/projected/e03e76bd-a14d-4bab-8fea-35b382ecabd7-kube-api-access-5842m\") pod \"multus-xtlx4\" (UID: \"e03e76bd-a14d-4bab-8fea-35b382ecabd7\") " pod="openshift-multus/multus-xtlx4"
Jul 28 12:53:09.405107 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.404727    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1-tmp\") pod \"tuned-xnfmf\" (UID: \"6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1\") " pod="openshift-cluster-node-tuning-operator/tuned-xnfmf"
Jul 28 12:53:09.405107 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.404755    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a874f52e-baf3-4227-b2ef-f66115db2ffb-etc-openvswitch\") pod \"ovnkube-node-nd4vq\" (UID: \"a874f52e-baf3-4227-b2ef-f66115db2ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:09.405107 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.404760    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e03e76bd-a14d-4bab-8fea-35b382ecabd7-etc-kubernetes\") pod \"multus-xtlx4\" (UID: \"e03e76bd-a14d-4bab-8fea-35b382ecabd7\") " pod="openshift-multus/multus-xtlx4"
Jul 28 12:53:09.405107 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.404784    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a874f52e-baf3-4227-b2ef-f66115db2ffb-host-run-ovn-kubernetes\") pod \"ovnkube-node-nd4vq\" (UID: \"a874f52e-baf3-4227-b2ef-f66115db2ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:09.405107 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.404808    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e03e76bd-a14d-4bab-8fea-35b382ecabd7-system-cni-dir\") pod \"multus-xtlx4\" (UID: \"e03e76bd-a14d-4bab-8fea-35b382ecabd7\") " pod="openshift-multus/multus-xtlx4"
Jul 28 12:53:09.405107 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.404838    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e03e76bd-a14d-4bab-8fea-35b382ecabd7-multus-conf-dir\") pod \"multus-xtlx4\" (UID: \"e03e76bd-a14d-4bab-8fea-35b382ecabd7\") " pod="openshift-multus/multus-xtlx4"
Jul 28 12:53:09.405107 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.404865    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a418b22c-fa2d-4d69-8a21-7e2209a892de-metrics-certs\") pod \"network-metrics-daemon-ps58v\" (UID: \"a418b22c-fa2d-4d69-8a21-7e2209a892de\") " pod="openshift-multus/network-metrics-daemon-ps58v"
Jul 28 12:53:09.405107 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.404895    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f562a438-3615-401c-9c89-a8e553a96877-sys-fs\") pod \"aws-ebs-csi-driver-node-csb84\" (UID: \"f562a438-3615-401c-9c89-a8e553a96877\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csb84"
Jul 28 12:53:09.405107 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.404923    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1-etc-sysctl-conf\") pod \"tuned-xnfmf\" (UID: \"6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1\") " pod="openshift-cluster-node-tuning-operator/tuned-xnfmf"
Jul 28 12:53:09.405107 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.404946    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1-etc-systemd\") pod \"tuned-xnfmf\" (UID: \"6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1\") " pod="openshift-cluster-node-tuning-operator/tuned-xnfmf"
Jul 28 12:53:09.405107 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.404974    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1-etc-tuned\") pod \"tuned-xnfmf\" (UID: \"6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1\") " pod="openshift-cluster-node-tuning-operator/tuned-xnfmf"
Jul 28 12:53:09.405107 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.405003    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e03e76bd-a14d-4bab-8fea-35b382ecabd7-host-run-netns\") pod \"multus-xtlx4\" (UID: \"e03e76bd-a14d-4bab-8fea-35b382ecabd7\") " pod="openshift-multus/multus-xtlx4"
Jul 28 12:53:09.405107 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.405031    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a874f52e-baf3-4227-b2ef-f66115db2ffb-log-socket\") pod \"ovnkube-node-nd4vq\" (UID: \"a874f52e-baf3-4227-b2ef-f66115db2ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:09.405881 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.405059    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/03193c25-6f01-4389-b250-6eae144b3298-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zf5gt\" (UID: \"03193c25-6f01-4389-b250-6eae144b3298\") " pod="openshift-multus/multus-additional-cni-plugins-zf5gt"
Jul 28 12:53:09.405881 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.405088    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1-etc-sysctl-d\") pod \"tuned-xnfmf\" (UID: \"6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1\") " pod="openshift-cluster-node-tuning-operator/tuned-xnfmf"
Jul 28 12:53:09.405881 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.405113    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1-run\") pod \"tuned-xnfmf\" (UID: \"6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1\") " pod="openshift-cluster-node-tuning-operator/tuned-xnfmf"
Jul 28 12:53:09.405881 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.405148    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3943de15-60a3-4e53-af9f-125cb8bae2ab-konnectivity-ca\") pod \"konnectivity-agent-cxz2b\" (UID: \"3943de15-60a3-4e53-af9f-125cb8bae2ab\") " pod="kube-system/konnectivity-agent-cxz2b"
Jul 28 12:53:09.405881 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.405183    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a874f52e-baf3-4227-b2ef-f66115db2ffb-host-cni-bin\") pod \"ovnkube-node-nd4vq\" (UID: \"a874f52e-baf3-4227-b2ef-f66115db2ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:09.405881 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.405213    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a874f52e-baf3-4227-b2ef-f66115db2ffb-env-overrides\") pod \"ovnkube-node-nd4vq\" (UID: \"a874f52e-baf3-4227-b2ef-f66115db2ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:09.405881 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.405240    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e03e76bd-a14d-4bab-8fea-35b382ecabd7-os-release\") pod \"multus-xtlx4\" (UID: \"e03e76bd-a14d-4bab-8fea-35b382ecabd7\") " pod="openshift-multus/multus-xtlx4"
Jul 28 12:53:09.405881 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.405263    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f562a438-3615-401c-9c89-a8e553a96877-device-dir\") pod \"aws-ebs-csi-driver-node-csb84\" (UID: \"f562a438-3615-401c-9c89-a8e553a96877\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csb84"
Jul 28 12:53:09.405881 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.405264    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e03e76bd-a14d-4bab-8fea-35b382ecabd7-cni-binary-copy\") pod \"multus-xtlx4\" (UID: \"e03e76bd-a14d-4bab-8fea-35b382ecabd7\") " pod="openshift-multus/multus-xtlx4"
Jul 28 12:53:09.405881 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.405292    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1-etc-sysconfig\") pod \"tuned-xnfmf\" (UID: \"6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1\") " pod="openshift-cluster-node-tuning-operator/tuned-xnfmf"
Jul 28 12:53:09.405881 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.405352    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1-etc-sysconfig\") pod \"tuned-xnfmf\" (UID: \"6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1\") " pod="openshift-cluster-node-tuning-operator/tuned-xnfmf"
Jul 28 12:53:09.405881 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.405363    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a89e422d-76c2-4e62-b9fb-4283d5fa3de9-hosts-file\") pod \"node-resolver-66b94\" (UID: \"a89e422d-76c2-4e62-b9fb-4283d5fa3de9\") " pod="openshift-dns/node-resolver-66b94"
Jul 28 12:53:09.405881 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.405397    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtqfq\" (UniqueName: \"kubernetes.io/projected/a89e422d-76c2-4e62-b9fb-4283d5fa3de9-kube-api-access-jtqfq\") pod \"node-resolver-66b94\" (UID: \"a89e422d-76c2-4e62-b9fb-4283d5fa3de9\") " pod="openshift-dns/node-resolver-66b94"
Jul 28 12:53:09.405881 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.405430    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a874f52e-baf3-4227-b2ef-f66115db2ffb-run-ovn\") pod \"ovnkube-node-nd4vq\" (UID: \"a874f52e-baf3-4227-b2ef-f66115db2ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:09.405881 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.405433    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/03193c25-6f01-4389-b250-6eae144b3298-system-cni-dir\") pod \"multus-additional-cni-plugins-zf5gt\" (UID: \"03193c25-6f01-4389-b250-6eae144b3298\") " pod="openshift-multus/multus-additional-cni-plugins-zf5gt"
Jul 28 12:53:09.405881 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.405563    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1-lib-modules\") pod \"tuned-xnfmf\" (UID: \"6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1\") " pod="openshift-cluster-node-tuning-operator/tuned-xnfmf"
Jul 28 12:53:09.405881 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.405660    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e03e76bd-a14d-4bab-8fea-35b382ecabd7-multus-socket-dir-parent\") pod \"multus-xtlx4\" (UID: \"e03e76bd-a14d-4bab-8fea-35b382ecabd7\") " pod="openshift-multus/multus-xtlx4"
Jul 28 12:53:09.405881 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.405753    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a89e422d-76c2-4e62-b9fb-4283d5fa3de9-hosts-file\") pod \"node-resolver-66b94\" (UID: \"a89e422d-76c2-4e62-b9fb-4283d5fa3de9\") " pod="openshift-dns/node-resolver-66b94"
Jul 28 12:53:09.406741 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.405994    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/03193c25-6f01-4389-b250-6eae144b3298-system-cni-dir\") pod \"multus-additional-cni-plugins-zf5gt\" (UID: \"03193c25-6f01-4389-b250-6eae144b3298\") " pod="openshift-multus/multus-additional-cni-plugins-zf5gt"
Jul 28 12:53:09.406741 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.406036    2574 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 12:53:09.406741 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.406101    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1-etc-sysctl-conf\") pod \"tuned-xnfmf\" (UID: \"6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1\") " pod="openshift-cluster-node-tuning-operator/tuned-xnfmf"
Jul 28 12:53:09.406741 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.406385    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1-run\") pod \"tuned-xnfmf\" (UID: \"6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1\") " pod="openshift-cluster-node-tuning-operator/tuned-xnfmf"
Jul 28 12:53:09.406741 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.406578    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a874f52e-baf3-4227-b2ef-f66115db2ffb-ovnkube-script-lib\") pod \"ovnkube-node-nd4vq\" (UID: \"a874f52e-baf3-4227-b2ef-f66115db2ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:09.406741 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.406582    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a874f52e-baf3-4227-b2ef-f66115db2ffb-log-socket\") pod \"ovnkube-node-nd4vq\" (UID: \"a874f52e-baf3-4227-b2ef-f66115db2ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:09.406741 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.406593    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e03e76bd-a14d-4bab-8fea-35b382ecabd7-hostroot\") pod \"multus-xtlx4\" (UID: \"e03e76bd-a14d-4bab-8fea-35b382ecabd7\") " pod="openshift-multus/multus-xtlx4"
Jul 28 12:53:09.406741 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.406607    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e03e76bd-a14d-4bab-8fea-35b382ecabd7-cnibin\") pod \"multus-xtlx4\" (UID: \"e03e76bd-a14d-4bab-8fea-35b382ecabd7\") " pod="openshift-multus/multus-xtlx4"
Jul 28 12:53:09.406741 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.406639    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a874f52e-baf3-4227-b2ef-f66115db2ffb-host-run-ovn-kubernetes\") pod \"ovnkube-node-nd4vq\" (UID: \"a874f52e-baf3-4227-b2ef-f66115db2ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:09.406741 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.406642    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e03e76bd-a14d-4bab-8fea-35b382ecabd7-os-release\") pod \"multus-xtlx4\" (UID: \"e03e76bd-a14d-4bab-8fea-35b382ecabd7\") " pod="openshift-multus/multus-xtlx4"
Jul 28 12:53:09.406741 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.406714    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a874f52e-baf3-4227-b2ef-f66115db2ffb-host-cni-bin\") pod \"ovnkube-node-nd4vq\" (UID: \"a874f52e-baf3-4227-b2ef-f66115db2ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:09.407277 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.406757    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9jb5\" (UniqueName: \"kubernetes.io/projected/03193c25-6f01-4389-b250-6eae144b3298-kube-api-access-c9jb5\") pod \"multus-additional-cni-plugins-zf5gt\" (UID: \"03193c25-6f01-4389-b250-6eae144b3298\") " pod="openshift-multus/multus-additional-cni-plugins-zf5gt"
Jul 28 12:53:09.407277 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.406799    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f562a438-3615-401c-9c89-a8e553a96877-kubelet-dir\") pod \"aws-ebs-csi-driver-node-csb84\" (UID: \"f562a438-3615-401c-9c89-a8e553a96877\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csb84"
Jul 28 12:53:09.407277 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.406827    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/46ca2ea9-2097-4729-8b59-52882a7ea6bf-serviceca\") pod \"node-ca-7n245\" (UID: \"46ca2ea9-2097-4729-8b59-52882a7ea6bf\") " pod="openshift-image-registry/node-ca-7n245"
Jul 28 12:53:09.407277 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.406853    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a874f52e-baf3-4227-b2ef-f66115db2ffb-host-cni-netd\") pod \"ovnkube-node-nd4vq\" (UID: \"a874f52e-baf3-4227-b2ef-f66115db2ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:09.407277 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.406875    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/03193c25-6f01-4389-b250-6eae144b3298-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zf5gt\" (UID: \"03193c25-6f01-4389-b250-6eae144b3298\") " pod="openshift-multus/multus-additional-cni-plugins-zf5gt"
Jul 28 12:53:09.407277 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.406904    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/03193c25-6f01-4389-b250-6eae144b3298-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zf5gt\" (UID: \"03193c25-6f01-4389-b250-6eae144b3298\") " pod="openshift-multus/multus-additional-cni-plugins-zf5gt"
Jul 28 12:53:09.407277 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.406930    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f562a438-3615-401c-9c89-a8e553a96877-etc-selinux\") pod \"aws-ebs-csi-driver-node-csb84\" (UID: \"f562a438-3615-401c-9c89-a8e553a96877\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csb84"
Jul 28 12:53:09.407277 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.406948    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3943de15-60a3-4e53-af9f-125cb8bae2ab-konnectivity-ca\") pod \"konnectivity-agent-cxz2b\" (UID: \"3943de15-60a3-4e53-af9f-125cb8bae2ab\") " pod="kube-system/konnectivity-agent-cxz2b"
Jul 28 12:53:09.407277 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.406960    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e03e76bd-a14d-4bab-8fea-35b382ecabd7-host-run-k8s-cni-cncf-io\") pod \"multus-xtlx4\" (UID: \"e03e76bd-a14d-4bab-8fea-35b382ecabd7\") " pod="openshift-multus/multus-xtlx4"
Jul 28 12:53:09.407277 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.406949    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f562a438-3615-401c-9c89-a8e553a96877-device-dir\") pod \"aws-ebs-csi-driver-node-csb84\" (UID: \"f562a438-3615-401c-9c89-a8e553a96877\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csb84"
Jul 28 12:53:09.407277 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.406985    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e03e76bd-a14d-4bab-8fea-35b382ecabd7-host-run-multus-certs\") pod \"multus-xtlx4\" (UID: \"e03e76bd-a14d-4bab-8fea-35b382ecabd7\") " pod="openshift-multus/multus-xtlx4"
Jul 28 12:53:09.407277 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.406992    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/03193c25-6f01-4389-b250-6eae144b3298-os-release\") pod \"multus-additional-cni-plugins-zf5gt\" (UID: \"03193c25-6f01-4389-b250-6eae144b3298\") " pod="openshift-multus/multus-additional-cni-plugins-zf5gt"
Jul 28 12:53:09.407277 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.407037    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2e5d8f8f-d54f-4126-97a1-4b91dc7d40f8-iptables-alerter-script\") pod \"iptables-alerter-bc4gl\" (UID: \"2e5d8f8f-d54f-4126-97a1-4b91dc7d40f8\") " pod="openshift-network-operator/iptables-alerter-bc4gl"
Jul 28 12:53:09.407277 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.407084    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e03e76bd-a14d-4bab-8fea-35b382ecabd7-host-run-k8s-cni-cncf-io\") pod \"multus-xtlx4\" (UID: \"e03e76bd-a14d-4bab-8fea-35b382ecabd7\") " pod="openshift-multus/multus-xtlx4"
Jul 28 12:53:09.407277 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.407117    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1-var-lib-kubelet\") pod \"tuned-xnfmf\" (UID: \"6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1\") " pod="openshift-cluster-node-tuning-operator/tuned-xnfmf"
Jul 28 12:53:09.407277 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.407134    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f562a438-3615-401c-9c89-a8e553a96877-sys-fs\") pod \"aws-ebs-csi-driver-node-csb84\" (UID: \"f562a438-3615-401c-9c89-a8e553a96877\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csb84"
Jul 28 12:53:09.407277 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.407151    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/46ca2ea9-2097-4729-8b59-52882a7ea6bf-host\") pod \"node-ca-7n245\" (UID: \"46ca2ea9-2097-4729-8b59-52882a7ea6bf\") " pod="openshift-image-registry/node-ca-7n245"
Jul 28 12:53:09.408095 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.407157    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e03e76bd-a14d-4bab-8fea-35b382ecabd7-multus-daemon-config\") pod \"multus-xtlx4\" (UID: \"e03e76bd-a14d-4bab-8fea-35b382ecabd7\") " pod="openshift-multus/multus-xtlx4"
Jul 28 12:53:09.408095 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.407178    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e03e76bd-a14d-4bab-8fea-35b382ecabd7-multus-conf-dir\") pod \"multus-xtlx4\" (UID: \"e03e76bd-a14d-4bab-8fea-35b382ecabd7\") " pod="openshift-multus/multus-xtlx4"
Jul 28 12:53:09.408095 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.407214    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a874f52e-baf3-4227-b2ef-f66115db2ffb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nd4vq\" (UID: \"a874f52e-baf3-4227-b2ef-f66115db2ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:09.408095 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.407253    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f562a438-3615-401c-9c89-a8e553a96877-etc-selinux\") pod \"aws-ebs-csi-driver-node-csb84\" (UID: \"f562a438-3615-401c-9c89-a8e553a96877\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csb84"
Jul 28 12:53:09.408095 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.407292    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1-etc-sysctl-d\") pod \"tuned-xnfmf\" (UID: \"6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1\") " pod="openshift-cluster-node-tuning-operator/tuned-xnfmf"
Jul 28 12:53:09.408095 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.407327    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e03e76bd-a14d-4bab-8fea-35b382ecabd7-host-run-multus-certs\") pod \"multus-xtlx4\" (UID: \"e03e76bd-a14d-4bab-8fea-35b382ecabd7\") " pod="openshift-multus/multus-xtlx4"
Jul 28 12:53:09.408095 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.407341    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a874f52e-baf3-4227-b2ef-f66115db2ffb-host-kubelet\") pod \"ovnkube-node-nd4vq\" (UID: \"a874f52e-baf3-4227-b2ef-f66115db2ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:09.408095 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.407379    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a874f52e-baf3-4227-b2ef-f66115db2ffb-host-cni-netd\") pod \"ovnkube-node-nd4vq\" (UID: \"a874f52e-baf3-4227-b2ef-f66115db2ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:09.408095 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.407439    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e03e76bd-a14d-4bab-8fea-35b382ecabd7-system-cni-dir\") pod \"multus-xtlx4\" (UID: \"e03e76bd-a14d-4bab-8fea-35b382ecabd7\") " pod="openshift-multus/multus-xtlx4"
Jul 28 12:53:09.408095 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.407489    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1-var-lib-kubelet\") pod \"tuned-xnfmf\" (UID: \"6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1\") " pod="openshift-cluster-node-tuning-operator/tuned-xnfmf"
Jul 28 12:53:09.408095 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.407494    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/03193c25-6f01-4389-b250-6eae144b3298-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zf5gt\" (UID: \"03193c25-6f01-4389-b250-6eae144b3298\") " pod="openshift-multus/multus-additional-cni-plugins-zf5gt"
Jul 28 12:53:09.408095 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.407538    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/03193c25-6f01-4389-b250-6eae144b3298-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zf5gt\" (UID: \"03193c25-6f01-4389-b250-6eae144b3298\") " pod="openshift-multus/multus-additional-cni-plugins-zf5gt"
Jul 28 12:53:09.408095 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.407541    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f562a438-3615-401c-9c89-a8e553a96877-kubelet-dir\") pod \"aws-ebs-csi-driver-node-csb84\" (UID: \"f562a438-3615-401c-9c89-a8e553a96877\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csb84"
Jul 28 12:53:09.408095 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.407576    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/46ca2ea9-2097-4729-8b59-52882a7ea6bf-host\") pod \"node-ca-7n245\" (UID: \"46ca2ea9-2097-4729-8b59-52882a7ea6bf\") " pod="openshift-image-registry/node-ca-7n245"
Jul 28 12:53:09.408095 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.407577    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2e5d8f8f-d54f-4126-97a1-4b91dc7d40f8-iptables-alerter-script\") pod \"iptables-alerter-bc4gl\" (UID: \"2e5d8f8f-d54f-4126-97a1-4b91dc7d40f8\") " pod="openshift-network-operator/iptables-alerter-bc4gl"
Jul 28 12:53:09.408095 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.407178    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a874f52e-baf3-4227-b2ef-f66115db2ffb-host-kubelet\") pod \"ovnkube-node-nd4vq\" (UID: \"a874f52e-baf3-4227-b2ef-f66115db2ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:09.408095 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.407668    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/03193c25-6f01-4389-b250-6eae144b3298-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zf5gt\" (UID: \"03193c25-6f01-4389-b250-6eae144b3298\") " pod="openshift-multus/multus-additional-cni-plugins-zf5gt"
Jul 28 12:53:09.408774 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.407663    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a874f52e-baf3-4227-b2ef-f66115db2ffb-env-overrides\") pod \"ovnkube-node-nd4vq\" (UID: \"a874f52e-baf3-4227-b2ef-f66115db2ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:09.408774 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.407817    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a874f52e-baf3-4227-b2ef-f66115db2ffb-etc-openvswitch\") pod \"ovnkube-node-nd4vq\" (UID: \"a874f52e-baf3-4227-b2ef-f66115db2ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:09.408774 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.408510    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e03e76bd-a14d-4bab-8fea-35b382ecabd7-host-var-lib-cni-bin\") pod \"multus-xtlx4\" (UID: \"e03e76bd-a14d-4bab-8fea-35b382ecabd7\") " pod="openshift-multus/multus-xtlx4"
Jul 28 12:53:09.408774 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.408514    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e03e76bd-a14d-4bab-8fea-35b382ecabd7-host-run-netns\") pod \"multus-xtlx4\" (UID: \"e03e76bd-a14d-4bab-8fea-35b382ecabd7\") " pod="openshift-multus/multus-xtlx4"
Jul 28 12:53:09.408774 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.408520    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1-etc-systemd\") pod \"tuned-xnfmf\" (UID: \"6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1\") " pod="openshift-cluster-node-tuning-operator/tuned-xnfmf"
Jul 28 12:53:09.408774 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:09.408573    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:53:09.408774 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.408630    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e03e76bd-a14d-4bab-8fea-35b382ecabd7-host-var-lib-kubelet\") pod \"multus-xtlx4\" (UID: \"e03e76bd-a14d-4bab-8fea-35b382ecabd7\") " pod="openshift-multus/multus-xtlx4"
Jul 28 12:53:09.408774 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.408648    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e03e76bd-a14d-4bab-8fea-35b382ecabd7-host-var-lib-cni-bin\") pod \"multus-xtlx4\" (UID: \"e03e76bd-a14d-4bab-8fea-35b382ecabd7\") " pod="openshift-multus/multus-xtlx4"
Jul 28 12:53:09.408774 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.408659    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da8b58f9-2528-4906-b7d2-aadea159b4f6-original-pull-secret\") pod \"global-pull-secret-syncer-bdgzk\" (UID: \"da8b58f9-2528-4906-b7d2-aadea159b4f6\") " pod="kube-system/global-pull-secret-syncer-bdgzk"
Jul 28 12:53:09.408774 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.408701    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e03e76bd-a14d-4bab-8fea-35b382ecabd7-host-var-lib-kubelet\") pod \"multus-xtlx4\" (UID: \"e03e76bd-a14d-4bab-8fea-35b382ecabd7\") " pod="openshift-multus/multus-xtlx4"
Jul 28 12:53:09.408774 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.408742    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f562a438-3615-401c-9c89-a8e553a96877-registration-dir\") pod \"aws-ebs-csi-driver-node-csb84\" (UID: \"f562a438-3615-401c-9c89-a8e553a96877\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csb84"
Jul 28 12:53:09.408774 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.408767    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3943de15-60a3-4e53-af9f-125cb8bae2ab-agent-certs\") pod \"konnectivity-agent-cxz2b\" (UID: \"3943de15-60a3-4e53-af9f-125cb8bae2ab\") " pod="kube-system/konnectivity-agent-cxz2b"
Jul 28 12:53:09.409379 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:09.408827    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a418b22c-fa2d-4d69-8a21-7e2209a892de-metrics-certs podName:a418b22c-fa2d-4d69-8a21-7e2209a892de nodeName:}" failed. No retries permitted until 2026-07-28 12:53:09.908787116 +0000 UTC m=+2.114173695 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a418b22c-fa2d-4d69-8a21-7e2209a892de-metrics-certs") pod "network-metrics-daemon-ps58v" (UID: "a418b22c-fa2d-4d69-8a21-7e2209a892de") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:53:09.409379 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.408851    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwtpx\" (UniqueName: \"kubernetes.io/projected/46ca2ea9-2097-4729-8b59-52882a7ea6bf-kube-api-access-jwtpx\") pod \"node-ca-7n245\" (UID: \"46ca2ea9-2097-4729-8b59-52882a7ea6bf\") " pod="openshift-image-registry/node-ca-7n245"
Jul 28 12:53:09.409379 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:09.408885    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:53:09.409379 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.408884    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/03193c25-6f01-4389-b250-6eae144b3298-cni-binary-copy\") pod \"multus-additional-cni-plugins-zf5gt\" (UID: \"03193c25-6f01-4389-b250-6eae144b3298\") " pod="openshift-multus/multus-additional-cni-plugins-zf5gt"
Jul 28 12:53:09.409379 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.408897    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/46ca2ea9-2097-4729-8b59-52882a7ea6bf-serviceca\") pod \"node-ca-7n245\" (UID: \"46ca2ea9-2097-4729-8b59-52882a7ea6bf\") " pod="openshift-image-registry/node-ca-7n245"
Jul 28 12:53:09.409379 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.408941    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f562a438-3615-401c-9c89-a8e553a96877-registration-dir\") pod \"aws-ebs-csi-driver-node-csb84\" (UID: \"f562a438-3615-401c-9c89-a8e553a96877\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csb84"
Jul 28 12:53:09.409379 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:09.409022    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da8b58f9-2528-4906-b7d2-aadea159b4f6-original-pull-secret podName:da8b58f9-2528-4906-b7d2-aadea159b4f6 nodeName:}" failed. No retries permitted until 2026-07-28 12:53:09.909008217 +0000 UTC m=+2.114394803 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/da8b58f9-2528-4906-b7d2-aadea159b4f6-original-pull-secret") pod "global-pull-secret-syncer-bdgzk" (UID: "da8b58f9-2528-4906-b7d2-aadea159b4f6") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:53:09.409379 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.409163    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1-etc-modprobe-d\") pod \"tuned-xnfmf\" (UID: \"6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1\") " pod="openshift-cluster-node-tuning-operator/tuned-xnfmf"
Jul 28 12:53:09.409379 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.409193    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9krd\" (UniqueName: \"kubernetes.io/projected/4807337c-67b7-4b80-bf38-58ecdd8d972c-kube-api-access-n9krd\") pod \"network-check-target-wk6vn\" (UID: \"4807337c-67b7-4b80-bf38-58ecdd8d972c\") " pod="openshift-network-diagnostics/network-check-target-wk6vn"
Jul 28 12:53:09.409379 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.409234    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a874f52e-baf3-4227-b2ef-f66115db2ffb-ovnkube-config\") pod \"ovnkube-node-nd4vq\" (UID: \"a874f52e-baf3-4227-b2ef-f66115db2ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:09.409379 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.409257    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e03e76bd-a14d-4bab-8fea-35b382ecabd7-multus-cni-dir\") pod \"multus-xtlx4\" (UID: \"e03e76bd-a14d-4bab-8fea-35b382ecabd7\") " pod="openshift-multus/multus-xtlx4"
Jul 28 12:53:09.409379 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.409265    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1-etc-modprobe-d\") pod \"tuned-xnfmf\" (UID: \"6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1\") " pod="openshift-cluster-node-tuning-operator/tuned-xnfmf"
Jul 28 12:53:09.409379 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.409281    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/da8b58f9-2528-4906-b7d2-aadea159b4f6-kubelet-config\") pod \"global-pull-secret-syncer-bdgzk\" (UID: \"da8b58f9-2528-4906-b7d2-aadea159b4f6\") " pod="kube-system/global-pull-secret-syncer-bdgzk"
Jul 28 12:53:09.409379 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.409322    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/03193c25-6f01-4389-b250-6eae144b3298-cnibin\") pod \"multus-additional-cni-plugins-zf5gt\" (UID: \"03193c25-6f01-4389-b250-6eae144b3298\") " pod="openshift-multus/multus-additional-cni-plugins-zf5gt"
Jul 28 12:53:09.409379 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.409333    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e03e76bd-a14d-4bab-8fea-35b382ecabd7-multus-cni-dir\") pod \"multus-xtlx4\" (UID: \"e03e76bd-a14d-4bab-8fea-35b382ecabd7\") " pod="openshift-multus/multus-xtlx4"
Jul 28 12:53:09.409379 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.409346    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x288h\" (UniqueName: \"kubernetes.io/projected/f562a438-3615-401c-9c89-a8e553a96877-kube-api-access-x288h\") pod \"aws-ebs-csi-driver-node-csb84\" (UID: \"f562a438-3615-401c-9c89-a8e553a96877\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csb84"
Jul 28 12:53:09.410260 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.409385    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a874f52e-baf3-4227-b2ef-f66115db2ffb-host-slash\") pod \"ovnkube-node-nd4vq\" (UID: \"a874f52e-baf3-4227-b2ef-f66115db2ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:09.410260 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.409387    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/03193c25-6f01-4389-b250-6eae144b3298-cni-binary-copy\") pod \"multus-additional-cni-plugins-zf5gt\" (UID: \"03193c25-6f01-4389-b250-6eae144b3298\") " pod="openshift-multus/multus-additional-cni-plugins-zf5gt"
Jul 28 12:53:09.410260 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.409410    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/da8b58f9-2528-4906-b7d2-aadea159b4f6-dbus\") pod \"global-pull-secret-syncer-bdgzk\" (UID: \"da8b58f9-2528-4906-b7d2-aadea159b4f6\") " pod="kube-system/global-pull-secret-syncer-bdgzk"
Jul 28 12:53:09.410260 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.409385    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/da8b58f9-2528-4906-b7d2-aadea159b4f6-kubelet-config\") pod \"global-pull-secret-syncer-bdgzk\" (UID: \"da8b58f9-2528-4906-b7d2-aadea159b4f6\") " pod="kube-system/global-pull-secret-syncer-bdgzk"
Jul 28 12:53:09.410260 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.409417    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/03193c25-6f01-4389-b250-6eae144b3298-cnibin\") pod \"multus-additional-cni-plugins-zf5gt\" (UID: \"03193c25-6f01-4389-b250-6eae144b3298\") " pod="openshift-multus/multus-additional-cni-plugins-zf5gt"
Jul 28 12:53:09.410260 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.409457    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a874f52e-baf3-4227-b2ef-f66115db2ffb-host-slash\") pod \"ovnkube-node-nd4vq\" (UID: \"a874f52e-baf3-4227-b2ef-f66115db2ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:09.410260 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.409486    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/da8b58f9-2528-4906-b7d2-aadea159b4f6-dbus\") pod \"global-pull-secret-syncer-bdgzk\" (UID: \"da8b58f9-2528-4906-b7d2-aadea159b4f6\") " pod="kube-system/global-pull-secret-syncer-bdgzk"
Jul 28 12:53:09.410260 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.409504    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1-etc-kubernetes\") pod \"tuned-xnfmf\" (UID: \"6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1\") " pod="openshift-cluster-node-tuning-operator/tuned-xnfmf"
Jul 28 12:53:09.410260 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.409531    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1-host\") pod \"tuned-xnfmf\" (UID: \"6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1\") " pod="openshift-cluster-node-tuning-operator/tuned-xnfmf"
Jul 28 12:53:09.410260 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.409553    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a874f52e-baf3-4227-b2ef-f66115db2ffb-systemd-units\") pod \"ovnkube-node-nd4vq\" (UID: \"a874f52e-baf3-4227-b2ef-f66115db2ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:09.410260 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.409565    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1-etc-kubernetes\") pod \"tuned-xnfmf\" (UID: \"6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1\") " pod="openshift-cluster-node-tuning-operator/tuned-xnfmf"
Jul 28 12:53:09.410260 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.409578    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e03e76bd-a14d-4bab-8fea-35b382ecabd7-host-var-lib-cni-multus\") pod \"multus-xtlx4\" (UID: \"e03e76bd-a14d-4bab-8fea-35b382ecabd7\") " pod="openshift-multus/multus-xtlx4"
Jul 28 12:53:09.410260 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.409604    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1-host\") pod \"tuned-xnfmf\" (UID: \"6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1\") " pod="openshift-cluster-node-tuning-operator/tuned-xnfmf"
Jul 28 12:53:09.410260 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.409618    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a874f52e-baf3-4227-b2ef-f66115db2ffb-systemd-units\") pod \"ovnkube-node-nd4vq\" (UID: \"a874f52e-baf3-4227-b2ef-f66115db2ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:09.410260 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.409661    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e03e76bd-a14d-4bab-8fea-35b382ecabd7-host-var-lib-cni-multus\") pod \"multus-xtlx4\" (UID: \"e03e76bd-a14d-4bab-8fea-35b382ecabd7\") " pod="openshift-multus/multus-xtlx4"
Jul 28 12:53:09.410260 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.410216    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1-tmp\") pod \"tuned-xnfmf\" (UID: \"6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1\") " pod="openshift-cluster-node-tuning-operator/tuned-xnfmf"
Jul 28 12:53:09.410260 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.410259    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a874f52e-baf3-4227-b2ef-f66115db2ffb-ovn-node-metrics-cert\") pod \"ovnkube-node-nd4vq\" (UID: \"a874f52e-baf3-4227-b2ef-f66115db2ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:09.411142 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.410567    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a874f52e-baf3-4227-b2ef-f66115db2ffb-ovnkube-config\") pod \"ovnkube-node-nd4vq\" (UID: \"a874f52e-baf3-4227-b2ef-f66115db2ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:09.411587 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.411564    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1-etc-tuned\") pod \"tuned-xnfmf\" (UID: \"6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1\") " pod="openshift-cluster-node-tuning-operator/tuned-xnfmf"
Jul 28 12:53:09.411661 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.411631    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3943de15-60a3-4e53-af9f-125cb8bae2ab-agent-certs\") pod \"konnectivity-agent-cxz2b\" (UID: \"3943de15-60a3-4e53-af9f-125cb8bae2ab\") " pod="kube-system/konnectivity-agent-cxz2b"
Jul 28 12:53:09.412179 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.412162    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmhqb\" (UniqueName: \"kubernetes.io/projected/a418b22c-fa2d-4d69-8a21-7e2209a892de-kube-api-access-jmhqb\") pod \"network-metrics-daemon-ps58v\" (UID: \"a418b22c-fa2d-4d69-8a21-7e2209a892de\") " pod="openshift-multus/network-metrics-daemon-ps58v"
Jul 28 12:53:09.412413 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.412397    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlh6d\" (UniqueName: \"kubernetes.io/projected/a874f52e-baf3-4227-b2ef-f66115db2ffb-kube-api-access-rlh6d\") pod \"ovnkube-node-nd4vq\" (UID: \"a874f52e-baf3-4227-b2ef-f66115db2ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:09.414631 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.414607    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5842m\" (UniqueName: \"kubernetes.io/projected/e03e76bd-a14d-4bab-8fea-35b382ecabd7-kube-api-access-5842m\") pod \"multus-xtlx4\" (UID: \"e03e76bd-a14d-4bab-8fea-35b382ecabd7\") " pod="openshift-multus/multus-xtlx4"
Jul 28 12:53:09.415277 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.415250    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxmqr\" (UniqueName: \"kubernetes.io/projected/6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1-kube-api-access-lxmqr\") pod \"tuned-xnfmf\" (UID: \"6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1\") " pod="openshift-cluster-node-tuning-operator/tuned-xnfmf"
Jul 28 12:53:09.415363 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.415275    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-666cc\" (UniqueName: \"kubernetes.io/projected/2e5d8f8f-d54f-4126-97a1-4b91dc7d40f8-kube-api-access-666cc\") pod \"iptables-alerter-bc4gl\" (UID: \"2e5d8f8f-d54f-4126-97a1-4b91dc7d40f8\") " pod="openshift-network-operator/iptables-alerter-bc4gl"
Jul 28 12:53:09.415363 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.415350    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtqfq\" (UniqueName: \"kubernetes.io/projected/a89e422d-76c2-4e62-b9fb-4283d5fa3de9-kube-api-access-jtqfq\") pod \"node-resolver-66b94\" (UID: \"a89e422d-76c2-4e62-b9fb-4283d5fa3de9\") " pod="openshift-dns/node-resolver-66b94"
Jul 28 12:53:09.415581 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:09.415568    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:53:09.415631 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:09.415583    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:53:09.415631 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:09.415592    2574 projected.go:196] Error preparing data for projected volume kube-api-access-n9krd for pod openshift-network-diagnostics/network-check-target-wk6vn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:53:09.415712 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:09.415641    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4807337c-67b7-4b80-bf38-58ecdd8d972c-kube-api-access-n9krd podName:4807337c-67b7-4b80-bf38-58ecdd8d972c nodeName:}" failed. No retries permitted until 2026-07-28 12:53:09.915629511 +0000 UTC m=+2.121016090 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-n9krd" (UniqueName: "kubernetes.io/projected/4807337c-67b7-4b80-bf38-58ecdd8d972c-kube-api-access-n9krd") pod "network-check-target-wk6vn" (UID: "4807337c-67b7-4b80-bf38-58ecdd8d972c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:53:09.417457 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.417439    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwtpx\" (UniqueName: \"kubernetes.io/projected/46ca2ea9-2097-4729-8b59-52882a7ea6bf-kube-api-access-jwtpx\") pod \"node-ca-7n245\" (UID: \"46ca2ea9-2097-4729-8b59-52882a7ea6bf\") " pod="openshift-image-registry/node-ca-7n245"
Jul 28 12:53:09.417457 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.417444    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9jb5\" (UniqueName: \"kubernetes.io/projected/03193c25-6f01-4389-b250-6eae144b3298-kube-api-access-c9jb5\") pod \"multus-additional-cni-plugins-zf5gt\" (UID: \"03193c25-6f01-4389-b250-6eae144b3298\") " pod="openshift-multus/multus-additional-cni-plugins-zf5gt"
Jul 28 12:53:09.417698 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.417672    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x288h\" (UniqueName: \"kubernetes.io/projected/f562a438-3615-401c-9c89-a8e553a96877-kube-api-access-x288h\") pod \"aws-ebs-csi-driver-node-csb84\" (UID: \"f562a438-3615-401c-9c89-a8e553a96877\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csb84"
Jul 28 12:53:09.548780 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.548733    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-66b94"
Jul 28 12:53:09.557753 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.557737    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xnfmf"
Jul 28 12:53:09.559638 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:09.559613    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda89e422d_76c2_4e62_b9fb_4283d5fa3de9.slice/crio-e1972eb28578d93ed0685f786b33512ba01ed001e07439cf0954faf55f5b859f WatchSource:0}: Error finding container e1972eb28578d93ed0685f786b33512ba01ed001e07439cf0954faf55f5b859f: Status 404 returned error can't find the container with id e1972eb28578d93ed0685f786b33512ba01ed001e07439cf0954faf55f5b859f
Jul 28 12:53:09.565050 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:09.565033    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f6527d2_58ba_4cf8_9eba_3db4d45ff3e1.slice/crio-787037fbc6ba200525baf5a4a095eb120d7976a25378a9dd40642268d41ba380 WatchSource:0}: Error finding container 787037fbc6ba200525baf5a4a095eb120d7976a25378a9dd40642268d41ba380: Status 404 returned error can't find the container with id 787037fbc6ba200525baf5a4a095eb120d7976a25378a9dd40642268d41ba380
Jul 28 12:53:09.589001 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.588984    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bc4gl"
Jul 28 12:53:09.591374 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:09.591354    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podead970784cc741b3c8d768dce86822b5.slice/crio-1a6ebfc36a4b9970edeba10a5cbca31915583c17551faf3c7a00d6a44c95b796 WatchSource:0}: Error finding container 1a6ebfc36a4b9970edeba10a5cbca31915583c17551faf3c7a00d6a44c95b796: Status 404 returned error can't find the container with id 1a6ebfc36a4b9970edeba10a5cbca31915583c17551faf3c7a00d6a44c95b796
Jul 28 12:53:09.596359 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:09.596336    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e5d8f8f_d54f_4126_97a1_4b91dc7d40f8.slice/crio-6ab8d8cac165d3764c66ecf72a0e2af3cc98810b461ddf3ee1f35b56af9e4595 WatchSource:0}: Error finding container 6ab8d8cac165d3764c66ecf72a0e2af3cc98810b461ddf3ee1f35b56af9e4595: Status 404 returned error can't find the container with id 6ab8d8cac165d3764c66ecf72a0e2af3cc98810b461ddf3ee1f35b56af9e4595
Jul 28 12:53:09.603565 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.603549    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cxz2b"
Jul 28 12:53:09.611971 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:09.611952    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3943de15_60a3_4e53_af9f_125cb8bae2ab.slice/crio-19c9ff55bc5055272964c6bbfdb4926d26d4fe5ce4a85c281342e06f9aa7fe91 WatchSource:0}: Error finding container 19c9ff55bc5055272964c6bbfdb4926d26d4fe5ce4a85c281342e06f9aa7fe91: Status 404 returned error can't find the container with id 19c9ff55bc5055272964c6bbfdb4926d26d4fe5ce4a85c281342e06f9aa7fe91
Jul 28 12:53:09.619672 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.619658    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7n245"
Jul 28 12:53:09.626706 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:09.626670    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46ca2ea9_2097_4729_8b59_52882a7ea6bf.slice/crio-51c41593aa47105de60552aad6529f30cb661ed2bd98301f7e8cd1745cdd393d WatchSource:0}: Error finding container 51c41593aa47105de60552aad6529f30cb661ed2bd98301f7e8cd1745cdd393d: Status 404 returned error can't find the container with id 51c41593aa47105de60552aad6529f30cb661ed2bd98301f7e8cd1745cdd393d
Jul 28 12:53:09.637671 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.637655    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zf5gt"
Jul 28 12:53:09.643450 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.643433    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:09.644772 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:09.644754    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03193c25_6f01_4389_b250_6eae144b3298.slice/crio-45bb42632fc200e5b03b3aa065e25178b41a971ff8c335c09f7859a2efb7171a WatchSource:0}: Error finding container 45bb42632fc200e5b03b3aa065e25178b41a971ff8c335c09f7859a2efb7171a: Status 404 returned error can't find the container with id 45bb42632fc200e5b03b3aa065e25178b41a971ff8c335c09f7859a2efb7171a
Jul 28 12:53:09.651447 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:09.651429    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda874f52e_baf3_4227_b2ef_f66115db2ffb.slice/crio-63b4c4217700dc6648c67ebc16f51ec61166e182970e2e6f1a8cb17736f8c204 WatchSource:0}: Error finding container 63b4c4217700dc6648c67ebc16f51ec61166e182970e2e6f1a8cb17736f8c204: Status 404 returned error can't find the container with id 63b4c4217700dc6648c67ebc16f51ec61166e182970e2e6f1a8cb17736f8c204
Jul 28 12:53:09.672445 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.672422    2574 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 12:53:09.677646 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.677631    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xtlx4"
Jul 28 12:53:09.682225 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.682209    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csb84"
Jul 28 12:53:09.686419 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:09.686402    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode03e76bd_a14d_4bab_8fea_35b382ecabd7.slice/crio-975dcae13b06c0e3eac119efd7b8e159930f54ff707f3aa5ccebe986d94f6187 WatchSource:0}: Error finding container 975dcae13b06c0e3eac119efd7b8e159930f54ff707f3aa5ccebe986d94f6187: Status 404 returned error can't find the container with id 975dcae13b06c0e3eac119efd7b8e159930f54ff707f3aa5ccebe986d94f6187
Jul 28 12:53:09.690626 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:09.690610    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf562a438_3615_401c_9c89_a8e553a96877.slice/crio-ec6af854096d991994afc9e3ed49130f758847fb79912f71d7db7f62d26d0470 WatchSource:0}: Error finding container ec6af854096d991994afc9e3ed49130f758847fb79912f71d7db7f62d26d0470: Status 404 returned error can't find the container with id ec6af854096d991994afc9e3ed49130f758847fb79912f71d7db7f62d26d0470
Jul 28 12:53:09.709628 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.709613    2574 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 12:53:09.915070 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.915011    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a418b22c-fa2d-4d69-8a21-7e2209a892de-metrics-certs\") pod \"network-metrics-daemon-ps58v\" (UID: \"a418b22c-fa2d-4d69-8a21-7e2209a892de\") " pod="openshift-multus/network-metrics-daemon-ps58v"
Jul 28 12:53:09.915070 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:09.915056    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da8b58f9-2528-4906-b7d2-aadea159b4f6-original-pull-secret\") pod \"global-pull-secret-syncer-bdgzk\" (UID: \"da8b58f9-2528-4906-b7d2-aadea159b4f6\") " pod="kube-system/global-pull-secret-syncer-bdgzk"
Jul 28 12:53:09.915211 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:09.915134    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:53:09.915211 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:09.915154    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:53:09.915211 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:09.915181    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a418b22c-fa2d-4d69-8a21-7e2209a892de-metrics-certs podName:a418b22c-fa2d-4d69-8a21-7e2209a892de nodeName:}" failed. No retries permitted until 2026-07-28 12:53:10.915168409 +0000 UTC m=+3.120554988 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a418b22c-fa2d-4d69-8a21-7e2209a892de-metrics-certs") pod "network-metrics-daemon-ps58v" (UID: "a418b22c-fa2d-4d69-8a21-7e2209a892de") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:53:09.915328 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:09.915235    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da8b58f9-2528-4906-b7d2-aadea159b4f6-original-pull-secret podName:da8b58f9-2528-4906-b7d2-aadea159b4f6 nodeName:}" failed. No retries permitted until 2026-07-28 12:53:10.915218418 +0000 UTC m=+3.120605202 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/da8b58f9-2528-4906-b7d2-aadea159b4f6-original-pull-secret") pod "global-pull-secret-syncer-bdgzk" (UID: "da8b58f9-2528-4906-b7d2-aadea159b4f6") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:53:10.016235 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:10.016209    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9krd\" (UniqueName: \"kubernetes.io/projected/4807337c-67b7-4b80-bf38-58ecdd8d972c-kube-api-access-n9krd\") pod \"network-check-target-wk6vn\" (UID: \"4807337c-67b7-4b80-bf38-58ecdd8d972c\") " pod="openshift-network-diagnostics/network-check-target-wk6vn"
Jul 28 12:53:10.016396 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:10.016376    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:53:10.016396 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:10.016399    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:53:10.016610 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:10.016411    2574 projected.go:196] Error preparing data for projected volume kube-api-access-n9krd for pod openshift-network-diagnostics/network-check-target-wk6vn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:53:10.016610 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:10.016463    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4807337c-67b7-4b80-bf38-58ecdd8d972c-kube-api-access-n9krd podName:4807337c-67b7-4b80-bf38-58ecdd8d972c nodeName:}" failed. No retries permitted until 2026-07-28 12:53:11.016447124 +0000 UTC m=+3.221833717 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-n9krd" (UniqueName: "kubernetes.io/projected/4807337c-67b7-4b80-bf38-58ecdd8d972c-kube-api-access-n9krd") pod "network-check-target-wk6vn" (UID: "4807337c-67b7-4b80-bf38-58ecdd8d972c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:53:10.261889 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:10.261770    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 12:48:09 +0000 UTC" deadline="2028-01-04 17:45:18.692578241 +0000 UTC"
Jul 28 12:53:10.261889 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:10.261804    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12604h52m8.430778713s"
Jul 28 12:53:10.308428 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:10.308329    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xtlx4" event={"ID":"e03e76bd-a14d-4bab-8fea-35b382ecabd7","Type":"ContainerStarted","Data":"975dcae13b06c0e3eac119efd7b8e159930f54ff707f3aa5ccebe986d94f6187"}
Jul 28 12:53:10.311902 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:10.311803    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq" event={"ID":"a874f52e-baf3-4227-b2ef-f66115db2ffb","Type":"ContainerStarted","Data":"63b4c4217700dc6648c67ebc16f51ec61166e182970e2e6f1a8cb17736f8c204"}
Jul 28 12:53:10.319108 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:10.319044    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zf5gt" event={"ID":"03193c25-6f01-4389-b250-6eae144b3298","Type":"ContainerStarted","Data":"45bb42632fc200e5b03b3aa065e25178b41a971ff8c335c09f7859a2efb7171a"}
Jul 28 12:53:10.326071 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:10.326007    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bc4gl" event={"ID":"2e5d8f8f-d54f-4126-97a1-4b91dc7d40f8","Type":"ContainerStarted","Data":"6ab8d8cac165d3764c66ecf72a0e2af3cc98810b461ddf3ee1f35b56af9e4595"}
Jul 28 12:53:10.330095 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:10.330069    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-158.ec2.internal" event={"ID":"ead970784cc741b3c8d768dce86822b5","Type":"ContainerStarted","Data":"1a6ebfc36a4b9970edeba10a5cbca31915583c17551faf3c7a00d6a44c95b796"}
Jul 28 12:53:10.340799 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:10.340777    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-66b94" event={"ID":"a89e422d-76c2-4e62-b9fb-4283d5fa3de9","Type":"ContainerStarted","Data":"e1972eb28578d93ed0685f786b33512ba01ed001e07439cf0954faf55f5b859f"}
Jul 28 12:53:10.346391 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:10.346297    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csb84" event={"ID":"f562a438-3615-401c-9c89-a8e553a96877","Type":"ContainerStarted","Data":"ec6af854096d991994afc9e3ed49130f758847fb79912f71d7db7f62d26d0470"}
Jul 28 12:53:10.350606 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:10.350584    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7n245" event={"ID":"46ca2ea9-2097-4729-8b59-52882a7ea6bf","Type":"ContainerStarted","Data":"51c41593aa47105de60552aad6529f30cb661ed2bd98301f7e8cd1745cdd393d"}
Jul 28 12:53:10.356001 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:10.355054    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cxz2b" event={"ID":"3943de15-60a3-4e53-af9f-125cb8bae2ab","Type":"ContainerStarted","Data":"19c9ff55bc5055272964c6bbfdb4926d26d4fe5ce4a85c281342e06f9aa7fe91"}
Jul 28 12:53:10.367047 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:10.367015    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xnfmf" event={"ID":"6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1","Type":"ContainerStarted","Data":"787037fbc6ba200525baf5a4a095eb120d7976a25378a9dd40642268d41ba380"}
Jul 28 12:53:10.373806 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:10.373782    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-158.ec2.internal" event={"ID":"6f72deeae9606046a5729c2df486eb15","Type":"ContainerStarted","Data":"80ed1b564500c13b6cdf4020b29c2bef46903a992ae59903a097a8a71214c2e0"}
Jul 28 12:53:10.427362 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:10.427072    2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 12:53:10.923096 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:10.923061    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a418b22c-fa2d-4d69-8a21-7e2209a892de-metrics-certs\") pod \"network-metrics-daemon-ps58v\" (UID: \"a418b22c-fa2d-4d69-8a21-7e2209a892de\") " pod="openshift-multus/network-metrics-daemon-ps58v"
Jul 28 12:53:10.923273 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:10.923129    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da8b58f9-2528-4906-b7d2-aadea159b4f6-original-pull-secret\") pod \"global-pull-secret-syncer-bdgzk\" (UID: \"da8b58f9-2528-4906-b7d2-aadea159b4f6\") " pod="kube-system/global-pull-secret-syncer-bdgzk"
Jul 28 12:53:10.923335 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:10.923294    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:53:10.923391 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:10.923353    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da8b58f9-2528-4906-b7d2-aadea159b4f6-original-pull-secret podName:da8b58f9-2528-4906-b7d2-aadea159b4f6 nodeName:}" failed. No retries permitted until 2026-07-28 12:53:12.923334912 +0000 UTC m=+5.128721496 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/da8b58f9-2528-4906-b7d2-aadea159b4f6-original-pull-secret") pod "global-pull-secret-syncer-bdgzk" (UID: "da8b58f9-2528-4906-b7d2-aadea159b4f6") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:53:10.923954 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:10.923754    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:53:10.923954 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:10.923809    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a418b22c-fa2d-4d69-8a21-7e2209a892de-metrics-certs podName:a418b22c-fa2d-4d69-8a21-7e2209a892de nodeName:}" failed. No retries permitted until 2026-07-28 12:53:12.923792136 +0000 UTC m=+5.129178718 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a418b22c-fa2d-4d69-8a21-7e2209a892de-metrics-certs") pod "network-metrics-daemon-ps58v" (UID: "a418b22c-fa2d-4d69-8a21-7e2209a892de") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:53:11.024233 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:11.024197    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9krd\" (UniqueName: \"kubernetes.io/projected/4807337c-67b7-4b80-bf38-58ecdd8d972c-kube-api-access-n9krd\") pod \"network-check-target-wk6vn\" (UID: \"4807337c-67b7-4b80-bf38-58ecdd8d972c\") " pod="openshift-network-diagnostics/network-check-target-wk6vn"
Jul 28 12:53:11.024398 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:11.024355    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:53:11.024398 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:11.024377    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:53:11.024524 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:11.024419    2574 projected.go:196] Error preparing data for projected volume kube-api-access-n9krd for pod openshift-network-diagnostics/network-check-target-wk6vn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:53:11.024524 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:11.024481    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4807337c-67b7-4b80-bf38-58ecdd8d972c-kube-api-access-n9krd podName:4807337c-67b7-4b80-bf38-58ecdd8d972c nodeName:}" failed. No retries permitted until 2026-07-28 12:53:13.024462546 +0000 UTC m=+5.229849141 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-n9krd" (UniqueName: "kubernetes.io/projected/4807337c-67b7-4b80-bf38-58ecdd8d972c-kube-api-access-n9krd") pod "network-check-target-wk6vn" (UID: "4807337c-67b7-4b80-bf38-58ecdd8d972c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:53:11.262047 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:11.261936    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 12:48:09 +0000 UTC" deadline="2028-01-29 17:07:27.652916202 +0000 UTC"
Jul 28 12:53:11.262047 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:11.261996    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13204h14m16.390924801s"
Jul 28 12:53:11.297945 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:11.297917    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdgzk"
Jul 28 12:53:11.298080 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:11.297917    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wk6vn"
Jul 28 12:53:11.298080 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:11.298061    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdgzk" podUID="da8b58f9-2528-4906-b7d2-aadea159b4f6"
Jul 28 12:53:11.298204 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:11.298110    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ps58v"
Jul 28 12:53:11.298204 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:11.298173    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wk6vn" podUID="4807337c-67b7-4b80-bf38-58ecdd8d972c"
Jul 28 12:53:11.298304 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:11.298245    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ps58v" podUID="a418b22c-fa2d-4d69-8a21-7e2209a892de"
Jul 28 12:53:12.247305 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:12.247092    2574 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 12:53:12.938167 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:12.938124    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da8b58f9-2528-4906-b7d2-aadea159b4f6-original-pull-secret\") pod \"global-pull-secret-syncer-bdgzk\" (UID: \"da8b58f9-2528-4906-b7d2-aadea159b4f6\") " pod="kube-system/global-pull-secret-syncer-bdgzk"
Jul 28 12:53:12.938609 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:12.938204    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a418b22c-fa2d-4d69-8a21-7e2209a892de-metrics-certs\") pod \"network-metrics-daemon-ps58v\" (UID: \"a418b22c-fa2d-4d69-8a21-7e2209a892de\") " pod="openshift-multus/network-metrics-daemon-ps58v"
Jul 28 12:53:12.938609 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:12.938321    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:53:12.938609 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:12.938383    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a418b22c-fa2d-4d69-8a21-7e2209a892de-metrics-certs podName:a418b22c-fa2d-4d69-8a21-7e2209a892de nodeName:}" failed. No retries permitted until 2026-07-28 12:53:16.938363715 +0000 UTC m=+9.143750296 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a418b22c-fa2d-4d69-8a21-7e2209a892de-metrics-certs") pod "network-metrics-daemon-ps58v" (UID: "a418b22c-fa2d-4d69-8a21-7e2209a892de") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:53:12.938801 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:12.938783    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:53:12.938912 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:12.938892    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da8b58f9-2528-4906-b7d2-aadea159b4f6-original-pull-secret podName:da8b58f9-2528-4906-b7d2-aadea159b4f6 nodeName:}" failed. No retries permitted until 2026-07-28 12:53:16.938875044 +0000 UTC m=+9.144261629 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/da8b58f9-2528-4906-b7d2-aadea159b4f6-original-pull-secret") pod "global-pull-secret-syncer-bdgzk" (UID: "da8b58f9-2528-4906-b7d2-aadea159b4f6") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:53:13.039026 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:13.038990    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9krd\" (UniqueName: \"kubernetes.io/projected/4807337c-67b7-4b80-bf38-58ecdd8d972c-kube-api-access-n9krd\") pod \"network-check-target-wk6vn\" (UID: \"4807337c-67b7-4b80-bf38-58ecdd8d972c\") " pod="openshift-network-diagnostics/network-check-target-wk6vn"
Jul 28 12:53:13.039238 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:13.039205    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:53:13.039238 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:13.039228    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:53:13.039238 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:13.039240    2574 projected.go:196] Error preparing data for projected volume kube-api-access-n9krd for pod openshift-network-diagnostics/network-check-target-wk6vn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:53:13.039410 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:13.039295    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4807337c-67b7-4b80-bf38-58ecdd8d972c-kube-api-access-n9krd podName:4807337c-67b7-4b80-bf38-58ecdd8d972c nodeName:}" failed. No retries permitted until 2026-07-28 12:53:17.039277624 +0000 UTC m=+9.244664207 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-n9krd" (UniqueName: "kubernetes.io/projected/4807337c-67b7-4b80-bf38-58ecdd8d972c-kube-api-access-n9krd") pod "network-check-target-wk6vn" (UID: "4807337c-67b7-4b80-bf38-58ecdd8d972c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:53:13.296808 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:13.296396    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wk6vn"
Jul 28 12:53:13.296808 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:13.296440    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ps58v"
Jul 28 12:53:13.296808 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:13.296519    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wk6vn" podUID="4807337c-67b7-4b80-bf38-58ecdd8d972c"
Jul 28 12:53:13.296808 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:13.296588    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ps58v" podUID="a418b22c-fa2d-4d69-8a21-7e2209a892de"
Jul 28 12:53:13.296808 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:13.296634    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdgzk"
Jul 28 12:53:13.296808 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:13.296739    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdgzk" podUID="da8b58f9-2528-4906-b7d2-aadea159b4f6"
Jul 28 12:53:15.296662 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:15.296628    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ps58v"
Jul 28 12:53:15.296662 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:15.296651    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdgzk"
Jul 28 12:53:15.297303 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:15.296827    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdgzk" podUID="da8b58f9-2528-4906-b7d2-aadea159b4f6"
Jul 28 12:53:15.297303 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:15.296840    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wk6vn"
Jul 28 12:53:15.297303 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:15.296947    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ps58v" podUID="a418b22c-fa2d-4d69-8a21-7e2209a892de"
Jul 28 12:53:15.297303 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:15.297044    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wk6vn" podUID="4807337c-67b7-4b80-bf38-58ecdd8d972c"
Jul 28 12:53:16.967872 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:16.967837    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a418b22c-fa2d-4d69-8a21-7e2209a892de-metrics-certs\") pod \"network-metrics-daemon-ps58v\" (UID: \"a418b22c-fa2d-4d69-8a21-7e2209a892de\") " pod="openshift-multus/network-metrics-daemon-ps58v"
Jul 28 12:53:16.968309 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:16.967902    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da8b58f9-2528-4906-b7d2-aadea159b4f6-original-pull-secret\") pod \"global-pull-secret-syncer-bdgzk\" (UID: \"da8b58f9-2528-4906-b7d2-aadea159b4f6\") " pod="kube-system/global-pull-secret-syncer-bdgzk"
Jul 28 12:53:16.968309 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:16.968050    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:53:16.968309 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:16.968111    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da8b58f9-2528-4906-b7d2-aadea159b4f6-original-pull-secret podName:da8b58f9-2528-4906-b7d2-aadea159b4f6 nodeName:}" failed. No retries permitted until 2026-07-28 12:53:24.9680931 +0000 UTC m=+17.173479693 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/da8b58f9-2528-4906-b7d2-aadea159b4f6-original-pull-secret") pod "global-pull-secret-syncer-bdgzk" (UID: "da8b58f9-2528-4906-b7d2-aadea159b4f6") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:53:16.968574 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:16.968545    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:53:16.968658 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:16.968605    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a418b22c-fa2d-4d69-8a21-7e2209a892de-metrics-certs podName:a418b22c-fa2d-4d69-8a21-7e2209a892de nodeName:}" failed. No retries permitted until 2026-07-28 12:53:24.968589073 +0000 UTC m=+17.173975652 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a418b22c-fa2d-4d69-8a21-7e2209a892de-metrics-certs") pod "network-metrics-daemon-ps58v" (UID: "a418b22c-fa2d-4d69-8a21-7e2209a892de") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:53:17.069266 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:17.069151    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9krd\" (UniqueName: \"kubernetes.io/projected/4807337c-67b7-4b80-bf38-58ecdd8d972c-kube-api-access-n9krd\") pod \"network-check-target-wk6vn\" (UID: \"4807337c-67b7-4b80-bf38-58ecdd8d972c\") " pod="openshift-network-diagnostics/network-check-target-wk6vn"
Jul 28 12:53:17.069413 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:17.069343    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:53:17.069413 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:17.069369    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:53:17.069413 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:17.069384    2574 projected.go:196] Error preparing data for projected volume kube-api-access-n9krd for pod openshift-network-diagnostics/network-check-target-wk6vn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:53:17.069603 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:17.069449    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4807337c-67b7-4b80-bf38-58ecdd8d972c-kube-api-access-n9krd podName:4807337c-67b7-4b80-bf38-58ecdd8d972c nodeName:}" failed. No retries permitted until 2026-07-28 12:53:25.069428805 +0000 UTC m=+17.274815387 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-n9krd" (UniqueName: "kubernetes.io/projected/4807337c-67b7-4b80-bf38-58ecdd8d972c-kube-api-access-n9krd") pod "network-check-target-wk6vn" (UID: "4807337c-67b7-4b80-bf38-58ecdd8d972c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:53:17.296736 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:17.296510    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdgzk"
Jul 28 12:53:17.296736 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:17.296519    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ps58v"
Jul 28 12:53:17.296736 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:17.296541    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wk6vn"
Jul 28 12:53:17.296736 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:17.296633    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdgzk" podUID="da8b58f9-2528-4906-b7d2-aadea159b4f6"
Jul 28 12:53:17.296736 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:17.296716    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wk6vn" podUID="4807337c-67b7-4b80-bf38-58ecdd8d972c"
Jul 28 12:53:17.297125 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:17.296843    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ps58v" podUID="a418b22c-fa2d-4d69-8a21-7e2209a892de"
Jul 28 12:53:19.296305 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:19.296225    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ps58v"
Jul 28 12:53:19.296305 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:19.296250    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wk6vn"
Jul 28 12:53:19.296934 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:19.296366    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ps58v" podUID="a418b22c-fa2d-4d69-8a21-7e2209a892de"
Jul 28 12:53:19.296934 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:19.296600    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wk6vn" podUID="4807337c-67b7-4b80-bf38-58ecdd8d972c"
Jul 28 12:53:19.296934 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:19.296228    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdgzk"
Jul 28 12:53:19.296934 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:19.296716    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdgzk" podUID="da8b58f9-2528-4906-b7d2-aadea159b4f6"
Jul 28 12:53:21.296708 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:21.296623    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ps58v"
Jul 28 12:53:21.296708 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:21.296634    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdgzk"
Jul 28 12:53:21.297116 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:21.296755    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wk6vn"
Jul 28 12:53:21.297116 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:21.296780    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ps58v" podUID="a418b22c-fa2d-4d69-8a21-7e2209a892de"
Jul 28 12:53:21.297116 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:21.296848    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wk6vn" podUID="4807337c-67b7-4b80-bf38-58ecdd8d972c"
Jul 28 12:53:21.297116 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:21.296922    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdgzk" podUID="da8b58f9-2528-4906-b7d2-aadea159b4f6"
Jul 28 12:53:23.296856 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:23.296821    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdgzk"
Jul 28 12:53:23.297300 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:23.296866    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ps58v"
Jul 28 12:53:23.297300 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:23.296866    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wk6vn"
Jul 28 12:53:23.297300 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:23.296944    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdgzk" podUID="da8b58f9-2528-4906-b7d2-aadea159b4f6"
Jul 28 12:53:23.297300 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:23.297067    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ps58v" podUID="a418b22c-fa2d-4d69-8a21-7e2209a892de"
Jul 28 12:53:23.297300 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:23.297164    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wk6vn" podUID="4807337c-67b7-4b80-bf38-58ecdd8d972c"
Jul 28 12:53:25.033604 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:25.033563    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da8b58f9-2528-4906-b7d2-aadea159b4f6-original-pull-secret\") pod \"global-pull-secret-syncer-bdgzk\" (UID: \"da8b58f9-2528-4906-b7d2-aadea159b4f6\") " pod="kube-system/global-pull-secret-syncer-bdgzk"
Jul 28 12:53:25.034105 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:25.033724    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a418b22c-fa2d-4d69-8a21-7e2209a892de-metrics-certs\") pod \"network-metrics-daemon-ps58v\" (UID: \"a418b22c-fa2d-4d69-8a21-7e2209a892de\") " pod="openshift-multus/network-metrics-daemon-ps58v"
Jul 28 12:53:25.034105 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:25.033743    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:53:25.034105 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:25.033822    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da8b58f9-2528-4906-b7d2-aadea159b4f6-original-pull-secret podName:da8b58f9-2528-4906-b7d2-aadea159b4f6 nodeName:}" failed. No retries permitted until 2026-07-28 12:53:41.033800328 +0000 UTC m=+33.239186922 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/da8b58f9-2528-4906-b7d2-aadea159b4f6-original-pull-secret") pod "global-pull-secret-syncer-bdgzk" (UID: "da8b58f9-2528-4906-b7d2-aadea159b4f6") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:53:25.034105 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:25.033836    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:53:25.034105 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:25.033888    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a418b22c-fa2d-4d69-8a21-7e2209a892de-metrics-certs podName:a418b22c-fa2d-4d69-8a21-7e2209a892de nodeName:}" failed. No retries permitted until 2026-07-28 12:53:41.03386985 +0000 UTC m=+33.239256434 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a418b22c-fa2d-4d69-8a21-7e2209a892de-metrics-certs") pod "network-metrics-daemon-ps58v" (UID: "a418b22c-fa2d-4d69-8a21-7e2209a892de") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:53:25.134499 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:25.134461    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9krd\" (UniqueName: \"kubernetes.io/projected/4807337c-67b7-4b80-bf38-58ecdd8d972c-kube-api-access-n9krd\") pod \"network-check-target-wk6vn\" (UID: \"4807337c-67b7-4b80-bf38-58ecdd8d972c\") " pod="openshift-network-diagnostics/network-check-target-wk6vn"
Jul 28 12:53:25.134661 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:25.134622    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:53:25.134661 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:25.134647    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:53:25.134661 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:25.134659    2574 projected.go:196] Error preparing data for projected volume kube-api-access-n9krd for pod openshift-network-diagnostics/network-check-target-wk6vn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:53:25.134851 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:25.134731    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4807337c-67b7-4b80-bf38-58ecdd8d972c-kube-api-access-n9krd podName:4807337c-67b7-4b80-bf38-58ecdd8d972c nodeName:}" failed. No retries permitted until 2026-07-28 12:53:41.134711784 +0000 UTC m=+33.340098378 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-n9krd" (UniqueName: "kubernetes.io/projected/4807337c-67b7-4b80-bf38-58ecdd8d972c-kube-api-access-n9krd") pod "network-check-target-wk6vn" (UID: "4807337c-67b7-4b80-bf38-58ecdd8d972c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:53:25.296445 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:25.296361    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdgzk"
Jul 28 12:53:25.296613 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:25.296364    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wk6vn"
Jul 28 12:53:25.296613 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:25.296495    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdgzk" podUID="da8b58f9-2528-4906-b7d2-aadea159b4f6"
Jul 28 12:53:25.296613 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:25.296365    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ps58v"
Jul 28 12:53:25.296613 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:25.296577    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wk6vn" podUID="4807337c-67b7-4b80-bf38-58ecdd8d972c"
Jul 28 12:53:25.296838 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:25.296653    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ps58v" podUID="a418b22c-fa2d-4d69-8a21-7e2209a892de"
Jul 28 12:53:27.297249 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:27.297216    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdgzk"
Jul 28 12:53:27.297249 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:27.297240    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wk6vn"
Jul 28 12:53:27.297249 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:27.297250    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ps58v"
Jul 28 12:53:27.297856 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:27.297356    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdgzk" podUID="da8b58f9-2528-4906-b7d2-aadea159b4f6"
Jul 28 12:53:27.297856 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:27.297432    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wk6vn" podUID="4807337c-67b7-4b80-bf38-58ecdd8d972c"
Jul 28 12:53:27.297856 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:27.297532    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ps58v" podUID="a418b22c-fa2d-4d69-8a21-7e2209a892de"
Jul 28 12:53:29.296929 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:29.296899    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ps58v"
Jul 28 12:53:29.296929 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:29.296920    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdgzk"
Jul 28 12:53:29.297365 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:29.296899    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wk6vn"
Jul 28 12:53:29.297365 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:29.296999    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ps58v" podUID="a418b22c-fa2d-4d69-8a21-7e2209a892de"
Jul 28 12:53:29.297365 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:29.297123    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wk6vn" podUID="4807337c-67b7-4b80-bf38-58ecdd8d972c"
Jul 28 12:53:29.297365 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:29.297222    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdgzk" podUID="da8b58f9-2528-4906-b7d2-aadea159b4f6"
Jul 28 12:53:30.427710 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:30.427039    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xtlx4" event={"ID":"e03e76bd-a14d-4bab-8fea-35b382ecabd7","Type":"ContainerStarted","Data":"2073641a31b644c5c9ba9196bcea6f4f99d7ea88c23dfaf4ff361d9dcf69a63f"}
Jul 28 12:53:30.432099 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:30.431979    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq" event={"ID":"a874f52e-baf3-4227-b2ef-f66115db2ffb","Type":"ContainerStarted","Data":"2f64a5adfe85326ac386ab586a60de8a3bd7783ee92112e06dd7b9b955cf7a92"}
Jul 28 12:53:30.432099 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:30.432058    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq" event={"ID":"a874f52e-baf3-4227-b2ef-f66115db2ffb","Type":"ContainerStarted","Data":"9888e92452b4c8a6b974e63072965954253db4da4bb02375b1c227f806d2b414"}
Jul 28 12:53:30.432319 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:30.432110    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq" event={"ID":"a874f52e-baf3-4227-b2ef-f66115db2ffb","Type":"ContainerStarted","Data":"9df8d2b951b65e6f995e3742ae8686a802f00a2bb221761fd3f1f1a3aa17afc8"}
Jul 28 12:53:30.432319 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:30.432160    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq" event={"ID":"a874f52e-baf3-4227-b2ef-f66115db2ffb","Type":"ContainerStarted","Data":"512e3cf59d6d8a9f402360eecb17e9cedb93f43d64cc000a3f61ac3987ff18d6"}
Jul 28 12:53:30.432319 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:30.432207    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq" event={"ID":"a874f52e-baf3-4227-b2ef-f66115db2ffb","Type":"ContainerStarted","Data":"60034a8c611eed9655e558cdf6275a8ee4ea002678973eaec6876bac792c8730"}
Jul 28 12:53:30.432319 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:30.432253    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq" event={"ID":"a874f52e-baf3-4227-b2ef-f66115db2ffb","Type":"ContainerStarted","Data":"a2ebd9ebaa7d1d2e02c7be36025772ec15b24da4f2e7deb2a425801aa3ecb072"}
Jul 28 12:53:30.433815 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:30.433783    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xnfmf" event={"ID":"6f6527d2-58ba-4cf8-9eba-3db4d45ff3e1","Type":"ContainerStarted","Data":"33bd67c5cffaa30414b556efee9b79d5eb58a12cdfdc0e6878ae8bff4f4c88f0"}
Jul 28 12:53:30.435888 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:30.435862    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-158.ec2.internal" event={"ID":"6f72deeae9606046a5729c2df486eb15","Type":"ContainerStarted","Data":"503f62509d77ab5940d1a08b1e52fb5ed81567802dd49a6648e9ec41094235ba"}
Jul 28 12:53:30.448055 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:30.448008    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-xtlx4" podStartSLOduration=2.540202801 podStartE2EDuration="22.447995061s" totalImagesPullingTime="19.90779226s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:53:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:53:30.447720459 +0000 UTC m=+22.653107060" watchObservedRunningTime="2026-07-28 12:53:30.447995061 +0000 UTC m=+22.653381662"
Jul 28 12:53:30.484486 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:30.484427    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-158.ec2.internal" podStartSLOduration=21.484407752 podStartE2EDuration="21.484407752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:53:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:53:30.462890362 +0000 UTC m=+22.668276964" watchObservedRunningTime="2026-07-28 12:53:30.484407752 +0000 UTC m=+22.689794353"
Jul 28 12:53:30.484660 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:30.484628    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-xnfmf" podStartSLOduration=2.488715338 podStartE2EDuration="22.484599605s" totalImagesPullingTime="19.995884267s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:53:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:53:30.48413851 +0000 UTC m=+22.689525115" watchObservedRunningTime="2026-07-28 12:53:30.484599605 +0000 UTC m=+22.689986216"
Jul 28 12:53:31.296801 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:31.296723    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ps58v"
Jul 28 12:53:31.296941 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:31.296727    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdgzk"
Jul 28 12:53:31.296941 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:31.296845    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ps58v" podUID="a418b22c-fa2d-4d69-8a21-7e2209a892de"
Jul 28 12:53:31.296941 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:31.296727    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wk6vn"
Jul 28 12:53:31.296941 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:31.296894    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdgzk" podUID="da8b58f9-2528-4906-b7d2-aadea159b4f6"
Jul 28 12:53:31.297145 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:31.296958    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wk6vn" podUID="4807337c-67b7-4b80-bf38-58ecdd8d972c"
Jul 28 12:53:31.438672 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:31.438642    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csb84" event={"ID":"f562a438-3615-401c-9c89-a8e553a96877","Type":"ContainerStarted","Data":"68a260f4c32a4abcffc151aaceb53fff71600d4326a9fe6460b44a9d09b9fd36"}
Jul 28 12:53:31.439983 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:31.439959    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7n245" event={"ID":"46ca2ea9-2097-4729-8b59-52882a7ea6bf","Type":"ContainerStarted","Data":"929e3cc35a64e8e52b442105a3a9342bce755ef2a5311c53b4bcf6f369c3d687"}
Jul 28 12:53:31.441340 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:31.441313    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cxz2b" event={"ID":"3943de15-60a3-4e53-af9f-125cb8bae2ab","Type":"ContainerStarted","Data":"3176827c4f8ca8df110c3e8f54f49292f411963cad8ed12cb98574e71cebfe81"}
Jul 28 12:53:31.442670 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:31.442649    2574 generic.go:356] "Generic (PLEG): container finished" podID="03193c25-6f01-4389-b250-6eae144b3298" containerID="cf1bebec00021faa25b5679cfd06559e31039525b7e055abcf41ce4052bf450c" exitCode=0
Jul 28 12:53:31.442797 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:31.442712    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zf5gt" event={"ID":"03193c25-6f01-4389-b250-6eae144b3298","Type":"ContainerDied","Data":"cf1bebec00021faa25b5679cfd06559e31039525b7e055abcf41ce4052bf450c"}
Jul 28 12:53:31.444134 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:31.444074    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bc4gl" event={"ID":"2e5d8f8f-d54f-4126-97a1-4b91dc7d40f8","Type":"ContainerStarted","Data":"1bd14e63c4f64f0d448f5474b5c4811589eb98c75d473c6187cd5541800eb505"}
Jul 28 12:53:31.445543 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:31.445514    2574 generic.go:356] "Generic (PLEG): container finished" podID="ead970784cc741b3c8d768dce86822b5" containerID="ae7d0d78bdd75c1343ba58d0af27c1eabbd6dd936ab2fa7c174adb207c2c6bf0" exitCode=0
Jul 28 12:53:31.445629 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:31.445598    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-158.ec2.internal" event={"ID":"ead970784cc741b3c8d768dce86822b5","Type":"ContainerDied","Data":"ae7d0d78bdd75c1343ba58d0af27c1eabbd6dd936ab2fa7c174adb207c2c6bf0"}
Jul 28 12:53:31.446991 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:31.446969    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-66b94" event={"ID":"a89e422d-76c2-4e62-b9fb-4283d5fa3de9","Type":"ContainerStarted","Data":"09f6b18938292becdfbd45e54862bf4ed3335c25482a9b4ec601ebe04e362ad1"}
Jul 28 12:53:31.460290 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:31.460249    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7n245" podStartSLOduration=3.545366859 podStartE2EDuration="23.460235238s" totalImagesPullingTime="19.914868379s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:53:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:53:31.460212875 +0000 UTC m=+23.665599475" watchObservedRunningTime="2026-07-28 12:53:31.460235238 +0000 UTC m=+23.665621840"
Jul 28 12:53:31.476230 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:31.476200    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-cxz2b" podStartSLOduration=3.570950651 podStartE2EDuration="23.476189545s" totalImagesPullingTime="19.905238894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:53:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:53:31.476137185 +0000 UTC m=+23.681523785" watchObservedRunningTime="2026-07-28 12:53:31.476189545 +0000 UTC m=+23.681576148"
Jul 28 12:53:31.537936 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:31.537902    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-66b94" podStartSLOduration=3.580662627 podStartE2EDuration="23.537889413s" totalImagesPullingTime="19.957226786s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:53:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:53:31.537245429 +0000 UTC m=+23.742632030" watchObservedRunningTime="2026-07-28 12:53:31.537889413 +0000 UTC m=+23.743276012"
Jul 28 12:53:31.553805 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:31.553746    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-bc4gl" podStartSLOduration=3.6337110299999997 podStartE2EDuration="23.553734826s" totalImagesPullingTime="19.920023796s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:53:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:53:31.553430121 +0000 UTC m=+23.758816723" watchObservedRunningTime="2026-07-28 12:53:31.553734826 +0000 UTC m=+23.759121426"
Jul 28 12:53:32.454662 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:32.454459    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-158.ec2.internal" event={"ID":"ead970784cc741b3c8d768dce86822b5","Type":"ContainerStarted","Data":"cc1dc41c30cae70ce803ee312024795cac6a18d29d7292ee5e98c4e98d3ded63"}
Jul 28 12:53:32.467261 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:32.467237    2574 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 12:53:32.484945 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:32.484892    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-158.ec2.internal" podStartSLOduration=23.484874177000002 podStartE2EDuration="23.484874177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:53:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:53:32.48447073 +0000 UTC m=+24.689857331" watchObservedRunningTime="2026-07-28 12:53:32.484874177 +0000 UTC m=+24.690260779"
Jul 28 12:53:33.272507 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:33.272412    2574 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T12:53:32.467257238Z","UUID":"6ac28fe4-6d21-4b6e-9ebb-e7977a833f85","Handler":null,"Name":"","Endpoint":""}
Jul 28 12:53:33.274408 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:33.274344    2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 12:53:33.274561 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:33.274441    2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 12:53:33.296526 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:33.296482    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wk6vn"
Jul 28 12:53:33.296722 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:33.296483    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdgzk"
Jul 28 12:53:33.296722 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:33.296603    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wk6vn" podUID="4807337c-67b7-4b80-bf38-58ecdd8d972c"
Jul 28 12:53:33.296722 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:33.296701    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdgzk" podUID="da8b58f9-2528-4906-b7d2-aadea159b4f6"
Jul 28 12:53:33.296722 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:33.296487    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ps58v"
Jul 28 12:53:33.296950 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:33.296803    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ps58v" podUID="a418b22c-fa2d-4d69-8a21-7e2209a892de"
Jul 28 12:53:33.460043 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:33.460004    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq" event={"ID":"a874f52e-baf3-4227-b2ef-f66115db2ffb","Type":"ContainerStarted","Data":"e8ce5ec203cf31e48cb689f2b464060cab2c648f18c6ad95e8693b07fac8446f"}
Jul 28 12:53:33.462026 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:33.461992    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csb84" event={"ID":"f562a438-3615-401c-9c89-a8e553a96877","Type":"ContainerStarted","Data":"d251e4f67322280be7552bf420988449f6b77d23d95aa4b1053e23a58b241f57"}
Jul 28 12:53:34.466740 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:34.466701    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csb84" event={"ID":"f562a438-3615-401c-9c89-a8e553a96877","Type":"ContainerStarted","Data":"2983fc665edc0be1cedefaa1bfc68249f955197ff294f602253c20eb39295bd9"}
Jul 28 12:53:34.489508 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:34.489456    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-csb84" podStartSLOduration=3.593149679 podStartE2EDuration="26.489439146s" totalImagesPullingTime="22.896289467s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:53:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:53:34.488874416 +0000 UTC m=+26.694261018" watchObservedRunningTime="2026-07-28 12:53:34.489439146 +0000 UTC m=+26.694825747"
Jul 28 12:53:35.215501 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:35.215465    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-cxz2b"
Jul 28 12:53:35.216080 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:35.216057    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-cxz2b"
Jul 28 12:53:35.297190 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:35.297160    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wk6vn"
Jul 28 12:53:35.297329 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:35.297164    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdgzk"
Jul 28 12:53:35.297329 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:35.297266    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wk6vn" podUID="4807337c-67b7-4b80-bf38-58ecdd8d972c"
Jul 28 12:53:35.297434 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:35.297165    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ps58v"
Jul 28 12:53:35.297434 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:35.297359    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdgzk" podUID="da8b58f9-2528-4906-b7d2-aadea159b4f6"
Jul 28 12:53:35.297434 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:35.297426    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ps58v" podUID="a418b22c-fa2d-4d69-8a21-7e2209a892de"
Jul 28 12:53:35.468923 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:35.468821    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-cxz2b"
Jul 28 12:53:35.469385 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:35.469283    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-cxz2b"
Jul 28 12:53:36.472388 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:36.472236    2574 generic.go:356] "Generic (PLEG): container finished" podID="03193c25-6f01-4389-b250-6eae144b3298" containerID="30f75b7ed21800ea11f2452483ca0abf91503068f47b25e0d6330be76d4e8512" exitCode=0
Jul 28 12:53:36.472827 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:36.472320    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zf5gt" event={"ID":"03193c25-6f01-4389-b250-6eae144b3298","Type":"ContainerDied","Data":"30f75b7ed21800ea11f2452483ca0abf91503068f47b25e0d6330be76d4e8512"}
Jul 28 12:53:36.475977 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:36.475957    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq" event={"ID":"a874f52e-baf3-4227-b2ef-f66115db2ffb","Type":"ContainerStarted","Data":"c648589b6611cf4bb27830843384938b8677f9bcae2df49e04bdfe91a942a661"}
Jul 28 12:53:36.476215 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:36.476199    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:36.476286 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:36.476255    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:36.476286 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:36.476284    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:36.490972 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:36.490953    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:36.491084 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:36.491072    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:53:37.296323 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:37.296270    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdgzk"
Jul 28 12:53:37.296323 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:37.296288    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wk6vn"
Jul 28 12:53:37.296323 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:37.296299    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ps58v"
Jul 28 12:53:37.296643 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:37.296385    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdgzk" podUID="da8b58f9-2528-4906-b7d2-aadea159b4f6"
Jul 28 12:53:37.296643 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:37.296502    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ps58v" podUID="a418b22c-fa2d-4d69-8a21-7e2209a892de"
Jul 28 12:53:37.296643 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:37.296579    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wk6vn" podUID="4807337c-67b7-4b80-bf38-58ecdd8d972c"
Jul 28 12:53:38.095955 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:38.095835    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq" podStartSLOduration=9.951449323 podStartE2EDuration="30.095817525s" totalImagesPullingTime="20.144368202s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:53:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:53:36.558213996 +0000 UTC m=+28.763600597" watchObservedRunningTime="2026-07-28 12:53:38.095817525 +0000 UTC m=+30.301204127"
Jul 28 12:53:38.096703 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:38.096591    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bdgzk"]
Jul 28 12:53:38.096770 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:38.096702    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdgzk"
Jul 28 12:53:38.096828 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:38.096809    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdgzk" podUID="da8b58f9-2528-4906-b7d2-aadea159b4f6"
Jul 28 12:53:38.099936 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:38.099910    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wk6vn"]
Jul 28 12:53:38.100076 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:38.099987    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wk6vn"
Jul 28 12:53:38.100142 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:38.100067    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wk6vn" podUID="4807337c-67b7-4b80-bf38-58ecdd8d972c"
Jul 28 12:53:38.100669 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:38.100648    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ps58v"]
Jul 28 12:53:38.100776 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:38.100740    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ps58v"
Jul 28 12:53:38.100833 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:38.100818    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ps58v" podUID="a418b22c-fa2d-4d69-8a21-7e2209a892de"
Jul 28 12:53:38.481409 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:38.481379    2574 generic.go:356] "Generic (PLEG): container finished" podID="03193c25-6f01-4389-b250-6eae144b3298" containerID="ab5a4aad860c8807b461b24202080b22b07042d3536fc1e4d5fd6fde61b5f8b5" exitCode=0
Jul 28 12:53:38.481546 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:38.481461    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zf5gt" event={"ID":"03193c25-6f01-4389-b250-6eae144b3298","Type":"ContainerDied","Data":"ab5a4aad860c8807b461b24202080b22b07042d3536fc1e4d5fd6fde61b5f8b5"}
Jul 28 12:53:39.297460 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:39.297247    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ps58v"
Jul 28 12:53:39.297791 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:39.297279    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wk6vn"
Jul 28 12:53:39.297791 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:39.297514    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ps58v" podUID="a418b22c-fa2d-4d69-8a21-7e2209a892de"
Jul 28 12:53:39.297791 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:39.297548    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wk6vn" podUID="4807337c-67b7-4b80-bf38-58ecdd8d972c"
Jul 28 12:53:40.296782 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:40.296755    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdgzk"
Jul 28 12:53:40.296904 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:40.296850    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdgzk" podUID="da8b58f9-2528-4906-b7d2-aadea159b4f6"
Jul 28 12:53:40.487467 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:40.487441    2574 generic.go:356] "Generic (PLEG): container finished" podID="03193c25-6f01-4389-b250-6eae144b3298" containerID="0b8367cf9903779cd1ab98535d70e926c8f3e3aff7d54d73862af177ada99a80" exitCode=0
Jul 28 12:53:40.487808 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:40.487483    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zf5gt" event={"ID":"03193c25-6f01-4389-b250-6eae144b3298","Type":"ContainerDied","Data":"0b8367cf9903779cd1ab98535d70e926c8f3e3aff7d54d73862af177ada99a80"}
Jul 28 12:53:41.069222 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:41.069199    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da8b58f9-2528-4906-b7d2-aadea159b4f6-original-pull-secret\") pod \"global-pull-secret-syncer-bdgzk\" (UID: \"da8b58f9-2528-4906-b7d2-aadea159b4f6\") " pod="kube-system/global-pull-secret-syncer-bdgzk"
Jul 28 12:53:41.069358 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:41.069242    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a418b22c-fa2d-4d69-8a21-7e2209a892de-metrics-certs\") pod \"network-metrics-daemon-ps58v\" (UID: \"a418b22c-fa2d-4d69-8a21-7e2209a892de\") " pod="openshift-multus/network-metrics-daemon-ps58v"
Jul 28 12:53:41.069358 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:41.069311    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:53:41.069358 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:41.069313    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:53:41.069358 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:41.069351    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a418b22c-fa2d-4d69-8a21-7e2209a892de-metrics-certs podName:a418b22c-fa2d-4d69-8a21-7e2209a892de nodeName:}" failed. No retries permitted until 2026-07-28 12:54:13.06933962 +0000 UTC m=+65.274726199 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a418b22c-fa2d-4d69-8a21-7e2209a892de-metrics-certs") pod "network-metrics-daemon-ps58v" (UID: "a418b22c-fa2d-4d69-8a21-7e2209a892de") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:53:41.069508 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:41.069363    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da8b58f9-2528-4906-b7d2-aadea159b4f6-original-pull-secret podName:da8b58f9-2528-4906-b7d2-aadea159b4f6 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:13.069357739 +0000 UTC m=+65.274744318 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/da8b58f9-2528-4906-b7d2-aadea159b4f6-original-pull-secret") pod "global-pull-secret-syncer-bdgzk" (UID: "da8b58f9-2528-4906-b7d2-aadea159b4f6") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:53:41.169833 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:41.169807    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9krd\" (UniqueName: \"kubernetes.io/projected/4807337c-67b7-4b80-bf38-58ecdd8d972c-kube-api-access-n9krd\") pod \"network-check-target-wk6vn\" (UID: \"4807337c-67b7-4b80-bf38-58ecdd8d972c\") " pod="openshift-network-diagnostics/network-check-target-wk6vn"
Jul 28 12:53:41.169999 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:41.169973    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:53:41.169999 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:41.170001    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:53:41.170152 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:41.170014    2574 projected.go:196] Error preparing data for projected volume kube-api-access-n9krd for pod openshift-network-diagnostics/network-check-target-wk6vn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:53:41.170152 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:41.170058    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4807337c-67b7-4b80-bf38-58ecdd8d972c-kube-api-access-n9krd podName:4807337c-67b7-4b80-bf38-58ecdd8d972c nodeName:}" failed. No retries permitted until 2026-07-28 12:54:13.170044377 +0000 UTC m=+65.375430957 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-n9krd" (UniqueName: "kubernetes.io/projected/4807337c-67b7-4b80-bf38-58ecdd8d972c-kube-api-access-n9krd") pod "network-check-target-wk6vn" (UID: "4807337c-67b7-4b80-bf38-58ecdd8d972c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:53:41.297055 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:41.297025    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ps58v"
Jul 28 12:53:41.297199 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:41.297025    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wk6vn"
Jul 28 12:53:41.297199 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:41.297165    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ps58v" podUID="a418b22c-fa2d-4d69-8a21-7e2209a892de"
Jul 28 12:53:41.297322 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:41.297200    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wk6vn" podUID="4807337c-67b7-4b80-bf38-58ecdd8d972c"
Jul 28 12:53:42.296791 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.296755    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdgzk"
Jul 28 12:53:42.297251 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:42.296888    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdgzk" podUID="da8b58f9-2528-4906-b7d2-aadea159b4f6"
Jul 28 12:53:42.574813 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.574780    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-158.ec2.internal" event="NodeReady"
Jul 28 12:53:42.574954 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.574923    2574 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 12:53:42.611815 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.611789    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-787757c757-c4mh9"]
Jul 28 12:53:42.615477 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.615454    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-787757c757-c4mh9"
Jul 28 12:53:42.618267 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.618244    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 28 12:53:42.618395 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.618298    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 28 12:53:42.618395 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.618250    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6hlfc\""
Jul 28 12:53:42.619617 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.619596    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 28 12:53:42.625037 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.625012    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-r8l5q"]
Jul 28 12:53:42.625278 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.625253    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 28 12:53:42.628936 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.628918    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-b9htm"]
Jul 28 12:53:42.629064 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.629046    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-r8l5q"
Jul 28 12:53:42.632049 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.631853    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 12:53:42.632049 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.631858    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4l68m\""
Jul 28 12:53:42.632049 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.631858    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 12:53:42.632749 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.632729    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-787757c757-c4mh9"]
Jul 28 12:53:42.632841 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.632817    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b9htm"
Jul 28 12:53:42.636167 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.635764    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 12:53:42.636167 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.635805    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 12:53:42.636167 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.635865    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vb57q\""
Jul 28 12:53:42.636167 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.636023    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 12:53:42.638698 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.638671    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-r8l5q"]
Jul 28 12:53:42.641944 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.641925    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b9htm"]
Jul 28 12:53:42.782977 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.782945    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxcfc\" (UniqueName: \"kubernetes.io/projected/f8bc5bca-7c04-47ec-b3dc-f6ce86345325-kube-api-access-pxcfc\") pod \"dns-default-r8l5q\" (UID: \"f8bc5bca-7c04-47ec-b3dc-f6ce86345325\") " pod="openshift-dns/dns-default-r8l5q"
Jul 28 12:53:42.783116 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.782987    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6d412255-c984-4473-af61-1271b6f141e5-trusted-ca\") pod \"image-registry-787757c757-c4mh9\" (UID: \"6d412255-c984-4473-af61-1271b6f141e5\") " pod="openshift-image-registry/image-registry-787757c757-c4mh9"
Jul 28 12:53:42.783116 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.783081    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6d412255-c984-4473-af61-1271b6f141e5-ca-trust-extracted\") pod \"image-registry-787757c757-c4mh9\" (UID: \"6d412255-c984-4473-af61-1271b6f141e5\") " pod="openshift-image-registry/image-registry-787757c757-c4mh9"
Jul 28 12:53:42.783240 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.783120    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b530194-44d2-4993-9d2e-5aae3188a18d-cert\") pod \"ingress-canary-b9htm\" (UID: \"8b530194-44d2-4993-9d2e-5aae3188a18d\") " pod="openshift-ingress-canary/ingress-canary-b9htm"
Jul 28 12:53:42.783240 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.783154    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r448l\" (UniqueName: \"kubernetes.io/projected/6d412255-c984-4473-af61-1271b6f141e5-kube-api-access-r448l\") pod \"image-registry-787757c757-c4mh9\" (UID: \"6d412255-c984-4473-af61-1271b6f141e5\") " pod="openshift-image-registry/image-registry-787757c757-c4mh9"
Jul 28 12:53:42.783240 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.783216    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8bc5bca-7c04-47ec-b3dc-f6ce86345325-metrics-tls\") pod \"dns-default-r8l5q\" (UID: \"f8bc5bca-7c04-47ec-b3dc-f6ce86345325\") " pod="openshift-dns/dns-default-r8l5q"
Jul 28 12:53:42.783383 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.783241    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6d412255-c984-4473-af61-1271b6f141e5-registry-certificates\") pod \"image-registry-787757c757-c4mh9\" (UID: \"6d412255-c984-4473-af61-1271b6f141e5\") " pod="openshift-image-registry/image-registry-787757c757-c4mh9"
Jul 28 12:53:42.783383 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.783264    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6d412255-c984-4473-af61-1271b6f141e5-image-registry-private-configuration\") pod \"image-registry-787757c757-c4mh9\" (UID: \"6d412255-c984-4473-af61-1271b6f141e5\") " pod="openshift-image-registry/image-registry-787757c757-c4mh9"
Jul 28 12:53:42.783383 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.783282    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6d412255-c984-4473-af61-1271b6f141e5-bound-sa-token\") pod \"image-registry-787757c757-c4mh9\" (UID: \"6d412255-c984-4473-af61-1271b6f141e5\") " pod="openshift-image-registry/image-registry-787757c757-c4mh9"
Jul 28 12:53:42.783383 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.783305    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zm6f\" (UniqueName: \"kubernetes.io/projected/8b530194-44d2-4993-9d2e-5aae3188a18d-kube-api-access-6zm6f\") pod \"ingress-canary-b9htm\" (UID: \"8b530194-44d2-4993-9d2e-5aae3188a18d\") " pod="openshift-ingress-canary/ingress-canary-b9htm"
Jul 28 12:53:42.783383 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.783353    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f8bc5bca-7c04-47ec-b3dc-f6ce86345325-config-volume\") pod \"dns-default-r8l5q\" (UID: \"f8bc5bca-7c04-47ec-b3dc-f6ce86345325\") " pod="openshift-dns/dns-default-r8l5q"
Jul 28 12:53:42.783383 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.783381    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8bc5bca-7c04-47ec-b3dc-f6ce86345325-tmp-dir\") pod \"dns-default-r8l5q\" (UID: \"f8bc5bca-7c04-47ec-b3dc-f6ce86345325\") " pod="openshift-dns/dns-default-r8l5q"
Jul 28 12:53:42.783672 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.783409    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d412255-c984-4473-af61-1271b6f141e5-registry-tls\") pod \"image-registry-787757c757-c4mh9\" (UID: \"6d412255-c984-4473-af61-1271b6f141e5\") " pod="openshift-image-registry/image-registry-787757c757-c4mh9"
Jul 28 12:53:42.783672 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.783424    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6d412255-c984-4473-af61-1271b6f141e5-installation-pull-secrets\") pod \"image-registry-787757c757-c4mh9\" (UID: \"6d412255-c984-4473-af61-1271b6f141e5\") " pod="openshift-image-registry/image-registry-787757c757-c4mh9"
Jul 28 12:53:42.794888 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.794860    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cxz2b_3943de15-60a3-4e53-af9f-125cb8bae2ab/konnectivity-agent/0.log" limit=1024
Jul 28 12:53:42.884224 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.884162    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6d412255-c984-4473-af61-1271b6f141e5-registry-certificates\") pod \"image-registry-787757c757-c4mh9\" (UID: \"6d412255-c984-4473-af61-1271b6f141e5\") " pod="openshift-image-registry/image-registry-787757c757-c4mh9"
Jul 28 12:53:42.884224 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.884192    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6d412255-c984-4473-af61-1271b6f141e5-image-registry-private-configuration\") pod \"image-registry-787757c757-c4mh9\" (UID: \"6d412255-c984-4473-af61-1271b6f141e5\") " pod="openshift-image-registry/image-registry-787757c757-c4mh9"
Jul 28 12:53:42.884224 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.884209    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6d412255-c984-4473-af61-1271b6f141e5-bound-sa-token\") pod \"image-registry-787757c757-c4mh9\" (UID: \"6d412255-c984-4473-af61-1271b6f141e5\") " pod="openshift-image-registry/image-registry-787757c757-c4mh9"
Jul 28 12:53:42.884488 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.884232    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zm6f\" (UniqueName: \"kubernetes.io/projected/8b530194-44d2-4993-9d2e-5aae3188a18d-kube-api-access-6zm6f\") pod \"ingress-canary-b9htm\" (UID: \"8b530194-44d2-4993-9d2e-5aae3188a18d\") " pod="openshift-ingress-canary/ingress-canary-b9htm"
Jul 28 12:53:42.884544 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.884490    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f8bc5bca-7c04-47ec-b3dc-f6ce86345325-config-volume\") pod \"dns-default-r8l5q\" (UID: \"f8bc5bca-7c04-47ec-b3dc-f6ce86345325\") " pod="openshift-dns/dns-default-r8l5q"
Jul 28 12:53:42.884544 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.884531    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8bc5bca-7c04-47ec-b3dc-f6ce86345325-tmp-dir\") pod \"dns-default-r8l5q\" (UID: \"f8bc5bca-7c04-47ec-b3dc-f6ce86345325\") " pod="openshift-dns/dns-default-r8l5q"
Jul 28 12:53:42.884648 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.884561    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d412255-c984-4473-af61-1271b6f141e5-registry-tls\") pod \"image-registry-787757c757-c4mh9\" (UID: \"6d412255-c984-4473-af61-1271b6f141e5\") " pod="openshift-image-registry/image-registry-787757c757-c4mh9"
Jul 28 12:53:42.884648 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.884587    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6d412255-c984-4473-af61-1271b6f141e5-installation-pull-secrets\") pod \"image-registry-787757c757-c4mh9\" (UID: \"6d412255-c984-4473-af61-1271b6f141e5\") " pod="openshift-image-registry/image-registry-787757c757-c4mh9"
Jul 28 12:53:42.884648 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.884634    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxcfc\" (UniqueName: \"kubernetes.io/projected/f8bc5bca-7c04-47ec-b3dc-f6ce86345325-kube-api-access-pxcfc\") pod \"dns-default-r8l5q\" (UID: \"f8bc5bca-7c04-47ec-b3dc-f6ce86345325\") " pod="openshift-dns/dns-default-r8l5q"
Jul 28 12:53:42.884811 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.884661    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6d412255-c984-4473-af61-1271b6f141e5-trusted-ca\") pod \"image-registry-787757c757-c4mh9\" (UID: \"6d412255-c984-4473-af61-1271b6f141e5\") " pod="openshift-image-registry/image-registry-787757c757-c4mh9"
Jul 28 12:53:42.884811 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.884723    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6d412255-c984-4473-af61-1271b6f141e5-ca-trust-extracted\") pod \"image-registry-787757c757-c4mh9\" (UID: \"6d412255-c984-4473-af61-1271b6f141e5\") " pod="openshift-image-registry/image-registry-787757c757-c4mh9"
Jul 28 12:53:42.884811 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.884747    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b530194-44d2-4993-9d2e-5aae3188a18d-cert\") pod \"ingress-canary-b9htm\" (UID: \"8b530194-44d2-4993-9d2e-5aae3188a18d\") " pod="openshift-ingress-canary/ingress-canary-b9htm"
Jul 28 12:53:42.884811 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.884776    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r448l\" (UniqueName: \"kubernetes.io/projected/6d412255-c984-4473-af61-1271b6f141e5-kube-api-access-r448l\") pod \"image-registry-787757c757-c4mh9\" (UID: \"6d412255-c984-4473-af61-1271b6f141e5\") " pod="openshift-image-registry/image-registry-787757c757-c4mh9"
Jul 28 12:53:42.884811 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.884809    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8bc5bca-7c04-47ec-b3dc-f6ce86345325-metrics-tls\") pod \"dns-default-r8l5q\" (UID: \"f8bc5bca-7c04-47ec-b3dc-f6ce86345325\") " pod="openshift-dns/dns-default-r8l5q"
Jul 28 12:53:42.885015 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.884834    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6d412255-c984-4473-af61-1271b6f141e5-registry-certificates\") pod \"image-registry-787757c757-c4mh9\" (UID: \"6d412255-c984-4473-af61-1271b6f141e5\") " pod="openshift-image-registry/image-registry-787757c757-c4mh9"
Jul 28 12:53:42.885015 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:42.884915    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 12:53:42.885015 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:42.884971    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8bc5bca-7c04-47ec-b3dc-f6ce86345325-metrics-tls podName:f8bc5bca-7c04-47ec-b3dc-f6ce86345325 nodeName:}" failed. No retries permitted until 2026-07-28 12:53:43.38495441 +0000 UTC m=+35.590340996 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f8bc5bca-7c04-47ec-b3dc-f6ce86345325-metrics-tls") pod "dns-default-r8l5q" (UID: "f8bc5bca-7c04-47ec-b3dc-f6ce86345325") : secret "dns-default-metrics-tls" not found
Jul 28 12:53:42.885252 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.885057    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f8bc5bca-7c04-47ec-b3dc-f6ce86345325-config-volume\") pod \"dns-default-r8l5q\" (UID: \"f8bc5bca-7c04-47ec-b3dc-f6ce86345325\") " pod="openshift-dns/dns-default-r8l5q"
Jul 28 12:53:42.885252 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:42.885094    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 12:53:42.885252 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:42.885112    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-787757c757-c4mh9: secret "image-registry-tls" not found
Jul 28 12:53:42.885252 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:42.885158    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d412255-c984-4473-af61-1271b6f141e5-registry-tls podName:6d412255-c984-4473-af61-1271b6f141e5 nodeName:}" failed. No retries permitted until 2026-07-28 12:53:43.385142996 +0000 UTC m=+35.590529576 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6d412255-c984-4473-af61-1271b6f141e5-registry-tls") pod "image-registry-787757c757-c4mh9" (UID: "6d412255-c984-4473-af61-1271b6f141e5") : secret "image-registry-tls" not found
Jul 28 12:53:42.885460 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.885396    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8bc5bca-7c04-47ec-b3dc-f6ce86345325-tmp-dir\") pod \"dns-default-r8l5q\" (UID: \"f8bc5bca-7c04-47ec-b3dc-f6ce86345325\") " pod="openshift-dns/dns-default-r8l5q"
Jul 28 12:53:42.885584 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.885557    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6d412255-c984-4473-af61-1271b6f141e5-ca-trust-extracted\") pod \"image-registry-787757c757-c4mh9\" (UID: \"6d412255-c984-4473-af61-1271b6f141e5\") " pod="openshift-image-registry/image-registry-787757c757-c4mh9"
Jul 28 12:53:42.886010 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.885989    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6d412255-c984-4473-af61-1271b6f141e5-trusted-ca\") pod \"image-registry-787757c757-c4mh9\" (UID: \"6d412255-c984-4473-af61-1271b6f141e5\") " pod="openshift-image-registry/image-registry-787757c757-c4mh9"
Jul 28 12:53:42.886736 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.886666    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cxz2b_3943de15-60a3-4e53-af9f-125cb8bae2ab/konnectivity-agent/0.log" limit=1024
Jul 28 12:53:42.889255 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.889233    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6d412255-c984-4473-af61-1271b6f141e5-installation-pull-secrets\") pod \"image-registry-787757c757-c4mh9\" (UID: \"6d412255-c984-4473-af61-1271b6f141e5\") " pod="openshift-image-registry/image-registry-787757c757-c4mh9"
Jul 28 12:53:42.889337 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.889236    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6d412255-c984-4473-af61-1271b6f141e5-image-registry-private-configuration\") pod \"image-registry-787757c757-c4mh9\" (UID: \"6d412255-c984-4473-af61-1271b6f141e5\") " pod="openshift-image-registry/image-registry-787757c757-c4mh9"
Jul 28 12:53:42.889337 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.889241    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b530194-44d2-4993-9d2e-5aae3188a18d-cert\") pod \"ingress-canary-b9htm\" (UID: \"8b530194-44d2-4993-9d2e-5aae3188a18d\") " pod="openshift-ingress-canary/ingress-canary-b9htm"
Jul 28 12:53:42.894018 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.893978    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zm6f\" (UniqueName: \"kubernetes.io/projected/8b530194-44d2-4993-9d2e-5aae3188a18d-kube-api-access-6zm6f\") pod \"ingress-canary-b9htm\" (UID: \"8b530194-44d2-4993-9d2e-5aae3188a18d\") " pod="openshift-ingress-canary/ingress-canary-b9htm"
Jul 28 12:53:42.894532 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.894510    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6d412255-c984-4473-af61-1271b6f141e5-bound-sa-token\") pod \"image-registry-787757c757-c4mh9\" (UID: \"6d412255-c984-4473-af61-1271b6f141e5\") " pod="openshift-image-registry/image-registry-787757c757-c4mh9"
Jul 28 12:53:42.894626 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.894560    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxcfc\" (UniqueName: \"kubernetes.io/projected/f8bc5bca-7c04-47ec-b3dc-f6ce86345325-kube-api-access-pxcfc\") pod \"dns-default-r8l5q\" (UID: \"f8bc5bca-7c04-47ec-b3dc-f6ce86345325\") " pod="openshift-dns/dns-default-r8l5q"
Jul 28 12:53:42.894761 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.894740    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r448l\" (UniqueName: \"kubernetes.io/projected/6d412255-c984-4473-af61-1271b6f141e5-kube-api-access-r448l\") pod \"image-registry-787757c757-c4mh9\" (UID: \"6d412255-c984-4473-af61-1271b6f141e5\") " pod="openshift-image-registry/image-registry-787757c757-c4mh9"
Jul 28 12:53:42.947757 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:42.947736    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b9htm"
Jul 28 12:53:43.089089 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:43.088877    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b9htm"]
Jul 28 12:53:43.296693 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:43.296647    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ps58v"
Jul 28 12:53:43.296859 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:43.296715    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wk6vn"
Jul 28 12:53:43.300222 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:43.300054    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 12:53:43.300222 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:43.300126    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 12:53:43.300222 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:43.300202    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 12:53:43.300222 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:43.300221    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sxfnt\""
Jul 28 12:53:43.300980 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:43.300961    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wvgpn\""
Jul 28 12:53:43.389236 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:43.389155    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8bc5bca-7c04-47ec-b3dc-f6ce86345325-metrics-tls\") pod \"dns-default-r8l5q\" (UID: \"f8bc5bca-7c04-47ec-b3dc-f6ce86345325\") " pod="openshift-dns/dns-default-r8l5q"
Jul 28 12:53:43.389377 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:43.389305    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d412255-c984-4473-af61-1271b6f141e5-registry-tls\") pod \"image-registry-787757c757-c4mh9\" (UID: \"6d412255-c984-4473-af61-1271b6f141e5\") " pod="openshift-image-registry/image-registry-787757c757-c4mh9"
Jul 28 12:53:43.389377 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:43.389331    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 12:53:43.389491 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:43.389390    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8bc5bca-7c04-47ec-b3dc-f6ce86345325-metrics-tls podName:f8bc5bca-7c04-47ec-b3dc-f6ce86345325 nodeName:}" failed. No retries permitted until 2026-07-28 12:53:44.389369263 +0000 UTC m=+36.594755843 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f8bc5bca-7c04-47ec-b3dc-f6ce86345325-metrics-tls") pod "dns-default-r8l5q" (UID: "f8bc5bca-7c04-47ec-b3dc-f6ce86345325") : secret "dns-default-metrics-tls" not found
Jul 28 12:53:43.389491 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:43.389479    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 12:53:43.389596 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:43.389496    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-787757c757-c4mh9: secret "image-registry-tls" not found
Jul 28 12:53:43.389596 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:43.389557    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d412255-c984-4473-af61-1271b6f141e5-registry-tls podName:6d412255-c984-4473-af61-1271b6f141e5 nodeName:}" failed. No retries permitted until 2026-07-28 12:53:44.389541019 +0000 UTC m=+36.594927598 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6d412255-c984-4473-af61-1271b6f141e5-registry-tls") pod "image-registry-787757c757-c4mh9" (UID: "6d412255-c984-4473-af61-1271b6f141e5") : secret "image-registry-tls" not found
Jul 28 12:53:44.296625 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:44.296587    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdgzk"
Jul 28 12:53:44.299900 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:44.299874    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 12:53:44.395391 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:44.395361    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8bc5bca-7c04-47ec-b3dc-f6ce86345325-metrics-tls\") pod \"dns-default-r8l5q\" (UID: \"f8bc5bca-7c04-47ec-b3dc-f6ce86345325\") " pod="openshift-dns/dns-default-r8l5q"
Jul 28 12:53:44.395548 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:44.395419    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d412255-c984-4473-af61-1271b6f141e5-registry-tls\") pod \"image-registry-787757c757-c4mh9\" (UID: \"6d412255-c984-4473-af61-1271b6f141e5\") " pod="openshift-image-registry/image-registry-787757c757-c4mh9"
Jul 28 12:53:44.395548 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:44.395513    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 12:53:44.395657 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:44.395555    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 12:53:44.395657 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:44.395567    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-787757c757-c4mh9: secret "image-registry-tls" not found
Jul 28 12:53:44.395657 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:44.395575    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8bc5bca-7c04-47ec-b3dc-f6ce86345325-metrics-tls podName:f8bc5bca-7c04-47ec-b3dc-f6ce86345325 nodeName:}" failed. No retries permitted until 2026-07-28 12:53:46.395560011 +0000 UTC m=+38.600946594 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f8bc5bca-7c04-47ec-b3dc-f6ce86345325-metrics-tls") pod "dns-default-r8l5q" (UID: "f8bc5bca-7c04-47ec-b3dc-f6ce86345325") : secret "dns-default-metrics-tls" not found
Jul 28 12:53:44.395657 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:44.395622    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d412255-c984-4473-af61-1271b6f141e5-registry-tls podName:6d412255-c984-4473-af61-1271b6f141e5 nodeName:}" failed. No retries permitted until 2026-07-28 12:53:46.395605251 +0000 UTC m=+38.600991853 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6d412255-c984-4473-af61-1271b6f141e5-registry-tls") pod "image-registry-787757c757-c4mh9" (UID: "6d412255-c984-4473-af61-1271b6f141e5") : secret "image-registry-tls" not found
Jul 28 12:53:44.477464 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:53:44.477432    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b530194_44d2_4993_9d2e_5aae3188a18d.slice/crio-0bcc984d5d5cc6e169dcfb37cb36d3627be41d2f2d6b80ad00cb925cf9da1e24 WatchSource:0}: Error finding container 0bcc984d5d5cc6e169dcfb37cb36d3627be41d2f2d6b80ad00cb925cf9da1e24: Status 404 returned error can't find the container with id 0bcc984d5d5cc6e169dcfb37cb36d3627be41d2f2d6b80ad00cb925cf9da1e24
Jul 28 12:53:44.498294 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:44.498259    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b9htm" event={"ID":"8b530194-44d2-4993-9d2e-5aae3188a18d","Type":"ContainerStarted","Data":"0bcc984d5d5cc6e169dcfb37cb36d3627be41d2f2d6b80ad00cb925cf9da1e24"}
Jul 28 12:53:45.504183 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:45.504152    2574 generic.go:356] "Generic (PLEG): container finished" podID="03193c25-6f01-4389-b250-6eae144b3298" containerID="febd169d453c979cf2b1b45a72d091d23b1f03b053365df5fb9dcbdbb93acb61" exitCode=0
Jul 28 12:53:45.504808 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:45.504202    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zf5gt" event={"ID":"03193c25-6f01-4389-b250-6eae144b3298","Type":"ContainerDied","Data":"febd169d453c979cf2b1b45a72d091d23b1f03b053365df5fb9dcbdbb93acb61"}
Jul 28 12:53:46.409840 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:46.409798    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d412255-c984-4473-af61-1271b6f141e5-registry-tls\") pod \"image-registry-787757c757-c4mh9\" (UID: \"6d412255-c984-4473-af61-1271b6f141e5\") " pod="openshift-image-registry/image-registry-787757c757-c4mh9"
Jul 28 12:53:46.410021 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:46.409894    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8bc5bca-7c04-47ec-b3dc-f6ce86345325-metrics-tls\") pod \"dns-default-r8l5q\" (UID: \"f8bc5bca-7c04-47ec-b3dc-f6ce86345325\") " pod="openshift-dns/dns-default-r8l5q"
Jul 28 12:53:46.410021 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:46.409968    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 12:53:46.410021 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:46.409988    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-787757c757-c4mh9: secret "image-registry-tls" not found
Jul 28 12:53:46.410177 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:46.410026    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 12:53:46.410177 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:46.410040    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d412255-c984-4473-af61-1271b6f141e5-registry-tls podName:6d412255-c984-4473-af61-1271b6f141e5 nodeName:}" failed. No retries permitted until 2026-07-28 12:53:50.410026282 +0000 UTC m=+42.615412861 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6d412255-c984-4473-af61-1271b6f141e5-registry-tls") pod "image-registry-787757c757-c4mh9" (UID: "6d412255-c984-4473-af61-1271b6f141e5") : secret "image-registry-tls" not found
Jul 28 12:53:46.410177 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:46.410066    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8bc5bca-7c04-47ec-b3dc-f6ce86345325-metrics-tls podName:f8bc5bca-7c04-47ec-b3dc-f6ce86345325 nodeName:}" failed. No retries permitted until 2026-07-28 12:53:50.410054265 +0000 UTC m=+42.615440849 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f8bc5bca-7c04-47ec-b3dc-f6ce86345325-metrics-tls") pod "dns-default-r8l5q" (UID: "f8bc5bca-7c04-47ec-b3dc-f6ce86345325") : secret "dns-default-metrics-tls" not found
Jul 28 12:53:46.510839 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:46.510808    2574 generic.go:356] "Generic (PLEG): container finished" podID="03193c25-6f01-4389-b250-6eae144b3298" containerID="91136ffb958ae560528b60029ee978cec6682965bea67fc7686d6a335cc32df9" exitCode=0
Jul 28 12:53:46.511348 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:46.510850    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zf5gt" event={"ID":"03193c25-6f01-4389-b250-6eae144b3298","Type":"ContainerDied","Data":"91136ffb958ae560528b60029ee978cec6682965bea67fc7686d6a335cc32df9"}
Jul 28 12:53:47.515051 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:47.515013    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b9htm" event={"ID":"8b530194-44d2-4993-9d2e-5aae3188a18d","Type":"ContainerStarted","Data":"4c8145ba0c7f7396de682648539fc97194a24482e1c0b6d42928409631162bfa"}
Jul 28 12:53:47.517960 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:47.517933    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zf5gt" event={"ID":"03193c25-6f01-4389-b250-6eae144b3298","Type":"ContainerStarted","Data":"7bd5aa998b76db33db30d0dada393585c4127c17a2e7a49f3258f1406851246f"}
Jul 28 12:53:47.531773 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:47.531739    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-b9htm" podStartSLOduration=3.116980825 podStartE2EDuration="5.531727177s" totalImagesPullingTime="2.414746352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:53:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:53:47.531662811 +0000 UTC m=+39.737049413" watchObservedRunningTime="2026-07-28 12:53:47.531727177 +0000 UTC m=+39.737113784"
Jul 28 12:53:47.555477 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:47.555440    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-zf5gt" podStartSLOduration=8.985266101 podStartE2EDuration="39.555432675s" totalImagesPullingTime="30.570166574s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:53:08 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:53:47.5540617 +0000 UTC m=+39.759448302" watchObservedRunningTime="2026-07-28 12:53:47.555432675 +0000 UTC m=+39.760819272"
Jul 28 12:53:50.438054 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:50.438025    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d412255-c984-4473-af61-1271b6f141e5-registry-tls\") pod \"image-registry-787757c757-c4mh9\" (UID: \"6d412255-c984-4473-af61-1271b6f141e5\") " pod="openshift-image-registry/image-registry-787757c757-c4mh9"
Jul 28 12:53:50.438426 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:50.438077    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8bc5bca-7c04-47ec-b3dc-f6ce86345325-metrics-tls\") pod \"dns-default-r8l5q\" (UID: \"f8bc5bca-7c04-47ec-b3dc-f6ce86345325\") " pod="openshift-dns/dns-default-r8l5q"
Jul 28 12:53:50.438426 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:50.438157    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 12:53:50.438426 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:50.438163    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 12:53:50.438426 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:50.438180    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-787757c757-c4mh9: secret "image-registry-tls" not found
Jul 28 12:53:50.438426 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:50.438203    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8bc5bca-7c04-47ec-b3dc-f6ce86345325-metrics-tls podName:f8bc5bca-7c04-47ec-b3dc-f6ce86345325 nodeName:}" failed. No retries permitted until 2026-07-28 12:53:58.438191604 +0000 UTC m=+50.643578186 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f8bc5bca-7c04-47ec-b3dc-f6ce86345325-metrics-tls") pod "dns-default-r8l5q" (UID: "f8bc5bca-7c04-47ec-b3dc-f6ce86345325") : secret "dns-default-metrics-tls" not found
Jul 28 12:53:50.438426 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:50.438220    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d412255-c984-4473-af61-1271b6f141e5-registry-tls podName:6d412255-c984-4473-af61-1271b6f141e5 nodeName:}" failed. No retries permitted until 2026-07-28 12:53:58.438208092 +0000 UTC m=+50.643594674 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6d412255-c984-4473-af61-1271b6f141e5-registry-tls") pod "image-registry-787757c757-c4mh9" (UID: "6d412255-c984-4473-af61-1271b6f141e5") : secret "image-registry-tls" not found
Jul 28 12:53:58.493072 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:58.493043    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8bc5bca-7c04-47ec-b3dc-f6ce86345325-metrics-tls\") pod \"dns-default-r8l5q\" (UID: \"f8bc5bca-7c04-47ec-b3dc-f6ce86345325\") " pod="openshift-dns/dns-default-r8l5q"
Jul 28 12:53:58.493417 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:53:58.493091    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d412255-c984-4473-af61-1271b6f141e5-registry-tls\") pod \"image-registry-787757c757-c4mh9\" (UID: \"6d412255-c984-4473-af61-1271b6f141e5\") " pod="openshift-image-registry/image-registry-787757c757-c4mh9"
Jul 28 12:53:58.493417 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:58.493173    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 12:53:58.493417 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:58.493225    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8bc5bca-7c04-47ec-b3dc-f6ce86345325-metrics-tls podName:f8bc5bca-7c04-47ec-b3dc-f6ce86345325 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:14.493210938 +0000 UTC m=+66.698597517 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f8bc5bca-7c04-47ec-b3dc-f6ce86345325-metrics-tls") pod "dns-default-r8l5q" (UID: "f8bc5bca-7c04-47ec-b3dc-f6ce86345325") : secret "dns-default-metrics-tls" not found
Jul 28 12:53:58.493417 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:58.493175    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 12:53:58.493417 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:58.493266    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-787757c757-c4mh9: secret "image-registry-tls" not found
Jul 28 12:53:58.493417 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:53:58.493312    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d412255-c984-4473-af61-1271b6f141e5-registry-tls podName:6d412255-c984-4473-af61-1271b6f141e5 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:14.493301317 +0000 UTC m=+66.698687897 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6d412255-c984-4473-af61-1271b6f141e5-registry-tls") pod "image-registry-787757c757-c4mh9" (UID: "6d412255-c984-4473-af61-1271b6f141e5") : secret "image-registry-tls" not found
Jul 28 12:54:08.498809 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:08.498785    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nd4vq"
Jul 28 12:54:13.088539 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:13.088495    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da8b58f9-2528-4906-b7d2-aadea159b4f6-original-pull-secret\") pod \"global-pull-secret-syncer-bdgzk\" (UID: \"da8b58f9-2528-4906-b7d2-aadea159b4f6\") " pod="kube-system/global-pull-secret-syncer-bdgzk"
Jul 28 12:54:13.089055 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:13.088565    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a418b22c-fa2d-4d69-8a21-7e2209a892de-metrics-certs\") pod \"network-metrics-daemon-ps58v\" (UID: \"a418b22c-fa2d-4d69-8a21-7e2209a892de\") " pod="openshift-multus/network-metrics-daemon-ps58v"
Jul 28 12:54:13.091652 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:13.091603    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 12:54:13.091782 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:13.091734    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 12:54:13.098928 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:54:13.098911    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 28 12:54:13.099020 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:54:13.098958    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a418b22c-fa2d-4d69-8a21-7e2209a892de-metrics-certs podName:a418b22c-fa2d-4d69-8a21-7e2209a892de nodeName:}" failed. No retries permitted until 2026-07-28 12:55:17.098944189 +0000 UTC m=+129.304330768 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a418b22c-fa2d-4d69-8a21-7e2209a892de-metrics-certs") pod "network-metrics-daemon-ps58v" (UID: "a418b22c-fa2d-4d69-8a21-7e2209a892de") : secret "metrics-daemon-secret" not found
Jul 28 12:54:13.102277 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:13.102260    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da8b58f9-2528-4906-b7d2-aadea159b4f6-original-pull-secret\") pod \"global-pull-secret-syncer-bdgzk\" (UID: \"da8b58f9-2528-4906-b7d2-aadea159b4f6\") " pod="kube-system/global-pull-secret-syncer-bdgzk"
Jul 28 12:54:13.108234 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:13.108215    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdgzk"
Jul 28 12:54:13.189105 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:13.189078    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9krd\" (UniqueName: \"kubernetes.io/projected/4807337c-67b7-4b80-bf38-58ecdd8d972c-kube-api-access-n9krd\") pod \"network-check-target-wk6vn\" (UID: \"4807337c-67b7-4b80-bf38-58ecdd8d972c\") " pod="openshift-network-diagnostics/network-check-target-wk6vn"
Jul 28 12:54:13.192300 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:13.192282    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 12:54:13.201924 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:13.201907    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 12:54:13.212622 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:13.212603    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9krd\" (UniqueName: \"kubernetes.io/projected/4807337c-67b7-4b80-bf38-58ecdd8d972c-kube-api-access-n9krd\") pod \"network-check-target-wk6vn\" (UID: \"4807337c-67b7-4b80-bf38-58ecdd8d972c\") " pod="openshift-network-diagnostics/network-check-target-wk6vn"
Jul 28 12:54:13.221054 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:13.220996    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bdgzk"]
Jul 28 12:54:13.223233 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:54:13.223212    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda8b58f9_2528_4906_b7d2_aadea159b4f6.slice/crio-8bf4ce07a459bbf3c2d685e49e1ab6d13c0b633a03346850f9c9a768586b32f5 WatchSource:0}: Error finding container 8bf4ce07a459bbf3c2d685e49e1ab6d13c0b633a03346850f9c9a768586b32f5: Status 404 returned error can't find the container with id 8bf4ce07a459bbf3c2d685e49e1ab6d13c0b633a03346850f9c9a768586b32f5
Jul 28 12:54:13.316171 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:13.316150    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sxfnt\""
Jul 28 12:54:13.324302 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:13.324287    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wk6vn"
Jul 28 12:54:13.435363 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:13.435335    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wk6vn"]
Jul 28 12:54:13.437879 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:54:13.437855    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4807337c_67b7_4b80_bf38_58ecdd8d972c.slice/crio-dedfb2b06d1857108473ba4f30b73c0ab0cf35029f47b485e3cb8902d2ef4093 WatchSource:0}: Error finding container dedfb2b06d1857108473ba4f30b73c0ab0cf35029f47b485e3cb8902d2ef4093: Status 404 returned error can't find the container with id dedfb2b06d1857108473ba4f30b73c0ab0cf35029f47b485e3cb8902d2ef4093
Jul 28 12:54:13.575060 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:13.575033    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wk6vn" event={"ID":"4807337c-67b7-4b80-bf38-58ecdd8d972c","Type":"ContainerStarted","Data":"dedfb2b06d1857108473ba4f30b73c0ab0cf35029f47b485e3cb8902d2ef4093"}
Jul 28 12:54:13.576007 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:13.575987    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bdgzk" event={"ID":"da8b58f9-2528-4906-b7d2-aadea159b4f6","Type":"ContainerStarted","Data":"8bf4ce07a459bbf3c2d685e49e1ab6d13c0b633a03346850f9c9a768586b32f5"}
Jul 28 12:54:14.499922 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:14.499791    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8bc5bca-7c04-47ec-b3dc-f6ce86345325-metrics-tls\") pod \"dns-default-r8l5q\" (UID: \"f8bc5bca-7c04-47ec-b3dc-f6ce86345325\") " pod="openshift-dns/dns-default-r8l5q"
Jul 28 12:54:14.499922 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:14.499854    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d412255-c984-4473-af61-1271b6f141e5-registry-tls\") pod \"image-registry-787757c757-c4mh9\" (UID: \"6d412255-c984-4473-af61-1271b6f141e5\") " pod="openshift-image-registry/image-registry-787757c757-c4mh9"
Jul 28 12:54:14.500523 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:54:14.499958    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 12:54:14.500523 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:54:14.500029    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8bc5bca-7c04-47ec-b3dc-f6ce86345325-metrics-tls podName:f8bc5bca-7c04-47ec-b3dc-f6ce86345325 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:46.500009895 +0000 UTC m=+98.705396488 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f8bc5bca-7c04-47ec-b3dc-f6ce86345325-metrics-tls") pod "dns-default-r8l5q" (UID: "f8bc5bca-7c04-47ec-b3dc-f6ce86345325") : secret "dns-default-metrics-tls" not found
Jul 28 12:54:14.500523 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:54:14.499961    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 12:54:14.500523 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:54:14.500060    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-787757c757-c4mh9: secret "image-registry-tls" not found
Jul 28 12:54:14.500523 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:54:14.500101    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d412255-c984-4473-af61-1271b6f141e5-registry-tls podName:6d412255-c984-4473-af61-1271b6f141e5 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:46.500088797 +0000 UTC m=+98.705475381 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6d412255-c984-4473-af61-1271b6f141e5-registry-tls") pod "image-registry-787757c757-c4mh9" (UID: "6d412255-c984-4473-af61-1271b6f141e5") : secret "image-registry-tls" not found
Jul 28 12:54:18.591196 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:18.591158    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wk6vn" event={"ID":"4807337c-67b7-4b80-bf38-58ecdd8d972c","Type":"ContainerStarted","Data":"716485a62d566e0bf8e3ec2882435ff36bd37d81d3761665b6c5faaf4438be81"}
Jul 28 12:54:18.591651 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:18.591279    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wk6vn"
Jul 28 12:54:18.592655 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:18.592633    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bdgzk" event={"ID":"da8b58f9-2528-4906-b7d2-aadea159b4f6","Type":"ContainerStarted","Data":"554c84454262be6d3f4847a7eb64f6a89dcab1555b62ec8d60f14a394f150b2c"}
Jul 28 12:54:18.608347 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:18.608312    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-wk6vn" podStartSLOduration=66.381736735 podStartE2EDuration="1m10.608300128s" totalImagesPullingTime="4.226563393s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:53:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:54:18.608104177 +0000 UTC m=+70.813490779" watchObservedRunningTime="2026-07-28 12:54:18.608300128 +0000 UTC m=+70.813686730"
Jul 28 12:54:18.623914 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:18.623876    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-bdgzk" podStartSLOduration=66.17991138 podStartE2EDuration="1m10.623864408s" totalImagesPullingTime="4.443953028s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:53:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:54:18.62307031 +0000 UTC m=+70.828456912" watchObservedRunningTime="2026-07-28 12:54:18.623864408 +0000 UTC m=+70.829251009"
Jul 28 12:54:39.021531 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:39.021401    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-x84d8"]
Jul 28 12:54:39.025950 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:39.025930    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-v6fhs"]
Jul 28 12:54:39.026073 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:39.026042    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-x84d8"
Jul 28 12:54:39.030118 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:39.030090    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 28 12:54:39.030279 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:39.030207    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 28 12:54:39.030526 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:39.030504    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 28 12:54:39.030624 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:39.030580    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 28 12:54:39.031148 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:39.031131    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-m4p54\""
Jul 28 12:54:39.032182 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:39.032165    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-v6fhs"
Jul 28 12:54:39.035053 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:39.035033    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 28 12:54:39.035140 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:39.035039    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-ljlwb\""
Jul 28 12:54:39.035480 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:39.035465    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 28 12:54:39.039608 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:39.039589    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-v6fhs"]
Jul 28 12:54:39.040654 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:39.040635    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-x84d8"]
Jul 28 12:54:39.163489 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:39.163467    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e686c7fc-a026-4f10-b033-38955eecea40-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-v6fhs\" (UID: \"e686c7fc-a026-4f10-b033-38955eecea40\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-v6fhs"
Jul 28 12:54:39.163489 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:39.163493    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d7908f8d-42b7-4a20-b5f6-c2bbf80af85d-config\") pod \"service-ca-operator-5d6c75f84f-x84d8\" (UID: \"d7908f8d-42b7-4a20-b5f6-c2bbf80af85d\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-x84d8"
Jul 28 12:54:39.163658 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:39.163530    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e686c7fc-a026-4f10-b033-38955eecea40-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-v6fhs\" (UID: \"e686c7fc-a026-4f10-b033-38955eecea40\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-v6fhs"
Jul 28 12:54:39.163658 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:39.163601    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r99f8\" (UniqueName: \"kubernetes.io/projected/d7908f8d-42b7-4a20-b5f6-c2bbf80af85d-kube-api-access-r99f8\") pod \"service-ca-operator-5d6c75f84f-x84d8\" (UID: \"d7908f8d-42b7-4a20-b5f6-c2bbf80af85d\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-x84d8"
Jul 28 12:54:39.163779 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:39.163663    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d7908f8d-42b7-4a20-b5f6-c2bbf80af85d-serving-cert\") pod \"service-ca-operator-5d6c75f84f-x84d8\" (UID: \"d7908f8d-42b7-4a20-b5f6-c2bbf80af85d\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-x84d8"
Jul 28 12:54:39.264861 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:39.264834    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d7908f8d-42b7-4a20-b5f6-c2bbf80af85d-serving-cert\") pod \"service-ca-operator-5d6c75f84f-x84d8\" (UID: \"d7908f8d-42b7-4a20-b5f6-c2bbf80af85d\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-x84d8"
Jul 28 12:54:39.264980 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:39.264880    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e686c7fc-a026-4f10-b033-38955eecea40-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-v6fhs\" (UID: \"e686c7fc-a026-4f10-b033-38955eecea40\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-v6fhs"
Jul 28 12:54:39.264980 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:39.264903    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d7908f8d-42b7-4a20-b5f6-c2bbf80af85d-config\") pod \"service-ca-operator-5d6c75f84f-x84d8\" (UID: \"d7908f8d-42b7-4a20-b5f6-c2bbf80af85d\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-x84d8"
Jul 28 12:54:39.264980 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:39.264950    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e686c7fc-a026-4f10-b033-38955eecea40-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-v6fhs\" (UID: \"e686c7fc-a026-4f10-b033-38955eecea40\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-v6fhs"
Jul 28 12:54:39.265136 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:39.264996    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r99f8\" (UniqueName: \"kubernetes.io/projected/d7908f8d-42b7-4a20-b5f6-c2bbf80af85d-kube-api-access-r99f8\") pod \"service-ca-operator-5d6c75f84f-x84d8\" (UID: \"d7908f8d-42b7-4a20-b5f6-c2bbf80af85d\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-x84d8"
Jul 28 12:54:39.265136 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:54:39.265077    2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 12:54:39.265230 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:54:39.265179    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e686c7fc-a026-4f10-b033-38955eecea40-networking-console-plugin-cert podName:e686c7fc-a026-4f10-b033-38955eecea40 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:39.765157746 +0000 UTC m=+91.970544335 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e686c7fc-a026-4f10-b033-38955eecea40-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-v6fhs" (UID: "e686c7fc-a026-4f10-b033-38955eecea40") : secret "networking-console-plugin-cert" not found
Jul 28 12:54:39.265443 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:39.265426    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d7908f8d-42b7-4a20-b5f6-c2bbf80af85d-config\") pod \"service-ca-operator-5d6c75f84f-x84d8\" (UID: \"d7908f8d-42b7-4a20-b5f6-c2bbf80af85d\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-x84d8"
Jul 28 12:54:39.265544 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:39.265527    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e686c7fc-a026-4f10-b033-38955eecea40-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-v6fhs\" (UID: \"e686c7fc-a026-4f10-b033-38955eecea40\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-v6fhs"
Jul 28 12:54:39.266932 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:39.266913    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d7908f8d-42b7-4a20-b5f6-c2bbf80af85d-serving-cert\") pod \"service-ca-operator-5d6c75f84f-x84d8\" (UID: \"d7908f8d-42b7-4a20-b5f6-c2bbf80af85d\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-x84d8"
Jul 28 12:54:39.273055 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:39.273010    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r99f8\" (UniqueName: \"kubernetes.io/projected/d7908f8d-42b7-4a20-b5f6-c2bbf80af85d-kube-api-access-r99f8\") pod \"service-ca-operator-5d6c75f84f-x84d8\" (UID: \"d7908f8d-42b7-4a20-b5f6-c2bbf80af85d\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-x84d8"
Jul 28 12:54:39.337503 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:39.337476    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-x84d8"
Jul 28 12:54:39.454404 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:39.454379    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-x84d8"]
Jul 28 12:54:39.457449 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:54:39.457424    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7908f8d_42b7_4a20_b5f6_c2bbf80af85d.slice/crio-14af360928832619386295aa24737d9954f2c482c650c39d89c6265f30c394d5 WatchSource:0}: Error finding container 14af360928832619386295aa24737d9954f2c482c650c39d89c6265f30c394d5: Status 404 returned error can't find the container with id 14af360928832619386295aa24737d9954f2c482c650c39d89c6265f30c394d5
Jul 28 12:54:39.640099 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:39.640036    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-x84d8" event={"ID":"d7908f8d-42b7-4a20-b5f6-c2bbf80af85d","Type":"ContainerStarted","Data":"14af360928832619386295aa24737d9954f2c482c650c39d89c6265f30c394d5"}
Jul 28 12:54:39.769146 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:39.769114    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e686c7fc-a026-4f10-b033-38955eecea40-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-v6fhs\" (UID: \"e686c7fc-a026-4f10-b033-38955eecea40\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-v6fhs"
Jul 28 12:54:39.769242 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:54:39.769235    2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 12:54:39.769296 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:54:39.769286    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e686c7fc-a026-4f10-b033-38955eecea40-networking-console-plugin-cert podName:e686c7fc-a026-4f10-b033-38955eecea40 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:40.769269058 +0000 UTC m=+92.974655647 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e686c7fc-a026-4f10-b033-38955eecea40-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-v6fhs" (UID: "e686c7fc-a026-4f10-b033-38955eecea40") : secret "networking-console-plugin-cert" not found
Jul 28 12:54:40.777933 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:40.777895    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e686c7fc-a026-4f10-b033-38955eecea40-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-v6fhs\" (UID: \"e686c7fc-a026-4f10-b033-38955eecea40\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-v6fhs"
Jul 28 12:54:40.778378 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:54:40.778036    2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 12:54:40.778378 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:54:40.778094    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e686c7fc-a026-4f10-b033-38955eecea40-networking-console-plugin-cert podName:e686c7fc-a026-4f10-b033-38955eecea40 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:42.778081173 +0000 UTC m=+94.983467755 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e686c7fc-a026-4f10-b033-38955eecea40-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-v6fhs" (UID: "e686c7fc-a026-4f10-b033-38955eecea40") : secret "networking-console-plugin-cert" not found
Jul 28 12:54:41.646076 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:41.646000    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-x84d8" event={"ID":"d7908f8d-42b7-4a20-b5f6-c2bbf80af85d","Type":"ContainerStarted","Data":"f04b9e751156e822e2469bd9ef202a1ab0e611b65ec0bbcebed58accc0727a90"}
Jul 28 12:54:41.664699 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:41.664638    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-x84d8" podStartSLOduration=0.895693525 podStartE2EDuration="2.664618783s" totalImagesPullingTime="1.768925258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:54:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:54:41.663302068 +0000 UTC m=+93.868688660" watchObservedRunningTime="2026-07-28 12:54:41.664618783 +0000 UTC m=+93.870005385"
Jul 28 12:54:42.367166 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:42.367133    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-27vf2"]
Jul 28 12:54:42.370187 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:42.370167    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-27vf2"
Jul 28 12:54:42.380115 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:42.380091    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-dl59l\""
Jul 28 12:54:42.380463 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:42.380448    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 12:54:42.380539 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:42.380525    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 12:54:42.380576 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:42.380534    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 12:54:42.381144 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:42.381131    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 12:54:42.391177 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:42.391157    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-27vf2"]
Jul 28 12:54:42.490354 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:42.490324    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f80dfaf5-fb64-4f4e-ab75-e1164768f586-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-27vf2\" (UID: \"f80dfaf5-fb64-4f4e-ab75-e1164768f586\") " pod="openshift-insights/insights-runtime-extractor-27vf2"
Jul 28 12:54:42.490454 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:42.490365    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzkld\" (UniqueName: \"kubernetes.io/projected/f80dfaf5-fb64-4f4e-ab75-e1164768f586-kube-api-access-xzkld\") pod \"insights-runtime-extractor-27vf2\" (UID: \"f80dfaf5-fb64-4f4e-ab75-e1164768f586\") " pod="openshift-insights/insights-runtime-extractor-27vf2"
Jul 28 12:54:42.490454 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:42.490395    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f80dfaf5-fb64-4f4e-ab75-e1164768f586-data-volume\") pod \"insights-runtime-extractor-27vf2\" (UID: \"f80dfaf5-fb64-4f4e-ab75-e1164768f586\") " pod="openshift-insights/insights-runtime-extractor-27vf2"
Jul 28 12:54:42.490526 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:42.490455    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f80dfaf5-fb64-4f4e-ab75-e1164768f586-crio-socket\") pod \"insights-runtime-extractor-27vf2\" (UID: \"f80dfaf5-fb64-4f4e-ab75-e1164768f586\") " pod="openshift-insights/insights-runtime-extractor-27vf2"
Jul 28 12:54:42.490526 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:42.490486    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f80dfaf5-fb64-4f4e-ab75-e1164768f586-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-27vf2\" (UID: \"f80dfaf5-fb64-4f4e-ab75-e1164768f586\") " pod="openshift-insights/insights-runtime-extractor-27vf2"
Jul 28 12:54:42.591348 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:42.591320    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f80dfaf5-fb64-4f4e-ab75-e1164768f586-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-27vf2\" (UID: \"f80dfaf5-fb64-4f4e-ab75-e1164768f586\") " pod="openshift-insights/insights-runtime-extractor-27vf2"
Jul 28 12:54:42.591348 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:54:42.591340    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 12:54:42.591504 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:42.591366    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzkld\" (UniqueName: \"kubernetes.io/projected/f80dfaf5-fb64-4f4e-ab75-e1164768f586-kube-api-access-xzkld\") pod \"insights-runtime-extractor-27vf2\" (UID: \"f80dfaf5-fb64-4f4e-ab75-e1164768f586\") " pod="openshift-insights/insights-runtime-extractor-27vf2"
Jul 28 12:54:42.591504 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:42.591398    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f80dfaf5-fb64-4f4e-ab75-e1164768f586-data-volume\") pod \"insights-runtime-extractor-27vf2\" (UID: \"f80dfaf5-fb64-4f4e-ab75-e1164768f586\") " pod="openshift-insights/insights-runtime-extractor-27vf2"
Jul 28 12:54:42.591504 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:54:42.591416    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f80dfaf5-fb64-4f4e-ab75-e1164768f586-kube-rbac-proxy-cm podName:f80dfaf5-fb64-4f4e-ab75-e1164768f586 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:43.091397484 +0000 UTC m=+95.296784066 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f80dfaf5-fb64-4f4e-ab75-e1164768f586-kube-rbac-proxy-cm") pod "insights-runtime-extractor-27vf2" (UID: "f80dfaf5-fb64-4f4e-ab75-e1164768f586") : configmap "kube-rbac-proxy" not found
Jul 28 12:54:42.591504 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:42.591460    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f80dfaf5-fb64-4f4e-ab75-e1164768f586-crio-socket\") pod \"insights-runtime-extractor-27vf2\" (UID: \"f80dfaf5-fb64-4f4e-ab75-e1164768f586\") " pod="openshift-insights/insights-runtime-extractor-27vf2"
Jul 28 12:54:42.591504 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:42.591487    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f80dfaf5-fb64-4f4e-ab75-e1164768f586-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-27vf2\" (UID: \"f80dfaf5-fb64-4f4e-ab75-e1164768f586\") " pod="openshift-insights/insights-runtime-extractor-27vf2"
Jul 28 12:54:42.591776 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:42.591594    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f80dfaf5-fb64-4f4e-ab75-e1164768f586-crio-socket\") pod \"insights-runtime-extractor-27vf2\" (UID: \"f80dfaf5-fb64-4f4e-ab75-e1164768f586\") " pod="openshift-insights/insights-runtime-extractor-27vf2"
Jul 28 12:54:42.591776 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:54:42.591611    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 12:54:42.591776 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:42.591636    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f80dfaf5-fb64-4f4e-ab75-e1164768f586-data-volume\") pod \"insights-runtime-extractor-27vf2\" (UID: \"f80dfaf5-fb64-4f4e-ab75-e1164768f586\") " pod="openshift-insights/insights-runtime-extractor-27vf2"
Jul 28 12:54:42.591776 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:54:42.591673    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f80dfaf5-fb64-4f4e-ab75-e1164768f586-insights-runtime-extractor-tls podName:f80dfaf5-fb64-4f4e-ab75-e1164768f586 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:43.091656749 +0000 UTC m=+95.297043337 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f80dfaf5-fb64-4f4e-ab75-e1164768f586-insights-runtime-extractor-tls") pod "insights-runtime-extractor-27vf2" (UID: "f80dfaf5-fb64-4f4e-ab75-e1164768f586") : secret "insights-runtime-extractor-tls" not found
Jul 28 12:54:42.600471 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:42.600449    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzkld\" (UniqueName: \"kubernetes.io/projected/f80dfaf5-fb64-4f4e-ab75-e1164768f586-kube-api-access-xzkld\") pod \"insights-runtime-extractor-27vf2\" (UID: \"f80dfaf5-fb64-4f4e-ab75-e1164768f586\") " pod="openshift-insights/insights-runtime-extractor-27vf2"
Jul 28 12:54:42.792381 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:42.792352    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e686c7fc-a026-4f10-b033-38955eecea40-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-v6fhs\" (UID: \"e686c7fc-a026-4f10-b033-38955eecea40\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-v6fhs"
Jul 28 12:54:42.792512 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:54:42.792458    2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 12:54:42.792512 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:54:42.792503    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e686c7fc-a026-4f10-b033-38955eecea40-networking-console-plugin-cert podName:e686c7fc-a026-4f10-b033-38955eecea40 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:46.792493013 +0000 UTC m=+98.997879591 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e686c7fc-a026-4f10-b033-38955eecea40-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-v6fhs" (UID: "e686c7fc-a026-4f10-b033-38955eecea40") : secret "networking-console-plugin-cert" not found
Jul 28 12:54:43.052398 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:43.052341    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-mxcxh"]
Jul 28 12:54:43.055248 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:43.055231    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-mxcxh"
Jul 28 12:54:43.058724 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:43.058706    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 28 12:54:43.058817 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:43.058728    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-cfzjj\""
Jul 28 12:54:43.059063 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:43.059044    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 28 12:54:43.066774 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:43.066756    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-mxcxh"]
Jul 28 12:54:43.095037 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:43.095015    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f80dfaf5-fb64-4f4e-ab75-e1164768f586-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-27vf2\" (UID: \"f80dfaf5-fb64-4f4e-ab75-e1164768f586\") " pod="openshift-insights/insights-runtime-extractor-27vf2"
Jul 28 12:54:43.095120 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:43.095075    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f80dfaf5-fb64-4f4e-ab75-e1164768f586-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-27vf2\" (UID: \"f80dfaf5-fb64-4f4e-ab75-e1164768f586\") " pod="openshift-insights/insights-runtime-extractor-27vf2"
Jul 28 12:54:43.095159 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:54:43.095148    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 12:54:43.095193 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:54:43.095165    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 12:54:43.095224 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:54:43.095210    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f80dfaf5-fb64-4f4e-ab75-e1164768f586-kube-rbac-proxy-cm podName:f80dfaf5-fb64-4f4e-ab75-e1164768f586 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:44.095194894 +0000 UTC m=+96.300581476 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f80dfaf5-fb64-4f4e-ab75-e1164768f586-kube-rbac-proxy-cm") pod "insights-runtime-extractor-27vf2" (UID: "f80dfaf5-fb64-4f4e-ab75-e1164768f586") : configmap "kube-rbac-proxy" not found
Jul 28 12:54:43.095224 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:54:43.095223    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f80dfaf5-fb64-4f4e-ab75-e1164768f586-insights-runtime-extractor-tls podName:f80dfaf5-fb64-4f4e-ab75-e1164768f586 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:44.095217127 +0000 UTC m=+96.300603705 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f80dfaf5-fb64-4f4e-ab75-e1164768f586-insights-runtime-extractor-tls") pod "insights-runtime-extractor-27vf2" (UID: "f80dfaf5-fb64-4f4e-ab75-e1164768f586") : secret "insights-runtime-extractor-tls" not found
Jul 28 12:54:43.196269 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:43.196241    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r8vq\" (UniqueName: \"kubernetes.io/projected/53b9048b-bb55-4753-b5a3-96b9aae769d9-kube-api-access-6r8vq\") pod \"migrator-6865c9967-mxcxh\" (UID: \"53b9048b-bb55-4753-b5a3-96b9aae769d9\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-mxcxh"
Jul 28 12:54:43.296766 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:43.296734    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r8vq\" (UniqueName: \"kubernetes.io/projected/53b9048b-bb55-4753-b5a3-96b9aae769d9-kube-api-access-6r8vq\") pod \"migrator-6865c9967-mxcxh\" (UID: \"53b9048b-bb55-4753-b5a3-96b9aae769d9\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-mxcxh"
Jul 28 12:54:43.305118 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:43.305061    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r8vq\" (UniqueName: \"kubernetes.io/projected/53b9048b-bb55-4753-b5a3-96b9aae769d9-kube-api-access-6r8vq\") pod \"migrator-6865c9967-mxcxh\" (UID: \"53b9048b-bb55-4753-b5a3-96b9aae769d9\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-mxcxh"
Jul 28 12:54:43.363743 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:43.363721    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-mxcxh"
Jul 28 12:54:43.480646 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:43.480620    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-mxcxh"]
Jul 28 12:54:43.483552 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:54:43.483530    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53b9048b_bb55_4753_b5a3_96b9aae769d9.slice/crio-3d0eaed03a64dfafb1cdc2b812b3fef6d4a3e898bd6923bf828a8df3e39bb423 WatchSource:0}: Error finding container 3d0eaed03a64dfafb1cdc2b812b3fef6d4a3e898bd6923bf828a8df3e39bb423: Status 404 returned error can't find the container with id 3d0eaed03a64dfafb1cdc2b812b3fef6d4a3e898bd6923bf828a8df3e39bb423
Jul 28 12:54:43.651282 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:43.651225    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-mxcxh" event={"ID":"53b9048b-bb55-4753-b5a3-96b9aae769d9","Type":"ContainerStarted","Data":"3d0eaed03a64dfafb1cdc2b812b3fef6d4a3e898bd6923bf828a8df3e39bb423"}
Jul 28 12:54:44.102993 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:44.102952    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f80dfaf5-fb64-4f4e-ab75-e1164768f586-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-27vf2\" (UID: \"f80dfaf5-fb64-4f4e-ab75-e1164768f586\") " pod="openshift-insights/insights-runtime-extractor-27vf2"
Jul 28 12:54:44.103175 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:44.103068    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f80dfaf5-fb64-4f4e-ab75-e1164768f586-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-27vf2\" (UID: \"f80dfaf5-fb64-4f4e-ab75-e1164768f586\") " pod="openshift-insights/insights-runtime-extractor-27vf2"
Jul 28 12:54:44.103175 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:54:44.103100    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 12:54:44.103175 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:54:44.103162    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 12:54:44.103290 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:54:44.103180    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f80dfaf5-fb64-4f4e-ab75-e1164768f586-insights-runtime-extractor-tls podName:f80dfaf5-fb64-4f4e-ab75-e1164768f586 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:46.103156104 +0000 UTC m=+98.308542700 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f80dfaf5-fb64-4f4e-ab75-e1164768f586-insights-runtime-extractor-tls") pod "insights-runtime-extractor-27vf2" (UID: "f80dfaf5-fb64-4f4e-ab75-e1164768f586") : secret "insights-runtime-extractor-tls" not found
Jul 28 12:54:44.103290 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:54:44.103213    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f80dfaf5-fb64-4f4e-ab75-e1164768f586-kube-rbac-proxy-cm podName:f80dfaf5-fb64-4f4e-ab75-e1164768f586 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:46.103197236 +0000 UTC m=+98.308583819 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f80dfaf5-fb64-4f4e-ab75-e1164768f586-kube-rbac-proxy-cm") pod "insights-runtime-extractor-27vf2" (UID: "f80dfaf5-fb64-4f4e-ab75-e1164768f586") : configmap "kube-rbac-proxy" not found
Jul 28 12:54:44.164698 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:44.164659    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-66b94_a89e422d-76c2-4e62-b9fb-4283d5fa3de9/dns-node-resolver/0.log"
Jul 28 12:54:44.656880 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:44.656854    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-mxcxh" event={"ID":"53b9048b-bb55-4753-b5a3-96b9aae769d9","Type":"ContainerStarted","Data":"c5259104ecc1e8419447fa76c16e7e134a341fd3a11c509de1aa0f8bb568ebc1"}
Jul 28 12:54:44.657250 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:44.656901    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-mxcxh" event={"ID":"53b9048b-bb55-4753-b5a3-96b9aae769d9","Type":"ContainerStarted","Data":"f157cd0d385e009af48d4e92962250e2d68961a9019d2f0e8ad92a52af6361d9"}
Jul 28 12:54:44.674396 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:44.674352    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-mxcxh" podStartSLOduration=0.681428527 podStartE2EDuration="1.674338586s" totalImagesPullingTime="992.910059ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:54:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:54:44.673440864 +0000 UTC m=+96.878827467" watchObservedRunningTime="2026-07-28 12:54:44.674338586 +0000 UTC m=+96.879725187"
Jul 28 12:54:45.563877 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:45.563853    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7n245_46ca2ea9-2097-4729-8b59-52882a7ea6bf/node-ca/0.log"
Jul 28 12:54:46.117199 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:46.117164    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f80dfaf5-fb64-4f4e-ab75-e1164768f586-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-27vf2\" (UID: \"f80dfaf5-fb64-4f4e-ab75-e1164768f586\") " pod="openshift-insights/insights-runtime-extractor-27vf2"
Jul 28 12:54:46.117581 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:46.117232    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f80dfaf5-fb64-4f4e-ab75-e1164768f586-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-27vf2\" (UID: \"f80dfaf5-fb64-4f4e-ab75-e1164768f586\") " pod="openshift-insights/insights-runtime-extractor-27vf2"
Jul 28 12:54:46.117581 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:54:46.117289    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 12:54:46.117581 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:54:46.117291    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 12:54:46.117581 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:54:46.117337    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f80dfaf5-fb64-4f4e-ab75-e1164768f586-kube-rbac-proxy-cm podName:f80dfaf5-fb64-4f4e-ab75-e1164768f586 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:50.117324732 +0000 UTC m=+102.322711314 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f80dfaf5-fb64-4f4e-ab75-e1164768f586-kube-rbac-proxy-cm") pod "insights-runtime-extractor-27vf2" (UID: "f80dfaf5-fb64-4f4e-ab75-e1164768f586") : configmap "kube-rbac-proxy" not found
Jul 28 12:54:46.117581 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:54:46.117349    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f80dfaf5-fb64-4f4e-ab75-e1164768f586-insights-runtime-extractor-tls podName:f80dfaf5-fb64-4f4e-ab75-e1164768f586 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:50.117343657 +0000 UTC m=+102.322730236 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f80dfaf5-fb64-4f4e-ab75-e1164768f586-insights-runtime-extractor-tls") pod "insights-runtime-extractor-27vf2" (UID: "f80dfaf5-fb64-4f4e-ab75-e1164768f586") : secret "insights-runtime-extractor-tls" not found
Jul 28 12:54:46.365348 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:46.365323    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-b9htm_8b530194-44d2-4993-9d2e-5aae3188a18d/serve-healthcheck-canary/0.log"
Jul 28 12:54:46.519342 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:46.519310    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8bc5bca-7c04-47ec-b3dc-f6ce86345325-metrics-tls\") pod \"dns-default-r8l5q\" (UID: \"f8bc5bca-7c04-47ec-b3dc-f6ce86345325\") " pod="openshift-dns/dns-default-r8l5q"
Jul 28 12:54:46.519473 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:46.519370    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d412255-c984-4473-af61-1271b6f141e5-registry-tls\") pod \"image-registry-787757c757-c4mh9\" (UID: \"6d412255-c984-4473-af61-1271b6f141e5\") " pod="openshift-image-registry/image-registry-787757c757-c4mh9"
Jul 28 12:54:46.519473 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:54:46.519452    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 12:54:46.519567 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:54:46.519510    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8bc5bca-7c04-47ec-b3dc-f6ce86345325-metrics-tls podName:f8bc5bca-7c04-47ec-b3dc-f6ce86345325 nodeName:}" failed. No retries permitted until 2026-07-28 12:55:50.519496513 +0000 UTC m=+162.724883092 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f8bc5bca-7c04-47ec-b3dc-f6ce86345325-metrics-tls") pod "dns-default-r8l5q" (UID: "f8bc5bca-7c04-47ec-b3dc-f6ce86345325") : secret "dns-default-metrics-tls" not found
Jul 28 12:54:46.519567 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:54:46.519457    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 12:54:46.519567 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:54:46.519549    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-787757c757-c4mh9: secret "image-registry-tls" not found
Jul 28 12:54:46.519669 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:54:46.519589    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d412255-c984-4473-af61-1271b6f141e5-registry-tls podName:6d412255-c984-4473-af61-1271b6f141e5 nodeName:}" failed. No retries permitted until 2026-07-28 12:55:50.519577782 +0000 UTC m=+162.724964360 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6d412255-c984-4473-af61-1271b6f141e5-registry-tls") pod "image-registry-787757c757-c4mh9" (UID: "6d412255-c984-4473-af61-1271b6f141e5") : secret "image-registry-tls" not found
Jul 28 12:54:46.820798 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:46.820737    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e686c7fc-a026-4f10-b033-38955eecea40-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-v6fhs\" (UID: \"e686c7fc-a026-4f10-b033-38955eecea40\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-v6fhs"
Jul 28 12:54:46.820907 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:54:46.820849    2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 12:54:46.820907 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:54:46.820894    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e686c7fc-a026-4f10-b033-38955eecea40-networking-console-plugin-cert podName:e686c7fc-a026-4f10-b033-38955eecea40 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:54.820883133 +0000 UTC m=+107.026269713 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e686c7fc-a026-4f10-b033-38955eecea40-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-v6fhs" (UID: "e686c7fc-a026-4f10-b033-38955eecea40") : secret "networking-console-plugin-cert" not found
Jul 28 12:54:49.597426 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:49.597395    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-wk6vn"
Jul 28 12:54:50.145213 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:50.145185    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f80dfaf5-fb64-4f4e-ab75-e1164768f586-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-27vf2\" (UID: \"f80dfaf5-fb64-4f4e-ab75-e1164768f586\") " pod="openshift-insights/insights-runtime-extractor-27vf2"
Jul 28 12:54:50.145453 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:50.145244    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f80dfaf5-fb64-4f4e-ab75-e1164768f586-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-27vf2\" (UID: \"f80dfaf5-fb64-4f4e-ab75-e1164768f586\") " pod="openshift-insights/insights-runtime-extractor-27vf2"
Jul 28 12:54:50.145453 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:54:50.145315    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 12:54:50.145453 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:54:50.145329    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 12:54:50.145453 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:54:50.145381    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f80dfaf5-fb64-4f4e-ab75-e1164768f586-kube-rbac-proxy-cm podName:f80dfaf5-fb64-4f4e-ab75-e1164768f586 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:58.145364287 +0000 UTC m=+110.350750866 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f80dfaf5-fb64-4f4e-ab75-e1164768f586-kube-rbac-proxy-cm") pod "insights-runtime-extractor-27vf2" (UID: "f80dfaf5-fb64-4f4e-ab75-e1164768f586") : configmap "kube-rbac-proxy" not found
Jul 28 12:54:50.145453 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:54:50.145395    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f80dfaf5-fb64-4f4e-ab75-e1164768f586-insights-runtime-extractor-tls podName:f80dfaf5-fb64-4f4e-ab75-e1164768f586 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:58.145389245 +0000 UTC m=+110.350775824 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f80dfaf5-fb64-4f4e-ab75-e1164768f586-insights-runtime-extractor-tls") pod "insights-runtime-extractor-27vf2" (UID: "f80dfaf5-fb64-4f4e-ab75-e1164768f586") : secret "insights-runtime-extractor-tls" not found
Jul 28 12:54:54.879397 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:54.879361    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e686c7fc-a026-4f10-b033-38955eecea40-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-v6fhs\" (UID: \"e686c7fc-a026-4f10-b033-38955eecea40\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-v6fhs"
Jul 28 12:54:54.879783 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:54:54.879503    2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 12:54:54.879783 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:54:54.879600    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e686c7fc-a026-4f10-b033-38955eecea40-networking-console-plugin-cert podName:e686c7fc-a026-4f10-b033-38955eecea40 nodeName:}" failed. No retries permitted until 2026-07-28 12:55:10.879584976 +0000 UTC m=+123.084971555 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e686c7fc-a026-4f10-b033-38955eecea40-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-v6fhs" (UID: "e686c7fc-a026-4f10-b033-38955eecea40") : secret "networking-console-plugin-cert" not found
Jul 28 12:54:58.202922 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:58.202886    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f80dfaf5-fb64-4f4e-ab75-e1164768f586-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-27vf2\" (UID: \"f80dfaf5-fb64-4f4e-ab75-e1164768f586\") " pod="openshift-insights/insights-runtime-extractor-27vf2"
Jul 28 12:54:58.203283 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:58.202961    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f80dfaf5-fb64-4f4e-ab75-e1164768f586-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-27vf2\" (UID: \"f80dfaf5-fb64-4f4e-ab75-e1164768f586\") " pod="openshift-insights/insights-runtime-extractor-27vf2"
Jul 28 12:54:58.203283 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:54:58.203037    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 12:54:58.203283 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:54:58.203106    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f80dfaf5-fb64-4f4e-ab75-e1164768f586-insights-runtime-extractor-tls podName:f80dfaf5-fb64-4f4e-ab75-e1164768f586 nodeName:}" failed. No retries permitted until 2026-07-28 12:55:14.203090821 +0000 UTC m=+126.408477400 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f80dfaf5-fb64-4f4e-ab75-e1164768f586-insights-runtime-extractor-tls") pod "insights-runtime-extractor-27vf2" (UID: "f80dfaf5-fb64-4f4e-ab75-e1164768f586") : secret "insights-runtime-extractor-tls" not found
Jul 28 12:54:58.203426 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:54:58.203408    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f80dfaf5-fb64-4f4e-ab75-e1164768f586-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-27vf2\" (UID: \"f80dfaf5-fb64-4f4e-ab75-e1164768f586\") " pod="openshift-insights/insights-runtime-extractor-27vf2"
Jul 28 12:55:07.425214 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:07.425183    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b6db54b97-q7qj7"]
Jul 28 12:55:07.428093 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:07.428072    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b6db54b97-q7qj7"
Jul 28 12:55:07.431962 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:07.431937    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 28 12:55:07.432074 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:07.431992    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 28 12:55:07.432074 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:07.432047    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 28 12:55:07.432268 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:07.432249    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 28 12:55:07.443999 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:07.443977    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b6db54b97-q7qj7"]
Jul 28 12:55:07.524870 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:07.524843    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-vj7z5"]
Jul 28 12:55:07.527572 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:07.527555    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d666b9745-dqx5g"]
Jul 28 12:55:07.527680 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:07.527666    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-vj7z5"
Jul 28 12:55:07.530210 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:07.530192    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d666b9745-dqx5g"
Jul 28 12:55:07.531275 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:07.531257    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 28 12:55:07.531382 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:07.531332    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 28 12:55:07.531470 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:07.531448    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-ft9jh\""
Jul 28 12:55:07.536119 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:07.536090    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 28 12:55:07.536435 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:07.536382    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 28 12:55:07.536435 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:07.536430    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-8pbqd\""
Jul 28 12:55:07.537561 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:07.537545    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 28 12:55:07.537561 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:07.537559    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 28 12:55:07.537734 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:07.537592    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 28 12:55:07.540757 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:07.540738    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-vj7z5"]
Jul 28 12:55:07.545036 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:07.545016    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d666b9745-dqx5g"]
Jul 28 12:55:07.565231 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:07.565212    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/5af7ab83-4d3d-483c-b7ec-e3c3ea7fe4a1-klusterlet-config\") pod \"klusterlet-addon-workmgr-6b6db54b97-q7qj7\" (UID: \"5af7ab83-4d3d-483c-b7ec-e3c3ea7fe4a1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b6db54b97-q7qj7"
Jul 28 12:55:07.565318 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:07.565238    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pnz5\" (UniqueName: \"kubernetes.io/projected/5af7ab83-4d3d-483c-b7ec-e3c3ea7fe4a1-kube-api-access-9pnz5\") pod \"klusterlet-addon-workmgr-6b6db54b97-q7qj7\" (UID: \"5af7ab83-4d3d-483c-b7ec-e3c3ea7fe4a1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b6db54b97-q7qj7"
Jul 28 12:55:07.565318 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:07.565262    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5af7ab83-4d3d-483c-b7ec-e3c3ea7fe4a1-tmp\") pod \"klusterlet-addon-workmgr-6b6db54b97-q7qj7\" (UID: \"5af7ab83-4d3d-483c-b7ec-e3c3ea7fe4a1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b6db54b97-q7qj7"
Jul 28 12:55:07.666281 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:07.666255    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9c66fb0e-db92-4d10-bd1e-6178c0f76cfa-console-serving-cert\") pod \"console-5d666b9745-dqx5g\" (UID: \"9c66fb0e-db92-4d10-bd1e-6178c0f76cfa\") " pod="openshift-console/console-5d666b9745-dqx5g"
Jul 28 12:55:07.666281 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:07.666280    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9c66fb0e-db92-4d10-bd1e-6178c0f76cfa-service-ca\") pod \"console-5d666b9745-dqx5g\" (UID: \"9c66fb0e-db92-4d10-bd1e-6178c0f76cfa\") " pod="openshift-console/console-5d666b9745-dqx5g"
Jul 28 12:55:07.666431 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:07.666312    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/5af7ab83-4d3d-483c-b7ec-e3c3ea7fe4a1-klusterlet-config\") pod \"klusterlet-addon-workmgr-6b6db54b97-q7qj7\" (UID: \"5af7ab83-4d3d-483c-b7ec-e3c3ea7fe4a1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b6db54b97-q7qj7"
Jul 28 12:55:07.666431 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:07.666330    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pnz5\" (UniqueName: \"kubernetes.io/projected/5af7ab83-4d3d-483c-b7ec-e3c3ea7fe4a1-kube-api-access-9pnz5\") pod \"klusterlet-addon-workmgr-6b6db54b97-q7qj7\" (UID: \"5af7ab83-4d3d-483c-b7ec-e3c3ea7fe4a1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b6db54b97-q7qj7"
Jul 28 12:55:07.666431 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:07.666381    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5af7ab83-4d3d-483c-b7ec-e3c3ea7fe4a1-tmp\") pod \"klusterlet-addon-workmgr-6b6db54b97-q7qj7\" (UID: \"5af7ab83-4d3d-483c-b7ec-e3c3ea7fe4a1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b6db54b97-q7qj7"
Jul 28 12:55:07.666431 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:07.666418    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9c66fb0e-db92-4d10-bd1e-6178c0f76cfa-console-oauth-config\") pod \"console-5d666b9745-dqx5g\" (UID: \"9c66fb0e-db92-4d10-bd1e-6178c0f76cfa\") " pod="openshift-console/console-5d666b9745-dqx5g"
Jul 28 12:55:07.666625 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:07.666435    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9c66fb0e-db92-4d10-bd1e-6178c0f76cfa-oauth-serving-cert\") pod \"console-5d666b9745-dqx5g\" (UID: \"9c66fb0e-db92-4d10-bd1e-6178c0f76cfa\") " pod="openshift-console/console-5d666b9745-dqx5g"
Jul 28 12:55:07.666625 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:07.666460    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9c66fb0e-db92-4d10-bd1e-6178c0f76cfa-console-config\") pod \"console-5d666b9745-dqx5g\" (UID: \"9c66fb0e-db92-4d10-bd1e-6178c0f76cfa\") " pod="openshift-console/console-5d666b9745-dqx5g"
Jul 28 12:55:07.666625 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:07.666488    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c76m\" (UniqueName: \"kubernetes.io/projected/9c66fb0e-db92-4d10-bd1e-6178c0f76cfa-kube-api-access-2c76m\") pod \"console-5d666b9745-dqx5g\" (UID: \"9c66fb0e-db92-4d10-bd1e-6178c0f76cfa\") " pod="openshift-console/console-5d666b9745-dqx5g"
Jul 28 12:55:07.666625 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:07.666582    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qgvv\" (UniqueName: \"kubernetes.io/projected/c626bd54-efbd-42bf-b1d5-0e761f15c5dd-kube-api-access-8qgvv\") pod \"downloads-7c97dc4555-vj7z5\" (UID: \"c626bd54-efbd-42bf-b1d5-0e761f15c5dd\") " pod="openshift-console/downloads-7c97dc4555-vj7z5"
Jul 28 12:55:07.666791 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:07.666712    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5af7ab83-4d3d-483c-b7ec-e3c3ea7fe4a1-tmp\") pod \"klusterlet-addon-workmgr-6b6db54b97-q7qj7\" (UID: \"5af7ab83-4d3d-483c-b7ec-e3c3ea7fe4a1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b6db54b97-q7qj7"
Jul 28 12:55:07.668740 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:07.668715    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/5af7ab83-4d3d-483c-b7ec-e3c3ea7fe4a1-klusterlet-config\") pod \"klusterlet-addon-workmgr-6b6db54b97-q7qj7\" (UID: \"5af7ab83-4d3d-483c-b7ec-e3c3ea7fe4a1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b6db54b97-q7qj7"
Jul 28 12:55:07.679322 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:07.679273    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pnz5\" (UniqueName: \"kubernetes.io/projected/5af7ab83-4d3d-483c-b7ec-e3c3ea7fe4a1-kube-api-access-9pnz5\") pod \"klusterlet-addon-workmgr-6b6db54b97-q7qj7\" (UID: \"5af7ab83-4d3d-483c-b7ec-e3c3ea7fe4a1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b6db54b97-q7qj7"
Jul 28 12:55:07.737228 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:07.737208    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b6db54b97-q7qj7"
Jul 28 12:55:07.767746 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:07.767724    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qgvv\" (UniqueName: \"kubernetes.io/projected/c626bd54-efbd-42bf-b1d5-0e761f15c5dd-kube-api-access-8qgvv\") pod \"downloads-7c97dc4555-vj7z5\" (UID: \"c626bd54-efbd-42bf-b1d5-0e761f15c5dd\") " pod="openshift-console/downloads-7c97dc4555-vj7z5"
Jul 28 12:55:07.767882 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:07.767858    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9c66fb0e-db92-4d10-bd1e-6178c0f76cfa-console-serving-cert\") pod \"console-5d666b9745-dqx5g\" (UID: \"9c66fb0e-db92-4d10-bd1e-6178c0f76cfa\") " pod="openshift-console/console-5d666b9745-dqx5g"
Jul 28 12:55:07.767962 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:07.767890    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9c66fb0e-db92-4d10-bd1e-6178c0f76cfa-service-ca\") pod \"console-5d666b9745-dqx5g\" (UID: \"9c66fb0e-db92-4d10-bd1e-6178c0f76cfa\") " pod="openshift-console/console-5d666b9745-dqx5g"
Jul 28 12:55:07.768017 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:07.767979    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9c66fb0e-db92-4d10-bd1e-6178c0f76cfa-console-oauth-config\") pod \"console-5d666b9745-dqx5g\" (UID: \"9c66fb0e-db92-4d10-bd1e-6178c0f76cfa\") " pod="openshift-console/console-5d666b9745-dqx5g"
Jul 28 12:55:07.768017 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:07.768003    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9c66fb0e-db92-4d10-bd1e-6178c0f76cfa-oauth-serving-cert\") pod \"console-5d666b9745-dqx5g\" (UID: \"9c66fb0e-db92-4d10-bd1e-6178c0f76cfa\") " pod="openshift-console/console-5d666b9745-dqx5g"
Jul 28 12:55:07.768103 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:07.768042    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9c66fb0e-db92-4d10-bd1e-6178c0f76cfa-console-config\") pod \"console-5d666b9745-dqx5g\" (UID: \"9c66fb0e-db92-4d10-bd1e-6178c0f76cfa\") " pod="openshift-console/console-5d666b9745-dqx5g"
Jul 28 12:55:07.768103 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:07.768086    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c76m\" (UniqueName: \"kubernetes.io/projected/9c66fb0e-db92-4d10-bd1e-6178c0f76cfa-kube-api-access-2c76m\") pod \"console-5d666b9745-dqx5g\" (UID: \"9c66fb0e-db92-4d10-bd1e-6178c0f76cfa\") " pod="openshift-console/console-5d666b9745-dqx5g"
Jul 28 12:55:07.768732 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:07.768707    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9c66fb0e-db92-4d10-bd1e-6178c0f76cfa-service-ca\") pod \"console-5d666b9745-dqx5g\" (UID: \"9c66fb0e-db92-4d10-bd1e-6178c0f76cfa\") " pod="openshift-console/console-5d666b9745-dqx5g"
Jul 28 12:55:07.768842 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:07.768790    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9c66fb0e-db92-4d10-bd1e-6178c0f76cfa-console-config\") pod \"console-5d666b9745-dqx5g\" (UID: \"9c66fb0e-db92-4d10-bd1e-6178c0f76cfa\") " pod="openshift-console/console-5d666b9745-dqx5g"
Jul 28 12:55:07.768842 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:07.768808    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9c66fb0e-db92-4d10-bd1e-6178c0f76cfa-oauth-serving-cert\") pod \"console-5d666b9745-dqx5g\" (UID: \"9c66fb0e-db92-4d10-bd1e-6178c0f76cfa\") " pod="openshift-console/console-5d666b9745-dqx5g"
Jul 28 12:55:07.770279 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:07.770255    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9c66fb0e-db92-4d10-bd1e-6178c0f76cfa-console-serving-cert\") pod \"console-5d666b9745-dqx5g\" (UID: \"9c66fb0e-db92-4d10-bd1e-6178c0f76cfa\") " pod="openshift-console/console-5d666b9745-dqx5g"
Jul 28 12:55:07.770374 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:07.770360    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9c66fb0e-db92-4d10-bd1e-6178c0f76cfa-console-oauth-config\") pod \"console-5d666b9745-dqx5g\" (UID: \"9c66fb0e-db92-4d10-bd1e-6178c0f76cfa\") " pod="openshift-console/console-5d666b9745-dqx5g"
Jul 28 12:55:07.778702 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:07.778666    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c76m\" (UniqueName: \"kubernetes.io/projected/9c66fb0e-db92-4d10-bd1e-6178c0f76cfa-kube-api-access-2c76m\") pod \"console-5d666b9745-dqx5g\" (UID: \"9c66fb0e-db92-4d10-bd1e-6178c0f76cfa\") " pod="openshift-console/console-5d666b9745-dqx5g"
Jul 28 12:55:07.779190 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:07.779171    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qgvv\" (UniqueName: \"kubernetes.io/projected/c626bd54-efbd-42bf-b1d5-0e761f15c5dd-kube-api-access-8qgvv\") pod \"downloads-7c97dc4555-vj7z5\" (UID: \"c626bd54-efbd-42bf-b1d5-0e761f15c5dd\") " pod="openshift-console/downloads-7c97dc4555-vj7z5"
Jul 28 12:55:07.837760 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:07.837735    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-vj7z5"
Jul 28 12:55:07.842868 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:07.842832    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d666b9745-dqx5g"
Jul 28 12:55:07.856296 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:07.856215    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b6db54b97-q7qj7"]
Jul 28 12:55:07.859152 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:55:07.859121    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5af7ab83_4d3d_483c_b7ec_e3c3ea7fe4a1.slice/crio-331cb189f250f6a4b4648188559467b9c78d7f8134dbfe40f17fc5540b37aafc WatchSource:0}: Error finding container 331cb189f250f6a4b4648188559467b9c78d7f8134dbfe40f17fc5540b37aafc: Status 404 returned error can't find the container with id 331cb189f250f6a4b4648188559467b9c78d7f8134dbfe40f17fc5540b37aafc
Jul 28 12:55:07.967009 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:07.966982    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d666b9745-dqx5g"]
Jul 28 12:55:07.969788 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:55:07.969765    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c66fb0e_db92_4d10_bd1e_6178c0f76cfa.slice/crio-725f5f9faa7701695788d245de1a9f4b78d9e74810c8bd21f189ab0577f73993 WatchSource:0}: Error finding container 725f5f9faa7701695788d245de1a9f4b78d9e74810c8bd21f189ab0577f73993: Status 404 returned error can't find the container with id 725f5f9faa7701695788d245de1a9f4b78d9e74810c8bd21f189ab0577f73993
Jul 28 12:55:07.979174 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:07.979155    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-vj7z5"]
Jul 28 12:55:07.981106 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:55:07.981087    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc626bd54_efbd_42bf_b1d5_0e761f15c5dd.slice/crio-64e185c72d39c791a99bb27f8a605860bda91fc3587f1b7447d8b7ba08e3f251 WatchSource:0}: Error finding container 64e185c72d39c791a99bb27f8a605860bda91fc3587f1b7447d8b7ba08e3f251: Status 404 returned error can't find the container with id 64e185c72d39c791a99bb27f8a605860bda91fc3587f1b7447d8b7ba08e3f251
Jul 28 12:55:08.729618 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:08.729572    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-vj7z5" event={"ID":"c626bd54-efbd-42bf-b1d5-0e761f15c5dd","Type":"ContainerStarted","Data":"64e185c72d39c791a99bb27f8a605860bda91fc3587f1b7447d8b7ba08e3f251"}
Jul 28 12:55:08.731076 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:08.731046    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d666b9745-dqx5g" event={"ID":"9c66fb0e-db92-4d10-bd1e-6178c0f76cfa","Type":"ContainerStarted","Data":"725f5f9faa7701695788d245de1a9f4b78d9e74810c8bd21f189ab0577f73993"}
Jul 28 12:55:08.733982 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:08.733953    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b6db54b97-q7qj7" event={"ID":"5af7ab83-4d3d-483c-b7ec-e3c3ea7fe4a1","Type":"ContainerStarted","Data":"331cb189f250f6a4b4648188559467b9c78d7f8134dbfe40f17fc5540b37aafc"}
Jul 28 12:55:10.895664 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:10.895411    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e686c7fc-a026-4f10-b033-38955eecea40-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-v6fhs\" (UID: \"e686c7fc-a026-4f10-b033-38955eecea40\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-v6fhs"
Jul 28 12:55:10.898645 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:10.898528    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e686c7fc-a026-4f10-b033-38955eecea40-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-v6fhs\" (UID: \"e686c7fc-a026-4f10-b033-38955eecea40\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-v6fhs"
Jul 28 12:55:11.146835 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:11.146561    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-ljlwb\""
Jul 28 12:55:11.154256 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:11.154229    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-v6fhs"
Jul 28 12:55:13.718761 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:13.718388    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-v6fhs"]
Jul 28 12:55:13.724611 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:55:13.724563    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode686c7fc_a026_4f10_b033_38955eecea40.slice/crio-edf01a1f3887acaee25368f885306c982750797443a8ac349b8e2e2a1339bca7 WatchSource:0}: Error finding container edf01a1f3887acaee25368f885306c982750797443a8ac349b8e2e2a1339bca7: Status 404 returned error can't find the container with id edf01a1f3887acaee25368f885306c982750797443a8ac349b8e2e2a1339bca7
Jul 28 12:55:13.757158 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:13.757123    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d666b9745-dqx5g" event={"ID":"9c66fb0e-db92-4d10-bd1e-6178c0f76cfa","Type":"ContainerStarted","Data":"c173fb2222e4c4f782335473141b0226cb22acb346aa68a61d5ef9d61506fcb1"}
Jul 28 12:55:13.758933 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:13.758899    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b6db54b97-q7qj7" event={"ID":"5af7ab83-4d3d-483c-b7ec-e3c3ea7fe4a1","Type":"ContainerStarted","Data":"59a8ad29fedeb36528c43592c2e9069ced2713d3d5e0c9f7804bb25bbafaa7ed"}
Jul 28 12:55:13.759059 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:13.759040    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b6db54b97-q7qj7"
Jul 28 12:55:13.760214 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:13.760188    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-v6fhs" event={"ID":"e686c7fc-a026-4f10-b033-38955eecea40","Type":"ContainerStarted","Data":"edf01a1f3887acaee25368f885306c982750797443a8ac349b8e2e2a1339bca7"}
Jul 28 12:55:13.760483 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:13.760462    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b6db54b97-q7qj7"
Jul 28 12:55:13.777155 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:13.777108    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5d666b9745-dqx5g" podStartSLOduration=1.163626384 podStartE2EDuration="6.777089705s" totalImagesPullingTime="5.613463321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:55:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:55:13.77685871 +0000 UTC m=+125.982245312" watchObservedRunningTime="2026-07-28 12:55:13.777089705 +0000 UTC m=+125.982476308"
Jul 28 12:55:13.796215 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:13.796174    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b6db54b97-q7qj7" podStartSLOduration=1.058917324 podStartE2EDuration="6.796158222s" totalImagesPullingTime="5.737240898s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:55:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:55:13.795113511 +0000 UTC m=+126.000500113" watchObservedRunningTime="2026-07-28 12:55:13.796158222 +0000 UTC m=+126.001544801"
Jul 28 12:55:14.226901 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:14.226867    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f80dfaf5-fb64-4f4e-ab75-e1164768f586-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-27vf2\" (UID: \"f80dfaf5-fb64-4f4e-ab75-e1164768f586\") " pod="openshift-insights/insights-runtime-extractor-27vf2"
Jul 28 12:55:14.229907 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:14.229879    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f80dfaf5-fb64-4f4e-ab75-e1164768f586-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-27vf2\" (UID: \"f80dfaf5-fb64-4f4e-ab75-e1164768f586\") " pod="openshift-insights/insights-runtime-extractor-27vf2"
Jul 28 12:55:14.481462 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:14.481393    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-dl59l\""
Jul 28 12:55:14.488729 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:14.488702    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-27vf2"
Jul 28 12:55:14.764988 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:14.764955    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-v6fhs" event={"ID":"e686c7fc-a026-4f10-b033-38955eecea40","Type":"ContainerStarted","Data":"f4f1977a3ee712e3e195fe315fb428c298e8abe01e9b384fd345d5c8396cd0d7"}
Jul 28 12:55:14.772753 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:14.772732    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-27vf2"]
Jul 28 12:55:14.776055 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:55:14.776028    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf80dfaf5_fb64_4f4e_ab75_e1164768f586.slice/crio-ee2570a2c783d7d4a84138dedc19626b6c5bf1d3b1dbe892fa66bb0d60dea16c WatchSource:0}: Error finding container ee2570a2c783d7d4a84138dedc19626b6c5bf1d3b1dbe892fa66bb0d60dea16c: Status 404 returned error can't find the container with id ee2570a2c783d7d4a84138dedc19626b6c5bf1d3b1dbe892fa66bb0d60dea16c
Jul 28 12:55:14.784809 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:14.784759    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-v6fhs" podStartSLOduration=34.82870037 podStartE2EDuration="35.78471889s" totalImagesPullingTime="956.01852ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:54:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:55:14.783652832 +0000 UTC m=+126.989039436" watchObservedRunningTime="2026-07-28 12:55:14.78471889 +0000 UTC m=+126.990105492"
Jul 28 12:55:15.770558 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:15.770519    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-27vf2" event={"ID":"f80dfaf5-fb64-4f4e-ab75-e1164768f586","Type":"ContainerStarted","Data":"ee2570a2c783d7d4a84138dedc19626b6c5bf1d3b1dbe892fa66bb0d60dea16c"}
Jul 28 12:55:15.841593 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:15.841565    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-pqmvv"]
Jul 28 12:55:15.844861 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:15.844839    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-pqmvv"
Jul 28 12:55:15.849656 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:15.849470    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 12:55:15.849656 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:15.849541    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 28 12:55:15.850210 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:15.850033    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 28 12:55:15.850829 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:15.850495    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 12:55:15.850829 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:15.850657    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 12:55:15.850829 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:15.850725    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-vq2qx\""
Jul 28 12:55:15.864406 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:15.864346    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-pqmvv"]
Jul 28 12:55:15.942369 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:15.942334    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/13c3f2a0-fa93-4793-808d-10ce77dafd69-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-pqmvv\" (UID: \"13c3f2a0-fa93-4793-808d-10ce77dafd69\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-pqmvv"
Jul 28 12:55:15.942502 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:15.942378    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbwrl\" (UniqueName: \"kubernetes.io/projected/13c3f2a0-fa93-4793-808d-10ce77dafd69-kube-api-access-rbwrl\") pod \"prometheus-operator-84d69f794d-pqmvv\" (UID: \"13c3f2a0-fa93-4793-808d-10ce77dafd69\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-pqmvv"
Jul 28 12:55:15.942502 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:15.942480    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/13c3f2a0-fa93-4793-808d-10ce77dafd69-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-pqmvv\" (UID: \"13c3f2a0-fa93-4793-808d-10ce77dafd69\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-pqmvv"
Jul 28 12:55:15.942609 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:15.942523    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/13c3f2a0-fa93-4793-808d-10ce77dafd69-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-pqmvv\" (UID: \"13c3f2a0-fa93-4793-808d-10ce77dafd69\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-pqmvv"
Jul 28 12:55:16.043378 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:16.043303    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/13c3f2a0-fa93-4793-808d-10ce77dafd69-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-pqmvv\" (UID: \"13c3f2a0-fa93-4793-808d-10ce77dafd69\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-pqmvv"
Jul 28 12:55:16.043378 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:16.043356    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/13c3f2a0-fa93-4793-808d-10ce77dafd69-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-pqmvv\" (UID: \"13c3f2a0-fa93-4793-808d-10ce77dafd69\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-pqmvv"
Jul 28 12:55:16.043560 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:55:16.043482    2574 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 28 12:55:16.043560 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:55:16.043535    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13c3f2a0-fa93-4793-808d-10ce77dafd69-prometheus-operator-tls podName:13c3f2a0-fa93-4793-808d-10ce77dafd69 nodeName:}" failed. No retries permitted until 2026-07-28 12:55:16.543517019 +0000 UTC m=+128.748903602 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/13c3f2a0-fa93-4793-808d-10ce77dafd69-prometheus-operator-tls") pod "prometheus-operator-84d69f794d-pqmvv" (UID: "13c3f2a0-fa93-4793-808d-10ce77dafd69") : secret "prometheus-operator-tls" not found
Jul 28 12:55:16.043670 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:16.043627    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/13c3f2a0-fa93-4793-808d-10ce77dafd69-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-pqmvv\" (UID: \"13c3f2a0-fa93-4793-808d-10ce77dafd69\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-pqmvv"
Jul 28 12:55:16.043789 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:16.043678    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbwrl\" (UniqueName: \"kubernetes.io/projected/13c3f2a0-fa93-4793-808d-10ce77dafd69-kube-api-access-rbwrl\") pod \"prometheus-operator-84d69f794d-pqmvv\" (UID: \"13c3f2a0-fa93-4793-808d-10ce77dafd69\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-pqmvv"
Jul 28 12:55:16.044375 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:16.044351    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/13c3f2a0-fa93-4793-808d-10ce77dafd69-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-pqmvv\" (UID: \"13c3f2a0-fa93-4793-808d-10ce77dafd69\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-pqmvv"
Jul 28 12:55:16.046358 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:16.046335    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/13c3f2a0-fa93-4793-808d-10ce77dafd69-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-pqmvv\" (UID: \"13c3f2a0-fa93-4793-808d-10ce77dafd69\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-pqmvv"
Jul 28 12:55:16.052884 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:16.052839    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbwrl\" (UniqueName: \"kubernetes.io/projected/13c3f2a0-fa93-4793-808d-10ce77dafd69-kube-api-access-rbwrl\") pod \"prometheus-operator-84d69f794d-pqmvv\" (UID: \"13c3f2a0-fa93-4793-808d-10ce77dafd69\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-pqmvv"
Jul 28 12:55:16.546802 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:16.546751    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/13c3f2a0-fa93-4793-808d-10ce77dafd69-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-pqmvv\" (UID: \"13c3f2a0-fa93-4793-808d-10ce77dafd69\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-pqmvv"
Jul 28 12:55:16.549914 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:16.549881    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/13c3f2a0-fa93-4793-808d-10ce77dafd69-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-pqmvv\" (UID: \"13c3f2a0-fa93-4793-808d-10ce77dafd69\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-pqmvv"
Jul 28 12:55:16.757548 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:16.757493    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-pqmvv"
Jul 28 12:55:16.938502 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:16.938445    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-pqmvv"]
Jul 28 12:55:16.942799 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:55:16.942772    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13c3f2a0_fa93_4793_808d_10ce77dafd69.slice/crio-d2de28c31357a3900097bf92b4d46cd40dbb9944952fd98d522202a5e4ec54ef WatchSource:0}: Error finding container d2de28c31357a3900097bf92b4d46cd40dbb9944952fd98d522202a5e4ec54ef: Status 404 returned error can't find the container with id d2de28c31357a3900097bf92b4d46cd40dbb9944952fd98d522202a5e4ec54ef
Jul 28 12:55:17.153074 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:17.152993    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a418b22c-fa2d-4d69-8a21-7e2209a892de-metrics-certs\") pod \"network-metrics-daemon-ps58v\" (UID: \"a418b22c-fa2d-4d69-8a21-7e2209a892de\") " pod="openshift-multus/network-metrics-daemon-ps58v"
Jul 28 12:55:17.155772 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:17.155747    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a418b22c-fa2d-4d69-8a21-7e2209a892de-metrics-certs\") pod \"network-metrics-daemon-ps58v\" (UID: \"a418b22c-fa2d-4d69-8a21-7e2209a892de\") " pod="openshift-multus/network-metrics-daemon-ps58v"
Jul 28 12:55:17.211460 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:17.211425    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wvgpn\""
Jul 28 12:55:17.219124 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:17.219102    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ps58v"
Jul 28 12:55:17.363286 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:17.363243    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ps58v"]
Jul 28 12:55:17.784368 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:17.784295    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-pqmvv" event={"ID":"13c3f2a0-fa93-4793-808d-10ce77dafd69","Type":"ContainerStarted","Data":"d2de28c31357a3900097bf92b4d46cd40dbb9944952fd98d522202a5e4ec54ef"}
Jul 28 12:55:17.843746 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:17.843713    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d666b9745-dqx5g"
Jul 28 12:55:17.843905 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:17.843780    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5d666b9745-dqx5g"
Jul 28 12:55:17.849651 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:17.849627    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5d666b9745-dqx5g"
Jul 28 12:55:18.793142 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:18.793083    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5d666b9745-dqx5g"
Jul 28 12:55:26.420178 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:55:26.420145    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda418b22c_fa2d_4d69_8a21_7e2209a892de.slice/crio-e1c2862e1b13648ea1b4637c8acdcf30c487fdf3e3cb138006b603587a6b91fb WatchSource:0}: Error finding container e1c2862e1b13648ea1b4637c8acdcf30c487fdf3e3cb138006b603587a6b91fb: Status 404 returned error can't find the container with id e1c2862e1b13648ea1b4637c8acdcf30c487fdf3e3cb138006b603587a6b91fb
Jul 28 12:55:26.818840 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:26.818800    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ps58v" event={"ID":"a418b22c-fa2d-4d69-8a21-7e2209a892de","Type":"ContainerStarted","Data":"e1c2862e1b13648ea1b4637c8acdcf30c487fdf3e3cb138006b603587a6b91fb"}
Jul 28 12:55:27.824309 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:27.824270    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-vj7z5" event={"ID":"c626bd54-efbd-42bf-b1d5-0e761f15c5dd","Type":"ContainerStarted","Data":"837eb72db8a10cd7f3efb672587a3a94cc080499146860e759134d9a007fa760"}
Jul 28 12:55:27.824807 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:27.824413    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-vj7z5"
Jul 28 12:55:27.825552 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:27.825515    2574 patch_prober.go:28] interesting pod/downloads-7c97dc4555-vj7z5 container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.132.0.14:8080/\": dial tcp 10.132.0.14:8080: connect: connection refused" start-of-body=
Jul 28 12:55:27.825670 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:27.825566    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-7c97dc4555-vj7z5" podUID="c626bd54-efbd-42bf-b1d5-0e761f15c5dd" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.14:8080/\": dial tcp 10.132.0.14:8080: connect: connection refused"
Jul 28 12:55:27.838507 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:27.838435    2574 patch_prober.go:28] interesting pod/downloads-7c97dc4555-vj7z5 container/download-server namespace/openshift-console: Liveness probe status=failure output="Get \"http://10.132.0.14:8080/\": dial tcp 10.132.0.14:8080: connect: connection refused" start-of-body=
Jul 28 12:55:27.838507 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:27.838482    2574 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-console/downloads-7c97dc4555-vj7z5" podUID="c626bd54-efbd-42bf-b1d5-0e761f15c5dd" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.14:8080/\": dial tcp 10.132.0.14:8080: connect: connection refused"
Jul 28 12:55:27.842767 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:27.842726    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-vj7z5" podStartSLOduration=1.237074903 podStartE2EDuration="20.842712973s" totalImagesPullingTime="19.60563807s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:55:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:55:27.842709196 +0000 UTC m=+140.048095796" watchObservedRunningTime="2026-07-28 12:55:27.842712973 +0000 UTC m=+140.048099606"
Jul 28 12:55:28.400787 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:28.400744    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d666b9745-dqx5g"]
Jul 28 12:55:28.830480 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:28.830393    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-27vf2" event={"ID":"f80dfaf5-fb64-4f4e-ab75-e1164768f586","Type":"ContainerStarted","Data":"e227583458fad3713547ee033df0ca1916771be249a89be31310782475c2a60a"}
Jul 28 12:55:28.833912 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:28.833086    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-pqmvv" event={"ID":"13c3f2a0-fa93-4793-808d-10ce77dafd69","Type":"ContainerStarted","Data":"0d5efa37d3770f6ce85796d921b9bdf8ec9c0ad2016b111506c002120d926556"}
Jul 28 12:55:28.833912 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:28.833141    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-pqmvv" event={"ID":"13c3f2a0-fa93-4793-808d-10ce77dafd69","Type":"ContainerStarted","Data":"ec0327822513ee13c0c972127c3fccff9432930a1d0b24e38a7a01c75c6c8825"}
Jul 28 12:55:28.836048 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:28.836015    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ps58v" event={"ID":"a418b22c-fa2d-4d69-8a21-7e2209a892de","Type":"ContainerStarted","Data":"7392a6b8e8511c4491f7b031a4acb3b7de573934cf4634a9de7ddfb00837e4cb"}
Jul 28 12:55:28.836162 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:28.836072    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ps58v" event={"ID":"a418b22c-fa2d-4d69-8a21-7e2209a892de","Type":"ContainerStarted","Data":"9172d95210a6fbc5454e8a428dafdf421a18c592ee069765cf75b16074077497"}
Jul 28 12:55:28.851466 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:28.851402    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-vj7z5"
Jul 28 12:55:28.852754 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:28.852705    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-pqmvv" podStartSLOduration=2.304167465 podStartE2EDuration="13.852673335s" totalImagesPullingTime="11.54850587s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:55:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:55:28.852451115 +0000 UTC m=+141.057837717" watchObservedRunningTime="2026-07-28 12:55:28.852673335 +0000 UTC m=+141.058059937"
Jul 28 12:55:28.873777 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:28.873732    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-ps58v" podStartSLOduration=138.7985499 podStartE2EDuration="2m20.873714715s" totalImagesPullingTime="2.075164815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:53:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:55:28.873715692 +0000 UTC m=+141.079102294" watchObservedRunningTime="2026-07-28 12:55:28.873714715 +0000 UTC m=+141.079101317"
Jul 28 12:55:29.316027 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:29.315672    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-787757c757-c4mh9"]
Jul 28 12:55:29.316027 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:55:29.315969    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-787757c757-c4mh9" podUID="6d412255-c984-4473-af61-1271b6f141e5"
Jul 28 12:55:29.841343 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:29.841312    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-787757c757-c4mh9"
Jul 28 12:55:29.867239 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:29.867181    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-787757c757-c4mh9"
Jul 28 12:55:29.964395 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:29.964361    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6d412255-c984-4473-af61-1271b6f141e5-image-registry-private-configuration\") pod \"6d412255-c984-4473-af61-1271b6f141e5\" (UID: \"6d412255-c984-4473-af61-1271b6f141e5\") "
Jul 28 12:55:29.964563 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:29.964449    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6d412255-c984-4473-af61-1271b6f141e5-ca-trust-extracted\") pod \"6d412255-c984-4473-af61-1271b6f141e5\" (UID: \"6d412255-c984-4473-af61-1271b6f141e5\") "
Jul 28 12:55:29.964563 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:29.964484    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6d412255-c984-4473-af61-1271b6f141e5-registry-certificates\") pod \"6d412255-c984-4473-af61-1271b6f141e5\" (UID: \"6d412255-c984-4473-af61-1271b6f141e5\") "
Jul 28 12:55:29.964563 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:29.964510    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6d412255-c984-4473-af61-1271b6f141e5-bound-sa-token\") pod \"6d412255-c984-4473-af61-1271b6f141e5\" (UID: \"6d412255-c984-4473-af61-1271b6f141e5\") "
Jul 28 12:55:29.964563 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:29.964542    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6d412255-c984-4473-af61-1271b6f141e5-installation-pull-secrets\") pod \"6d412255-c984-4473-af61-1271b6f141e5\" (UID: \"6d412255-c984-4473-af61-1271b6f141e5\") "
Jul 28 12:55:29.964792 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:29.964571    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6d412255-c984-4473-af61-1271b6f141e5-trusted-ca\") pod \"6d412255-c984-4473-af61-1271b6f141e5\" (UID: \"6d412255-c984-4473-af61-1271b6f141e5\") "
Jul 28 12:55:29.964792 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:29.964597    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r448l\" (UniqueName: \"kubernetes.io/projected/6d412255-c984-4473-af61-1271b6f141e5-kube-api-access-r448l\") pod \"6d412255-c984-4473-af61-1271b6f141e5\" (UID: \"6d412255-c984-4473-af61-1271b6f141e5\") "
Jul 28 12:55:29.966292 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:29.966264    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d412255-c984-4473-af61-1271b6f141e5-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "6d412255-c984-4473-af61-1271b6f141e5" (UID: "6d412255-c984-4473-af61-1271b6f141e5"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:55:29.966574 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:29.966469    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d412255-c984-4473-af61-1271b6f141e5-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "6d412255-c984-4473-af61-1271b6f141e5" (UID: "6d412255-c984-4473-af61-1271b6f141e5"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:55:29.966712 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:29.966671    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d412255-c984-4473-af61-1271b6f141e5-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "6d412255-c984-4473-af61-1271b6f141e5" (UID: "6d412255-c984-4473-af61-1271b6f141e5"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:55:29.971882 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:29.971819    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d412255-c984-4473-af61-1271b6f141e5-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "6d412255-c984-4473-af61-1271b6f141e5" (UID: "6d412255-c984-4473-af61-1271b6f141e5"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:55:29.971882 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:29.971827    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d412255-c984-4473-af61-1271b6f141e5-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "6d412255-c984-4473-af61-1271b6f141e5" (UID: "6d412255-c984-4473-af61-1271b6f141e5"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:55:29.972005 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:29.971980    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d412255-c984-4473-af61-1271b6f141e5-kube-api-access-r448l" (OuterVolumeSpecName: "kube-api-access-r448l") pod "6d412255-c984-4473-af61-1271b6f141e5" (UID: "6d412255-c984-4473-af61-1271b6f141e5"). InnerVolumeSpecName "kube-api-access-r448l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:55:29.972811 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:29.972775    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d412255-c984-4473-af61-1271b6f141e5-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "6d412255-c984-4473-af61-1271b6f141e5" (UID: "6d412255-c984-4473-af61-1271b6f141e5"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:55:30.066077 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:30.066043    2574 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6d412255-c984-4473-af61-1271b6f141e5-installation-pull-secrets\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 12:55:30.066077 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:30.066081    2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6d412255-c984-4473-af61-1271b6f141e5-trusted-ca\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 12:55:30.066294 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:30.066093    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r448l\" (UniqueName: \"kubernetes.io/projected/6d412255-c984-4473-af61-1271b6f141e5-kube-api-access-r448l\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 12:55:30.066294 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:30.066106    2574 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6d412255-c984-4473-af61-1271b6f141e5-image-registry-private-configuration\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 12:55:30.066294 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:30.066119    2574 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6d412255-c984-4473-af61-1271b6f141e5-ca-trust-extracted\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 12:55:30.066294 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:30.066131    2574 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6d412255-c984-4473-af61-1271b6f141e5-registry-certificates\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 12:55:30.066294 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:30.066142    2574 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6d412255-c984-4473-af61-1271b6f141e5-bound-sa-token\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 12:55:30.846008 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:30.845970    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-787757c757-c4mh9"
Jul 28 12:55:30.904474 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:30.901676    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-787757c757-c4mh9"]
Jul 28 12:55:30.914872 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:30.914841    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-787757c757-c4mh9"]
Jul 28 12:55:30.975887 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:30.975853    2574 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d412255-c984-4473-af61-1271b6f141e5-registry-tls\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 12:55:31.217505 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.217270    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-h47vh"]
Jul 28 12:55:31.271560 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.271516    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-h47vh"]
Jul 28 12:55:31.271560 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.271563    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-h585r"]
Jul 28 12:55:31.272617 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.272595    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-h47vh"
Jul 28 12:55:31.277667 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.277386    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 28 12:55:31.277667 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.277464    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 28 12:55:31.277876 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.277742    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-64gpz\""
Jul 28 12:55:31.292914 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.292891    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-h585r"]
Jul 28 12:55:31.293149 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.293111    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-h585r"
Jul 28 12:55:31.300995 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.300976    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 28 12:55:31.301105 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.301090    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 28 12:55:31.302505 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.302486    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-t8frk\""
Jul 28 12:55:31.302629 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.302533    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 28 12:55:31.378890 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.378856    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wxxg\" (UniqueName: \"kubernetes.io/projected/9fefb33c-bdac-47fb-9d68-bbc531bc37cc-kube-api-access-8wxxg\") pod \"openshift-state-metrics-b8f49f5c6-h47vh\" (UID: \"9fefb33c-bdac-47fb-9d68-bbc531bc37cc\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-h47vh"
Jul 28 12:55:31.379018 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.378904    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61f6e666-8451-4f07-bcc9-cbe18de924f1-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-h585r\" (UID: \"61f6e666-8451-4f07-bcc9-cbe18de924f1\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-h585r"
Jul 28 12:55:31.379018 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.378973    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9fefb33c-bdac-47fb-9d68-bbc531bc37cc-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-h47vh\" (UID: \"9fefb33c-bdac-47fb-9d68-bbc531bc37cc\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-h47vh"
Jul 28 12:55:31.379132 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.379019    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/61f6e666-8451-4f07-bcc9-cbe18de924f1-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-h585r\" (UID: \"61f6e666-8451-4f07-bcc9-cbe18de924f1\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-h585r"
Jul 28 12:55:31.379132 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.379077    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8dxf\" (UniqueName: \"kubernetes.io/projected/61f6e666-8451-4f07-bcc9-cbe18de924f1-kube-api-access-f8dxf\") pod \"kube-state-metrics-56d9f944b4-h585r\" (UID: \"61f6e666-8451-4f07-bcc9-cbe18de924f1\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-h585r"
Jul 28 12:55:31.379229 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.379127    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61f6e666-8451-4f07-bcc9-cbe18de924f1-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-h585r\" (UID: \"61f6e666-8451-4f07-bcc9-cbe18de924f1\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-h585r"
Jul 28 12:55:31.379229 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.379197    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9fefb33c-bdac-47fb-9d68-bbc531bc37cc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-h47vh\" (UID: \"9fefb33c-bdac-47fb-9d68-bbc531bc37cc\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-h47vh"
Jul 28 12:55:31.379311 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.379227    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/61f6e666-8451-4f07-bcc9-cbe18de924f1-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-h585r\" (UID: \"61f6e666-8451-4f07-bcc9-cbe18de924f1\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-h585r"
Jul 28 12:55:31.379358 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.379307    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9fefb33c-bdac-47fb-9d68-bbc531bc37cc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-h47vh\" (UID: \"9fefb33c-bdac-47fb-9d68-bbc531bc37cc\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-h47vh"
Jul 28 12:55:31.379412 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.379371    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/61f6e666-8451-4f07-bcc9-cbe18de924f1-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-h585r\" (UID: \"61f6e666-8451-4f07-bcc9-cbe18de924f1\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-h585r"
Jul 28 12:55:31.398477 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.398451    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-nlkwm"]
Jul 28 12:55:31.420997 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.420969    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nlkwm"
Jul 28 12:55:31.438622 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.438599    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-trl9r\""
Jul 28 12:55:31.439112 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.439089    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 12:55:31.439220 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.439121    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 12:55:31.475404 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.475346    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 12:55:31.480648 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.480627    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9fefb33c-bdac-47fb-9d68-bbc531bc37cc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-h47vh\" (UID: \"9fefb33c-bdac-47fb-9d68-bbc531bc37cc\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-h47vh"
Jul 28 12:55:31.480762 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.480654    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/61f6e666-8451-4f07-bcc9-cbe18de924f1-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-h585r\" (UID: \"61f6e666-8451-4f07-bcc9-cbe18de924f1\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-h585r"
Jul 28 12:55:31.480762 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.480727    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1099cd90-ebb3-4cc5-b316-17a0bb480a75-sys\") pod \"node-exporter-nlkwm\" (UID: \"1099cd90-ebb3-4cc5-b316-17a0bb480a75\") " pod="openshift-monitoring/node-exporter-nlkwm"
Jul 28 12:55:31.480762 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.480755    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1099cd90-ebb3-4cc5-b316-17a0bb480a75-node-exporter-wtmp\") pod \"node-exporter-nlkwm\" (UID: \"1099cd90-ebb3-4cc5-b316-17a0bb480a75\") " pod="openshift-monitoring/node-exporter-nlkwm"
Jul 28 12:55:31.480909 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.480774    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1099cd90-ebb3-4cc5-b316-17a0bb480a75-node-exporter-textfile\") pod \"node-exporter-nlkwm\" (UID: \"1099cd90-ebb3-4cc5-b316-17a0bb480a75\") " pod="openshift-monitoring/node-exporter-nlkwm"
Jul 28 12:55:31.480909 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.480802    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9fefb33c-bdac-47fb-9d68-bbc531bc37cc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-h47vh\" (UID: \"9fefb33c-bdac-47fb-9d68-bbc531bc37cc\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-h47vh"
Jul 28 12:55:31.480909 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.480822    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1099cd90-ebb3-4cc5-b316-17a0bb480a75-root\") pod \"node-exporter-nlkwm\" (UID: \"1099cd90-ebb3-4cc5-b316-17a0bb480a75\") " pod="openshift-monitoring/node-exporter-nlkwm"
Jul 28 12:55:31.481050 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.480932    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klpts\" (UniqueName: \"kubernetes.io/projected/1099cd90-ebb3-4cc5-b316-17a0bb480a75-kube-api-access-klpts\") pod \"node-exporter-nlkwm\" (UID: \"1099cd90-ebb3-4cc5-b316-17a0bb480a75\") " pod="openshift-monitoring/node-exporter-nlkwm"
Jul 28 12:55:31.481050 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.480981    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/61f6e666-8451-4f07-bcc9-cbe18de924f1-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-h585r\" (UID: \"61f6e666-8451-4f07-bcc9-cbe18de924f1\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-h585r"
Jul 28 12:55:31.481050 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.481007    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1099cd90-ebb3-4cc5-b316-17a0bb480a75-metrics-client-ca\") pod \"node-exporter-nlkwm\" (UID: \"1099cd90-ebb3-4cc5-b316-17a0bb480a75\") " pod="openshift-monitoring/node-exporter-nlkwm"
Jul 28 12:55:31.481050 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.481038    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wxxg\" (UniqueName: \"kubernetes.io/projected/9fefb33c-bdac-47fb-9d68-bbc531bc37cc-kube-api-access-8wxxg\") pod \"openshift-state-metrics-b8f49f5c6-h47vh\" (UID: \"9fefb33c-bdac-47fb-9d68-bbc531bc37cc\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-h47vh"
Jul 28 12:55:31.481229 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.481062    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1099cd90-ebb3-4cc5-b316-17a0bb480a75-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nlkwm\" (UID: \"1099cd90-ebb3-4cc5-b316-17a0bb480a75\") " pod="openshift-monitoring/node-exporter-nlkwm"
Jul 28 12:55:31.481229 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.481135    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61f6e666-8451-4f07-bcc9-cbe18de924f1-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-h585r\" (UID: \"61f6e666-8451-4f07-bcc9-cbe18de924f1\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-h585r"
Jul 28 12:55:31.481229 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.481177    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1099cd90-ebb3-4cc5-b316-17a0bb480a75-node-exporter-tls\") pod \"node-exporter-nlkwm\" (UID: \"1099cd90-ebb3-4cc5-b316-17a0bb480a75\") " pod="openshift-monitoring/node-exporter-nlkwm"
Jul 28 12:55:31.481229 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.481210    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9fefb33c-bdac-47fb-9d68-bbc531bc37cc-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-h47vh\" (UID: \"9fefb33c-bdac-47fb-9d68-bbc531bc37cc\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-h47vh"
Jul 28 12:55:31.481428 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.481242    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/61f6e666-8451-4f07-bcc9-cbe18de924f1-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-h585r\" (UID: \"61f6e666-8451-4f07-bcc9-cbe18de924f1\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-h585r"
Jul 28 12:55:31.481428 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.481276    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8dxf\" (UniqueName: \"kubernetes.io/projected/61f6e666-8451-4f07-bcc9-cbe18de924f1-kube-api-access-f8dxf\") pod \"kube-state-metrics-56d9f944b4-h585r\" (UID: \"61f6e666-8451-4f07-bcc9-cbe18de924f1\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-h585r"
Jul 28 12:55:31.481428 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.481312    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1099cd90-ebb3-4cc5-b316-17a0bb480a75-node-exporter-accelerators-collector-config\") pod \"node-exporter-nlkwm\" (UID: \"1099cd90-ebb3-4cc5-b316-17a0bb480a75\") " pod="openshift-monitoring/node-exporter-nlkwm"
Jul 28 12:55:31.481428 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.481343    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61f6e666-8451-4f07-bcc9-cbe18de924f1-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-h585r\" (UID: \"61f6e666-8451-4f07-bcc9-cbe18de924f1\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-h585r"
Jul 28 12:55:31.481637 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.481613    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/61f6e666-8451-4f07-bcc9-cbe18de924f1-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-h585r\" (UID: \"61f6e666-8451-4f07-bcc9-cbe18de924f1\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-h585r"
Jul 28 12:55:31.481773 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.481745    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/61f6e666-8451-4f07-bcc9-cbe18de924f1-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-h585r\" (UID: \"61f6e666-8451-4f07-bcc9-cbe18de924f1\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-h585r"
Jul 28 12:55:31.482082 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.482056    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61f6e666-8451-4f07-bcc9-cbe18de924f1-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-h585r\" (UID: \"61f6e666-8451-4f07-bcc9-cbe18de924f1\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-h585r"
Jul 28 12:55:31.483961 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.483938    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/61f6e666-8451-4f07-bcc9-cbe18de924f1-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-h585r\" (UID: \"61f6e666-8451-4f07-bcc9-cbe18de924f1\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-h585r"
Jul 28 12:55:31.484390 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.484334    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61f6e666-8451-4f07-bcc9-cbe18de924f1-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-h585r\" (UID: \"61f6e666-8451-4f07-bcc9-cbe18de924f1\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-h585r"
Jul 28 12:55:31.490706 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.490641    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9fefb33c-bdac-47fb-9d68-bbc531bc37cc-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-h47vh\" (UID: \"9fefb33c-bdac-47fb-9d68-bbc531bc37cc\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-h47vh"
Jul 28 12:55:31.494875 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.494850    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9fefb33c-bdac-47fb-9d68-bbc531bc37cc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-h47vh\" (UID: \"9fefb33c-bdac-47fb-9d68-bbc531bc37cc\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-h47vh"
Jul 28 12:55:31.496528 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.496495    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9fefb33c-bdac-47fb-9d68-bbc531bc37cc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-h47vh\" (UID: \"9fefb33c-bdac-47fb-9d68-bbc531bc37cc\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-h47vh"
Jul 28 12:55:31.510719 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.510661    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8dxf\" (UniqueName: \"kubernetes.io/projected/61f6e666-8451-4f07-bcc9-cbe18de924f1-kube-api-access-f8dxf\") pod \"kube-state-metrics-56d9f944b4-h585r\" (UID: \"61f6e666-8451-4f07-bcc9-cbe18de924f1\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-h585r"
Jul 28 12:55:31.539896 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.539864    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wxxg\" (UniqueName: \"kubernetes.io/projected/9fefb33c-bdac-47fb-9d68-bbc531bc37cc-kube-api-access-8wxxg\") pod \"openshift-state-metrics-b8f49f5c6-h47vh\" (UID: \"9fefb33c-bdac-47fb-9d68-bbc531bc37cc\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-h47vh"
Jul 28 12:55:31.582329 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.582301    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1099cd90-ebb3-4cc5-b316-17a0bb480a75-root\") pod \"node-exporter-nlkwm\" (UID: \"1099cd90-ebb3-4cc5-b316-17a0bb480a75\") " pod="openshift-monitoring/node-exporter-nlkwm"
Jul 28 12:55:31.582457 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.582343    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klpts\" (UniqueName: \"kubernetes.io/projected/1099cd90-ebb3-4cc5-b316-17a0bb480a75-kube-api-access-klpts\") pod \"node-exporter-nlkwm\" (UID: \"1099cd90-ebb3-4cc5-b316-17a0bb480a75\") " pod="openshift-monitoring/node-exporter-nlkwm"
Jul 28 12:55:31.582457 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.582376    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1099cd90-ebb3-4cc5-b316-17a0bb480a75-metrics-client-ca\") pod \"node-exporter-nlkwm\" (UID: \"1099cd90-ebb3-4cc5-b316-17a0bb480a75\") " pod="openshift-monitoring/node-exporter-nlkwm"
Jul 28 12:55:31.582457 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.582408    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1099cd90-ebb3-4cc5-b316-17a0bb480a75-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nlkwm\" (UID: \"1099cd90-ebb3-4cc5-b316-17a0bb480a75\") " pod="openshift-monitoring/node-exporter-nlkwm"
Jul 28 12:55:31.582457 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.582429    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1099cd90-ebb3-4cc5-b316-17a0bb480a75-root\") pod \"node-exporter-nlkwm\" (UID: \"1099cd90-ebb3-4cc5-b316-17a0bb480a75\") " pod="openshift-monitoring/node-exporter-nlkwm"
Jul 28 12:55:31.583042 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.582444    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1099cd90-ebb3-4cc5-b316-17a0bb480a75-node-exporter-tls\") pod \"node-exporter-nlkwm\" (UID: \"1099cd90-ebb3-4cc5-b316-17a0bb480a75\") " pod="openshift-monitoring/node-exporter-nlkwm"
Jul 28 12:55:31.583170 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.583091    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1099cd90-ebb3-4cc5-b316-17a0bb480a75-node-exporter-accelerators-collector-config\") pod \"node-exporter-nlkwm\" (UID: \"1099cd90-ebb3-4cc5-b316-17a0bb480a75\") " pod="openshift-monitoring/node-exporter-nlkwm"
Jul 28 12:55:31.583234 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.583169    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1099cd90-ebb3-4cc5-b316-17a0bb480a75-sys\") pod \"node-exporter-nlkwm\" (UID: \"1099cd90-ebb3-4cc5-b316-17a0bb480a75\") " pod="openshift-monitoring/node-exporter-nlkwm"
Jul 28 12:55:31.583234 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.583221    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1099cd90-ebb3-4cc5-b316-17a0bb480a75-sys\") pod \"node-exporter-nlkwm\" (UID: \"1099cd90-ebb3-4cc5-b316-17a0bb480a75\") " pod="openshift-monitoring/node-exporter-nlkwm"
Jul 28 12:55:31.583345 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.583254    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1099cd90-ebb3-4cc5-b316-17a0bb480a75-node-exporter-wtmp\") pod \"node-exporter-nlkwm\" (UID: \"1099cd90-ebb3-4cc5-b316-17a0bb480a75\") " pod="openshift-monitoring/node-exporter-nlkwm"
Jul 28 12:55:31.583345 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.583286    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1099cd90-ebb3-4cc5-b316-17a0bb480a75-node-exporter-textfile\") pod \"node-exporter-nlkwm\" (UID: \"1099cd90-ebb3-4cc5-b316-17a0bb480a75\") " pod="openshift-monitoring/node-exporter-nlkwm"
Jul 28 12:55:31.583465 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.583445    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1099cd90-ebb3-4cc5-b316-17a0bb480a75-metrics-client-ca\") pod \"node-exporter-nlkwm\" (UID: \"1099cd90-ebb3-4cc5-b316-17a0bb480a75\") " pod="openshift-monitoring/node-exporter-nlkwm"
Jul 28 12:55:31.583574 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.583554    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1099cd90-ebb3-4cc5-b316-17a0bb480a75-node-exporter-textfile\") pod \"node-exporter-nlkwm\" (UID: \"1099cd90-ebb3-4cc5-b316-17a0bb480a75\") " pod="openshift-monitoring/node-exporter-nlkwm"
Jul 28 12:55:31.583638 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.583579    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1099cd90-ebb3-4cc5-b316-17a0bb480a75-node-exporter-wtmp\") pod \"node-exporter-nlkwm\" (UID: \"1099cd90-ebb3-4cc5-b316-17a0bb480a75\") " pod="openshift-monitoring/node-exporter-nlkwm"
Jul 28 12:55:31.583699 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.583650    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1099cd90-ebb3-4cc5-b316-17a0bb480a75-node-exporter-accelerators-collector-config\") pod \"node-exporter-nlkwm\" (UID: \"1099cd90-ebb3-4cc5-b316-17a0bb480a75\") " pod="openshift-monitoring/node-exporter-nlkwm"
Jul 28 12:55:31.585098 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.585074    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1099cd90-ebb3-4cc5-b316-17a0bb480a75-node-exporter-tls\") pod \"node-exporter-nlkwm\" (UID: \"1099cd90-ebb3-4cc5-b316-17a0bb480a75\") " pod="openshift-monitoring/node-exporter-nlkwm"
Jul 28 12:55:31.585715 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.585670    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1099cd90-ebb3-4cc5-b316-17a0bb480a75-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nlkwm\" (UID: \"1099cd90-ebb3-4cc5-b316-17a0bb480a75\") " pod="openshift-monitoring/node-exporter-nlkwm"
Jul 28 12:55:31.587841 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.587729    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-h47vh"
Jul 28 12:55:31.604879 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.604856    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-h585r"
Jul 28 12:55:31.613590 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.613572    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klpts\" (UniqueName: \"kubernetes.io/projected/1099cd90-ebb3-4cc5-b316-17a0bb480a75-kube-api-access-klpts\") pod \"node-exporter-nlkwm\" (UID: \"1099cd90-ebb3-4cc5-b316-17a0bb480a75\") " pod="openshift-monitoring/node-exporter-nlkwm"
Jul 28 12:55:31.740872 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:31.740605    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nlkwm"
Jul 28 12:55:32.303121 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:32.303085    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d412255-c984-4473-af61-1271b6f141e5" path="/var/lib/kubelet/pods/6d412255-c984-4473-af61-1271b6f141e5/volumes"
Jul 28 12:55:32.370576 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:32.370543    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 12:55:32.422724 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:32.422696    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 12:55:32.423072 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:32.422857    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:55:32.427281 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:32.426032    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 28 12:55:32.427281 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:32.426413    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 28 12:55:32.427281 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:32.426679    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 28 12:55:32.427281 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:32.426916    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 28 12:55:32.427281 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:32.427117    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 28 12:55:32.427281 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:32.427220    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 28 12:55:32.427770 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:32.427367    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-bvx4x\""
Jul 28 12:55:32.427770 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:32.427542    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 28 12:55:32.427770 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:32.427762    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 28 12:55:32.431846 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:32.430806    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 28 12:55:32.489835 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:32.489798    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4d592775-753a-47d3-bc13-ff297afd9f51-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4d592775-753a-47d3-bc13-ff297afd9f51\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:55:32.489984 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:32.489859    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbjqv\" (UniqueName: \"kubernetes.io/projected/4d592775-753a-47d3-bc13-ff297afd9f51-kube-api-access-sbjqv\") pod \"alertmanager-main-0\" (UID: \"4d592775-753a-47d3-bc13-ff297afd9f51\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:55:32.489984 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:32.489876    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4d592775-753a-47d3-bc13-ff297afd9f51-config-out\") pod \"alertmanager-main-0\" (UID: \"4d592775-753a-47d3-bc13-ff297afd9f51\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:55:32.489984 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:32.489898    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4d592775-753a-47d3-bc13-ff297afd9f51-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4d592775-753a-47d3-bc13-ff297afd9f51\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:55:32.489984 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:32.489920    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4d592775-753a-47d3-bc13-ff297afd9f51-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4d592775-753a-47d3-bc13-ff297afd9f51\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:55:32.489984 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:32.489950    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4d592775-753a-47d3-bc13-ff297afd9f51-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4d592775-753a-47d3-bc13-ff297afd9f51\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:55:32.490220 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:32.490000    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4d592775-753a-47d3-bc13-ff297afd9f51-config-volume\") pod \"alertmanager-main-0\" (UID: \"4d592775-753a-47d3-bc13-ff297afd9f51\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:55:32.490220 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:32.490026    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4d592775-753a-47d3-bc13-ff297afd9f51-web-config\") pod \"alertmanager-main-0\" (UID: \"4d592775-753a-47d3-bc13-ff297afd9f51\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:55:32.490220 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:32.490046    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d592775-753a-47d3-bc13-ff297afd9f51-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4d592775-753a-47d3-bc13-ff297afd9f51\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:55:32.490220 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:32.490066    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4d592775-753a-47d3-bc13-ff297afd9f51-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4d592775-753a-47d3-bc13-ff297afd9f51\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:55:32.490220 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:32.490090    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d592775-753a-47d3-bc13-ff297afd9f51-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4d592775-753a-47d3-bc13-ff297afd9f51\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:55:32.490220 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:32.490119    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4d592775-753a-47d3-bc13-ff297afd9f51-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4d592775-753a-47d3-bc13-ff297afd9f51\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:55:32.490220 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:32.490149    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4d592775-753a-47d3-bc13-ff297afd9f51-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4d592775-753a-47d3-bc13-ff297afd9f51\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:55:32.591220 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:32.591135    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4d592775-753a-47d3-bc13-ff297afd9f51-config-volume\") pod \"alertmanager-main-0\" (UID: \"4d592775-753a-47d3-bc13-ff297afd9f51\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:55:32.591220 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:32.591184    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4d592775-753a-47d3-bc13-ff297afd9f51-web-config\") pod \"alertmanager-main-0\" (UID: \"4d592775-753a-47d3-bc13-ff297afd9f51\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:55:32.591220 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:32.591212    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d592775-753a-47d3-bc13-ff297afd9f51-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4d592775-753a-47d3-bc13-ff297afd9f51\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:55:32.592856 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:32.591476    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4d592775-753a-47d3-bc13-ff297afd9f51-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4d592775-753a-47d3-bc13-ff297afd9f51\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:55:32.592856 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:32.591519    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d592775-753a-47d3-bc13-ff297afd9f51-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4d592775-753a-47d3-bc13-ff297afd9f51\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:55:32.592856 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:32.591552    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4d592775-753a-47d3-bc13-ff297afd9f51-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4d592775-753a-47d3-bc13-ff297afd9f51\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:55:32.592856 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:32.591586    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4d592775-753a-47d3-bc13-ff297afd9f51-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4d592775-753a-47d3-bc13-ff297afd9f51\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:55:32.592856 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:32.591630    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4d592775-753a-47d3-bc13-ff297afd9f51-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4d592775-753a-47d3-bc13-ff297afd9f51\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:55:32.592856 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:32.591713    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbjqv\" (UniqueName: \"kubernetes.io/projected/4d592775-753a-47d3-bc13-ff297afd9f51-kube-api-access-sbjqv\") pod \"alertmanager-main-0\" (UID: \"4d592775-753a-47d3-bc13-ff297afd9f51\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:55:32.592856 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:32.591739    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4d592775-753a-47d3-bc13-ff297afd9f51-config-out\") pod \"alertmanager-main-0\" (UID: \"4d592775-753a-47d3-bc13-ff297afd9f51\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:55:32.592856 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:32.591768    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4d592775-753a-47d3-bc13-ff297afd9f51-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4d592775-753a-47d3-bc13-ff297afd9f51\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:55:32.592856 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:32.591791    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4d592775-753a-47d3-bc13-ff297afd9f51-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4d592775-753a-47d3-bc13-ff297afd9f51\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:55:32.592856 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:32.591832    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4d592775-753a-47d3-bc13-ff297afd9f51-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4d592775-753a-47d3-bc13-ff297afd9f51\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:55:32.594440 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:32.593010    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4d592775-753a-47d3-bc13-ff297afd9f51-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4d592775-753a-47d3-bc13-ff297afd9f51\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:55:32.594934 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:32.594650    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4d592775-753a-47d3-bc13-ff297afd9f51-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4d592775-753a-47d3-bc13-ff297afd9f51\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:55:32.595127 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:32.595102    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d592775-753a-47d3-bc13-ff297afd9f51-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4d592775-753a-47d3-bc13-ff297afd9f51\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:55:32.595189 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:32.595134    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4d592775-753a-47d3-bc13-ff297afd9f51-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4d592775-753a-47d3-bc13-ff297afd9f51\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:55:32.596132 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:32.596106    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4d592775-753a-47d3-bc13-ff297afd9f51-web-config\") pod \"alertmanager-main-0\" (UID: \"4d592775-753a-47d3-bc13-ff297afd9f51\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:55:32.596216 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:32.596183    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d592775-753a-47d3-bc13-ff297afd9f51-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4d592775-753a-47d3-bc13-ff297afd9f51\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:55:32.599460 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:32.599372    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4d592775-753a-47d3-bc13-ff297afd9f51-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4d592775-753a-47d3-bc13-ff297afd9f51\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:55:32.599460 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:32.599411    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4d592775-753a-47d3-bc13-ff297afd9f51-config-volume\") pod \"alertmanager-main-0\" (UID: \"4d592775-753a-47d3-bc13-ff297afd9f51\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:55:32.600145 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:32.600105    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4d592775-753a-47d3-bc13-ff297afd9f51-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4d592775-753a-47d3-bc13-ff297afd9f51\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:55:32.600145 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:32.600122    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4d592775-753a-47d3-bc13-ff297afd9f51-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4d592775-753a-47d3-bc13-ff297afd9f51\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:55:32.600502 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:32.600464    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4d592775-753a-47d3-bc13-ff297afd9f51-config-out\") pod \"alertmanager-main-0\" (UID: \"4d592775-753a-47d3-bc13-ff297afd9f51\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:55:32.603282 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:32.603229    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4d592775-753a-47d3-bc13-ff297afd9f51-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4d592775-753a-47d3-bc13-ff297afd9f51\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:55:32.609301 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:32.609280    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbjqv\" (UniqueName: \"kubernetes.io/projected/4d592775-753a-47d3-bc13-ff297afd9f51-kube-api-access-sbjqv\") pod \"alertmanager-main-0\" (UID: \"4d592775-753a-47d3-bc13-ff297afd9f51\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:55:32.742870 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:32.742835    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:55:33.168941 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:55:33.168876    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1099cd90_ebb3_4cc5_b316_17a0bb480a75.slice/crio-3a202f043335419a9fd9c2a42a376e4005ea001c469cbfa03c1bcdfaa53edafc WatchSource:0}: Error finding container 3a202f043335419a9fd9c2a42a376e4005ea001c469cbfa03c1bcdfaa53edafc: Status 404 returned error can't find the container with id 3a202f043335419a9fd9c2a42a376e4005ea001c469cbfa03c1bcdfaa53edafc
Jul 28 12:55:33.322554 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:33.322525    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-h585r"]
Jul 28 12:55:33.324934 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:55:33.324906    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61f6e666_8451_4f07_bcc9_cbe18de924f1.slice/crio-5e742d75f3c0efc19aeb2634e770e509879f5b957b40b4f70b1fd7cd8f526bee WatchSource:0}: Error finding container 5e742d75f3c0efc19aeb2634e770e509879f5b957b40b4f70b1fd7cd8f526bee: Status 404 returned error can't find the container with id 5e742d75f3c0efc19aeb2634e770e509879f5b957b40b4f70b1fd7cd8f526bee
Jul 28 12:55:33.555642 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:33.555554    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-h47vh"]
Jul 28 12:55:33.559566 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:55:33.559538    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fefb33c_bdac_47fb_9d68_bbc531bc37cc.slice/crio-50f89aafc088bde1f48cf71823c5b451f59fbe754d7e21daa396ba34244b143b WatchSource:0}: Error finding container 50f89aafc088bde1f48cf71823c5b451f59fbe754d7e21daa396ba34244b143b: Status 404 returned error can't find the container with id 50f89aafc088bde1f48cf71823c5b451f59fbe754d7e21daa396ba34244b143b
Jul 28 12:55:33.565278 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:33.565251    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 12:55:33.566309 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:55:33.566271    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d592775_753a_47d3_bc13_ff297afd9f51.slice/crio-b2322766eeb4775cf92b447db2a30d0f6154abdc576a3f41c090416b97f1ede2 WatchSource:0}: Error finding container b2322766eeb4775cf92b447db2a30d0f6154abdc576a3f41c090416b97f1ede2: Status 404 returned error can't find the container with id b2322766eeb4775cf92b447db2a30d0f6154abdc576a3f41c090416b97f1ede2
Jul 28 12:55:33.863813 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:33.863717    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nlkwm" event={"ID":"1099cd90-ebb3-4cc5-b316-17a0bb480a75","Type":"ContainerStarted","Data":"3a202f043335419a9fd9c2a42a376e4005ea001c469cbfa03c1bcdfaa53edafc"}
Jul 28 12:55:33.866152 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:33.866079    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4d592775-753a-47d3-bc13-ff297afd9f51","Type":"ContainerStarted","Data":"b2322766eeb4775cf92b447db2a30d0f6154abdc576a3f41c090416b97f1ede2"}
Jul 28 12:55:33.870133 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:33.870061    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-27vf2" event={"ID":"f80dfaf5-fb64-4f4e-ab75-e1164768f586","Type":"ContainerStarted","Data":"afaebcf42dfd46fd7a868043c1ffff7d265dd89aec46d148455d018ad1aec616"}
Jul 28 12:55:33.872899 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:33.872876    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-h47vh" event={"ID":"9fefb33c-bdac-47fb-9d68-bbc531bc37cc","Type":"ContainerStarted","Data":"35e7599d0cb3b285351c840c21d7937dae96fc8f8b420e40091a72221bbcf6f0"}
Jul 28 12:55:33.873001 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:33.872929    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-h47vh" event={"ID":"9fefb33c-bdac-47fb-9d68-bbc531bc37cc","Type":"ContainerStarted","Data":"76b9e95636e5c130a859e67e3a10a32dd8ca0f4ea908732bad00bb9f50211838"}
Jul 28 12:55:33.873001 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:33.872960    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-h47vh" event={"ID":"9fefb33c-bdac-47fb-9d68-bbc531bc37cc","Type":"ContainerStarted","Data":"50f89aafc088bde1f48cf71823c5b451f59fbe754d7e21daa396ba34244b143b"}
Jul 28 12:55:33.875021 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:33.874990    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-h585r" event={"ID":"61f6e666-8451-4f07-bcc9-cbe18de924f1","Type":"ContainerStarted","Data":"5e742d75f3c0efc19aeb2634e770e509879f5b957b40b4f70b1fd7cd8f526bee"}
Jul 28 12:55:36.004385 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:36.004357    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-8qxp6"]
Jul 28 12:55:36.022090 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:36.020984    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-8qxp6"]
Jul 28 12:55:36.022090 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:36.021092    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-8qxp6"
Jul 28 12:55:36.023996 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:36.023975    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 28 12:55:36.024113 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:36.023995    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-v4s4r\""
Jul 28 12:55:36.129743 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:36.129701    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng7bb\" (UniqueName: \"kubernetes.io/projected/cd34484f-2793-4e67-89eb-586d70b97eb4-kube-api-access-ng7bb\") pod \"monitoring-plugin-68b7fc9574-8qxp6\" (UID: \"cd34484f-2793-4e67-89eb-586d70b97eb4\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-8qxp6"
Jul 28 12:55:36.129893 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:36.129756    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cd34484f-2793-4e67-89eb-586d70b97eb4-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-8qxp6\" (UID: \"cd34484f-2793-4e67-89eb-586d70b97eb4\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-8qxp6"
Jul 28 12:55:36.230795 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:36.230740    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng7bb\" (UniqueName: \"kubernetes.io/projected/cd34484f-2793-4e67-89eb-586d70b97eb4-kube-api-access-ng7bb\") pod \"monitoring-plugin-68b7fc9574-8qxp6\" (UID: \"cd34484f-2793-4e67-89eb-586d70b97eb4\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-8qxp6"
Jul 28 12:55:36.231255 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:36.231176    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cd34484f-2793-4e67-89eb-586d70b97eb4-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-8qxp6\" (UID: \"cd34484f-2793-4e67-89eb-586d70b97eb4\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-8qxp6"
Jul 28 12:55:36.248141 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:36.234770    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cd34484f-2793-4e67-89eb-586d70b97eb4-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-8qxp6\" (UID: \"cd34484f-2793-4e67-89eb-586d70b97eb4\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-8qxp6"
Jul 28 12:55:36.248141 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:36.240354    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng7bb\" (UniqueName: \"kubernetes.io/projected/cd34484f-2793-4e67-89eb-586d70b97eb4-kube-api-access-ng7bb\") pod \"monitoring-plugin-68b7fc9574-8qxp6\" (UID: \"cd34484f-2793-4e67-89eb-586d70b97eb4\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-8qxp6"
Jul 28 12:55:36.342331 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:36.342046    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-8qxp6"
Jul 28 12:55:36.614539 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:36.614466    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-8qxp6"]
Jul 28 12:55:36.890329 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:36.890276    2574 generic.go:356] "Generic (PLEG): container finished" podID="1099cd90-ebb3-4cc5-b316-17a0bb480a75" containerID="ee7ccedde21fbff91bcd99f0bf26bfb2eebf121eaa48b1c4054eef825b59c681" exitCode=0
Jul 28 12:55:36.890461 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:36.890383    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nlkwm" event={"ID":"1099cd90-ebb3-4cc5-b316-17a0bb480a75","Type":"ContainerDied","Data":"ee7ccedde21fbff91bcd99f0bf26bfb2eebf121eaa48b1c4054eef825b59c681"}
Jul 28 12:55:36.892406 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:36.892380    2574 generic.go:356] "Generic (PLEG): container finished" podID="4d592775-753a-47d3-bc13-ff297afd9f51" containerID="e4797a92595cf739bb114c08b91636290166975bf4b864e3714f284cd2eb31e0" exitCode=0
Jul 28 12:55:36.892505 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:36.892481    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4d592775-753a-47d3-bc13-ff297afd9f51","Type":"ContainerDied","Data":"e4797a92595cf739bb114c08b91636290166975bf4b864e3714f284cd2eb31e0"}
Jul 28 12:55:36.895659 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:36.895607    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-8qxp6" event={"ID":"cd34484f-2793-4e67-89eb-586d70b97eb4","Type":"ContainerStarted","Data":"6cade7037b051b695bd0b5672cb2a0d6afb3ccbe03230d73c43ddd02b062502b"}
Jul 28 12:55:36.898590 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:36.898530    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-h47vh" event={"ID":"9fefb33c-bdac-47fb-9d68-bbc531bc37cc","Type":"ContainerStarted","Data":"4e0afec43e89fd8ba37b8a01e8ffbce7e04250368b3a8aedfd9e3c9a3a461d7d"}
Jul 28 12:55:36.901149 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:36.901045    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-h585r" event={"ID":"61f6e666-8451-4f07-bcc9-cbe18de924f1","Type":"ContainerStarted","Data":"61f54962a8accb9c4c5ebff77e403a7c22a789a9161047dde96b4d12e0e2b4e8"}
Jul 28 12:55:36.901149 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:36.901095    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-h585r" event={"ID":"61f6e666-8451-4f07-bcc9-cbe18de924f1","Type":"ContainerStarted","Data":"0586ee206b02d8ff805f3b1dcf3ad907fd2b57bcd72cfccb51cc930a7612deb6"}
Jul 28 12:55:36.929923 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:36.929831    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-h47vh" podStartSLOduration=3.043918402 podStartE2EDuration="5.929808912s" totalImagesPullingTime="2.88589051s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:55:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:55:36.927707972 +0000 UTC m=+149.133094598" watchObservedRunningTime="2026-07-28 12:55:36.929808912 +0000 UTC m=+149.135195493"
Jul 28 12:55:37.910877 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:37.910764    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nlkwm" event={"ID":"1099cd90-ebb3-4cc5-b316-17a0bb480a75","Type":"ContainerStarted","Data":"facbb337a49097804013404927b1bd8a7b5b43edddd2a629fafec644bc5c55a7"}
Jul 28 12:55:37.911843 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:37.910849    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nlkwm" event={"ID":"1099cd90-ebb3-4cc5-b316-17a0bb480a75","Type":"ContainerStarted","Data":"9b46c3475e638956285d34344cb3f93b00dfab1c0bb4d57c95ad3064d02f3da9"}
Jul 28 12:55:37.914532 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:37.914495    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-h585r" event={"ID":"61f6e666-8451-4f07-bcc9-cbe18de924f1","Type":"ContainerStarted","Data":"e1ec3fbf56b1ae6f3cb7cdc52feec9619d2cec1008509f1c160d3897fea6aeac"}
Jul 28 12:55:37.935871 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:37.934984    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-nlkwm" podStartSLOduration=3.99960723 podStartE2EDuration="6.934963183s" totalImagesPullingTime="2.829621257s" totalInitContainerRuntime="105.734696ms" isStatefulPod=true podCreationTimestamp="2026-07-28 12:55:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:55:37.933469907 +0000 UTC m=+150.138856532" watchObservedRunningTime="2026-07-28 12:55:37.934963183 +0000 UTC m=+150.140349788"
Jul 28 12:55:37.958102 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:37.956595    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-h585r" podStartSLOduration=4.283887438 podStartE2EDuration="6.956575477s" totalImagesPullingTime="2.672688039s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:55:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:55:37.954865575 +0000 UTC m=+150.160252177" watchObservedRunningTime="2026-07-28 12:55:37.956575477 +0000 UTC m=+150.161962080"
Jul 28 12:55:40.927553 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:40.927523    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-8qxp6" event={"ID":"cd34484f-2793-4e67-89eb-586d70b97eb4","Type":"ContainerStarted","Data":"1bdb39cc26fc6c216520b540f09debe7630530973d120beba4ddd45c6eed5157"}
Jul 28 12:55:40.927998 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:40.927580    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-8qxp6"
Jul 28 12:55:40.929788 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:40.929756    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-27vf2" event={"ID":"f80dfaf5-fb64-4f4e-ab75-e1164768f586","Type":"ContainerStarted","Data":"0423ce9fb27f771c477c1351682a6d789b6eb34f6489a2a00de7b5879ff2d6fc"}
Jul 28 12:55:40.933212 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:40.933195    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-8qxp6"
Jul 28 12:55:40.945068 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:40.945016    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-8qxp6" podStartSLOduration=1.890102073 podStartE2EDuration="5.944998671s" totalImagesPullingTime="4.054896598s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:55:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:55:40.943156952 +0000 UTC m=+153.148543554" watchObservedRunningTime="2026-07-28 12:55:40.944998671 +0000 UTC m=+153.150385273"
Jul 28 12:55:40.976078 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:40.976019    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-27vf2" podStartSLOduration=33.188041584 podStartE2EDuration="58.975995242s" totalImagesPullingTime="25.787953658s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:54:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:55:40.974885871 +0000 UTC m=+153.180272475" watchObservedRunningTime="2026-07-28 12:55:40.975995242 +0000 UTC m=+153.181381844"
Jul 28 12:55:41.938675 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:41.938005    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4d592775-753a-47d3-bc13-ff297afd9f51","Type":"ContainerStarted","Data":"8a6d1ffbef1102e109d87c2e5352367c087ed5a5c9ea9bf84587a9c94162f67a"}
Jul 28 12:55:41.938675 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:41.938117    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4d592775-753a-47d3-bc13-ff297afd9f51","Type":"ContainerStarted","Data":"6404efc81aaddf56c9bf9baa731b63fbc1cabeeb24f32551ed16f79d817e09f7"}
Jul 28 12:55:41.938675 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:41.938156    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4d592775-753a-47d3-bc13-ff297afd9f51","Type":"ContainerStarted","Data":"ebb829fdf118b9b7399ac60a08a6fe8c6ac7e3cacac71fe494fabca94ef10da8"}
Jul 28 12:55:41.938675 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:41.938193    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4d592775-753a-47d3-bc13-ff297afd9f51","Type":"ContainerStarted","Data":"025989bdd527cd61ded68c94d6fce86ff7c4e4f68d131ef6b1e9d0827ecac01d"}
Jul 28 12:55:41.938675 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:41.938231    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4d592775-753a-47d3-bc13-ff297afd9f51","Type":"ContainerStarted","Data":"6b3a5d6b9efe3f195c9d0d3e78307ecaad38469eb6cc503696b57feedad92555"}
Jul 28 12:55:42.945991 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:42.945959    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4d592775-753a-47d3-bc13-ff297afd9f51","Type":"ContainerStarted","Data":"17c5298497ece6aa4b9e5d87c31964eed14bf31de2a11c0d6033e0fb26f84a27"}
Jul 28 12:55:42.979478 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:42.979404    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.250655701 podStartE2EDuration="10.979388826s" totalImagesPullingTime="7.701033597s" totalInitContainerRuntime="27.699528ms" isStatefulPod=false podCreationTimestamp="2026-07-28 12:55:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:55:42.97743028 +0000 UTC m=+155.182816920" watchObservedRunningTime="2026-07-28 12:55:42.979388826 +0000 UTC m=+155.184775444"
Jul 28 12:55:45.640992 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:55:45.640952    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-r8l5q" podUID="f8bc5bca-7c04-47ec-b3dc-f6ce86345325"
Jul 28 12:55:45.956598 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:45.956573    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-r8l5q"
Jul 28 12:55:48.334443 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:48.334416    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-b9htm_8b530194-44d2-4993-9d2e-5aae3188a18d/serve-healthcheck-canary/0.log"
Jul 28 12:55:50.553083 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:50.553055    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8bc5bca-7c04-47ec-b3dc-f6ce86345325-metrics-tls\") pod \"dns-default-r8l5q\" (UID: \"f8bc5bca-7c04-47ec-b3dc-f6ce86345325\") " pod="openshift-dns/dns-default-r8l5q"
Jul 28 12:55:50.555212 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:50.555191    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8bc5bca-7c04-47ec-b3dc-f6ce86345325-metrics-tls\") pod \"dns-default-r8l5q\" (UID: \"f8bc5bca-7c04-47ec-b3dc-f6ce86345325\") " pod="openshift-dns/dns-default-r8l5q"
Jul 28 12:55:50.759788 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:50.759763    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4l68m\""
Jul 28 12:55:50.767498 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:50.767482    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-r8l5q"
Jul 28 12:55:50.923958 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:50.923936    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-r8l5q"]
Jul 28 12:55:50.926328 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:55:50.926301    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8bc5bca_7c04_47ec_b3dc_f6ce86345325.slice/crio-2f86aa860c465c874030c12401141d9e45413c798d5d5129ba62aa7e4a408cf0 WatchSource:0}: Error finding container 2f86aa860c465c874030c12401141d9e45413c798d5d5129ba62aa7e4a408cf0: Status 404 returned error can't find the container with id 2f86aa860c465c874030c12401141d9e45413c798d5d5129ba62aa7e4a408cf0
Jul 28 12:55:50.976666 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:50.976638    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r8l5q" event={"ID":"f8bc5bca-7c04-47ec-b3dc-f6ce86345325","Type":"ContainerStarted","Data":"2f86aa860c465c874030c12401141d9e45413c798d5d5129ba62aa7e4a408cf0"}
Jul 28 12:55:53.426567 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:53.426514    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5d666b9745-dqx5g" podUID="9c66fb0e-db92-4d10-bd1e-6178c0f76cfa" containerName="console" containerID="cri-o://c173fb2222e4c4f782335473141b0226cb22acb346aa68a61d5ef9d61506fcb1" gracePeriod=15
Jul 28 12:55:53.692367 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:53.692345    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d666b9745-dqx5g"
Jul 28 12:55:53.778052 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:53.778019    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9c66fb0e-db92-4d10-bd1e-6178c0f76cfa-console-serving-cert\") pod \"9c66fb0e-db92-4d10-bd1e-6178c0f76cfa\" (UID: \"9c66fb0e-db92-4d10-bd1e-6178c0f76cfa\") "
Jul 28 12:55:53.778207 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:53.778066    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2c76m\" (UniqueName: \"kubernetes.io/projected/9c66fb0e-db92-4d10-bd1e-6178c0f76cfa-kube-api-access-2c76m\") pod \"9c66fb0e-db92-4d10-bd1e-6178c0f76cfa\" (UID: \"9c66fb0e-db92-4d10-bd1e-6178c0f76cfa\") "
Jul 28 12:55:53.778207 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:53.778093    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9c66fb0e-db92-4d10-bd1e-6178c0f76cfa-console-config\") pod \"9c66fb0e-db92-4d10-bd1e-6178c0f76cfa\" (UID: \"9c66fb0e-db92-4d10-bd1e-6178c0f76cfa\") "
Jul 28 12:55:53.778207 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:53.778120    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9c66fb0e-db92-4d10-bd1e-6178c0f76cfa-service-ca\") pod \"9c66fb0e-db92-4d10-bd1e-6178c0f76cfa\" (UID: \"9c66fb0e-db92-4d10-bd1e-6178c0f76cfa\") "
Jul 28 12:55:53.778207 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:53.778153    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9c66fb0e-db92-4d10-bd1e-6178c0f76cfa-oauth-serving-cert\") pod \"9c66fb0e-db92-4d10-bd1e-6178c0f76cfa\" (UID: \"9c66fb0e-db92-4d10-bd1e-6178c0f76cfa\") "
Jul 28 12:55:53.778207 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:53.778202    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9c66fb0e-db92-4d10-bd1e-6178c0f76cfa-console-oauth-config\") pod \"9c66fb0e-db92-4d10-bd1e-6178c0f76cfa\" (UID: \"9c66fb0e-db92-4d10-bd1e-6178c0f76cfa\") "
Jul 28 12:55:53.778593 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:53.778568    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c66fb0e-db92-4d10-bd1e-6178c0f76cfa-console-config" (OuterVolumeSpecName: "console-config") pod "9c66fb0e-db92-4d10-bd1e-6178c0f76cfa" (UID: "9c66fb0e-db92-4d10-bd1e-6178c0f76cfa"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:55:53.778702 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:53.778565    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c66fb0e-db92-4d10-bd1e-6178c0f76cfa-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9c66fb0e-db92-4d10-bd1e-6178c0f76cfa" (UID: "9c66fb0e-db92-4d10-bd1e-6178c0f76cfa"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:55:53.778702 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:53.778576    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c66fb0e-db92-4d10-bd1e-6178c0f76cfa-service-ca" (OuterVolumeSpecName: "service-ca") pod "9c66fb0e-db92-4d10-bd1e-6178c0f76cfa" (UID: "9c66fb0e-db92-4d10-bd1e-6178c0f76cfa"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:55:53.780343 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:53.780320    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c66fb0e-db92-4d10-bd1e-6178c0f76cfa-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9c66fb0e-db92-4d10-bd1e-6178c0f76cfa" (UID: "9c66fb0e-db92-4d10-bd1e-6178c0f76cfa"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:55:53.780462 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:53.780353    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c66fb0e-db92-4d10-bd1e-6178c0f76cfa-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9c66fb0e-db92-4d10-bd1e-6178c0f76cfa" (UID: "9c66fb0e-db92-4d10-bd1e-6178c0f76cfa"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:55:53.780462 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:53.780377    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c66fb0e-db92-4d10-bd1e-6178c0f76cfa-kube-api-access-2c76m" (OuterVolumeSpecName: "kube-api-access-2c76m") pod "9c66fb0e-db92-4d10-bd1e-6178c0f76cfa" (UID: "9c66fb0e-db92-4d10-bd1e-6178c0f76cfa"). InnerVolumeSpecName "kube-api-access-2c76m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:55:53.879370 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:53.879346    2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9c66fb0e-db92-4d10-bd1e-6178c0f76cfa-console-oauth-config\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 12:55:53.879370 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:53.879366    2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9c66fb0e-db92-4d10-bd1e-6178c0f76cfa-console-serving-cert\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 12:55:53.879511 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:53.879375    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c76m\" (UniqueName: \"kubernetes.io/projected/9c66fb0e-db92-4d10-bd1e-6178c0f76cfa-kube-api-access-2c76m\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 12:55:53.879511 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:53.879384    2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9c66fb0e-db92-4d10-bd1e-6178c0f76cfa-console-config\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 12:55:53.879511 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:53.879392    2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9c66fb0e-db92-4d10-bd1e-6178c0f76cfa-service-ca\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 12:55:53.879511 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:53.879400    2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9c66fb0e-db92-4d10-bd1e-6178c0f76cfa-oauth-serving-cert\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 12:55:53.990071 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:53.990041    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r8l5q" event={"ID":"f8bc5bca-7c04-47ec-b3dc-f6ce86345325","Type":"ContainerStarted","Data":"186ec63d9fc203213a2b9d9f1064292282f64af36c14f1674e9bc2ca56c84fd9"}
Jul 28 12:55:53.990219 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:53.990117    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r8l5q" event={"ID":"f8bc5bca-7c04-47ec-b3dc-f6ce86345325","Type":"ContainerStarted","Data":"7b344de55cf6008173aea002fc819be1c0a00f99fc18c43c953d908f378b1f15"}
Jul 28 12:55:53.990219 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:53.990156    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-r8l5q"
Jul 28 12:55:53.991403 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:53.991384    2574 generic.go:356] "Generic (PLEG): container finished" podID="9c66fb0e-db92-4d10-bd1e-6178c0f76cfa" containerID="c173fb2222e4c4f782335473141b0226cb22acb346aa68a61d5ef9d61506fcb1" exitCode=0
Jul 28 12:55:53.991489 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:53.991407    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d666b9745-dqx5g" event={"ID":"9c66fb0e-db92-4d10-bd1e-6178c0f76cfa","Type":"ContainerDied","Data":"c173fb2222e4c4f782335473141b0226cb22acb346aa68a61d5ef9d61506fcb1"}
Jul 28 12:55:53.991489 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:53.991432    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d666b9745-dqx5g"
Jul 28 12:55:53.991489 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:53.991443    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d666b9745-dqx5g" event={"ID":"9c66fb0e-db92-4d10-bd1e-6178c0f76cfa","Type":"ContainerDied","Data":"725f5f9faa7701695788d245de1a9f4b78d9e74810c8bd21f189ab0577f73993"}
Jul 28 12:55:53.991489 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:53.991464    2574 scope.go:117] "RemoveContainer" containerID="c173fb2222e4c4f782335473141b0226cb22acb346aa68a61d5ef9d61506fcb1"
Jul 28 12:55:54.000368 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:54.000348    2574 scope.go:117] "RemoveContainer" containerID="c173fb2222e4c4f782335473141b0226cb22acb346aa68a61d5ef9d61506fcb1"
Jul 28 12:55:54.000596 ip-10-0-138-158 kubenswrapper[2574]: E0728 12:55:54.000576    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c173fb2222e4c4f782335473141b0226cb22acb346aa68a61d5ef9d61506fcb1\": container with ID starting with c173fb2222e4c4f782335473141b0226cb22acb346aa68a61d5ef9d61506fcb1 not found: ID does not exist" containerID="c173fb2222e4c4f782335473141b0226cb22acb346aa68a61d5ef9d61506fcb1"
Jul 28 12:55:54.000638 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:54.000603    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c173fb2222e4c4f782335473141b0226cb22acb346aa68a61d5ef9d61506fcb1"} err="failed to get container status \"c173fb2222e4c4f782335473141b0226cb22acb346aa68a61d5ef9d61506fcb1\": rpc error: code = NotFound desc = could not find container \"c173fb2222e4c4f782335473141b0226cb22acb346aa68a61d5ef9d61506fcb1\": container with ID starting with c173fb2222e4c4f782335473141b0226cb22acb346aa68a61d5ef9d61506fcb1 not found: ID does not exist"
Jul 28 12:55:54.011551 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:54.011514    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-r8l5q" podStartSLOduration=129.679097229 podStartE2EDuration="2m12.011502727s" totalImagesPullingTime="2.332405498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:53:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:55:54.009962928 +0000 UTC m=+166.215349529" watchObservedRunningTime="2026-07-28 12:55:54.011502727 +0000 UTC m=+166.216889328"
Jul 28 12:55:54.024987 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:54.024956    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d666b9745-dqx5g"]
Jul 28 12:55:54.027646 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:54.027624    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5d666b9745-dqx5g"]
Jul 28 12:55:54.303306 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:55:54.303280    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c66fb0e-db92-4d10-bd1e-6178c0f76cfa" path="/var/lib/kubelet/pods/9c66fb0e-db92-4d10-bd1e-6178c0f76cfa/volumes"
Jul 28 12:56:03.998834 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:56:03.998762    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-r8l5q"
Jul 28 12:56:17.087191 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:56:17.087157    2574 generic.go:356] "Generic (PLEG): container finished" podID="d7908f8d-42b7-4a20-b5f6-c2bbf80af85d" containerID="f04b9e751156e822e2469bd9ef202a1ab0e611b65ec0bbcebed58accc0727a90" exitCode=0
Jul 28 12:56:17.087596 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:56:17.087200    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-x84d8" event={"ID":"d7908f8d-42b7-4a20-b5f6-c2bbf80af85d","Type":"ContainerDied","Data":"f04b9e751156e822e2469bd9ef202a1ab0e611b65ec0bbcebed58accc0727a90"}
Jul 28 12:56:17.087596 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:56:17.087494    2574 scope.go:117] "RemoveContainer" containerID="f04b9e751156e822e2469bd9ef202a1ab0e611b65ec0bbcebed58accc0727a90"
Jul 28 12:56:18.092973 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:56:18.092943    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-x84d8" event={"ID":"d7908f8d-42b7-4a20-b5f6-c2bbf80af85d","Type":"ContainerStarted","Data":"8d2d8ed1a6cbd4a2e650dcbc7b0ab27cfcd0ada518910efa773459e845be3701"}
Jul 28 12:57:02.345826 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:57:02.345797    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-579744d899-t8mz8"]
Jul 28 12:57:02.346286 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:57:02.346098    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c66fb0e-db92-4d10-bd1e-6178c0f76cfa" containerName="console"
Jul 28 12:57:02.346286 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:57:02.346112    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c66fb0e-db92-4d10-bd1e-6178c0f76cfa" containerName="console"
Jul 28 12:57:02.346286 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:57:02.346262    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c66fb0e-db92-4d10-bd1e-6178c0f76cfa" containerName="console"
Jul 28 12:57:02.349041 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:57:02.349026    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-579744d899-t8mz8"
Jul 28 12:57:02.351822 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:57:02.351796    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 28 12:57:02.353322 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:57:02.353307    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 28 12:57:02.354997 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:57:02.354979    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 28 12:57:02.355091 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:57:02.355011    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-8pbqd\""
Jul 28 12:57:02.355224 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:57:02.355204    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 28 12:57:02.355344 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:57:02.355313    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 28 12:57:02.364260 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:57:02.364242    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 28 12:57:02.366398 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:57:02.366379    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-579744d899-t8mz8"]
Jul 28 12:57:02.455830 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:57:02.455804    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d1cbde83-9f5b-4b2b-980a-821eda44aff1-service-ca\") pod \"console-579744d899-t8mz8\" (UID: \"d1cbde83-9f5b-4b2b-980a-821eda44aff1\") " pod="openshift-console/console-579744d899-t8mz8"
Jul 28 12:57:02.455935 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:57:02.455840    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d1cbde83-9f5b-4b2b-980a-821eda44aff1-console-oauth-config\") pod \"console-579744d899-t8mz8\" (UID: \"d1cbde83-9f5b-4b2b-980a-821eda44aff1\") " pod="openshift-console/console-579744d899-t8mz8"
Jul 28 12:57:02.455935 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:57:02.455857    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d1cbde83-9f5b-4b2b-980a-821eda44aff1-console-config\") pod \"console-579744d899-t8mz8\" (UID: \"d1cbde83-9f5b-4b2b-980a-821eda44aff1\") " pod="openshift-console/console-579744d899-t8mz8"
Jul 28 12:57:02.455935 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:57:02.455877    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1cbde83-9f5b-4b2b-980a-821eda44aff1-trusted-ca-bundle\") pod \"console-579744d899-t8mz8\" (UID: \"d1cbde83-9f5b-4b2b-980a-821eda44aff1\") " pod="openshift-console/console-579744d899-t8mz8"
Jul 28 12:57:02.455935 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:57:02.455901    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d1cbde83-9f5b-4b2b-980a-821eda44aff1-oauth-serving-cert\") pod \"console-579744d899-t8mz8\" (UID: \"d1cbde83-9f5b-4b2b-980a-821eda44aff1\") " pod="openshift-console/console-579744d899-t8mz8"
Jul 28 12:57:02.455935 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:57:02.455933    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d1cbde83-9f5b-4b2b-980a-821eda44aff1-console-serving-cert\") pod \"console-579744d899-t8mz8\" (UID: \"d1cbde83-9f5b-4b2b-980a-821eda44aff1\") " pod="openshift-console/console-579744d899-t8mz8"
Jul 28 12:57:02.456105 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:57:02.455971    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2268m\" (UniqueName: \"kubernetes.io/projected/d1cbde83-9f5b-4b2b-980a-821eda44aff1-kube-api-access-2268m\") pod \"console-579744d899-t8mz8\" (UID: \"d1cbde83-9f5b-4b2b-980a-821eda44aff1\") " pod="openshift-console/console-579744d899-t8mz8"
Jul 28 12:57:02.557047 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:57:02.557016    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d1cbde83-9f5b-4b2b-980a-821eda44aff1-console-serving-cert\") pod \"console-579744d899-t8mz8\" (UID: \"d1cbde83-9f5b-4b2b-980a-821eda44aff1\") " pod="openshift-console/console-579744d899-t8mz8"
Jul 28 12:57:02.557142 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:57:02.557059    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2268m\" (UniqueName: \"kubernetes.io/projected/d1cbde83-9f5b-4b2b-980a-821eda44aff1-kube-api-access-2268m\") pod \"console-579744d899-t8mz8\" (UID: \"d1cbde83-9f5b-4b2b-980a-821eda44aff1\") " pod="openshift-console/console-579744d899-t8mz8"
Jul 28 12:57:02.557142 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:57:02.557081    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d1cbde83-9f5b-4b2b-980a-821eda44aff1-service-ca\") pod \"console-579744d899-t8mz8\" (UID: \"d1cbde83-9f5b-4b2b-980a-821eda44aff1\") " pod="openshift-console/console-579744d899-t8mz8"
Jul 28 12:57:02.557142 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:57:02.557122    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d1cbde83-9f5b-4b2b-980a-821eda44aff1-console-oauth-config\") pod \"console-579744d899-t8mz8\" (UID: \"d1cbde83-9f5b-4b2b-980a-821eda44aff1\") " pod="openshift-console/console-579744d899-t8mz8"
Jul 28 12:57:02.557274 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:57:02.557146    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d1cbde83-9f5b-4b2b-980a-821eda44aff1-console-config\") pod \"console-579744d899-t8mz8\" (UID: \"d1cbde83-9f5b-4b2b-980a-821eda44aff1\") " pod="openshift-console/console-579744d899-t8mz8"
Jul 28 12:57:02.557274 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:57:02.557176    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1cbde83-9f5b-4b2b-980a-821eda44aff1-trusted-ca-bundle\") pod \"console-579744d899-t8mz8\" (UID: \"d1cbde83-9f5b-4b2b-980a-821eda44aff1\") " pod="openshift-console/console-579744d899-t8mz8"
Jul 28 12:57:02.557274 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:57:02.557198    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d1cbde83-9f5b-4b2b-980a-821eda44aff1-oauth-serving-cert\") pod \"console-579744d899-t8mz8\" (UID: \"d1cbde83-9f5b-4b2b-980a-821eda44aff1\") " pod="openshift-console/console-579744d899-t8mz8"
Jul 28 12:57:02.557859 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:57:02.557838    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d1cbde83-9f5b-4b2b-980a-821eda44aff1-console-config\") pod \"console-579744d899-t8mz8\" (UID: \"d1cbde83-9f5b-4b2b-980a-821eda44aff1\") " pod="openshift-console/console-579744d899-t8mz8"
Jul 28 12:57:02.557967 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:57:02.557949    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d1cbde83-9f5b-4b2b-980a-821eda44aff1-service-ca\") pod \"console-579744d899-t8mz8\" (UID: \"d1cbde83-9f5b-4b2b-980a-821eda44aff1\") " pod="openshift-console/console-579744d899-t8mz8"
Jul 28 12:57:02.558111 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:57:02.558091    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d1cbde83-9f5b-4b2b-980a-821eda44aff1-oauth-serving-cert\") pod \"console-579744d899-t8mz8\" (UID: \"d1cbde83-9f5b-4b2b-980a-821eda44aff1\") " pod="openshift-console/console-579744d899-t8mz8"
Jul 28 12:57:02.558169 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:57:02.558150    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1cbde83-9f5b-4b2b-980a-821eda44aff1-trusted-ca-bundle\") pod \"console-579744d899-t8mz8\" (UID: \"d1cbde83-9f5b-4b2b-980a-821eda44aff1\") " pod="openshift-console/console-579744d899-t8mz8"
Jul 28 12:57:02.559563 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:57:02.559538    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d1cbde83-9f5b-4b2b-980a-821eda44aff1-console-serving-cert\") pod \"console-579744d899-t8mz8\" (UID: \"d1cbde83-9f5b-4b2b-980a-821eda44aff1\") " pod="openshift-console/console-579744d899-t8mz8"
Jul 28 12:57:02.559638 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:57:02.559621    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d1cbde83-9f5b-4b2b-980a-821eda44aff1-console-oauth-config\") pod \"console-579744d899-t8mz8\" (UID: \"d1cbde83-9f5b-4b2b-980a-821eda44aff1\") " pod="openshift-console/console-579744d899-t8mz8"
Jul 28 12:57:02.569182 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:57:02.569163    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2268m\" (UniqueName: \"kubernetes.io/projected/d1cbde83-9f5b-4b2b-980a-821eda44aff1-kube-api-access-2268m\") pod \"console-579744d899-t8mz8\" (UID: \"d1cbde83-9f5b-4b2b-980a-821eda44aff1\") " pod="openshift-console/console-579744d899-t8mz8"
Jul 28 12:57:02.658415 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:57:02.658358    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-579744d899-t8mz8"
Jul 28 12:57:02.783547 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:57:02.783516    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-579744d899-t8mz8"]
Jul 28 12:57:02.787081 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:57:02.787052    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1cbde83_9f5b_4b2b_980a_821eda44aff1.slice/crio-e70e5fde3aadcf28cb644e595bb4de7d0d0e712e4138fdba5424ffb3b2348d35 WatchSource:0}: Error finding container e70e5fde3aadcf28cb644e595bb4de7d0d0e712e4138fdba5424ffb3b2348d35: Status 404 returned error can't find the container with id e70e5fde3aadcf28cb644e595bb4de7d0d0e712e4138fdba5424ffb3b2348d35
Jul 28 12:57:03.261215 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:57:03.261186    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-579744d899-t8mz8" event={"ID":"d1cbde83-9f5b-4b2b-980a-821eda44aff1","Type":"ContainerStarted","Data":"c9a4593d056b816244300d9910119f2645e5fca8abb3e62fdfd010e99d5bbb59"}
Jul 28 12:57:03.261384 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:57:03.261232    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-579744d899-t8mz8" event={"ID":"d1cbde83-9f5b-4b2b-980a-821eda44aff1","Type":"ContainerStarted","Data":"e70e5fde3aadcf28cb644e595bb4de7d0d0e712e4138fdba5424ffb3b2348d35"}
Jul 28 12:57:03.282313 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:57:03.282253    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-579744d899-t8mz8" podStartSLOduration=1.282230535 podStartE2EDuration="1.282230535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:57:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:57:03.28221423 +0000 UTC m=+235.487600832" watchObservedRunningTime="2026-07-28 12:57:03.282230535 +0000 UTC m=+235.487617137"
Jul 28 12:57:12.658797 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:57:12.658765    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-579744d899-t8mz8"
Jul 28 12:57:12.659220 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:57:12.658817    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-579744d899-t8mz8"
Jul 28 12:57:12.663156 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:57:12.663136    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-579744d899-t8mz8"
Jul 28 12:57:13.300467 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:57:13.300442    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-579744d899-t8mz8"
Jul 28 12:58:08.224856 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:58:08.224832    2574 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 12:59:55.144230 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:59:55.144196    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5sxt9r"]
Jul 28 12:59:55.147668 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:59:55.147650    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5sxt9r"
Jul 28 12:59:55.150647 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:59:55.150630    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 28 12:59:55.150746 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:59:55.150642    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 28 12:59:55.150746 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:59:55.150642    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-bh6j8\""
Jul 28 12:59:55.163788 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:59:55.163768    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prw2x\" (UniqueName: \"kubernetes.io/projected/92e0e9ad-007b-4657-9933-01b4d5eff080-kube-api-access-prw2x\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5sxt9r\" (UID: \"92e0e9ad-007b-4657-9933-01b4d5eff080\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5sxt9r"
Jul 28 12:59:55.163908 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:59:55.163811    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/92e0e9ad-007b-4657-9933-01b4d5eff080-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5sxt9r\" (UID: \"92e0e9ad-007b-4657-9933-01b4d5eff080\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5sxt9r"
Jul 28 12:59:55.163908 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:59:55.163833    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/92e0e9ad-007b-4657-9933-01b4d5eff080-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5sxt9r\" (UID: \"92e0e9ad-007b-4657-9933-01b4d5eff080\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5sxt9r"
Jul 28 12:59:55.164111 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:59:55.164094    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5sxt9r"]
Jul 28 12:59:55.264337 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:59:55.264317    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/92e0e9ad-007b-4657-9933-01b4d5eff080-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5sxt9r\" (UID: \"92e0e9ad-007b-4657-9933-01b4d5eff080\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5sxt9r"
Jul 28 12:59:55.264428 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:59:55.264379    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prw2x\" (UniqueName: \"kubernetes.io/projected/92e0e9ad-007b-4657-9933-01b4d5eff080-kube-api-access-prw2x\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5sxt9r\" (UID: \"92e0e9ad-007b-4657-9933-01b4d5eff080\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5sxt9r"
Jul 28 12:59:55.264471 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:59:55.264425    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/92e0e9ad-007b-4657-9933-01b4d5eff080-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5sxt9r\" (UID: \"92e0e9ad-007b-4657-9933-01b4d5eff080\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5sxt9r"
Jul 28 12:59:55.264778 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:59:55.264756    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/92e0e9ad-007b-4657-9933-01b4d5eff080-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5sxt9r\" (UID: \"92e0e9ad-007b-4657-9933-01b4d5eff080\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5sxt9r"
Jul 28 12:59:55.264778 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:59:55.264771    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/92e0e9ad-007b-4657-9933-01b4d5eff080-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5sxt9r\" (UID: \"92e0e9ad-007b-4657-9933-01b4d5eff080\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5sxt9r"
Jul 28 12:59:55.273109 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:59:55.273065    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prw2x\" (UniqueName: \"kubernetes.io/projected/92e0e9ad-007b-4657-9933-01b4d5eff080-kube-api-access-prw2x\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5sxt9r\" (UID: \"92e0e9ad-007b-4657-9933-01b4d5eff080\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5sxt9r"
Jul 28 12:59:55.456420 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:59:55.456401    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5sxt9r"
Jul 28 12:59:55.579950 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:59:55.579918    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5sxt9r"]
Jul 28 12:59:55.582877 ip-10-0-138-158 kubenswrapper[2574]: W0728 12:59:55.582852    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92e0e9ad_007b_4657_9933_01b4d5eff080.slice/crio-43533af416f69f4d0b6eda7bc78c19409ca221dcd49834651855554b33b158ae WatchSource:0}: Error finding container 43533af416f69f4d0b6eda7bc78c19409ca221dcd49834651855554b33b158ae: Status 404 returned error can't find the container with id 43533af416f69f4d0b6eda7bc78c19409ca221dcd49834651855554b33b158ae
Jul 28 12:59:55.584842 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:59:55.584828    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 12:59:55.883297 ip-10-0-138-158 kubenswrapper[2574]: I0728 12:59:55.883226    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5sxt9r" event={"ID":"92e0e9ad-007b-4657-9933-01b4d5eff080","Type":"ContainerStarted","Data":"43533af416f69f4d0b6eda7bc78c19409ca221dcd49834651855554b33b158ae"}
Jul 28 13:00:01.911478 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:01.911443    2574 generic.go:356] "Generic (PLEG): container finished" podID="92e0e9ad-007b-4657-9933-01b4d5eff080" containerID="9e93d31752eaa38c14f5848a04a0232a90a6620a0bb6cfcc3257c1cfc2aa0207" exitCode=0
Jul 28 13:00:01.911980 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:01.911510    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5sxt9r" event={"ID":"92e0e9ad-007b-4657-9933-01b4d5eff080","Type":"ContainerDied","Data":"9e93d31752eaa38c14f5848a04a0232a90a6620a0bb6cfcc3257c1cfc2aa0207"}
Jul 28 13:00:06.933367 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:06.933335    2574 generic.go:356] "Generic (PLEG): container finished" podID="92e0e9ad-007b-4657-9933-01b4d5eff080" containerID="4af2a7534bc829f52d1196380bacca0c9d26af169b0693cf245b4fd43cf4b162" exitCode=0
Jul 28 13:00:06.933795 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:06.933400    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5sxt9r" event={"ID":"92e0e9ad-007b-4657-9933-01b4d5eff080","Type":"ContainerDied","Data":"4af2a7534bc829f52d1196380bacca0c9d26af169b0693cf245b4fd43cf4b162"}
Jul 28 13:00:13.966842 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:13.966811    2574 generic.go:356] "Generic (PLEG): container finished" podID="92e0e9ad-007b-4657-9933-01b4d5eff080" containerID="71c942d9bc10d09c40e831117d186b94f7da34120c6eee07fe452ed3821d7173" exitCode=0
Jul 28 13:00:13.967231 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:13.966879    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5sxt9r" event={"ID":"92e0e9ad-007b-4657-9933-01b4d5eff080","Type":"ContainerDied","Data":"71c942d9bc10d09c40e831117d186b94f7da34120c6eee07fe452ed3821d7173"}
Jul 28 13:00:15.087958 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:15.087930    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5sxt9r"
Jul 28 13:00:15.227426 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:15.227369    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-prw2x\" (UniqueName: \"kubernetes.io/projected/92e0e9ad-007b-4657-9933-01b4d5eff080-kube-api-access-prw2x\") pod \"92e0e9ad-007b-4657-9933-01b4d5eff080\" (UID: \"92e0e9ad-007b-4657-9933-01b4d5eff080\") "
Jul 28 13:00:15.227426 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:15.227409    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/92e0e9ad-007b-4657-9933-01b4d5eff080-util\") pod \"92e0e9ad-007b-4657-9933-01b4d5eff080\" (UID: \"92e0e9ad-007b-4657-9933-01b4d5eff080\") "
Jul 28 13:00:15.227554 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:15.227456    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/92e0e9ad-007b-4657-9933-01b4d5eff080-bundle\") pod \"92e0e9ad-007b-4657-9933-01b4d5eff080\" (UID: \"92e0e9ad-007b-4657-9933-01b4d5eff080\") "
Jul 28 13:00:15.228169 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:15.228138    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92e0e9ad-007b-4657-9933-01b4d5eff080-bundle" (OuterVolumeSpecName: "bundle") pod "92e0e9ad-007b-4657-9933-01b4d5eff080" (UID: "92e0e9ad-007b-4657-9933-01b4d5eff080"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:00:15.229512 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:15.229483    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92e0e9ad-007b-4657-9933-01b4d5eff080-kube-api-access-prw2x" (OuterVolumeSpecName: "kube-api-access-prw2x") pod "92e0e9ad-007b-4657-9933-01b4d5eff080" (UID: "92e0e9ad-007b-4657-9933-01b4d5eff080"). InnerVolumeSpecName "kube-api-access-prw2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:00:15.231400 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:15.231379    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92e0e9ad-007b-4657-9933-01b4d5eff080-util" (OuterVolumeSpecName: "util") pod "92e0e9ad-007b-4657-9933-01b4d5eff080" (UID: "92e0e9ad-007b-4657-9933-01b4d5eff080"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:00:15.328395 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:15.328371    2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/92e0e9ad-007b-4657-9933-01b4d5eff080-bundle\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:00:15.328395 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:15.328393    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prw2x\" (UniqueName: \"kubernetes.io/projected/92e0e9ad-007b-4657-9933-01b4d5eff080-kube-api-access-prw2x\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:00:15.328522 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:15.328406    2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/92e0e9ad-007b-4657-9933-01b4d5eff080-util\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:00:15.976556 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:15.976527    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5sxt9r"
Jul 28 13:00:15.976770 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:15.976532    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5sxt9r" event={"ID":"92e0e9ad-007b-4657-9933-01b4d5eff080","Type":"ContainerDied","Data":"43533af416f69f4d0b6eda7bc78c19409ca221dcd49834651855554b33b158ae"}
Jul 28 13:00:15.976770 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:15.976644    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43533af416f69f4d0b6eda7bc78c19409ca221dcd49834651855554b33b158ae"
Jul 28 13:00:23.050644 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:23.050609    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-shvpt"]
Jul 28 13:00:23.051109 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:23.050905    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92e0e9ad-007b-4657-9933-01b4d5eff080" containerName="pull"
Jul 28 13:00:23.051109 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:23.050915    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="92e0e9ad-007b-4657-9933-01b4d5eff080" containerName="pull"
Jul 28 13:00:23.051109 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:23.050925    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92e0e9ad-007b-4657-9933-01b4d5eff080" containerName="util"
Jul 28 13:00:23.051109 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:23.050930    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="92e0e9ad-007b-4657-9933-01b4d5eff080" containerName="util"
Jul 28 13:00:23.051109 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:23.050945    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92e0e9ad-007b-4657-9933-01b4d5eff080" containerName="extract"
Jul 28 13:00:23.051109 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:23.050951    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="92e0e9ad-007b-4657-9933-01b4d5eff080" containerName="extract"
Jul 28 13:00:23.051109 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:23.051001    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="92e0e9ad-007b-4657-9933-01b4d5eff080" containerName="extract"
Jul 28 13:00:23.054360 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:23.054340    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-shvpt"
Jul 28 13:00:23.057312 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:23.057286    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 28 13:00:23.057453 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:23.057395    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-4t9zk\""
Jul 28 13:00:23.057523 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:23.057460    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 28 13:00:23.057633 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:23.057617    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 28 13:00:23.068332 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:23.068312    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-shvpt"]
Jul 28 13:00:23.186647 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:23.186620    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/11a16ae5-d1f7-411c-9abc-478eccc8d82b-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-shvpt\" (UID: \"11a16ae5-d1f7-411c-9abc-478eccc8d82b\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-shvpt"
Jul 28 13:00:23.186775 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:23.186666    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58qql\" (UniqueName: \"kubernetes.io/projected/11a16ae5-d1f7-411c-9abc-478eccc8d82b-kube-api-access-58qql\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-shvpt\" (UID: \"11a16ae5-d1f7-411c-9abc-478eccc8d82b\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-shvpt"
Jul 28 13:00:23.287870 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:23.287847    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58qql\" (UniqueName: \"kubernetes.io/projected/11a16ae5-d1f7-411c-9abc-478eccc8d82b-kube-api-access-58qql\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-shvpt\" (UID: \"11a16ae5-d1f7-411c-9abc-478eccc8d82b\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-shvpt"
Jul 28 13:00:23.287952 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:23.287895    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/11a16ae5-d1f7-411c-9abc-478eccc8d82b-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-shvpt\" (UID: \"11a16ae5-d1f7-411c-9abc-478eccc8d82b\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-shvpt"
Jul 28 13:00:23.290385 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:23.290365    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/11a16ae5-d1f7-411c-9abc-478eccc8d82b-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-shvpt\" (UID: \"11a16ae5-d1f7-411c-9abc-478eccc8d82b\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-shvpt"
Jul 28 13:00:23.296361 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:23.296343    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58qql\" (UniqueName: \"kubernetes.io/projected/11a16ae5-d1f7-411c-9abc-478eccc8d82b-kube-api-access-58qql\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-shvpt\" (UID: \"11a16ae5-d1f7-411c-9abc-478eccc8d82b\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-shvpt"
Jul 28 13:00:23.363857 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:23.363808    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-shvpt"
Jul 28 13:00:23.491102 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:23.491076    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-shvpt"]
Jul 28 13:00:23.493742 ip-10-0-138-158 kubenswrapper[2574]: W0728 13:00:23.493715    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11a16ae5_d1f7_411c_9abc_478eccc8d82b.slice/crio-e86ca46fba32d7ca37e23bfa816040b052ef5fa99515341b4cc5e475f374b19a WatchSource:0}: Error finding container e86ca46fba32d7ca37e23bfa816040b052ef5fa99515341b4cc5e475f374b19a: Status 404 returned error can't find the container with id e86ca46fba32d7ca37e23bfa816040b052ef5fa99515341b4cc5e475f374b19a
Jul 28 13:00:24.008557 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:24.008528    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-shvpt" event={"ID":"11a16ae5-d1f7-411c-9abc-478eccc8d82b","Type":"ContainerStarted","Data":"e86ca46fba32d7ca37e23bfa816040b052ef5fa99515341b4cc5e475f374b19a"}
Jul 28 13:00:27.024389 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:27.024355    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-shvpt" event={"ID":"11a16ae5-d1f7-411c-9abc-478eccc8d82b","Type":"ContainerStarted","Data":"017b9bc383b8198ec4276a6d2ab32942f0f1857c23952f6ece1591e1a72f431c"}
Jul 28 13:00:27.024761 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:27.024470    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-shvpt"
Jul 28 13:00:27.048067 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:27.048011    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-shvpt" podStartSLOduration=1.079158259 podStartE2EDuration="4.047997225s" totalImagesPullingTime="2.968838966s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:00:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:00:27.046312909 +0000 UTC m=+439.251699510" watchObservedRunningTime="2026-07-28 13:00:27.047997225 +0000 UTC m=+439.253383826"
Jul 28 13:00:27.350935 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:27.350858    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-77f648d694-5rzk6"]
Jul 28 13:00:27.353013 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:27.352995    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-5rzk6"
Jul 28 13:00:27.355853 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:27.355831    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jul 28 13:00:27.356000 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:27.355899    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-8c2d6\""
Jul 28 13:00:27.356077 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:27.356063    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 28 13:00:27.363369 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:27.363346    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-5rzk6"]
Jul 28 13:00:27.422217 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:27.422182    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sjn5\" (UniqueName: \"kubernetes.io/projected/49eabdeb-bfe0-4b55-b413-5eacd8fffb16-kube-api-access-5sjn5\") pod \"keda-operator-77f648d694-5rzk6\" (UID: \"49eabdeb-bfe0-4b55-b413-5eacd8fffb16\") " pod="openshift-keda/keda-operator-77f648d694-5rzk6"
Jul 28 13:00:27.422354 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:27.422234    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/49eabdeb-bfe0-4b55-b413-5eacd8fffb16-cabundle0\") pod \"keda-operator-77f648d694-5rzk6\" (UID: \"49eabdeb-bfe0-4b55-b413-5eacd8fffb16\") " pod="openshift-keda/keda-operator-77f648d694-5rzk6"
Jul 28 13:00:27.422354 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:27.422317    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/49eabdeb-bfe0-4b55-b413-5eacd8fffb16-certificates\") pod \"keda-operator-77f648d694-5rzk6\" (UID: \"49eabdeb-bfe0-4b55-b413-5eacd8fffb16\") " pod="openshift-keda/keda-operator-77f648d694-5rzk6"
Jul 28 13:00:27.523104 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:27.523078    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/49eabdeb-bfe0-4b55-b413-5eacd8fffb16-cabundle0\") pod \"keda-operator-77f648d694-5rzk6\" (UID: \"49eabdeb-bfe0-4b55-b413-5eacd8fffb16\") " pod="openshift-keda/keda-operator-77f648d694-5rzk6"
Jul 28 13:00:27.523318 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:27.523129    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/49eabdeb-bfe0-4b55-b413-5eacd8fffb16-certificates\") pod \"keda-operator-77f648d694-5rzk6\" (UID: \"49eabdeb-bfe0-4b55-b413-5eacd8fffb16\") " pod="openshift-keda/keda-operator-77f648d694-5rzk6"
Jul 28 13:00:27.523318 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:27.523199    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sjn5\" (UniqueName: \"kubernetes.io/projected/49eabdeb-bfe0-4b55-b413-5eacd8fffb16-kube-api-access-5sjn5\") pod \"keda-operator-77f648d694-5rzk6\" (UID: \"49eabdeb-bfe0-4b55-b413-5eacd8fffb16\") " pod="openshift-keda/keda-operator-77f648d694-5rzk6"
Jul 28 13:00:27.523431 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:00:27.523328    2574 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Jul 28 13:00:27.523431 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:00:27.523354    2574 secret.go:281] references non-existent secret key: ca.crt
Jul 28 13:00:27.523431 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:00:27.523366    2574 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 28 13:00:27.523431 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:00:27.523383    2574 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-5rzk6: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 28 13:00:27.523601 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:00:27.523455    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49eabdeb-bfe0-4b55-b413-5eacd8fffb16-certificates podName:49eabdeb-bfe0-4b55-b413-5eacd8fffb16 nodeName:}" failed. No retries permitted until 2026-07-28 13:00:28.023434649 +0000 UTC m=+440.228821232 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/49eabdeb-bfe0-4b55-b413-5eacd8fffb16-certificates") pod "keda-operator-77f648d694-5rzk6" (UID: "49eabdeb-bfe0-4b55-b413-5eacd8fffb16") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 28 13:00:27.523738 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:27.523721    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/49eabdeb-bfe0-4b55-b413-5eacd8fffb16-cabundle0\") pod \"keda-operator-77f648d694-5rzk6\" (UID: \"49eabdeb-bfe0-4b55-b413-5eacd8fffb16\") " pod="openshift-keda/keda-operator-77f648d694-5rzk6"
Jul 28 13:00:27.532992 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:27.532973    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sjn5\" (UniqueName: \"kubernetes.io/projected/49eabdeb-bfe0-4b55-b413-5eacd8fffb16-kube-api-access-5sjn5\") pod \"keda-operator-77f648d694-5rzk6\" (UID: \"49eabdeb-bfe0-4b55-b413-5eacd8fffb16\") " pod="openshift-keda/keda-operator-77f648d694-5rzk6"
Jul 28 13:00:27.565852 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:27.565826    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-nd4fg"]
Jul 28 13:00:27.567896 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:27.567878    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-nd4fg"
Jul 28 13:00:27.570994 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:27.570977    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Jul 28 13:00:27.581315 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:27.580792    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-nd4fg"]
Jul 28 13:00:27.724523 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:27.724502    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/bec07b2c-991b-4017-b28c-f0c32dcd0c81-certificates\") pod \"keda-metrics-apiserver-55557df8c-nd4fg\" (UID: \"bec07b2c-991b-4017-b28c-f0c32dcd0c81\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-nd4fg"
Jul 28 13:00:27.724630 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:27.724542    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd22w\" (UniqueName: \"kubernetes.io/projected/bec07b2c-991b-4017-b28c-f0c32dcd0c81-kube-api-access-bd22w\") pod \"keda-metrics-apiserver-55557df8c-nd4fg\" (UID: \"bec07b2c-991b-4017-b28c-f0c32dcd0c81\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-nd4fg"
Jul 28 13:00:27.724630 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:27.724571    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/bec07b2c-991b-4017-b28c-f0c32dcd0c81-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-nd4fg\" (UID: \"bec07b2c-991b-4017-b28c-f0c32dcd0c81\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-nd4fg"
Jul 28 13:00:27.825538 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:27.825515    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/bec07b2c-991b-4017-b28c-f0c32dcd0c81-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-nd4fg\" (UID: \"bec07b2c-991b-4017-b28c-f0c32dcd0c81\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-nd4fg"
Jul 28 13:00:27.825639 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:27.825580    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/bec07b2c-991b-4017-b28c-f0c32dcd0c81-certificates\") pod \"keda-metrics-apiserver-55557df8c-nd4fg\" (UID: \"bec07b2c-991b-4017-b28c-f0c32dcd0c81\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-nd4fg"
Jul 28 13:00:27.825639 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:27.825614    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd22w\" (UniqueName: \"kubernetes.io/projected/bec07b2c-991b-4017-b28c-f0c32dcd0c81-kube-api-access-bd22w\") pod \"keda-metrics-apiserver-55557df8c-nd4fg\" (UID: \"bec07b2c-991b-4017-b28c-f0c32dcd0c81\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-nd4fg"
Jul 28 13:00:27.825753 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:00:27.825655    2574 secret.go:281] references non-existent secret key: tls.crt
Jul 28 13:00:27.825753 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:00:27.825672    2574 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Jul 28 13:00:27.825753 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:00:27.825706    2574 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-55557df8c-nd4fg: references non-existent secret key: tls.crt
Jul 28 13:00:27.825753 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:00:27.825752    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bec07b2c-991b-4017-b28c-f0c32dcd0c81-certificates podName:bec07b2c-991b-4017-b28c-f0c32dcd0c81 nodeName:}" failed. No retries permitted until 2026-07-28 13:00:28.325738169 +0000 UTC m=+440.531124748 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/bec07b2c-991b-4017-b28c-f0c32dcd0c81-certificates") pod "keda-metrics-apiserver-55557df8c-nd4fg" (UID: "bec07b2c-991b-4017-b28c-f0c32dcd0c81") : references non-existent secret key: tls.crt
Jul 28 13:00:27.825894 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:27.825873    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/bec07b2c-991b-4017-b28c-f0c32dcd0c81-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-nd4fg\" (UID: \"bec07b2c-991b-4017-b28c-f0c32dcd0c81\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-nd4fg"
Jul 28 13:00:27.834146 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:27.834127    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd22w\" (UniqueName: \"kubernetes.io/projected/bec07b2c-991b-4017-b28c-f0c32dcd0c81-kube-api-access-bd22w\") pod \"keda-metrics-apiserver-55557df8c-nd4fg\" (UID: \"bec07b2c-991b-4017-b28c-f0c32dcd0c81\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-nd4fg"
Jul 28 13:00:28.027906 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:28.027821    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/49eabdeb-bfe0-4b55-b413-5eacd8fffb16-certificates\") pod \"keda-operator-77f648d694-5rzk6\" (UID: \"49eabdeb-bfe0-4b55-b413-5eacd8fffb16\") " pod="openshift-keda/keda-operator-77f648d694-5rzk6"
Jul 28 13:00:28.028309 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:00:28.027963    2574 secret.go:281] references non-existent secret key: ca.crt
Jul 28 13:00:28.028309 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:00:28.027978    2574 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 28 13:00:28.028309 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:00:28.027987    2574 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-5rzk6: references non-existent secret key: ca.crt
Jul 28 13:00:28.028309 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:00:28.028048    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49eabdeb-bfe0-4b55-b413-5eacd8fffb16-certificates podName:49eabdeb-bfe0-4b55-b413-5eacd8fffb16 nodeName:}" failed. No retries permitted until 2026-07-28 13:00:29.028032022 +0000 UTC m=+441.233418602 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/49eabdeb-bfe0-4b55-b413-5eacd8fffb16-certificates") pod "keda-operator-77f648d694-5rzk6" (UID: "49eabdeb-bfe0-4b55-b413-5eacd8fffb16") : references non-existent secret key: ca.crt
Jul 28 13:00:28.329964 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:28.329884    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/bec07b2c-991b-4017-b28c-f0c32dcd0c81-certificates\") pod \"keda-metrics-apiserver-55557df8c-nd4fg\" (UID: \"bec07b2c-991b-4017-b28c-f0c32dcd0c81\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-nd4fg"
Jul 28 13:00:28.332546 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:28.332522    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/bec07b2c-991b-4017-b28c-f0c32dcd0c81-certificates\") pod \"keda-metrics-apiserver-55557df8c-nd4fg\" (UID: \"bec07b2c-991b-4017-b28c-f0c32dcd0c81\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-nd4fg"
Jul 28 13:00:28.478699 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:28.478652    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-nd4fg"
Jul 28 13:00:28.627409 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:28.627382    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-nd4fg"]
Jul 28 13:00:29.034165 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:29.034127    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-nd4fg" event={"ID":"bec07b2c-991b-4017-b28c-f0c32dcd0c81","Type":"ContainerStarted","Data":"a3415fb6255ee26e3cc10b83c1bdc5cc9a73d7ebaa4847113ffffc74c4a734dc"}
Jul 28 13:00:29.035451 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:29.035433    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/49eabdeb-bfe0-4b55-b413-5eacd8fffb16-certificates\") pod \"keda-operator-77f648d694-5rzk6\" (UID: \"49eabdeb-bfe0-4b55-b413-5eacd8fffb16\") " pod="openshift-keda/keda-operator-77f648d694-5rzk6"
Jul 28 13:00:29.037802 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:29.037782    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/49eabdeb-bfe0-4b55-b413-5eacd8fffb16-certificates\") pod \"keda-operator-77f648d694-5rzk6\" (UID: \"49eabdeb-bfe0-4b55-b413-5eacd8fffb16\") " pod="openshift-keda/keda-operator-77f648d694-5rzk6"
Jul 28 13:00:29.163954 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:29.163930    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-5rzk6"
Jul 28 13:00:29.310502 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:29.310478    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-5rzk6"]
Jul 28 13:00:29.312976 ip-10-0-138-158 kubenswrapper[2574]: W0728 13:00:29.312947    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49eabdeb_bfe0_4b55_b413_5eacd8fffb16.slice/crio-7da4bd1d3a1407984a3864ec741278946a5ba50885b480728b071f9e0140237a WatchSource:0}: Error finding container 7da4bd1d3a1407984a3864ec741278946a5ba50885b480728b071f9e0140237a: Status 404 returned error can't find the container with id 7da4bd1d3a1407984a3864ec741278946a5ba50885b480728b071f9e0140237a
Jul 28 13:00:30.044730 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:30.044696    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-5rzk6" event={"ID":"49eabdeb-bfe0-4b55-b413-5eacd8fffb16","Type":"ContainerStarted","Data":"7da4bd1d3a1407984a3864ec741278946a5ba50885b480728b071f9e0140237a"}
Jul 28 13:00:33.061668 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:33.061627    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-5rzk6" event={"ID":"49eabdeb-bfe0-4b55-b413-5eacd8fffb16","Type":"ContainerStarted","Data":"d4ce9abe2c93f5e9134cffee9c18c40c15df1c33623033587e59ce1c4e8a0b4b"}
Jul 28 13:00:33.062141 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:33.061762    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-77f648d694-5rzk6"
Jul 28 13:00:33.081642 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:33.081589    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-77f648d694-5rzk6" podStartSLOduration=2.799320552 podStartE2EDuration="6.081572704s" totalImagesPullingTime="3.282252152s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:00:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:00:33.080809841 +0000 UTC m=+445.286196450" watchObservedRunningTime="2026-07-28 13:00:33.081572704 +0000 UTC m=+445.286959306"
Jul 28 13:00:35.072481 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:35.072447    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-nd4fg" event={"ID":"bec07b2c-991b-4017-b28c-f0c32dcd0c81","Type":"ContainerStarted","Data":"4aff8922f1e801bd8de94a357962d770f9d7faac4d74fcf15812d967e6d89f6d"}
Jul 28 13:00:35.072860 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:35.072581    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-nd4fg"
Jul 28 13:00:35.089335 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:35.089290    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-55557df8c-nd4fg" podStartSLOduration=2.34010429 podStartE2EDuration="8.089275052s" totalImagesPullingTime="5.749170762s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:00:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:00:35.088494167 +0000 UTC m=+447.293880791" watchObservedRunningTime="2026-07-28 13:00:35.089275052 +0000 UTC m=+447.294661653"
Jul 28 13:00:46.081573 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:46.081546    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-nd4fg"
Jul 28 13:00:48.030527 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:48.030502    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-shvpt"
Jul 28 13:00:54.069199 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:54.069169    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-77f648d694-5rzk6"
Jul 28 13:00:59.613541 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:00:59.613510    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-579744d899-t8mz8"]
Jul 28 13:01:24.631894 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:01:24.631856    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-579744d899-t8mz8" podUID="d1cbde83-9f5b-4b2b-980a-821eda44aff1" containerName="console" containerID="cri-o://c9a4593d056b816244300d9910119f2645e5fca8abb3e62fdfd010e99d5bbb59" gracePeriod=15
Jul 28 13:01:24.871720 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:01:24.871699    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-579744d899-t8mz8"
Jul 28 13:01:24.934602 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:01:24.934576    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d1cbde83-9f5b-4b2b-980a-821eda44aff1-service-ca\") pod \"d1cbde83-9f5b-4b2b-980a-821eda44aff1\" (UID: \"d1cbde83-9f5b-4b2b-980a-821eda44aff1\") "
Jul 28 13:01:24.934756 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:01:24.934611    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d1cbde83-9f5b-4b2b-980a-821eda44aff1-oauth-serving-cert\") pod \"d1cbde83-9f5b-4b2b-980a-821eda44aff1\" (UID: \"d1cbde83-9f5b-4b2b-980a-821eda44aff1\") "
Jul 28 13:01:24.934756 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:01:24.934629    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2268m\" (UniqueName: \"kubernetes.io/projected/d1cbde83-9f5b-4b2b-980a-821eda44aff1-kube-api-access-2268m\") pod \"d1cbde83-9f5b-4b2b-980a-821eda44aff1\" (UID: \"d1cbde83-9f5b-4b2b-980a-821eda44aff1\") "
Jul 28 13:01:24.934756 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:01:24.934656    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d1cbde83-9f5b-4b2b-980a-821eda44aff1-console-config\") pod \"d1cbde83-9f5b-4b2b-980a-821eda44aff1\" (UID: \"d1cbde83-9f5b-4b2b-980a-821eda44aff1\") "
Jul 28 13:01:24.934932 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:01:24.934814    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d1cbde83-9f5b-4b2b-980a-821eda44aff1-console-serving-cert\") pod \"d1cbde83-9f5b-4b2b-980a-821eda44aff1\" (UID: \"d1cbde83-9f5b-4b2b-980a-821eda44aff1\") "
Jul 28 13:01:24.934932 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:01:24.934866    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d1cbde83-9f5b-4b2b-980a-821eda44aff1-console-oauth-config\") pod \"d1cbde83-9f5b-4b2b-980a-821eda44aff1\" (UID: \"d1cbde83-9f5b-4b2b-980a-821eda44aff1\") "
Jul 28 13:01:24.934932 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:01:24.934926    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1cbde83-9f5b-4b2b-980a-821eda44aff1-trusted-ca-bundle\") pod \"d1cbde83-9f5b-4b2b-980a-821eda44aff1\" (UID: \"d1cbde83-9f5b-4b2b-980a-821eda44aff1\") "
Jul 28 13:01:24.935296 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:01:24.935274    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1cbde83-9f5b-4b2b-980a-821eda44aff1-console-config" (OuterVolumeSpecName: "console-config") pod "d1cbde83-9f5b-4b2b-980a-821eda44aff1" (UID: "d1cbde83-9f5b-4b2b-980a-821eda44aff1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:01:24.935416 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:01:24.935291    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1cbde83-9f5b-4b2b-980a-821eda44aff1-service-ca" (OuterVolumeSpecName: "service-ca") pod "d1cbde83-9f5b-4b2b-980a-821eda44aff1" (UID: "d1cbde83-9f5b-4b2b-980a-821eda44aff1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:01:24.935416 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:01:24.935320    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1cbde83-9f5b-4b2b-980a-821eda44aff1-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d1cbde83-9f5b-4b2b-980a-821eda44aff1" (UID: "d1cbde83-9f5b-4b2b-980a-821eda44aff1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:01:24.935517 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:01:24.935451    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1cbde83-9f5b-4b2b-980a-821eda44aff1-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d1cbde83-9f5b-4b2b-980a-821eda44aff1" (UID: "d1cbde83-9f5b-4b2b-980a-821eda44aff1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:01:24.936899 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:01:24.936876    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1cbde83-9f5b-4b2b-980a-821eda44aff1-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d1cbde83-9f5b-4b2b-980a-821eda44aff1" (UID: "d1cbde83-9f5b-4b2b-980a-821eda44aff1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:01:24.937180 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:01:24.937157    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1cbde83-9f5b-4b2b-980a-821eda44aff1-kube-api-access-2268m" (OuterVolumeSpecName: "kube-api-access-2268m") pod "d1cbde83-9f5b-4b2b-980a-821eda44aff1" (UID: "d1cbde83-9f5b-4b2b-980a-821eda44aff1"). InnerVolumeSpecName "kube-api-access-2268m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:01:24.937260 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:01:24.937171    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1cbde83-9f5b-4b2b-980a-821eda44aff1-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d1cbde83-9f5b-4b2b-980a-821eda44aff1" (UID: "d1cbde83-9f5b-4b2b-980a-821eda44aff1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:01:25.036194 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:01:25.036172    2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d1cbde83-9f5b-4b2b-980a-821eda44aff1-service-ca\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:01:25.036194 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:01:25.036192    2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d1cbde83-9f5b-4b2b-980a-821eda44aff1-oauth-serving-cert\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:01:25.036317 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:01:25.036203    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2268m\" (UniqueName: \"kubernetes.io/projected/d1cbde83-9f5b-4b2b-980a-821eda44aff1-kube-api-access-2268m\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:01:25.036317 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:01:25.036212    2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d1cbde83-9f5b-4b2b-980a-821eda44aff1-console-config\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:01:25.036317 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:01:25.036222    2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d1cbde83-9f5b-4b2b-980a-821eda44aff1-console-serving-cert\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:01:25.036317 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:01:25.036230    2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d1cbde83-9f5b-4b2b-980a-821eda44aff1-console-oauth-config\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:01:25.036317 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:01:25.036238    2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1cbde83-9f5b-4b2b-980a-821eda44aff1-trusted-ca-bundle\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:01:25.285679 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:01:25.285620    2574 generic.go:356] "Generic (PLEG): container finished" podID="d1cbde83-9f5b-4b2b-980a-821eda44aff1" containerID="c9a4593d056b816244300d9910119f2645e5fca8abb3e62fdfd010e99d5bbb59" exitCode=0
Jul 28 13:01:25.285679 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:01:25.285660    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-579744d899-t8mz8"
Jul 28 13:01:25.285828 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:01:25.285707    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-579744d899-t8mz8" event={"ID":"d1cbde83-9f5b-4b2b-980a-821eda44aff1","Type":"ContainerDied","Data":"c9a4593d056b816244300d9910119f2645e5fca8abb3e62fdfd010e99d5bbb59"}
Jul 28 13:01:25.285828 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:01:25.285755    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-579744d899-t8mz8" event={"ID":"d1cbde83-9f5b-4b2b-980a-821eda44aff1","Type":"ContainerDied","Data":"e70e5fde3aadcf28cb644e595bb4de7d0d0e712e4138fdba5424ffb3b2348d35"}
Jul 28 13:01:25.285828 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:01:25.285777    2574 scope.go:117] "RemoveContainer" containerID="c9a4593d056b816244300d9910119f2645e5fca8abb3e62fdfd010e99d5bbb59"
Jul 28 13:01:25.294384 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:01:25.294367    2574 scope.go:117] "RemoveContainer" containerID="c9a4593d056b816244300d9910119f2645e5fca8abb3e62fdfd010e99d5bbb59"
Jul 28 13:01:25.294616 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:01:25.294597    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9a4593d056b816244300d9910119f2645e5fca8abb3e62fdfd010e99d5bbb59\": container with ID starting with c9a4593d056b816244300d9910119f2645e5fca8abb3e62fdfd010e99d5bbb59 not found: ID does not exist" containerID="c9a4593d056b816244300d9910119f2645e5fca8abb3e62fdfd010e99d5bbb59"
Jul 28 13:01:25.294675 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:01:25.294621    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9a4593d056b816244300d9910119f2645e5fca8abb3e62fdfd010e99d5bbb59"} err="failed to get container status \"c9a4593d056b816244300d9910119f2645e5fca8abb3e62fdfd010e99d5bbb59\": rpc error: code = NotFound desc = could not find container \"c9a4593d056b816244300d9910119f2645e5fca8abb3e62fdfd010e99d5bbb59\": container with ID starting with c9a4593d056b816244300d9910119f2645e5fca8abb3e62fdfd010e99d5bbb59 not found: ID does not exist"
Jul 28 13:01:25.305964 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:01:25.305945    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-579744d899-t8mz8"]
Jul 28 13:01:25.310048 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:01:25.310030    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-579744d899-t8mz8"]
Jul 28 13:01:26.301786 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:01:26.301755    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1cbde83-9f5b-4b2b-980a-821eda44aff1" path="/var/lib/kubelet/pods/d1cbde83-9f5b-4b2b-980a-821eda44aff1/volumes"
Jul 28 13:01:50.208761 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:01:50.208727    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-84fc959947-hwnfs"]
Jul 28 13:01:50.210145 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:01:50.210117    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1cbde83-9f5b-4b2b-980a-821eda44aff1" containerName="console"
Jul 28 13:01:50.210316 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:01:50.210302    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1cbde83-9f5b-4b2b-980a-821eda44aff1" containerName="console"
Jul 28 13:01:50.210614 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:01:50.210586    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1cbde83-9f5b-4b2b-980a-821eda44aff1" containerName="console"
Jul 28 13:01:50.214334 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:01:50.214315    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-84fc959947-hwnfs"
Jul 28 13:01:50.217127 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:01:50.217107    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 28 13:01:50.217650 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:01:50.217637    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 28 13:01:50.218820 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:01:50.218804    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-cgqw4\""
Jul 28 13:01:50.218918 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:01:50.218809    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 28 13:01:50.234475 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:01:50.234456    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-84fc959947-hwnfs"]
Jul 28 13:01:50.322040 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:01:50.322010    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/264cd878-2814-4ff8-9a3e-3d8f40e01535-cert\") pod \"llmisvc-controller-manager-84fc959947-hwnfs\" (UID: \"264cd878-2814-4ff8-9a3e-3d8f40e01535\") " pod="kserve/llmisvc-controller-manager-84fc959947-hwnfs"
Jul 28 13:01:50.322158 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:01:50.322071    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fnrj\" (UniqueName: \"kubernetes.io/projected/264cd878-2814-4ff8-9a3e-3d8f40e01535-kube-api-access-4fnrj\") pod \"llmisvc-controller-manager-84fc959947-hwnfs\" (UID: \"264cd878-2814-4ff8-9a3e-3d8f40e01535\") " pod="kserve/llmisvc-controller-manager-84fc959947-hwnfs"
Jul 28 13:01:50.422862 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:01:50.422838    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/264cd878-2814-4ff8-9a3e-3d8f40e01535-cert\") pod \"llmisvc-controller-manager-84fc959947-hwnfs\" (UID: \"264cd878-2814-4ff8-9a3e-3d8f40e01535\") " pod="kserve/llmisvc-controller-manager-84fc959947-hwnfs"
Jul 28 13:01:50.422967 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:01:50.422908    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fnrj\" (UniqueName: \"kubernetes.io/projected/264cd878-2814-4ff8-9a3e-3d8f40e01535-kube-api-access-4fnrj\") pod \"llmisvc-controller-manager-84fc959947-hwnfs\" (UID: \"264cd878-2814-4ff8-9a3e-3d8f40e01535\") " pod="kserve/llmisvc-controller-manager-84fc959947-hwnfs"
Jul 28 13:01:50.423034 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:01:50.422973    2574 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 28 13:01:50.423091 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:01:50.423035    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/264cd878-2814-4ff8-9a3e-3d8f40e01535-cert podName:264cd878-2814-4ff8-9a3e-3d8f40e01535 nodeName:}" failed. No retries permitted until 2026-07-28 13:01:50.923016526 +0000 UTC m=+523.128403107 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/264cd878-2814-4ff8-9a3e-3d8f40e01535-cert") pod "llmisvc-controller-manager-84fc959947-hwnfs" (UID: "264cd878-2814-4ff8-9a3e-3d8f40e01535") : secret "llmisvc-webhook-server-cert" not found
Jul 28 13:01:50.464370 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:01:50.464316    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fnrj\" (UniqueName: \"kubernetes.io/projected/264cd878-2814-4ff8-9a3e-3d8f40e01535-kube-api-access-4fnrj\") pod \"llmisvc-controller-manager-84fc959947-hwnfs\" (UID: \"264cd878-2814-4ff8-9a3e-3d8f40e01535\") " pod="kserve/llmisvc-controller-manager-84fc959947-hwnfs"
Jul 28 13:01:50.927028 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:01:50.926956    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/264cd878-2814-4ff8-9a3e-3d8f40e01535-cert\") pod \"llmisvc-controller-manager-84fc959947-hwnfs\" (UID: \"264cd878-2814-4ff8-9a3e-3d8f40e01535\") " pod="kserve/llmisvc-controller-manager-84fc959947-hwnfs"
Jul 28 13:01:50.929313 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:01:50.929292    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/264cd878-2814-4ff8-9a3e-3d8f40e01535-cert\") pod \"llmisvc-controller-manager-84fc959947-hwnfs\" (UID: \"264cd878-2814-4ff8-9a3e-3d8f40e01535\") " pod="kserve/llmisvc-controller-manager-84fc959947-hwnfs"
Jul 28 13:01:51.123841 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:01:51.123817    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-84fc959947-hwnfs"
Jul 28 13:01:51.266837 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:01:51.266814    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-84fc959947-hwnfs"]
Jul 28 13:01:51.269134 ip-10-0-138-158 kubenswrapper[2574]: W0728 13:01:51.269107    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod264cd878_2814_4ff8_9a3e_3d8f40e01535.slice/crio-49d66af9c749c9d21ffff294dbd52969ceb18fd31a33008543bc8d662883a536 WatchSource:0}: Error finding container 49d66af9c749c9d21ffff294dbd52969ceb18fd31a33008543bc8d662883a536: Status 404 returned error can't find the container with id 49d66af9c749c9d21ffff294dbd52969ceb18fd31a33008543bc8d662883a536
Jul 28 13:01:51.393352 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:01:51.393325    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-84fc959947-hwnfs" event={"ID":"264cd878-2814-4ff8-9a3e-3d8f40e01535","Type":"ContainerStarted","Data":"49d66af9c749c9d21ffff294dbd52969ceb18fd31a33008543bc8d662883a536"}
Jul 28 13:01:55.411718 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:01:55.411671    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-84fc959947-hwnfs" event={"ID":"264cd878-2814-4ff8-9a3e-3d8f40e01535","Type":"ContainerStarted","Data":"cd542c9dc78326d146afead88b23303be6ddaacc61c36ac913ff0cdad9831b69"}
Jul 28 13:01:55.412107 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:01:55.411756    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-84fc959947-hwnfs"
Jul 28 13:01:55.430224 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:01:55.430182    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-84fc959947-hwnfs" podStartSLOduration=2.308043172 podStartE2EDuration="5.430168422s" totalImagesPullingTime="3.12212525s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:01:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:01:55.429508407 +0000 UTC m=+527.634895008" watchObservedRunningTime="2026-07-28 13:01:55.430168422 +0000 UTC m=+527.635555023"
Jul 28 13:02:26.417797 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:02:26.417770    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-84fc959947-hwnfs"
Jul 28 13:03:24.103823 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:03:24.103792    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-shspn"]
Jul 28 13:03:24.107067 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:03:24.107051    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-shspn"
Jul 28 13:03:24.109862 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:03:24.109842    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 28 13:03:24.109968 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:03:24.109871    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-q9h84\""
Jul 28 13:03:24.115485 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:03:24.115465    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-shspn"]
Jul 28 13:03:24.165339 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:03:24.165318    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87e4fd7e-2d34-470a-a8a1-01634e2db6a2-model-cache\") pod \"s3-init-shspn\" (UID: \"87e4fd7e-2d34-470a-a8a1-01634e2db6a2\") " pod="kserve/s3-init-shspn"
Jul 28 13:03:24.165443 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:03:24.165345    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9kqv\" (UniqueName: \"kubernetes.io/projected/87e4fd7e-2d34-470a-a8a1-01634e2db6a2-kube-api-access-f9kqv\") pod \"s3-init-shspn\" (UID: \"87e4fd7e-2d34-470a-a8a1-01634e2db6a2\") " pod="kserve/s3-init-shspn"
Jul 28 13:03:24.266604 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:03:24.266584    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87e4fd7e-2d34-470a-a8a1-01634e2db6a2-model-cache\") pod \"s3-init-shspn\" (UID: \"87e4fd7e-2d34-470a-a8a1-01634e2db6a2\") " pod="kserve/s3-init-shspn"
Jul 28 13:03:24.266701 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:03:24.266612    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9kqv\" (UniqueName: \"kubernetes.io/projected/87e4fd7e-2d34-470a-a8a1-01634e2db6a2-kube-api-access-f9kqv\") pod \"s3-init-shspn\" (UID: \"87e4fd7e-2d34-470a-a8a1-01634e2db6a2\") " pod="kserve/s3-init-shspn"
Jul 28 13:03:24.266922 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:03:24.266906    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87e4fd7e-2d34-470a-a8a1-01634e2db6a2-model-cache\") pod \"s3-init-shspn\" (UID: \"87e4fd7e-2d34-470a-a8a1-01634e2db6a2\") " pod="kserve/s3-init-shspn"
Jul 28 13:03:24.275972 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:03:24.275948    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9kqv\" (UniqueName: \"kubernetes.io/projected/87e4fd7e-2d34-470a-a8a1-01634e2db6a2-kube-api-access-f9kqv\") pod \"s3-init-shspn\" (UID: \"87e4fd7e-2d34-470a-a8a1-01634e2db6a2\") " pod="kserve/s3-init-shspn"
Jul 28 13:03:24.428404 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:03:24.428335    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-shspn"
Jul 28 13:03:24.559252 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:03:24.559194    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-shspn"]
Jul 28 13:03:24.561755 ip-10-0-138-158 kubenswrapper[2574]: W0728 13:03:24.561723    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod87e4fd7e_2d34_470a_a8a1_01634e2db6a2.slice/crio-06d118bd705e2564412b944a94050d3be35a919cb8486245f871fd575090c28a WatchSource:0}: Error finding container 06d118bd705e2564412b944a94050d3be35a919cb8486245f871fd575090c28a: Status 404 returned error can't find the container with id 06d118bd705e2564412b944a94050d3be35a919cb8486245f871fd575090c28a
Jul 28 13:03:24.782259 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:03:24.782232    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-shspn" event={"ID":"87e4fd7e-2d34-470a-a8a1-01634e2db6a2","Type":"ContainerStarted","Data":"06d118bd705e2564412b944a94050d3be35a919cb8486245f871fd575090c28a"}
Jul 28 13:03:28.805638 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:03:28.805604    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-shspn" event={"ID":"87e4fd7e-2d34-470a-a8a1-01634e2db6a2","Type":"ContainerStarted","Data":"f49ee2a29b5852a2cc8083fd4692f15ebb2e65575180ad97ed2a162639a60d56"}
Jul 28 13:03:30.815351 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:03:30.815282    2574 generic.go:356] "Generic (PLEG): container finished" podID="87e4fd7e-2d34-470a-a8a1-01634e2db6a2" containerID="f49ee2a29b5852a2cc8083fd4692f15ebb2e65575180ad97ed2a162639a60d56" exitCode=0
Jul 28 13:03:30.815722 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:03:30.815360    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-shspn" event={"ID":"87e4fd7e-2d34-470a-a8a1-01634e2db6a2","Type":"ContainerDied","Data":"f49ee2a29b5852a2cc8083fd4692f15ebb2e65575180ad97ed2a162639a60d56"}
Jul 28 13:03:35.843504 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:03:35.843469    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-shspn" event={"ID":"87e4fd7e-2d34-470a-a8a1-01634e2db6a2","Type":"ContainerStarted","Data":"ab7048df59b695b94890f90cd954feff942fed457cd074479b426098f2f0e653"}
Jul 28 13:03:35.862211 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:03:35.862161    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-shspn" podStartSLOduration=2.051236282 podStartE2EDuration="11.862146566s" totalImagesPullingTime="7.7953366s" totalInitContainerRuntime="2.015573684s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:03:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:03:35.861823271 +0000 UTC m=+628.067209898" watchObservedRunningTime="2026-07-28 13:03:35.862146566 +0000 UTC m=+628.067533166"
Jul 28 13:04:40.110938 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:04:40.110905    2574 generic.go:356] "Generic (PLEG): container finished" podID="87e4fd7e-2d34-470a-a8a1-01634e2db6a2" containerID="ab7048df59b695b94890f90cd954feff942fed457cd074479b426098f2f0e653" exitCode=0
Jul 28 13:04:40.111346 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:04:40.110980    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-shspn" event={"ID":"87e4fd7e-2d34-470a-a8a1-01634e2db6a2","Type":"ContainerDied","Data":"ab7048df59b695b94890f90cd954feff942fed457cd074479b426098f2f0e653"}
Jul 28 13:04:41.241871 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:04:41.241844    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-shspn"
Jul 28 13:04:41.341759 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:04:41.341738    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87e4fd7e-2d34-470a-a8a1-01634e2db6a2-model-cache\") pod \"87e4fd7e-2d34-470a-a8a1-01634e2db6a2\" (UID: \"87e4fd7e-2d34-470a-a8a1-01634e2db6a2\") "
Jul 28 13:04:41.341860 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:04:41.341802    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f9kqv\" (UniqueName: \"kubernetes.io/projected/87e4fd7e-2d34-470a-a8a1-01634e2db6a2-kube-api-access-f9kqv\") pod \"87e4fd7e-2d34-470a-a8a1-01634e2db6a2\" (UID: \"87e4fd7e-2d34-470a-a8a1-01634e2db6a2\") "
Jul 28 13:04:41.343839 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:04:41.343815    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87e4fd7e-2d34-470a-a8a1-01634e2db6a2-kube-api-access-f9kqv" (OuterVolumeSpecName: "kube-api-access-f9kqv") pod "87e4fd7e-2d34-470a-a8a1-01634e2db6a2" (UID: "87e4fd7e-2d34-470a-a8a1-01634e2db6a2"). InnerVolumeSpecName "kube-api-access-f9kqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:04:41.364387 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:04:41.364363    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87e4fd7e-2d34-470a-a8a1-01634e2db6a2-model-cache" (OuterVolumeSpecName: "model-cache") pod "87e4fd7e-2d34-470a-a8a1-01634e2db6a2" (UID: "87e4fd7e-2d34-470a-a8a1-01634e2db6a2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:04:41.442808 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:04:41.442790    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87e4fd7e-2d34-470a-a8a1-01634e2db6a2-model-cache\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:04:41.442808 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:04:41.442807    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9kqv\" (UniqueName: \"kubernetes.io/projected/87e4fd7e-2d34-470a-a8a1-01634e2db6a2-kube-api-access-f9kqv\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:04:42.120938 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:04:42.120908    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-shspn" event={"ID":"87e4fd7e-2d34-470a-a8a1-01634e2db6a2","Type":"ContainerDied","Data":"06d118bd705e2564412b944a94050d3be35a919cb8486245f871fd575090c28a"}
Jul 28 13:04:42.121089 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:04:42.120940    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-shspn"
Jul 28 13:04:42.121089 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:04:42.120943    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06d118bd705e2564412b944a94050d3be35a919cb8486245f871fd575090c28a"
Jul 28 13:04:49.833851 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:04:49.833816    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-v6w9j"]
Jul 28 13:04:49.834285 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:04:49.834255    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87e4fd7e-2d34-470a-a8a1-01634e2db6a2" containerName="s3-init"
Jul 28 13:04:49.834285 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:04:49.834272    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="87e4fd7e-2d34-470a-a8a1-01634e2db6a2" containerName="s3-init"
Jul 28 13:04:49.834400 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:04:49.834309    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87e4fd7e-2d34-470a-a8a1-01634e2db6a2" containerName="download-hf-model"
Jul 28 13:04:49.834400 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:04:49.834319    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="87e4fd7e-2d34-470a-a8a1-01634e2db6a2" containerName="download-hf-model"
Jul 28 13:04:49.834505 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:04:49.834401    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="87e4fd7e-2d34-470a-a8a1-01634e2db6a2" containerName="s3-init"
Jul 28 13:04:49.837808 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:04:49.837788    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-v6w9j"
Jul 28 13:04:49.840383 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:04:49.840362    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Jul 28 13:04:49.840469 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:04:49.840447    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-q9h84\""
Jul 28 13:04:49.844900 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:04:49.844759    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-v6w9j"]
Jul 28 13:04:49.903884 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:04:49.903858    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04e39b3c-cec6-4dd2-bc7e-e82734cae114-model-cache\") pod \"s3-tls-init-custom-v6w9j\" (UID: \"04e39b3c-cec6-4dd2-bc7e-e82734cae114\") " pod="kserve/s3-tls-init-custom-v6w9j"
Jul 28 13:04:49.903988 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:04:49.903926    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpwtl\" (UniqueName: \"kubernetes.io/projected/04e39b3c-cec6-4dd2-bc7e-e82734cae114-kube-api-access-hpwtl\") pod \"s3-tls-init-custom-v6w9j\" (UID: \"04e39b3c-cec6-4dd2-bc7e-e82734cae114\") " pod="kserve/s3-tls-init-custom-v6w9j"
Jul 28 13:04:50.004909 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:04:50.004883    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04e39b3c-cec6-4dd2-bc7e-e82734cae114-model-cache\") pod \"s3-tls-init-custom-v6w9j\" (UID: \"04e39b3c-cec6-4dd2-bc7e-e82734cae114\") " pod="kserve/s3-tls-init-custom-v6w9j"
Jul 28 13:04:50.005000 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:04:50.004950    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpwtl\" (UniqueName: \"kubernetes.io/projected/04e39b3c-cec6-4dd2-bc7e-e82734cae114-kube-api-access-hpwtl\") pod \"s3-tls-init-custom-v6w9j\" (UID: \"04e39b3c-cec6-4dd2-bc7e-e82734cae114\") " pod="kserve/s3-tls-init-custom-v6w9j"
Jul 28 13:04:50.005292 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:04:50.005274    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04e39b3c-cec6-4dd2-bc7e-e82734cae114-model-cache\") pod \"s3-tls-init-custom-v6w9j\" (UID: \"04e39b3c-cec6-4dd2-bc7e-e82734cae114\") " pod="kserve/s3-tls-init-custom-v6w9j"
Jul 28 13:04:50.012866 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:04:50.012842    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpwtl\" (UniqueName: \"kubernetes.io/projected/04e39b3c-cec6-4dd2-bc7e-e82734cae114-kube-api-access-hpwtl\") pod \"s3-tls-init-custom-v6w9j\" (UID: \"04e39b3c-cec6-4dd2-bc7e-e82734cae114\") " pod="kserve/s3-tls-init-custom-v6w9j"
Jul 28 13:04:50.156943 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:04:50.156881    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-v6w9j"
Jul 28 13:04:50.283572 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:04:50.283548    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-v6w9j"]
Jul 28 13:04:50.286209 ip-10-0-138-158 kubenswrapper[2574]: W0728 13:04:50.286184    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod04e39b3c_cec6_4dd2_bc7e_e82734cae114.slice/crio-8757211278d504e2f769fdf89c15e7ea5bd520dbfa22f9e15ceb732bb54b0692 WatchSource:0}: Error finding container 8757211278d504e2f769fdf89c15e7ea5bd520dbfa22f9e15ceb732bb54b0692: Status 404 returned error can't find the container with id 8757211278d504e2f769fdf89c15e7ea5bd520dbfa22f9e15ceb732bb54b0692
Jul 28 13:04:51.158812 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:04:51.158767    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-v6w9j" event={"ID":"04e39b3c-cec6-4dd2-bc7e-e82734cae114","Type":"ContainerStarted","Data":"a1da13ece4f299f969fb2c759e153f53087dde7c6612c1ccaf7351aa2c85122b"}
Jul 28 13:04:51.158812 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:04:51.158818    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-v6w9j" event={"ID":"04e39b3c-cec6-4dd2-bc7e-e82734cae114","Type":"ContainerStarted","Data":"8757211278d504e2f769fdf89c15e7ea5bd520dbfa22f9e15ceb732bb54b0692"}
Jul 28 13:04:53.168988 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:04:53.168958    2574 generic.go:356] "Generic (PLEG): container finished" podID="04e39b3c-cec6-4dd2-bc7e-e82734cae114" containerID="a1da13ece4f299f969fb2c759e153f53087dde7c6612c1ccaf7351aa2c85122b" exitCode=0
Jul 28 13:04:53.169352 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:04:53.169023    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-v6w9j" event={"ID":"04e39b3c-cec6-4dd2-bc7e-e82734cae114","Type":"ContainerDied","Data":"a1da13ece4f299f969fb2c759e153f53087dde7c6612c1ccaf7351aa2c85122b"}
Jul 28 13:04:54.174869 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:04:54.174833    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-v6w9j" event={"ID":"04e39b3c-cec6-4dd2-bc7e-e82734cae114","Type":"ContainerStarted","Data":"f8227e0972e74e663ec57a46426ebab207988b1b5a353f217951b6be780bb1d9"}
Jul 28 13:04:54.192303 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:04:54.192244    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-v6w9j" podStartSLOduration=3.288508389 podStartE2EDuration="5.192230318s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.903721929s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:04:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:04:54.190723936 +0000 UTC m=+706.396110674" watchObservedRunningTime="2026-07-28 13:04:54.192230318 +0000 UTC m=+706.397616919"
Jul 28 13:04:59.197914 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:04:59.197884    2574 generic.go:356] "Generic (PLEG): container finished" podID="04e39b3c-cec6-4dd2-bc7e-e82734cae114" containerID="f8227e0972e74e663ec57a46426ebab207988b1b5a353f217951b6be780bb1d9" exitCode=0
Jul 28 13:04:59.198316 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:04:59.197922    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-v6w9j" event={"ID":"04e39b3c-cec6-4dd2-bc7e-e82734cae114","Type":"ContainerDied","Data":"f8227e0972e74e663ec57a46426ebab207988b1b5a353f217951b6be780bb1d9"}
Jul 28 13:05:00.339204 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:00.339181    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-v6w9j"
Jul 28 13:05:00.395902 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:00.395873    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04e39b3c-cec6-4dd2-bc7e-e82734cae114-model-cache\") pod \"04e39b3c-cec6-4dd2-bc7e-e82734cae114\" (UID: \"04e39b3c-cec6-4dd2-bc7e-e82734cae114\") "
Jul 28 13:05:00.396018 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:00.395951    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hpwtl\" (UniqueName: \"kubernetes.io/projected/04e39b3c-cec6-4dd2-bc7e-e82734cae114-kube-api-access-hpwtl\") pod \"04e39b3c-cec6-4dd2-bc7e-e82734cae114\" (UID: \"04e39b3c-cec6-4dd2-bc7e-e82734cae114\") "
Jul 28 13:05:00.397834 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:00.397807    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04e39b3c-cec6-4dd2-bc7e-e82734cae114-kube-api-access-hpwtl" (OuterVolumeSpecName: "kube-api-access-hpwtl") pod "04e39b3c-cec6-4dd2-bc7e-e82734cae114" (UID: "04e39b3c-cec6-4dd2-bc7e-e82734cae114"). InnerVolumeSpecName "kube-api-access-hpwtl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:05:00.418341 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:00.418318    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04e39b3c-cec6-4dd2-bc7e-e82734cae114-model-cache" (OuterVolumeSpecName: "model-cache") pod "04e39b3c-cec6-4dd2-bc7e-e82734cae114" (UID: "04e39b3c-cec6-4dd2-bc7e-e82734cae114"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:05:00.497336 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:00.497287    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04e39b3c-cec6-4dd2-bc7e-e82734cae114-model-cache\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:05:00.497336 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:00.497307    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpwtl\" (UniqueName: \"kubernetes.io/projected/04e39b3c-cec6-4dd2-bc7e-e82734cae114-kube-api-access-hpwtl\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:05:01.207630 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:01.207608    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-v6w9j"
Jul 28 13:05:01.207790 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:01.207655    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-v6w9j" event={"ID":"04e39b3c-cec6-4dd2-bc7e-e82734cae114","Type":"ContainerDied","Data":"8757211278d504e2f769fdf89c15e7ea5bd520dbfa22f9e15ceb732bb54b0692"}
Jul 28 13:05:01.207790 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:01.207704    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8757211278d504e2f769fdf89c15e7ea5bd520dbfa22f9e15ceb732bb54b0692"
Jul 28 13:05:04.193641 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:04.193603    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-hgsct"]
Jul 28 13:05:04.194051 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:04.193974    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04e39b3c-cec6-4dd2-bc7e-e82734cae114" containerName="download-hf-model"
Jul 28 13:05:04.194051 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:04.193986    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="04e39b3c-cec6-4dd2-bc7e-e82734cae114" containerName="download-hf-model"
Jul 28 13:05:04.194051 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:04.193996    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04e39b3c-cec6-4dd2-bc7e-e82734cae114" containerName="s3-tls-init-custom"
Jul 28 13:05:04.194051 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:04.194001    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="04e39b3c-cec6-4dd2-bc7e-e82734cae114" containerName="s3-tls-init-custom"
Jul 28 13:05:04.194176 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:04.194057    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="04e39b3c-cec6-4dd2-bc7e-e82734cae114" containerName="s3-tls-init-custom"
Jul 28 13:05:04.197059 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:04.197042    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-hgsct"
Jul 28 13:05:04.199540 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:04.199521    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Jul 28 13:05:04.199540 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:04.199528    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-q9h84\""
Jul 28 13:05:04.204756 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:04.204729    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-hgsct"]
Jul 28 13:05:04.327946 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:04.327910    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/657bad82-d976-492f-9230-b6a889dcfade-model-cache\") pod \"s3-tls-init-serving-hgsct\" (UID: \"657bad82-d976-492f-9230-b6a889dcfade\") " pod="kserve/s3-tls-init-serving-hgsct"
Jul 28 13:05:04.327946 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:04.327949    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhd2l\" (UniqueName: \"kubernetes.io/projected/657bad82-d976-492f-9230-b6a889dcfade-kube-api-access-jhd2l\") pod \"s3-tls-init-serving-hgsct\" (UID: \"657bad82-d976-492f-9230-b6a889dcfade\") " pod="kserve/s3-tls-init-serving-hgsct"
Jul 28 13:05:04.428819 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:04.428774    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/657bad82-d976-492f-9230-b6a889dcfade-model-cache\") pod \"s3-tls-init-serving-hgsct\" (UID: \"657bad82-d976-492f-9230-b6a889dcfade\") " pod="kserve/s3-tls-init-serving-hgsct"
Jul 28 13:05:04.428819 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:04.428824    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhd2l\" (UniqueName: \"kubernetes.io/projected/657bad82-d976-492f-9230-b6a889dcfade-kube-api-access-jhd2l\") pod \"s3-tls-init-serving-hgsct\" (UID: \"657bad82-d976-492f-9230-b6a889dcfade\") " pod="kserve/s3-tls-init-serving-hgsct"
Jul 28 13:05:04.429150 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:04.429125    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/657bad82-d976-492f-9230-b6a889dcfade-model-cache\") pod \"s3-tls-init-serving-hgsct\" (UID: \"657bad82-d976-492f-9230-b6a889dcfade\") " pod="kserve/s3-tls-init-serving-hgsct"
Jul 28 13:05:04.437778 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:04.437757    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhd2l\" (UniqueName: \"kubernetes.io/projected/657bad82-d976-492f-9230-b6a889dcfade-kube-api-access-jhd2l\") pod \"s3-tls-init-serving-hgsct\" (UID: \"657bad82-d976-492f-9230-b6a889dcfade\") " pod="kserve/s3-tls-init-serving-hgsct"
Jul 28 13:05:04.523158 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:04.523070    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-hgsct"
Jul 28 13:05:04.656958 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:04.656879    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-hgsct"]
Jul 28 13:05:04.659745 ip-10-0-138-158 kubenswrapper[2574]: W0728 13:05:04.659719    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod657bad82_d976_492f_9230_b6a889dcfade.slice/crio-0551340ffc458e286235c883cddf019742f01c8f926f3ddc015853e00cc0e1ad WatchSource:0}: Error finding container 0551340ffc458e286235c883cddf019742f01c8f926f3ddc015853e00cc0e1ad: Status 404 returned error can't find the container with id 0551340ffc458e286235c883cddf019742f01c8f926f3ddc015853e00cc0e1ad
Jul 28 13:05:05.228530 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:05.228489    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-hgsct" event={"ID":"657bad82-d976-492f-9230-b6a889dcfade","Type":"ContainerStarted","Data":"4b77dfe6f70a21c7263a6ba6a35c3929c2e06416b97a22dbf19a3a2942919119"}
Jul 28 13:05:05.228530 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:05.228532    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-hgsct" event={"ID":"657bad82-d976-492f-9230-b6a889dcfade","Type":"ContainerStarted","Data":"0551340ffc458e286235c883cddf019742f01c8f926f3ddc015853e00cc0e1ad"}
Jul 28 13:05:07.239948 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:07.239915    2574 generic.go:356] "Generic (PLEG): container finished" podID="657bad82-d976-492f-9230-b6a889dcfade" containerID="4b77dfe6f70a21c7263a6ba6a35c3929c2e06416b97a22dbf19a3a2942919119" exitCode=0
Jul 28 13:05:07.240331 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:07.239990    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-hgsct" event={"ID":"657bad82-d976-492f-9230-b6a889dcfade","Type":"ContainerDied","Data":"4b77dfe6f70a21c7263a6ba6a35c3929c2e06416b97a22dbf19a3a2942919119"}
Jul 28 13:05:08.246623 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:08.246585    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-hgsct" event={"ID":"657bad82-d976-492f-9230-b6a889dcfade","Type":"ContainerStarted","Data":"e15421bd753f454d5b680966686f928afe9cc385329141b7a231eb0740d9f74b"}
Jul 28 13:05:08.264868 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:08.264817    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-hgsct" podStartSLOduration=2.2316769499999998 podStartE2EDuration="4.264799265s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.033122315s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:05:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:05:08.263106338 +0000 UTC m=+720.468492942" watchObservedRunningTime="2026-07-28 13:05:08.264799265 +0000 UTC m=+720.470185912"
Jul 28 13:05:14.276522 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:14.276486    2574 generic.go:356] "Generic (PLEG): container finished" podID="657bad82-d976-492f-9230-b6a889dcfade" containerID="e15421bd753f454d5b680966686f928afe9cc385329141b7a231eb0740d9f74b" exitCode=0
Jul 28 13:05:14.276876 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:14.276555    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-hgsct" event={"ID":"657bad82-d976-492f-9230-b6a889dcfade","Type":"ContainerDied","Data":"e15421bd753f454d5b680966686f928afe9cc385329141b7a231eb0740d9f74b"}
Jul 28 13:05:15.421365 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:15.421340    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-hgsct"
Jul 28 13:05:15.533894 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:15.533829    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhd2l\" (UniqueName: \"kubernetes.io/projected/657bad82-d976-492f-9230-b6a889dcfade-kube-api-access-jhd2l\") pod \"657bad82-d976-492f-9230-b6a889dcfade\" (UID: \"657bad82-d976-492f-9230-b6a889dcfade\") "
Jul 28 13:05:15.533894 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:15.533883    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/657bad82-d976-492f-9230-b6a889dcfade-model-cache\") pod \"657bad82-d976-492f-9230-b6a889dcfade\" (UID: \"657bad82-d976-492f-9230-b6a889dcfade\") "
Jul 28 13:05:15.535876 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:15.535852    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/657bad82-d976-492f-9230-b6a889dcfade-kube-api-access-jhd2l" (OuterVolumeSpecName: "kube-api-access-jhd2l") pod "657bad82-d976-492f-9230-b6a889dcfade" (UID: "657bad82-d976-492f-9230-b6a889dcfade"). InnerVolumeSpecName "kube-api-access-jhd2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:05:15.556443 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:15.556416    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/657bad82-d976-492f-9230-b6a889dcfade-model-cache" (OuterVolumeSpecName: "model-cache") pod "657bad82-d976-492f-9230-b6a889dcfade" (UID: "657bad82-d976-492f-9230-b6a889dcfade"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:05:15.635204 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:15.635180    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhd2l\" (UniqueName: \"kubernetes.io/projected/657bad82-d976-492f-9230-b6a889dcfade-kube-api-access-jhd2l\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:05:15.635204 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:15.635201    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/657bad82-d976-492f-9230-b6a889dcfade-model-cache\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:05:16.286107 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:16.286079    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-hgsct"
Jul 28 13:05:16.286107 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:16.286081    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-hgsct" event={"ID":"657bad82-d976-492f-9230-b6a889dcfade","Type":"ContainerDied","Data":"0551340ffc458e286235c883cddf019742f01c8f926f3ddc015853e00cc0e1ad"}
Jul 28 13:05:16.286314 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:16.286120    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0551340ffc458e286235c883cddf019742f01c8f926f3ddc015853e00cc0e1ad"
Jul 28 13:05:30.726112 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:30.726076    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc"]
Jul 28 13:05:30.726570 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:30.726548    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="657bad82-d976-492f-9230-b6a889dcfade" containerName="s3-tls-init-serving"
Jul 28 13:05:30.726643 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:30.726575    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="657bad82-d976-492f-9230-b6a889dcfade" containerName="s3-tls-init-serving"
Jul 28 13:05:30.726643 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:30.726619    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="657bad82-d976-492f-9230-b6a889dcfade" containerName="download-hf-model"
Jul 28 13:05:30.726643 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:30.726628    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="657bad82-d976-492f-9230-b6a889dcfade" containerName="download-hf-model"
Jul 28 13:05:30.726830 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:30.726730    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="657bad82-d976-492f-9230-b6a889dcfade" containerName="s3-tls-init-serving"
Jul 28 13:05:30.730322 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:30.730301    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc"
Jul 28 13:05:30.732933 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:30.732906    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:05:30.732933 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:30.732908    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Jul 28 13:05:30.733102 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:30.732942    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rxfwd\""
Jul 28 13:05:30.733102 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:30.732959    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:05:30.733102 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:30.732916    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Jul 28 13:05:30.733946 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:30.733928    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:05:30.734038 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:30.734024    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 13:05:30.743774 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:30.743756    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc"]
Jul 28 13:05:30.743887 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:30.743865    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae40e035-42d4-407d-b5f3-aa16744d90ba-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc\" (UID: \"ae40e035-42d4-407d-b5f3-aa16744d90ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc"
Jul 28 13:05:30.743958 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:30.743906    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae40e035-42d4-407d-b5f3-aa16744d90ba-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc\" (UID: \"ae40e035-42d4-407d-b5f3-aa16744d90ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc"
Jul 28 13:05:30.744014 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:30.744000    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn5sq\" (UniqueName: \"kubernetes.io/projected/ae40e035-42d4-407d-b5f3-aa16744d90ba-kube-api-access-mn5sq\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc\" (UID: \"ae40e035-42d4-407d-b5f3-aa16744d90ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc"
Jul 28 13:05:30.744082 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:30.744064    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae40e035-42d4-407d-b5f3-aa16744d90ba-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc\" (UID: \"ae40e035-42d4-407d-b5f3-aa16744d90ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc"
Jul 28 13:05:30.744137 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:30.744104    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae40e035-42d4-407d-b5f3-aa16744d90ba-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc\" (UID: \"ae40e035-42d4-407d-b5f3-aa16744d90ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc"
Jul 28 13:05:30.844719 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:30.844679    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn5sq\" (UniqueName: \"kubernetes.io/projected/ae40e035-42d4-407d-b5f3-aa16744d90ba-kube-api-access-mn5sq\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc\" (UID: \"ae40e035-42d4-407d-b5f3-aa16744d90ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc"
Jul 28 13:05:30.844815 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:30.844740    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae40e035-42d4-407d-b5f3-aa16744d90ba-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc\" (UID: \"ae40e035-42d4-407d-b5f3-aa16744d90ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc"
Jul 28 13:05:30.844815 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:30.844774    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae40e035-42d4-407d-b5f3-aa16744d90ba-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc\" (UID: \"ae40e035-42d4-407d-b5f3-aa16744d90ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc"
Jul 28 13:05:30.844882 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:30.844823    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae40e035-42d4-407d-b5f3-aa16744d90ba-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc\" (UID: \"ae40e035-42d4-407d-b5f3-aa16744d90ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc"
Jul 28 13:05:30.844882 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:30.844848    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae40e035-42d4-407d-b5f3-aa16744d90ba-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc\" (UID: \"ae40e035-42d4-407d-b5f3-aa16744d90ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc"
Jul 28 13:05:30.845193 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:30.845173    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae40e035-42d4-407d-b5f3-aa16744d90ba-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc\" (UID: \"ae40e035-42d4-407d-b5f3-aa16744d90ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc"
Jul 28 13:05:30.845493 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:30.845475    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae40e035-42d4-407d-b5f3-aa16744d90ba-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc\" (UID: \"ae40e035-42d4-407d-b5f3-aa16744d90ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc"
Jul 28 13:05:30.845563 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:30.845474    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae40e035-42d4-407d-b5f3-aa16744d90ba-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc\" (UID: \"ae40e035-42d4-407d-b5f3-aa16744d90ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc"
Jul 28 13:05:30.846970 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:30.846951    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae40e035-42d4-407d-b5f3-aa16744d90ba-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc\" (UID: \"ae40e035-42d4-407d-b5f3-aa16744d90ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc"
Jul 28 13:05:30.853203 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:30.853183    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn5sq\" (UniqueName: \"kubernetes.io/projected/ae40e035-42d4-407d-b5f3-aa16744d90ba-kube-api-access-mn5sq\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc\" (UID: \"ae40e035-42d4-407d-b5f3-aa16744d90ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc"
Jul 28 13:05:31.041220 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:31.041166    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc"
Jul 28 13:05:31.170194 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:31.170165    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc"]
Jul 28 13:05:31.172770 ip-10-0-138-158 kubenswrapper[2574]: W0728 13:05:31.172742    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae40e035_42d4_407d_b5f3_aa16744d90ba.slice/crio-0dc56bd54ecf910a8e66532c5bf566ead5337cae4043f48f17bb3c6dc73a42a0 WatchSource:0}: Error finding container 0dc56bd54ecf910a8e66532c5bf566ead5337cae4043f48f17bb3c6dc73a42a0: Status 404 returned error can't find the container with id 0dc56bd54ecf910a8e66532c5bf566ead5337cae4043f48f17bb3c6dc73a42a0
Jul 28 13:05:31.355078 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:31.354994    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc" event={"ID":"ae40e035-42d4-407d-b5f3-aa16744d90ba","Type":"ContainerStarted","Data":"662a0e8f8458f159e0983942d0d072e961426da662984664b48b1018ea47ff99"}
Jul 28 13:05:31.355206 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:31.355096    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc" event={"ID":"ae40e035-42d4-407d-b5f3-aa16744d90ba","Type":"ContainerStarted","Data":"0dc56bd54ecf910a8e66532c5bf566ead5337cae4043f48f17bb3c6dc73a42a0"}
Jul 28 13:05:35.372741 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:35.372663    2574 generic.go:356] "Generic (PLEG): container finished" podID="ae40e035-42d4-407d-b5f3-aa16744d90ba" containerID="662a0e8f8458f159e0983942d0d072e961426da662984664b48b1018ea47ff99" exitCode=0
Jul 28 13:05:35.372741 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:35.372725    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc" event={"ID":"ae40e035-42d4-407d-b5f3-aa16744d90ba","Type":"ContainerDied","Data":"662a0e8f8458f159e0983942d0d072e961426da662984664b48b1018ea47ff99"}
Jul 28 13:05:35.373831 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:35.373815    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 13:05:48.443485 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:48.443457    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc" event={"ID":"ae40e035-42d4-407d-b5f3-aa16744d90ba","Type":"ContainerStarted","Data":"30a6f8289cfd847bfb099e816c88299b08d1cb9fef18d67af38a138f2f0211de"}
Jul 28 13:05:51.467050 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:51.467008    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc" event={"ID":"ae40e035-42d4-407d-b5f3-aa16744d90ba","Type":"ContainerStarted","Data":"fb95185f66e7714129f1a1217ebbceda462ba88fa313dc6b2ff1f92c8f539286"}
Jul 28 13:05:53.478201 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:53.478170    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc" event={"ID":"ae40e035-42d4-407d-b5f3-aa16744d90ba","Type":"ContainerStarted","Data":"bc94257297e9386f01a0f25f48559d5f8ca3049b439f6117964e2876f7ae902c"}
Jul 28 13:05:53.478627 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:53.478423    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc"
Jul 28 13:05:53.478627 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:53.478480    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc"
Jul 28 13:05:53.478627 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:53.478509    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc"
Jul 28 13:05:53.479808 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:53.479782    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc" podUID="ae40e035-42d4-407d-b5f3-aa16744d90ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 28 13:05:53.480447 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:53.480424    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc" podUID="ae40e035-42d4-407d-b5f3-aa16744d90ba" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 13:05:53.499437 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:53.499387    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc" podStartSLOduration=2.274923458 podStartE2EDuration="23.499369724s" totalImagesPullingTime="17.523105548s" totalInitContainerRuntime="3.701340718s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:05:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:05:53.496889946 +0000 UTC m=+765.702276546" watchObservedRunningTime="2026-07-28 13:05:53.499369724 +0000 UTC m=+765.704756325"
Jul 28 13:05:54.482631 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:54.482592    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc" podUID="ae40e035-42d4-407d-b5f3-aa16744d90ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 28 13:05:54.483169 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:54.483146    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc" podUID="ae40e035-42d4-407d-b5f3-aa16744d90ba" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 13:05:59.487966 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:59.487936    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc"
Jul 28 13:05:59.488548 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:59.488517    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc" podUID="ae40e035-42d4-407d-b5f3-aa16744d90ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 28 13:05:59.488929 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:05:59.488902    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc" podUID="ae40e035-42d4-407d-b5f3-aa16744d90ba" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 13:06:09.488798 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:06:09.488761    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc" podUID="ae40e035-42d4-407d-b5f3-aa16744d90ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 28 13:06:09.489261 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:06:09.489238    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc" podUID="ae40e035-42d4-407d-b5f3-aa16744d90ba" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 13:06:19.488923 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:06:19.488875    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc" podUID="ae40e035-42d4-407d-b5f3-aa16744d90ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 28 13:06:19.489414 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:06:19.489373    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc" podUID="ae40e035-42d4-407d-b5f3-aa16744d90ba" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 13:06:29.488380 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:06:29.488341    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc" podUID="ae40e035-42d4-407d-b5f3-aa16744d90ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 28 13:06:29.488841 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:06:29.488818    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc" podUID="ae40e035-42d4-407d-b5f3-aa16744d90ba" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 13:06:39.489300 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:06:39.489223    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc" podUID="ae40e035-42d4-407d-b5f3-aa16744d90ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 28 13:06:39.489713 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:06:39.489667    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc" podUID="ae40e035-42d4-407d-b5f3-aa16744d90ba" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 13:06:49.488730 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:06:49.488664    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc" podUID="ae40e035-42d4-407d-b5f3-aa16744d90ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 28 13:06:49.489154 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:06:49.489127    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc" podUID="ae40e035-42d4-407d-b5f3-aa16744d90ba" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 13:06:59.489054 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:06:59.489021    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc"
Jul 28 13:06:59.489649 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:06:59.489628    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc"
Jul 28 13:07:05.880344 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:05.880313    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc"]
Jul 28 13:07:05.880740 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:05.880612    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc" podUID="ae40e035-42d4-407d-b5f3-aa16744d90ba" containerName="kserve-container" containerID="cri-o://30a6f8289cfd847bfb099e816c88299b08d1cb9fef18d67af38a138f2f0211de" gracePeriod=30
Jul 28 13:07:05.880740 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:05.880642    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc" podUID="ae40e035-42d4-407d-b5f3-aa16744d90ba" containerName="agent" containerID="cri-o://bc94257297e9386f01a0f25f48559d5f8ca3049b439f6117964e2876f7ae902c" gracePeriod=30
Jul 28 13:07:05.880740 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:05.880664    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc" podUID="ae40e035-42d4-407d-b5f3-aa16744d90ba" containerName="kube-rbac-proxy" containerID="cri-o://fb95185f66e7714129f1a1217ebbceda462ba88fa313dc6b2ff1f92c8f539286" gracePeriod=30
Jul 28 13:07:05.898797 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:05.898774    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg"]
Jul 28 13:07:05.902385 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:05.902363    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg"
Jul 28 13:07:05.904836 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:05.904815    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Jul 28 13:07:05.904937 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:05.904815    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Jul 28 13:07:05.915123 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:05.915102    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg"]
Jul 28 13:07:05.965025 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:05.965001    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stlgc\" (UniqueName: \"kubernetes.io/projected/cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e-kube-api-access-stlgc\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg\" (UID: \"cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg"
Jul 28 13:07:05.965112 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:05.965044    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg\" (UID: \"cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg"
Jul 28 13:07:05.965112 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:05.965064    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg\" (UID: \"cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg"
Jul 28 13:07:05.965186 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:05.965125    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg\" (UID: \"cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg"
Jul 28 13:07:05.965186 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:05.965173    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg\" (UID: \"cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg"
Jul 28 13:07:06.065828 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:06.065798    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stlgc\" (UniqueName: \"kubernetes.io/projected/cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e-kube-api-access-stlgc\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg\" (UID: \"cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg"
Jul 28 13:07:06.065968 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:06.065843    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg\" (UID: \"cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg"
Jul 28 13:07:06.065968 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:06.065859    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg\" (UID: \"cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg"
Jul 28 13:07:06.065968 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:06.065878    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg\" (UID: \"cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg"
Jul 28 13:07:06.065968 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:06.065935    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg\" (UID: \"cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg"
Jul 28 13:07:06.066395 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:06.066375    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg\" (UID: \"cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg"
Jul 28 13:07:06.066546 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:06.066530    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg\" (UID: \"cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg"
Jul 28 13:07:06.066622 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:06.066601    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg\" (UID: \"cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg"
Jul 28 13:07:06.068204 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:06.068184    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg\" (UID: \"cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg"
Jul 28 13:07:06.073150 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:06.073131    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stlgc\" (UniqueName: \"kubernetes.io/projected/cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e-kube-api-access-stlgc\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg\" (UID: \"cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg"
Jul 28 13:07:06.213064 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:06.213043    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg"
Jul 28 13:07:06.343718 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:06.343692    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg"]
Jul 28 13:07:06.345021 ip-10-0-138-158 kubenswrapper[2574]: W0728 13:07:06.344994    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb7c3735_5e9b_42bc_80c9_6e26a2b02c9e.slice/crio-5b5f2a251c8508281bef8831740e7a2d5ab4923e9d511cbb01e234ba50bb6405 WatchSource:0}: Error finding container 5b5f2a251c8508281bef8831740e7a2d5ab4923e9d511cbb01e234ba50bb6405: Status 404 returned error can't find the container with id 5b5f2a251c8508281bef8831740e7a2d5ab4923e9d511cbb01e234ba50bb6405
Jul 28 13:07:06.801038 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:06.801001    2574 generic.go:356] "Generic (PLEG): container finished" podID="ae40e035-42d4-407d-b5f3-aa16744d90ba" containerID="fb95185f66e7714129f1a1217ebbceda462ba88fa313dc6b2ff1f92c8f539286" exitCode=2
Jul 28 13:07:06.801223 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:06.801072    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc" event={"ID":"ae40e035-42d4-407d-b5f3-aa16744d90ba","Type":"ContainerDied","Data":"fb95185f66e7714129f1a1217ebbceda462ba88fa313dc6b2ff1f92c8f539286"}
Jul 28 13:07:06.802450 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:06.802421    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg" event={"ID":"cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e","Type":"ContainerStarted","Data":"644fd2252ac8a3760143147959dba860340bb901122e03776c1f1d45705c71c6"}
Jul 28 13:07:06.802563 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:06.802475    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg" event={"ID":"cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e","Type":"ContainerStarted","Data":"5b5f2a251c8508281bef8831740e7a2d5ab4923e9d511cbb01e234ba50bb6405"}
Jul 28 13:07:09.483037 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:09.482999    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc" podUID="ae40e035-42d4-407d-b5f3-aa16744d90ba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Jul 28 13:07:09.488752 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:09.488721    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc" podUID="ae40e035-42d4-407d-b5f3-aa16744d90ba" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 13:07:09.489901 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:09.489878    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc" podUID="ae40e035-42d4-407d-b5f3-aa16744d90ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 28 13:07:09.818764 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:09.818740    2574 generic.go:356] "Generic (PLEG): container finished" podID="ae40e035-42d4-407d-b5f3-aa16744d90ba" containerID="30a6f8289cfd847bfb099e816c88299b08d1cb9fef18d67af38a138f2f0211de" exitCode=0
Jul 28 13:07:09.818875 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:09.818786    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc" event={"ID":"ae40e035-42d4-407d-b5f3-aa16744d90ba","Type":"ContainerDied","Data":"30a6f8289cfd847bfb099e816c88299b08d1cb9fef18d67af38a138f2f0211de"}
Jul 28 13:07:10.825207 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:10.825174    2574 generic.go:356] "Generic (PLEG): container finished" podID="cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e" containerID="644fd2252ac8a3760143147959dba860340bb901122e03776c1f1d45705c71c6" exitCode=0
Jul 28 13:07:10.825574 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:10.825246    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg" event={"ID":"cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e","Type":"ContainerDied","Data":"644fd2252ac8a3760143147959dba860340bb901122e03776c1f1d45705c71c6"}
Jul 28 13:07:11.831882 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:11.831850    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg" event={"ID":"cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e","Type":"ContainerStarted","Data":"a98cc8a636be3e532f66e8d8de7cd5dc51e1bfba60c069925fa0c2a03d108e29"}
Jul 28 13:07:11.832284 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:11.831913    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg" event={"ID":"cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e","Type":"ContainerStarted","Data":"2389b110d7f66692f622810199091be45f4d49fd7f8452f449a0b01875a808c1"}
Jul 28 13:07:11.832284 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:11.831939    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg" event={"ID":"cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e","Type":"ContainerStarted","Data":"26dac160b2dd04ea1eeb3311c9de98a0b4376c489acfc396af6ad50cc7fc5291"}
Jul 28 13:07:11.832284 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:11.832236    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg"
Jul 28 13:07:11.832392 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:11.832309    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg"
Jul 28 13:07:11.832392 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:11.832346    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg"
Jul 28 13:07:11.833772 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:11.833746    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg" podUID="cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:5000: connect: connection refused"
Jul 28 13:07:11.834438 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:11.834416    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg" podUID="cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 13:07:11.854514 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:11.854468    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg" podStartSLOduration=3.221376559 podStartE2EDuration="6.854454081s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.633077522s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:07:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:07:11.851807485 +0000 UTC m=+844.057194087" watchObservedRunningTime="2026-07-28 13:07:11.854454081 +0000 UTC m=+844.059840681"
Jul 28 13:07:12.836880 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:12.836842    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg" podUID="cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:5000: connect: connection refused"
Jul 28 13:07:12.837298 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:12.837212    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg" podUID="cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 13:07:14.483224 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:14.483186    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc" podUID="ae40e035-42d4-407d-b5f3-aa16744d90ba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Jul 28 13:07:17.841310 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:17.841284    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg"
Jul 28 13:07:17.841919 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:17.841883    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg" podUID="cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:5000: connect: connection refused"
Jul 28 13:07:17.842232 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:17.842205    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg" podUID="cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 13:07:19.483211 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:19.483170    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc" podUID="ae40e035-42d4-407d-b5f3-aa16744d90ba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Jul 28 13:07:19.483596 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:19.483297    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc"
Jul 28 13:07:19.488969 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:19.488942    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc" podUID="ae40e035-42d4-407d-b5f3-aa16744d90ba" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 13:07:19.490068 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:19.490044    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc" podUID="ae40e035-42d4-407d-b5f3-aa16744d90ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 28 13:07:24.483030 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:24.482994    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc" podUID="ae40e035-42d4-407d-b5f3-aa16744d90ba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Jul 28 13:07:27.842466 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:27.842427    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg" podUID="cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:5000: connect: connection refused"
Jul 28 13:07:27.843013 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:27.842991    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg" podUID="cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 13:07:29.483678 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:29.483632    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc" podUID="ae40e035-42d4-407d-b5f3-aa16744d90ba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Jul 28 13:07:29.489469 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:29.489426    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc" podUID="ae40e035-42d4-407d-b5f3-aa16744d90ba" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 13:07:29.489633 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:29.489541    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc"
Jul 28 13:07:29.489784 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:29.489759    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc" podUID="ae40e035-42d4-407d-b5f3-aa16744d90ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 28 13:07:29.489901 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:29.489888    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc"
Jul 28 13:07:34.483230 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:34.483191    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc" podUID="ae40e035-42d4-407d-b5f3-aa16744d90ba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Jul 28 13:07:35.945323 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:35.945289    2574 generic.go:356] "Generic (PLEG): container finished" podID="ae40e035-42d4-407d-b5f3-aa16744d90ba" containerID="bc94257297e9386f01a0f25f48559d5f8ca3049b439f6117964e2876f7ae902c" exitCode=0
Jul 28 13:07:35.945665 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:35.945355    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc" event={"ID":"ae40e035-42d4-407d-b5f3-aa16744d90ba","Type":"ContainerDied","Data":"bc94257297e9386f01a0f25f48559d5f8ca3049b439f6117964e2876f7ae902c"}
Jul 28 13:07:36.030304 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:36.030283    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc"
Jul 28 13:07:36.095757 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:36.095731    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae40e035-42d4-407d-b5f3-aa16744d90ba-kube-rbac-proxy-sar-config\") pod \"ae40e035-42d4-407d-b5f3-aa16744d90ba\" (UID: \"ae40e035-42d4-407d-b5f3-aa16744d90ba\") "
Jul 28 13:07:36.095870 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:36.095771    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae40e035-42d4-407d-b5f3-aa16744d90ba-kserve-provision-location\") pod \"ae40e035-42d4-407d-b5f3-aa16744d90ba\" (UID: \"ae40e035-42d4-407d-b5f3-aa16744d90ba\") "
Jul 28 13:07:36.095870 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:36.095805    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mn5sq\" (UniqueName: \"kubernetes.io/projected/ae40e035-42d4-407d-b5f3-aa16744d90ba-kube-api-access-mn5sq\") pod \"ae40e035-42d4-407d-b5f3-aa16744d90ba\" (UID: \"ae40e035-42d4-407d-b5f3-aa16744d90ba\") "
Jul 28 13:07:36.095870 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:36.095848    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae40e035-42d4-407d-b5f3-aa16744d90ba-cabundle-cert\") pod \"ae40e035-42d4-407d-b5f3-aa16744d90ba\" (UID: \"ae40e035-42d4-407d-b5f3-aa16744d90ba\") "
Jul 28 13:07:36.096079 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:36.095871    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae40e035-42d4-407d-b5f3-aa16744d90ba-proxy-tls\") pod \"ae40e035-42d4-407d-b5f3-aa16744d90ba\" (UID: \"ae40e035-42d4-407d-b5f3-aa16744d90ba\") "
Jul 28 13:07:36.096079 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:36.096061    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae40e035-42d4-407d-b5f3-aa16744d90ba-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ae40e035-42d4-407d-b5f3-aa16744d90ba" (UID: "ae40e035-42d4-407d-b5f3-aa16744d90ba"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:07:36.096208 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:36.096172    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae40e035-42d4-407d-b5f3-aa16744d90ba-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ae40e035-42d4-407d-b5f3-aa16744d90ba" (UID: "ae40e035-42d4-407d-b5f3-aa16744d90ba"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:07:36.096320 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:36.096241    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae40e035-42d4-407d-b5f3-aa16744d90ba-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ae40e035-42d4-407d-b5f3-aa16744d90ba" (UID: "ae40e035-42d4-407d-b5f3-aa16744d90ba"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:07:36.097945 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:36.097917    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae40e035-42d4-407d-b5f3-aa16744d90ba-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ae40e035-42d4-407d-b5f3-aa16744d90ba" (UID: "ae40e035-42d4-407d-b5f3-aa16744d90ba"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:07:36.098039 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:36.098019    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae40e035-42d4-407d-b5f3-aa16744d90ba-kube-api-access-mn5sq" (OuterVolumeSpecName: "kube-api-access-mn5sq") pod "ae40e035-42d4-407d-b5f3-aa16744d90ba" (UID: "ae40e035-42d4-407d-b5f3-aa16744d90ba"). InnerVolumeSpecName "kube-api-access-mn5sq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:07:36.196584 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:36.196563    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae40e035-42d4-407d-b5f3-aa16744d90ba-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:07:36.196584 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:36.196584    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae40e035-42d4-407d-b5f3-aa16744d90ba-kserve-provision-location\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:07:36.196724 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:36.196593    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mn5sq\" (UniqueName: \"kubernetes.io/projected/ae40e035-42d4-407d-b5f3-aa16744d90ba-kube-api-access-mn5sq\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:07:36.196724 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:36.196602    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae40e035-42d4-407d-b5f3-aa16744d90ba-cabundle-cert\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:07:36.196724 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:36.196612    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae40e035-42d4-407d-b5f3-aa16744d90ba-proxy-tls\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:07:36.952184 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:36.952158    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc"
Jul 28 13:07:36.952569 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:36.952160    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc" event={"ID":"ae40e035-42d4-407d-b5f3-aa16744d90ba","Type":"ContainerDied","Data":"0dc56bd54ecf910a8e66532c5bf566ead5337cae4043f48f17bb3c6dc73a42a0"}
Jul 28 13:07:36.952569 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:36.952306    2574 scope.go:117] "RemoveContainer" containerID="bc94257297e9386f01a0f25f48559d5f8ca3049b439f6117964e2876f7ae902c"
Jul 28 13:07:36.961213 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:36.961192    2574 scope.go:117] "RemoveContainer" containerID="fb95185f66e7714129f1a1217ebbceda462ba88fa313dc6b2ff1f92c8f539286"
Jul 28 13:07:36.968127 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:36.968111    2574 scope.go:117] "RemoveContainer" containerID="30a6f8289cfd847bfb099e816c88299b08d1cb9fef18d67af38a138f2f0211de"
Jul 28 13:07:36.973334 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:36.973309    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc"]
Jul 28 13:07:36.977041 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:36.977022    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-dt7rc"]
Jul 28 13:07:36.977111 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:36.977042    2574 scope.go:117] "RemoveContainer" containerID="662a0e8f8458f159e0983942d0d072e961426da662984664b48b1018ea47ff99"
Jul 28 13:07:37.842199 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:37.842162    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg" podUID="cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:5000: connect: connection refused"
Jul 28 13:07:37.842581 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:37.842564    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg" podUID="cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 13:07:38.302747 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:38.302711    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae40e035-42d4-407d-b5f3-aa16744d90ba" path="/var/lib/kubelet/pods/ae40e035-42d4-407d-b5f3-aa16744d90ba/volumes"
Jul 28 13:07:47.842051 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:47.841971    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg" podUID="cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:5000: connect: connection refused"
Jul 28 13:07:47.842412 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:47.842397    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg" podUID="cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 13:07:57.842333 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:57.842299    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg" podUID="cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:5000: connect: connection refused"
Jul 28 13:07:57.842750 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:07:57.842722    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg" podUID="cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 13:08:07.842046 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:07.841964    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg" podUID="cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:5000: connect: connection refused"
Jul 28 13:08:07.842531 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:07.842326    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg" podUID="cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 13:08:17.842125 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:17.842089    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg" podUID="cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:5000: connect: connection refused"
Jul 28 13:08:17.842557 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:17.842534    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg" podUID="cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 13:08:21.297616 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:21.297588    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg"
Jul 28 13:08:21.298011 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:21.297843    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg"
Jul 28 13:08:31.009562 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:31.009531    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg"]
Jul 28 13:08:31.010019 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:31.009852    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg" podUID="cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e" containerName="kserve-container" containerID="cri-o://26dac160b2dd04ea1eeb3311c9de98a0b4376c489acfc396af6ad50cc7fc5291" gracePeriod=30
Jul 28 13:08:31.010019 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:31.009904    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg" podUID="cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e" containerName="kube-rbac-proxy" containerID="cri-o://2389b110d7f66692f622810199091be45f4d49fd7f8452f449a0b01875a808c1" gracePeriod=30
Jul 28 13:08:31.010149 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:31.010066    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg" podUID="cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e" containerName="agent" containerID="cri-o://a98cc8a636be3e532f66e8d8de7cd5dc51e1bfba60c069925fa0c2a03d108e29" gracePeriod=30
Jul 28 13:08:31.076910 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:31.076884    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-r5f4w"]
Jul 28 13:08:31.077231 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:31.077220    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae40e035-42d4-407d-b5f3-aa16744d90ba" containerName="storage-initializer"
Jul 28 13:08:31.077273 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:31.077233    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae40e035-42d4-407d-b5f3-aa16744d90ba" containerName="storage-initializer"
Jul 28 13:08:31.077273 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:31.077242    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae40e035-42d4-407d-b5f3-aa16744d90ba" containerName="agent"
Jul 28 13:08:31.077273 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:31.077248    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae40e035-42d4-407d-b5f3-aa16744d90ba" containerName="agent"
Jul 28 13:08:31.077273 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:31.077260    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae40e035-42d4-407d-b5f3-aa16744d90ba" containerName="kserve-container"
Jul 28 13:08:31.077273 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:31.077266    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae40e035-42d4-407d-b5f3-aa16744d90ba" containerName="kserve-container"
Jul 28 13:08:31.077476 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:31.077281    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae40e035-42d4-407d-b5f3-aa16744d90ba" containerName="kube-rbac-proxy"
Jul 28 13:08:31.077476 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:31.077286    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae40e035-42d4-407d-b5f3-aa16744d90ba" containerName="kube-rbac-proxy"
Jul 28 13:08:31.077476 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:31.077336    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae40e035-42d4-407d-b5f3-aa16744d90ba" containerName="kserve-container"
Jul 28 13:08:31.077476 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:31.077346    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae40e035-42d4-407d-b5f3-aa16744d90ba" containerName="agent"
Jul 28 13:08:31.077476 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:31.077354    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae40e035-42d4-407d-b5f3-aa16744d90ba" containerName="kube-rbac-proxy"
Jul 28 13:08:31.081733 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:31.081716    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-r5f4w"
Jul 28 13:08:31.084232 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:31.084212    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Jul 28 13:08:31.084334 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:31.084237    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Jul 28 13:08:31.091325 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:31.091307    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-r5f4w"]
Jul 28 13:08:31.099853 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:31.099831    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/63f28749-51c9-4c2a-82c8-58c8476bf1b9-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-r5f4w\" (UID: \"63f28749-51c9-4c2a-82c8-58c8476bf1b9\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-r5f4w"
Jul 28 13:08:31.099958 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:31.099858    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbjj7\" (UniqueName: \"kubernetes.io/projected/63f28749-51c9-4c2a-82c8-58c8476bf1b9-kube-api-access-jbjj7\") pod \"message-dumper-predictor-6575bcfdfd-r5f4w\" (UID: \"63f28749-51c9-4c2a-82c8-58c8476bf1b9\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-r5f4w"
Jul 28 13:08:31.099958 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:31.099888    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63f28749-51c9-4c2a-82c8-58c8476bf1b9-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-r5f4w\" (UID: \"63f28749-51c9-4c2a-82c8-58c8476bf1b9\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-r5f4w"
Jul 28 13:08:31.199625 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:31.199601    2574 generic.go:356] "Generic (PLEG): container finished" podID="cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e" containerID="2389b110d7f66692f622810199091be45f4d49fd7f8452f449a0b01875a808c1" exitCode=2
Jul 28 13:08:31.199782 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:31.199707    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg" event={"ID":"cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e","Type":"ContainerDied","Data":"2389b110d7f66692f622810199091be45f4d49fd7f8452f449a0b01875a808c1"}
Jul 28 13:08:31.200945 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:31.200924    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/63f28749-51c9-4c2a-82c8-58c8476bf1b9-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-r5f4w\" (UID: \"63f28749-51c9-4c2a-82c8-58c8476bf1b9\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-r5f4w"
Jul 28 13:08:31.201002 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:31.200954    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbjj7\" (UniqueName: \"kubernetes.io/projected/63f28749-51c9-4c2a-82c8-58c8476bf1b9-kube-api-access-jbjj7\") pod \"message-dumper-predictor-6575bcfdfd-r5f4w\" (UID: \"63f28749-51c9-4c2a-82c8-58c8476bf1b9\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-r5f4w"
Jul 28 13:08:31.201002 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:31.200981    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63f28749-51c9-4c2a-82c8-58c8476bf1b9-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-r5f4w\" (UID: \"63f28749-51c9-4c2a-82c8-58c8476bf1b9\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-r5f4w"
Jul 28 13:08:31.201102 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:08:31.201063    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-predictor-serving-cert: secret "message-dumper-predictor-serving-cert" not found
Jul 28 13:08:31.201148 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:08:31.201124    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63f28749-51c9-4c2a-82c8-58c8476bf1b9-proxy-tls podName:63f28749-51c9-4c2a-82c8-58c8476bf1b9 nodeName:}" failed. No retries permitted until 2026-07-28 13:08:31.70110433 +0000 UTC m=+923.906490909 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/63f28749-51c9-4c2a-82c8-58c8476bf1b9-proxy-tls") pod "message-dumper-predictor-6575bcfdfd-r5f4w" (UID: "63f28749-51c9-4c2a-82c8-58c8476bf1b9") : secret "message-dumper-predictor-serving-cert" not found
Jul 28 13:08:31.201549 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:31.201533    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/63f28749-51c9-4c2a-82c8-58c8476bf1b9-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-r5f4w\" (UID: \"63f28749-51c9-4c2a-82c8-58c8476bf1b9\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-r5f4w"
Jul 28 13:08:31.211889 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:31.211863    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbjj7\" (UniqueName: \"kubernetes.io/projected/63f28749-51c9-4c2a-82c8-58c8476bf1b9-kube-api-access-jbjj7\") pod \"message-dumper-predictor-6575bcfdfd-r5f4w\" (UID: \"63f28749-51c9-4c2a-82c8-58c8476bf1b9\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-r5f4w"
Jul 28 13:08:31.297185 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:31.297110    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg" podUID="cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:5000: connect: connection refused"
Jul 28 13:08:31.297550 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:31.297527    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg" podUID="cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 13:08:31.704462 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:31.704430    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63f28749-51c9-4c2a-82c8-58c8476bf1b9-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-r5f4w\" (UID: \"63f28749-51c9-4c2a-82c8-58c8476bf1b9\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-r5f4w"
Jul 28 13:08:31.706814 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:31.706788    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63f28749-51c9-4c2a-82c8-58c8476bf1b9-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-r5f4w\" (UID: \"63f28749-51c9-4c2a-82c8-58c8476bf1b9\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-r5f4w"
Jul 28 13:08:31.992127 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:31.992037    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-r5f4w"
Jul 28 13:08:32.116400 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:32.116374    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-r5f4w"]
Jul 28 13:08:32.118027 ip-10-0-138-158 kubenswrapper[2574]: W0728 13:08:32.118000    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63f28749_51c9_4c2a_82c8_58c8476bf1b9.slice/crio-4e22ab88bbc089f7bad0fb087617b454bbfe1e6a5328d8227714769a5826c4b2 WatchSource:0}: Error finding container 4e22ab88bbc089f7bad0fb087617b454bbfe1e6a5328d8227714769a5826c4b2: Status 404 returned error can't find the container with id 4e22ab88bbc089f7bad0fb087617b454bbfe1e6a5328d8227714769a5826c4b2
Jul 28 13:08:32.205616 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:32.205592    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-r5f4w" event={"ID":"63f28749-51c9-4c2a-82c8-58c8476bf1b9","Type":"ContainerStarted","Data":"4e22ab88bbc089f7bad0fb087617b454bbfe1e6a5328d8227714769a5826c4b2"}
Jul 28 13:08:32.837939 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:32.837903    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg" podUID="cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Jul 28 13:08:34.216448 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:34.216415    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-r5f4w" event={"ID":"63f28749-51c9-4c2a-82c8-58c8476bf1b9","Type":"ContainerStarted","Data":"23cdc54e5bdf0ef5c2d88c1864ee50315c664ac7eaddced0ef6830cca96ba797"}
Jul 28 13:08:34.216830 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:34.216471    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-r5f4w" event={"ID":"63f28749-51c9-4c2a-82c8-58c8476bf1b9","Type":"ContainerStarted","Data":"82d9a75ef36b11010446dde8fd584a701e9a36ae00ffbeb3eb66289ca3c9d00f"}
Jul 28 13:08:34.216830 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:34.216496    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-r5f4w"
Jul 28 13:08:34.234754 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:34.234709    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-r5f4w" podStartSLOduration=1.9358997470000001 podStartE2EDuration="3.234694913s" totalImagesPullingTime="1.298795166s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:08:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:08:34.232789564 +0000 UTC m=+926.438176169" watchObservedRunningTime="2026-07-28 13:08:34.234694913 +0000 UTC m=+926.440081505"
Jul 28 13:08:35.221576 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:35.221541    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-r5f4w"
Jul 28 13:08:35.222833 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:35.222809    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-r5f4w"
Jul 28 13:08:36.228842 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:36.228809    2574 generic.go:356] "Generic (PLEG): container finished" podID="cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e" containerID="26dac160b2dd04ea1eeb3311c9de98a0b4376c489acfc396af6ad50cc7fc5291" exitCode=0
Jul 28 13:08:36.229273 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:36.228883    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg" event={"ID":"cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e","Type":"ContainerDied","Data":"26dac160b2dd04ea1eeb3311c9de98a0b4376c489acfc396af6ad50cc7fc5291"}
Jul 28 13:08:37.836947 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:37.836914    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg" podUID="cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Jul 28 13:08:41.297503 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:41.297466    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg" podUID="cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:5000: connect: connection refused"
Jul 28 13:08:41.298079 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:41.297805    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg" podUID="cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 13:08:42.238448 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:42.238419    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-r5f4w"
Jul 28 13:08:42.837761 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:42.837728    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg" podUID="cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Jul 28 13:08:42.838135 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:42.837856    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg"
Jul 28 13:08:47.837373 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:47.837283    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg" podUID="cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Jul 28 13:08:51.132009 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:51.131977    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb"]
Jul 28 13:08:51.135990 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:51.135969    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb"
Jul 28 13:08:51.138594 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:51.138571    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Jul 28 13:08:51.138703 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:51.138580    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Jul 28 13:08:51.148853 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:51.148828    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb"]
Jul 28 13:08:51.241764 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:51.241733    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dac46039-0356-4ad8-abd8-7c5f179a5a48-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-6b6746bb7-99mhb\" (UID: \"dac46039-0356-4ad8-abd8-7c5f179a5a48\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb"
Jul 28 13:08:51.241896 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:51.241772    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dac46039-0356-4ad8-abd8-7c5f179a5a48-kserve-provision-location\") pod \"isvc-logger-predictor-6b6746bb7-99mhb\" (UID: \"dac46039-0356-4ad8-abd8-7c5f179a5a48\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb"
Jul 28 13:08:51.241896 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:51.241814    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dac46039-0356-4ad8-abd8-7c5f179a5a48-cabundle-cert\") pod \"isvc-logger-predictor-6b6746bb7-99mhb\" (UID: \"dac46039-0356-4ad8-abd8-7c5f179a5a48\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb"
Jul 28 13:08:51.241896 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:51.241844    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dac46039-0356-4ad8-abd8-7c5f179a5a48-proxy-tls\") pod \"isvc-logger-predictor-6b6746bb7-99mhb\" (UID: \"dac46039-0356-4ad8-abd8-7c5f179a5a48\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb"
Jul 28 13:08:51.241896 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:51.241870    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch9gl\" (UniqueName: \"kubernetes.io/projected/dac46039-0356-4ad8-abd8-7c5f179a5a48-kube-api-access-ch9gl\") pod \"isvc-logger-predictor-6b6746bb7-99mhb\" (UID: \"dac46039-0356-4ad8-abd8-7c5f179a5a48\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb"
Jul 28 13:08:51.297210 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:51.297169    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg" podUID="cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:5000: connect: connection refused"
Jul 28 13:08:51.297343 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:51.297316    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg"
Jul 28 13:08:51.297535 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:51.297496    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg" podUID="cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 13:08:51.297649 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:51.297633    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg"
Jul 28 13:08:51.342889 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:51.342865    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dac46039-0356-4ad8-abd8-7c5f179a5a48-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-6b6746bb7-99mhb\" (UID: \"dac46039-0356-4ad8-abd8-7c5f179a5a48\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb"
Jul 28 13:08:51.343001 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:51.342898    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dac46039-0356-4ad8-abd8-7c5f179a5a48-kserve-provision-location\") pod \"isvc-logger-predictor-6b6746bb7-99mhb\" (UID: \"dac46039-0356-4ad8-abd8-7c5f179a5a48\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb"
Jul 28 13:08:51.343001 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:51.342926    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dac46039-0356-4ad8-abd8-7c5f179a5a48-cabundle-cert\") pod \"isvc-logger-predictor-6b6746bb7-99mhb\" (UID: \"dac46039-0356-4ad8-abd8-7c5f179a5a48\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb"
Jul 28 13:08:51.343001 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:51.342952    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dac46039-0356-4ad8-abd8-7c5f179a5a48-proxy-tls\") pod \"isvc-logger-predictor-6b6746bb7-99mhb\" (UID: \"dac46039-0356-4ad8-abd8-7c5f179a5a48\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb"
Jul 28 13:08:51.343001 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:51.342987    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch9gl\" (UniqueName: \"kubernetes.io/projected/dac46039-0356-4ad8-abd8-7c5f179a5a48-kube-api-access-ch9gl\") pod \"isvc-logger-predictor-6b6746bb7-99mhb\" (UID: \"dac46039-0356-4ad8-abd8-7c5f179a5a48\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb"
Jul 28 13:08:51.343348 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:51.343324    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dac46039-0356-4ad8-abd8-7c5f179a5a48-kserve-provision-location\") pod \"isvc-logger-predictor-6b6746bb7-99mhb\" (UID: \"dac46039-0356-4ad8-abd8-7c5f179a5a48\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb"
Jul 28 13:08:51.343621 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:51.343601    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dac46039-0356-4ad8-abd8-7c5f179a5a48-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-6b6746bb7-99mhb\" (UID: \"dac46039-0356-4ad8-abd8-7c5f179a5a48\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb"
Jul 28 13:08:51.343738 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:51.343722    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dac46039-0356-4ad8-abd8-7c5f179a5a48-cabundle-cert\") pod \"isvc-logger-predictor-6b6746bb7-99mhb\" (UID: \"dac46039-0356-4ad8-abd8-7c5f179a5a48\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb"
Jul 28 13:08:51.345383 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:51.345366    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dac46039-0356-4ad8-abd8-7c5f179a5a48-proxy-tls\") pod \"isvc-logger-predictor-6b6746bb7-99mhb\" (UID: \"dac46039-0356-4ad8-abd8-7c5f179a5a48\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb"
Jul 28 13:08:51.352042 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:51.352018    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch9gl\" (UniqueName: \"kubernetes.io/projected/dac46039-0356-4ad8-abd8-7c5f179a5a48-kube-api-access-ch9gl\") pod \"isvc-logger-predictor-6b6746bb7-99mhb\" (UID: \"dac46039-0356-4ad8-abd8-7c5f179a5a48\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb"
Jul 28 13:08:51.446080 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:51.446056    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb"
Jul 28 13:08:51.575145 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:51.575113    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb"]
Jul 28 13:08:51.578168 ip-10-0-138-158 kubenswrapper[2574]: W0728 13:08:51.578135    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddac46039_0356_4ad8_abd8_7c5f179a5a48.slice/crio-8a78c79bf6554742906b6eb650f40cb7bd423428a6a726e6cc2989621b70e343 WatchSource:0}: Error finding container 8a78c79bf6554742906b6eb650f40cb7bd423428a6a726e6cc2989621b70e343: Status 404 returned error can't find the container with id 8a78c79bf6554742906b6eb650f40cb7bd423428a6a726e6cc2989621b70e343
Jul 28 13:08:52.304004 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:52.303977    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb" event={"ID":"dac46039-0356-4ad8-abd8-7c5f179a5a48","Type":"ContainerStarted","Data":"e273ea01f108087c0edbae0b36f750569bd6c0a16f4c420a1012cb50dbbc866b"}
Jul 28 13:08:52.304452 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:52.304038    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb" event={"ID":"dac46039-0356-4ad8-abd8-7c5f179a5a48","Type":"ContainerStarted","Data":"8a78c79bf6554742906b6eb650f40cb7bd423428a6a726e6cc2989621b70e343"}
Jul 28 13:08:52.837457 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:52.837420    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg" podUID="cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Jul 28 13:08:56.323184 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:56.323151    2574 generic.go:356] "Generic (PLEG): container finished" podID="dac46039-0356-4ad8-abd8-7c5f179a5a48" containerID="e273ea01f108087c0edbae0b36f750569bd6c0a16f4c420a1012cb50dbbc866b" exitCode=0
Jul 28 13:08:56.323525 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:56.323224    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb" event={"ID":"dac46039-0356-4ad8-abd8-7c5f179a5a48","Type":"ContainerDied","Data":"e273ea01f108087c0edbae0b36f750569bd6c0a16f4c420a1012cb50dbbc866b"}
Jul 28 13:08:57.330552 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:57.330519    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb" event={"ID":"dac46039-0356-4ad8-abd8-7c5f179a5a48","Type":"ContainerStarted","Data":"232fa783d669009b095c1dc3a74a253a26ae8640f49fbdbd265a7bf36ab9c34b"}
Jul 28 13:08:57.330969 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:57.330587    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb" event={"ID":"dac46039-0356-4ad8-abd8-7c5f179a5a48","Type":"ContainerStarted","Data":"a06a896ebc22c24414b8413d7218e6e3191cd9a1694320e646e54348ad68f2d8"}
Jul 28 13:08:57.330969 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:57.330614    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb" event={"ID":"dac46039-0356-4ad8-abd8-7c5f179a5a48","Type":"ContainerStarted","Data":"7e52c19198b3f170d38ef72e5738aa4190a5138af80770a973184c17143d33d5"}
Jul 28 13:08:57.330969 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:57.330935    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb"
Jul 28 13:08:57.331094 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:57.331015    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb"
Jul 28 13:08:57.331094 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:57.331046    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb"
Jul 28 13:08:57.332566 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:57.332540    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb" podUID="dac46039-0356-4ad8-abd8-7c5f179a5a48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 28 13:08:57.333177 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:57.333153    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb" podUID="dac46039-0356-4ad8-abd8-7c5f179a5a48" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 13:08:57.351869 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:57.351831    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb" podStartSLOduration=2.65445235 podStartE2EDuration="6.351817607s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.697365257s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:08:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:08:57.35027099 +0000 UTC m=+949.555657593" watchObservedRunningTime="2026-07-28 13:08:57.351817607 +0000 UTC m=+949.557204208"
Jul 28 13:08:57.837360 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:57.837326    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg" podUID="cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Jul 28 13:08:58.334749 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:58.334716    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb" podUID="dac46039-0356-4ad8-abd8-7c5f179a5a48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 28 13:08:58.335139 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:08:58.335116    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb" podUID="dac46039-0356-4ad8-abd8-7c5f179a5a48" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 13:09:01.297571 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:09:01.297491    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg" podUID="cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:5000: connect: connection refused"
Jul 28 13:09:01.297571 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:09:01.297545    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg" podUID="cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e" containerName="agent" probeResult="failure" output="Get \"http://10.132.0.31:9081/\": dial tcp 10.132.0.31:9081: connect: connection refused"
Jul 28 13:09:01.351939 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:09:01.351908    2574 generic.go:356] "Generic (PLEG): container finished" podID="cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e" containerID="a98cc8a636be3e532f66e8d8de7cd5dc51e1bfba60c069925fa0c2a03d108e29" exitCode=0
Jul 28 13:09:01.352057 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:09:01.351972    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg" event={"ID":"cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e","Type":"ContainerDied","Data":"a98cc8a636be3e532f66e8d8de7cd5dc51e1bfba60c069925fa0c2a03d108e29"}
Jul 28 13:09:01.660023 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:09:01.659998    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg"
Jul 28 13:09:01.726555 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:09:01.726531    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e-kserve-provision-location\") pod \"cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e\" (UID: \"cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e\") "
Jul 28 13:09:01.726647 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:09:01.726563    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e-kube-rbac-proxy-sar-config\") pod \"cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e\" (UID: \"cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e\") "
Jul 28 13:09:01.726647 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:09:01.726603    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e-cabundle-cert\") pod \"cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e\" (UID: \"cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e\") "
Jul 28 13:09:01.726647 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:09:01.726630    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-stlgc\" (UniqueName: \"kubernetes.io/projected/cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e-kube-api-access-stlgc\") pod \"cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e\" (UID: \"cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e\") "
Jul 28 13:09:01.726796 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:09:01.726650    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e-proxy-tls\") pod \"cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e\" (UID: \"cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e\") "
Jul 28 13:09:01.726880 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:09:01.726857    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e" (UID: "cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:09:01.726984 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:09:01.726966    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e-kserve-provision-location\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:09:01.727026 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:09:01.726994    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e" (UID: "cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:09:01.727061 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:09:01.727019    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e" (UID: "cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:09:01.728670 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:09:01.728654    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e" (UID: "cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:09:01.728792 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:09:01.728774    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e-kube-api-access-stlgc" (OuterVolumeSpecName: "kube-api-access-stlgc") pod "cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e" (UID: "cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e"). InnerVolumeSpecName "kube-api-access-stlgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:09:01.827645 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:09:01.827557    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:09:01.827645 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:09:01.827593    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e-cabundle-cert\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:09:01.827645 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:09:01.827603    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stlgc\" (UniqueName: \"kubernetes.io/projected/cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e-kube-api-access-stlgc\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:09:01.827645 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:09:01.827612    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e-proxy-tls\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:09:02.359887 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:09:02.359861    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg"
Jul 28 13:09:02.359887 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:09:02.359865    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg" event={"ID":"cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e","Type":"ContainerDied","Data":"5b5f2a251c8508281bef8831740e7a2d5ab4923e9d511cbb01e234ba50bb6405"}
Jul 28 13:09:02.360269 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:09:02.359967    2574 scope.go:117] "RemoveContainer" containerID="a98cc8a636be3e532f66e8d8de7cd5dc51e1bfba60c069925fa0c2a03d108e29"
Jul 28 13:09:02.368357 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:09:02.368327    2574 scope.go:117] "RemoveContainer" containerID="2389b110d7f66692f622810199091be45f4d49fd7f8452f449a0b01875a808c1"
Jul 28 13:09:02.375523 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:09:02.375506    2574 scope.go:117] "RemoveContainer" containerID="26dac160b2dd04ea1eeb3311c9de98a0b4376c489acfc396af6ad50cc7fc5291"
Jul 28 13:09:02.379185 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:09:02.379166    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg"]
Jul 28 13:09:02.383372 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:09:02.383353    2574 scope.go:117] "RemoveContainer" containerID="644fd2252ac8a3760143147959dba860340bb901122e03776c1f1d45705c71c6"
Jul 28 13:09:02.384591 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:09:02.384569    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-lgpdg"]
Jul 28 13:09:03.339026 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:09:03.338997    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb"
Jul 28 13:09:03.339606 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:09:03.339577    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb" podUID="dac46039-0356-4ad8-abd8-7c5f179a5a48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 28 13:09:03.339917 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:09:03.339890    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb" podUID="dac46039-0356-4ad8-abd8-7c5f179a5a48" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 13:09:04.301648 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:09:04.301615    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e" path="/var/lib/kubelet/pods/cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e/volumes"
Jul 28 13:09:13.339523 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:09:13.339486    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb" podUID="dac46039-0356-4ad8-abd8-7c5f179a5a48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 28 13:09:13.340060 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:09:13.339965    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb" podUID="dac46039-0356-4ad8-abd8-7c5f179a5a48" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 13:09:23.340405 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:09:23.340368    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb" podUID="dac46039-0356-4ad8-abd8-7c5f179a5a48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 28 13:09:23.340914 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:09:23.340891    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb" podUID="dac46039-0356-4ad8-abd8-7c5f179a5a48" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 13:09:33.340296 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:09:33.340213    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb" podUID="dac46039-0356-4ad8-abd8-7c5f179a5a48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 28 13:09:33.340746 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:09:33.340725    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb" podUID="dac46039-0356-4ad8-abd8-7c5f179a5a48" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 13:09:43.339936 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:09:43.339898    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb" podUID="dac46039-0356-4ad8-abd8-7c5f179a5a48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 28 13:09:43.341834 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:09:43.340415    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb" podUID="dac46039-0356-4ad8-abd8-7c5f179a5a48" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 13:09:53.340260 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:09:53.340220    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb" podUID="dac46039-0356-4ad8-abd8-7c5f179a5a48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 28 13:09:53.340769 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:09:53.340672    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb" podUID="dac46039-0356-4ad8-abd8-7c5f179a5a48" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 13:10:03.339699 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:03.339641    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb" podUID="dac46039-0356-4ad8-abd8-7c5f179a5a48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 28 13:10:03.340138 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:03.340108    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb" podUID="dac46039-0356-4ad8-abd8-7c5f179a5a48" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 13:10:13.340887 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:13.340847    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb"
Jul 28 13:10:13.341273 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:13.340962    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb"
Jul 28 13:10:35.460260 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:35.460231    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-6575bcfdfd-r5f4w_63f28749-51c9-4c2a-82c8-58c8476bf1b9/kserve-container/0.log"
Jul 28 13:10:35.645955 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:35.645921    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb"]
Jul 28 13:10:35.646230 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:35.646198    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb" podUID="dac46039-0356-4ad8-abd8-7c5f179a5a48" containerName="kserve-container" containerID="cri-o://7e52c19198b3f170d38ef72e5738aa4190a5138af80770a973184c17143d33d5" gracePeriod=30
Jul 28 13:10:35.646370 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:35.646221    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb" podUID="dac46039-0356-4ad8-abd8-7c5f179a5a48" containerName="kube-rbac-proxy" containerID="cri-o://a06a896ebc22c24414b8413d7218e6e3191cd9a1694320e646e54348ad68f2d8" gracePeriod=30
Jul 28 13:10:35.646445 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:35.646235    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb" podUID="dac46039-0356-4ad8-abd8-7c5f179a5a48" containerName="agent" containerID="cri-o://232fa783d669009b095c1dc3a74a253a26ae8640f49fbdbd265a7bf36ab9c34b" gracePeriod=30
Jul 28 13:10:35.744846 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:35.744774    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-r5f4w"]
Jul 28 13:10:35.745031 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:35.744995    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-r5f4w" podUID="63f28749-51c9-4c2a-82c8-58c8476bf1b9" containerName="kserve-container" containerID="cri-o://82d9a75ef36b11010446dde8fd584a701e9a36ae00ffbeb3eb66289ca3c9d00f" gracePeriod=30
Jul 28 13:10:35.745091 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:35.745033    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-r5f4w" podUID="63f28749-51c9-4c2a-82c8-58c8476bf1b9" containerName="kube-rbac-proxy" containerID="cri-o://23cdc54e5bdf0ef5c2d88c1864ee50315c664ac7eaddced0ef6830cca96ba797" gracePeriod=30
Jul 28 13:10:35.774019 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:35.773994    2574 generic.go:356] "Generic (PLEG): container finished" podID="dac46039-0356-4ad8-abd8-7c5f179a5a48" containerID="a06a896ebc22c24414b8413d7218e6e3191cd9a1694320e646e54348ad68f2d8" exitCode=2
Jul 28 13:10:35.774114 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:35.774060    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb" event={"ID":"dac46039-0356-4ad8-abd8-7c5f179a5a48","Type":"ContainerDied","Data":"a06a896ebc22c24414b8413d7218e6e3191cd9a1694320e646e54348ad68f2d8"}
Jul 28 13:10:35.804408 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:35.804386    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9j57w"]
Jul 28 13:10:35.804749 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:35.804736    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e" containerName="kube-rbac-proxy"
Jul 28 13:10:35.804801 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:35.804750    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e" containerName="kube-rbac-proxy"
Jul 28 13:10:35.804801 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:35.804765    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e" containerName="storage-initializer"
Jul 28 13:10:35.804801 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:35.804770    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e" containerName="storage-initializer"
Jul 28 13:10:35.804801 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:35.804777    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e" containerName="kserve-container"
Jul 28 13:10:35.804801 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:35.804782    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e" containerName="kserve-container"
Jul 28 13:10:35.804801 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:35.804790    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e" containerName="agent"
Jul 28 13:10:35.804801 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:35.804794    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e" containerName="agent"
Jul 28 13:10:35.805010 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:35.804844    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e" containerName="kserve-container"
Jul 28 13:10:35.805010 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:35.804854    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e" containerName="kube-rbac-proxy"
Jul 28 13:10:35.805010 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:35.804863    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb7c3735-5e9b-42bc-80c9-6e26a2b02c9e" containerName="agent"
Jul 28 13:10:35.807863 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:35.807849    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9j57w"
Jul 28 13:10:35.810406 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:35.810383    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Jul 28 13:10:35.810479 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:35.810450    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Jul 28 13:10:35.818908 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:35.818880    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9j57w"]
Jul 28 13:10:35.905453 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:35.905424    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f40f9bce-5596-41b9-9dea-c192431f2fda-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-9j57w\" (UID: \"f40f9bce-5596-41b9-9dea-c192431f2fda\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9j57w"
Jul 28 13:10:35.905637 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:35.905460    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bljdh\" (UniqueName: \"kubernetes.io/projected/f40f9bce-5596-41b9-9dea-c192431f2fda-kube-api-access-bljdh\") pod \"isvc-autogluon-v1-predictor-574b5b7495-9j57w\" (UID: \"f40f9bce-5596-41b9-9dea-c192431f2fda\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9j57w"
Jul 28 13:10:35.905637 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:35.905490    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f40f9bce-5596-41b9-9dea-c192431f2fda-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-9j57w\" (UID: \"f40f9bce-5596-41b9-9dea-c192431f2fda\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9j57w"
Jul 28 13:10:35.905637 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:35.905580    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f40f9bce-5596-41b9-9dea-c192431f2fda-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-9j57w\" (UID: \"f40f9bce-5596-41b9-9dea-c192431f2fda\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9j57w"
Jul 28 13:10:35.905637 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:35.905625    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f40f9bce-5596-41b9-9dea-c192431f2fda-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-9j57w\" (UID: \"f40f9bce-5596-41b9-9dea-c192431f2fda\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9j57w"
Jul 28 13:10:35.987938 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:35.987916    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-r5f4w"
Jul 28 13:10:36.008173 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:36.007105    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f40f9bce-5596-41b9-9dea-c192431f2fda-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-9j57w\" (UID: \"f40f9bce-5596-41b9-9dea-c192431f2fda\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9j57w"
Jul 28 13:10:36.008173 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:36.007155    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bljdh\" (UniqueName: \"kubernetes.io/projected/f40f9bce-5596-41b9-9dea-c192431f2fda-kube-api-access-bljdh\") pod \"isvc-autogluon-v1-predictor-574b5b7495-9j57w\" (UID: \"f40f9bce-5596-41b9-9dea-c192431f2fda\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9j57w"
Jul 28 13:10:36.008173 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:36.007205    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f40f9bce-5596-41b9-9dea-c192431f2fda-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-9j57w\" (UID: \"f40f9bce-5596-41b9-9dea-c192431f2fda\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9j57w"
Jul 28 13:10:36.008173 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:36.007264    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f40f9bce-5596-41b9-9dea-c192431f2fda-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-9j57w\" (UID: \"f40f9bce-5596-41b9-9dea-c192431f2fda\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9j57w"
Jul 28 13:10:36.008173 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:36.007328    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f40f9bce-5596-41b9-9dea-c192431f2fda-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-9j57w\" (UID: \"f40f9bce-5596-41b9-9dea-c192431f2fda\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9j57w"
Jul 28 13:10:36.008173 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:36.007746    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f40f9bce-5596-41b9-9dea-c192431f2fda-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-9j57w\" (UID: \"f40f9bce-5596-41b9-9dea-c192431f2fda\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9j57w"
Jul 28 13:10:36.013728 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:36.009743    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f40f9bce-5596-41b9-9dea-c192431f2fda-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-9j57w\" (UID: \"f40f9bce-5596-41b9-9dea-c192431f2fda\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9j57w"
Jul 28 13:10:36.013728 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:36.010321    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f40f9bce-5596-41b9-9dea-c192431f2fda-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-9j57w\" (UID: \"f40f9bce-5596-41b9-9dea-c192431f2fda\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9j57w"
Jul 28 13:10:36.015963 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:36.015934    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f40f9bce-5596-41b9-9dea-c192431f2fda-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-9j57w\" (UID: \"f40f9bce-5596-41b9-9dea-c192431f2fda\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9j57w"
Jul 28 13:10:36.017019 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:36.016999    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bljdh\" (UniqueName: \"kubernetes.io/projected/f40f9bce-5596-41b9-9dea-c192431f2fda-kube-api-access-bljdh\") pod \"isvc-autogluon-v1-predictor-574b5b7495-9j57w\" (UID: \"f40f9bce-5596-41b9-9dea-c192431f2fda\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9j57w"
Jul 28 13:10:36.108270 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:36.108251    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63f28749-51c9-4c2a-82c8-58c8476bf1b9-proxy-tls\") pod \"63f28749-51c9-4c2a-82c8-58c8476bf1b9\" (UID: \"63f28749-51c9-4c2a-82c8-58c8476bf1b9\") "
Jul 28 13:10:36.108357 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:36.108299    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/63f28749-51c9-4c2a-82c8-58c8476bf1b9-kube-rbac-proxy-sar-config\") pod \"63f28749-51c9-4c2a-82c8-58c8476bf1b9\" (UID: \"63f28749-51c9-4c2a-82c8-58c8476bf1b9\") "
Jul 28 13:10:36.108410 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:36.108355    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jbjj7\" (UniqueName: \"kubernetes.io/projected/63f28749-51c9-4c2a-82c8-58c8476bf1b9-kube-api-access-jbjj7\") pod \"63f28749-51c9-4c2a-82c8-58c8476bf1b9\" (UID: \"63f28749-51c9-4c2a-82c8-58c8476bf1b9\") "
Jul 28 13:10:36.108669 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:36.108646    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63f28749-51c9-4c2a-82c8-58c8476bf1b9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "63f28749-51c9-4c2a-82c8-58c8476bf1b9" (UID: "63f28749-51c9-4c2a-82c8-58c8476bf1b9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:10:36.110002 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:36.109984    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63f28749-51c9-4c2a-82c8-58c8476bf1b9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "63f28749-51c9-4c2a-82c8-58c8476bf1b9" (UID: "63f28749-51c9-4c2a-82c8-58c8476bf1b9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:10:36.110210 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:36.110191    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63f28749-51c9-4c2a-82c8-58c8476bf1b9-kube-api-access-jbjj7" (OuterVolumeSpecName: "kube-api-access-jbjj7") pod "63f28749-51c9-4c2a-82c8-58c8476bf1b9" (UID: "63f28749-51c9-4c2a-82c8-58c8476bf1b9"). InnerVolumeSpecName "kube-api-access-jbjj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:10:36.121347 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:36.121330    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9j57w"
Jul 28 13:10:36.209319 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:36.209289    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbjj7\" (UniqueName: \"kubernetes.io/projected/63f28749-51c9-4c2a-82c8-58c8476bf1b9-kube-api-access-jbjj7\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:10:36.209429 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:36.209325    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63f28749-51c9-4c2a-82c8-58c8476bf1b9-proxy-tls\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:10:36.209429 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:36.209342    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/63f28749-51c9-4c2a-82c8-58c8476bf1b9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:10:36.248594 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:36.248569    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9j57w"]
Jul 28 13:10:36.249854 ip-10-0-138-158 kubenswrapper[2574]: W0728 13:10:36.249827    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf40f9bce_5596_41b9_9dea_c192431f2fda.slice/crio-886f79083ee7b8dc0a856d120a11898513b62f32dfb875b49855e156f3e293c8 WatchSource:0}: Error finding container 886f79083ee7b8dc0a856d120a11898513b62f32dfb875b49855e156f3e293c8: Status 404 returned error can't find the container with id 886f79083ee7b8dc0a856d120a11898513b62f32dfb875b49855e156f3e293c8
Jul 28 13:10:36.780401 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:36.780369    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9j57w" event={"ID":"f40f9bce-5596-41b9-9dea-c192431f2fda","Type":"ContainerStarted","Data":"29f9d892733a3a30cafd7ed3d70d8d7e776e86e2c3efb615b558b1f5435edd39"}
Jul 28 13:10:36.780799 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:36.780446    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9j57w" event={"ID":"f40f9bce-5596-41b9-9dea-c192431f2fda","Type":"ContainerStarted","Data":"886f79083ee7b8dc0a856d120a11898513b62f32dfb875b49855e156f3e293c8"}
Jul 28 13:10:36.781979 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:36.781957    2574 generic.go:356] "Generic (PLEG): container finished" podID="63f28749-51c9-4c2a-82c8-58c8476bf1b9" containerID="23cdc54e5bdf0ef5c2d88c1864ee50315c664ac7eaddced0ef6830cca96ba797" exitCode=2
Jul 28 13:10:36.781979 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:36.781975    2574 generic.go:356] "Generic (PLEG): container finished" podID="63f28749-51c9-4c2a-82c8-58c8476bf1b9" containerID="82d9a75ef36b11010446dde8fd584a701e9a36ae00ffbeb3eb66289ca3c9d00f" exitCode=2
Jul 28 13:10:36.782131 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:36.781993    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-r5f4w" event={"ID":"63f28749-51c9-4c2a-82c8-58c8476bf1b9","Type":"ContainerDied","Data":"23cdc54e5bdf0ef5c2d88c1864ee50315c664ac7eaddced0ef6830cca96ba797"}
Jul 28 13:10:36.782131 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:36.782015    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-r5f4w"
Jul 28 13:10:36.782131 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:36.782065    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-r5f4w" event={"ID":"63f28749-51c9-4c2a-82c8-58c8476bf1b9","Type":"ContainerDied","Data":"82d9a75ef36b11010446dde8fd584a701e9a36ae00ffbeb3eb66289ca3c9d00f"}
Jul 28 13:10:36.782131 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:36.782099    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-r5f4w" event={"ID":"63f28749-51c9-4c2a-82c8-58c8476bf1b9","Type":"ContainerDied","Data":"4e22ab88bbc089f7bad0fb087617b454bbfe1e6a5328d8227714769a5826c4b2"}
Jul 28 13:10:36.782131 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:36.782120    2574 scope.go:117] "RemoveContainer" containerID="23cdc54e5bdf0ef5c2d88c1864ee50315c664ac7eaddced0ef6830cca96ba797"
Jul 28 13:10:36.790906 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:36.790891    2574 scope.go:117] "RemoveContainer" containerID="82d9a75ef36b11010446dde8fd584a701e9a36ae00ffbeb3eb66289ca3c9d00f"
Jul 28 13:10:36.799353 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:36.799321    2574 scope.go:117] "RemoveContainer" containerID="23cdc54e5bdf0ef5c2d88c1864ee50315c664ac7eaddced0ef6830cca96ba797"
Jul 28 13:10:36.799804 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:10:36.799770    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23cdc54e5bdf0ef5c2d88c1864ee50315c664ac7eaddced0ef6830cca96ba797\": container with ID starting with 23cdc54e5bdf0ef5c2d88c1864ee50315c664ac7eaddced0ef6830cca96ba797 not found: ID does not exist" containerID="23cdc54e5bdf0ef5c2d88c1864ee50315c664ac7eaddced0ef6830cca96ba797"
Jul 28 13:10:36.799916 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:36.799810    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23cdc54e5bdf0ef5c2d88c1864ee50315c664ac7eaddced0ef6830cca96ba797"} err="failed to get container status \"23cdc54e5bdf0ef5c2d88c1864ee50315c664ac7eaddced0ef6830cca96ba797\": rpc error: code = NotFound desc = could not find container \"23cdc54e5bdf0ef5c2d88c1864ee50315c664ac7eaddced0ef6830cca96ba797\": container with ID starting with 23cdc54e5bdf0ef5c2d88c1864ee50315c664ac7eaddced0ef6830cca96ba797 not found: ID does not exist"
Jul 28 13:10:36.799916 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:36.799832    2574 scope.go:117] "RemoveContainer" containerID="82d9a75ef36b11010446dde8fd584a701e9a36ae00ffbeb3eb66289ca3c9d00f"
Jul 28 13:10:36.800283 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:10:36.800255    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82d9a75ef36b11010446dde8fd584a701e9a36ae00ffbeb3eb66289ca3c9d00f\": container with ID starting with 82d9a75ef36b11010446dde8fd584a701e9a36ae00ffbeb3eb66289ca3c9d00f not found: ID does not exist" containerID="82d9a75ef36b11010446dde8fd584a701e9a36ae00ffbeb3eb66289ca3c9d00f"
Jul 28 13:10:36.800383 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:36.800290    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82d9a75ef36b11010446dde8fd584a701e9a36ae00ffbeb3eb66289ca3c9d00f"} err="failed to get container status \"82d9a75ef36b11010446dde8fd584a701e9a36ae00ffbeb3eb66289ca3c9d00f\": rpc error: code = NotFound desc = could not find container \"82d9a75ef36b11010446dde8fd584a701e9a36ae00ffbeb3eb66289ca3c9d00f\": container with ID starting with 82d9a75ef36b11010446dde8fd584a701e9a36ae00ffbeb3eb66289ca3c9d00f not found: ID does not exist"
Jul 28 13:10:36.800383 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:36.800313    2574 scope.go:117] "RemoveContainer" containerID="23cdc54e5bdf0ef5c2d88c1864ee50315c664ac7eaddced0ef6830cca96ba797"
Jul 28 13:10:36.800581 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:36.800558    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23cdc54e5bdf0ef5c2d88c1864ee50315c664ac7eaddced0ef6830cca96ba797"} err="failed to get container status \"23cdc54e5bdf0ef5c2d88c1864ee50315c664ac7eaddced0ef6830cca96ba797\": rpc error: code = NotFound desc = could not find container \"23cdc54e5bdf0ef5c2d88c1864ee50315c664ac7eaddced0ef6830cca96ba797\": container with ID starting with 23cdc54e5bdf0ef5c2d88c1864ee50315c664ac7eaddced0ef6830cca96ba797 not found: ID does not exist"
Jul 28 13:10:36.800721 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:36.800585    2574 scope.go:117] "RemoveContainer" containerID="82d9a75ef36b11010446dde8fd584a701e9a36ae00ffbeb3eb66289ca3c9d00f"
Jul 28 13:10:36.800915 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:36.800890    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82d9a75ef36b11010446dde8fd584a701e9a36ae00ffbeb3eb66289ca3c9d00f"} err="failed to get container status \"82d9a75ef36b11010446dde8fd584a701e9a36ae00ffbeb3eb66289ca3c9d00f\": rpc error: code = NotFound desc = could not find container \"82d9a75ef36b11010446dde8fd584a701e9a36ae00ffbeb3eb66289ca3c9d00f\": container with ID starting with 82d9a75ef36b11010446dde8fd584a701e9a36ae00ffbeb3eb66289ca3c9d00f not found: ID does not exist"
Jul 28 13:10:36.812923 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:36.812895    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-r5f4w"]
Jul 28 13:10:36.816321 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:36.816302    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-r5f4w"]
Jul 28 13:10:38.301876 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:38.301846    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63f28749-51c9-4c2a-82c8-58c8476bf1b9" path="/var/lib/kubelet/pods/63f28749-51c9-4c2a-82c8-58c8476bf1b9/volumes"
Jul 28 13:10:38.335474 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:38.335447    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb" podUID="dac46039-0356-4ad8-abd8-7c5f179a5a48" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Jul 28 13:10:39.799071 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:39.799044    2574 generic.go:356] "Generic (PLEG): container finished" podID="dac46039-0356-4ad8-abd8-7c5f179a5a48" containerID="7e52c19198b3f170d38ef72e5738aa4190a5138af80770a973184c17143d33d5" exitCode=0
Jul 28 13:10:39.799389 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:39.799116    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb" event={"ID":"dac46039-0356-4ad8-abd8-7c5f179a5a48","Type":"ContainerDied","Data":"7e52c19198b3f170d38ef72e5738aa4190a5138af80770a973184c17143d33d5"}
Jul 28 13:10:43.335621 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:43.335581    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb" podUID="dac46039-0356-4ad8-abd8-7c5f179a5a48" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Jul 28 13:10:43.339983 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:43.339945    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb" podUID="dac46039-0356-4ad8-abd8-7c5f179a5a48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 28 13:10:43.340316 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:43.340290    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb" podUID="dac46039-0356-4ad8-abd8-7c5f179a5a48" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 13:10:48.335746 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:48.335709    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb" podUID="dac46039-0356-4ad8-abd8-7c5f179a5a48" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Jul 28 13:10:48.336082 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:48.335806    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb"
Jul 28 13:10:53.335252 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:53.335217    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb" podUID="dac46039-0356-4ad8-abd8-7c5f179a5a48" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Jul 28 13:10:53.339652 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:53.339621    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb" podUID="dac46039-0356-4ad8-abd8-7c5f179a5a48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 28 13:10:53.340049 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:53.340027    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb" podUID="dac46039-0356-4ad8-abd8-7c5f179a5a48" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 13:10:58.335746 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:10:58.335705    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb" podUID="dac46039-0356-4ad8-abd8-7c5f179a5a48" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Jul 28 13:11:03.334898 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:11:03.334814    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb" podUID="dac46039-0356-4ad8-abd8-7c5f179a5a48" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Jul 28 13:11:03.340368 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:11:03.340345    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb" podUID="dac46039-0356-4ad8-abd8-7c5f179a5a48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 28 13:11:03.340489 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:11:03.340473    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb"
Jul 28 13:11:03.340948 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:11:03.340925    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb" podUID="dac46039-0356-4ad8-abd8-7c5f179a5a48" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 13:11:03.341025 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:11:03.341017    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb"
Jul 28 13:11:04.910537 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:11:04.910500    2574 generic.go:356] "Generic (PLEG): container finished" podID="f40f9bce-5596-41b9-9dea-c192431f2fda" containerID="29f9d892733a3a30cafd7ed3d70d8d7e776e86e2c3efb615b558b1f5435edd39" exitCode=0
Jul 28 13:11:04.910900 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:11:04.910571    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9j57w" event={"ID":"f40f9bce-5596-41b9-9dea-c192431f2fda","Type":"ContainerDied","Data":"29f9d892733a3a30cafd7ed3d70d8d7e776e86e2c3efb615b558b1f5435edd39"}
Jul 28 13:11:04.911962 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:11:04.911948    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 13:11:05.837591 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:11:05.837562    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb"
Jul 28 13:11:05.921036 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:11:05.920929    2574 generic.go:356] "Generic (PLEG): container finished" podID="dac46039-0356-4ad8-abd8-7c5f179a5a48" containerID="232fa783d669009b095c1dc3a74a253a26ae8640f49fbdbd265a7bf36ab9c34b" exitCode=0
Jul 28 13:11:05.921036 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:11:05.921016    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb"
Jul 28 13:11:05.921036 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:11:05.921015    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb" event={"ID":"dac46039-0356-4ad8-abd8-7c5f179a5a48","Type":"ContainerDied","Data":"232fa783d669009b095c1dc3a74a253a26ae8640f49fbdbd265a7bf36ab9c34b"}
Jul 28 13:11:05.921662 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:11:05.921090    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb" event={"ID":"dac46039-0356-4ad8-abd8-7c5f179a5a48","Type":"ContainerDied","Data":"8a78c79bf6554742906b6eb650f40cb7bd423428a6a726e6cc2989621b70e343"}
Jul 28 13:11:05.921662 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:11:05.921138    2574 scope.go:117] "RemoveContainer" containerID="232fa783d669009b095c1dc3a74a253a26ae8640f49fbdbd265a7bf36ab9c34b"
Jul 28 13:11:05.934625 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:11:05.934528    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dac46039-0356-4ad8-abd8-7c5f179a5a48-kserve-provision-location\") pod \"dac46039-0356-4ad8-abd8-7c5f179a5a48\" (UID: \"dac46039-0356-4ad8-abd8-7c5f179a5a48\") "
Jul 28 13:11:05.934625 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:11:05.934569    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dac46039-0356-4ad8-abd8-7c5f179a5a48-proxy-tls\") pod \"dac46039-0356-4ad8-abd8-7c5f179a5a48\" (UID: \"dac46039-0356-4ad8-abd8-7c5f179a5a48\") "
Jul 28 13:11:05.934625 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:11:05.934612    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ch9gl\" (UniqueName: \"kubernetes.io/projected/dac46039-0356-4ad8-abd8-7c5f179a5a48-kube-api-access-ch9gl\") pod \"dac46039-0356-4ad8-abd8-7c5f179a5a48\" (UID: \"dac46039-0356-4ad8-abd8-7c5f179a5a48\") "
Jul 28 13:11:05.934854 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:11:05.934646    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dac46039-0356-4ad8-abd8-7c5f179a5a48-kube-rbac-proxy-sar-config\") pod \"dac46039-0356-4ad8-abd8-7c5f179a5a48\" (UID: \"dac46039-0356-4ad8-abd8-7c5f179a5a48\") "
Jul 28 13:11:05.934854 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:11:05.934733    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dac46039-0356-4ad8-abd8-7c5f179a5a48-cabundle-cert\") pod \"dac46039-0356-4ad8-abd8-7c5f179a5a48\" (UID: \"dac46039-0356-4ad8-abd8-7c5f179a5a48\") "
Jul 28 13:11:05.934854 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:11:05.934812    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dac46039-0356-4ad8-abd8-7c5f179a5a48-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dac46039-0356-4ad8-abd8-7c5f179a5a48" (UID: "dac46039-0356-4ad8-abd8-7c5f179a5a48"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:11:05.935108 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:11:05.935057    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dac46039-0356-4ad8-abd8-7c5f179a5a48-kserve-provision-location\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:11:05.935108 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:11:05.935100    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dac46039-0356-4ad8-abd8-7c5f179a5a48-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dac46039-0356-4ad8-abd8-7c5f179a5a48" (UID: "dac46039-0356-4ad8-abd8-7c5f179a5a48"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:11:05.935373 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:11:05.935340    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dac46039-0356-4ad8-abd8-7c5f179a5a48-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dac46039-0356-4ad8-abd8-7c5f179a5a48" (UID: "dac46039-0356-4ad8-abd8-7c5f179a5a48"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:11:05.939027 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:11:05.938991    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dac46039-0356-4ad8-abd8-7c5f179a5a48-kube-api-access-ch9gl" (OuterVolumeSpecName: "kube-api-access-ch9gl") pod "dac46039-0356-4ad8-abd8-7c5f179a5a48" (UID: "dac46039-0356-4ad8-abd8-7c5f179a5a48"). InnerVolumeSpecName "kube-api-access-ch9gl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:11:05.939368 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:11:05.939341    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dac46039-0356-4ad8-abd8-7c5f179a5a48-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dac46039-0356-4ad8-abd8-7c5f179a5a48" (UID: "dac46039-0356-4ad8-abd8-7c5f179a5a48"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:11:06.035995 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:11:06.035957    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dac46039-0356-4ad8-abd8-7c5f179a5a48-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:11:06.035995 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:11:06.035995    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dac46039-0356-4ad8-abd8-7c5f179a5a48-cabundle-cert\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:11:06.036186 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:11:06.036011    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dac46039-0356-4ad8-abd8-7c5f179a5a48-proxy-tls\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:11:06.036186 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:11:06.036021    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch9gl\" (UniqueName: \"kubernetes.io/projected/dac46039-0356-4ad8-abd8-7c5f179a5a48-kube-api-access-ch9gl\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:11:06.245975 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:11:06.245945    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb"]
Jul 28 13:11:06.250650 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:11:06.250626    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-99mhb"]
Jul 28 13:11:06.306900 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:11:06.306870    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dac46039-0356-4ad8-abd8-7c5f179a5a48" path="/var/lib/kubelet/pods/dac46039-0356-4ad8-abd8-7c5f179a5a48/volumes"
Jul 28 13:11:07.053008 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:11:07.052641    2574 scope.go:117] "RemoveContainer" containerID="a06a896ebc22c24414b8413d7218e6e3191cd9a1694320e646e54348ad68f2d8"
Jul 28 13:11:07.199957 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:11:07.199930    2574 scope.go:117] "RemoveContainer" containerID="7e52c19198b3f170d38ef72e5738aa4190a5138af80770a973184c17143d33d5"
Jul 28 13:11:07.302419 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:11:07.302383    2574 scope.go:117] "RemoveContainer" containerID="e273ea01f108087c0edbae0b36f750569bd6c0a16f4c420a1012cb50dbbc866b"
Jul 28 13:11:07.312563 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:11:07.312533    2574 scope.go:117] "RemoveContainer" containerID="232fa783d669009b095c1dc3a74a253a26ae8640f49fbdbd265a7bf36ab9c34b"
Jul 28 13:11:07.312881 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:11:07.312848    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"232fa783d669009b095c1dc3a74a253a26ae8640f49fbdbd265a7bf36ab9c34b\": container with ID starting with 232fa783d669009b095c1dc3a74a253a26ae8640f49fbdbd265a7bf36ab9c34b not found: ID does not exist" containerID="232fa783d669009b095c1dc3a74a253a26ae8640f49fbdbd265a7bf36ab9c34b"
Jul 28 13:11:07.312960 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:11:07.312884    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"232fa783d669009b095c1dc3a74a253a26ae8640f49fbdbd265a7bf36ab9c34b"} err="failed to get container status \"232fa783d669009b095c1dc3a74a253a26ae8640f49fbdbd265a7bf36ab9c34b\": rpc error: code = NotFound desc = could not find container \"232fa783d669009b095c1dc3a74a253a26ae8640f49fbdbd265a7bf36ab9c34b\": container with ID starting with 232fa783d669009b095c1dc3a74a253a26ae8640f49fbdbd265a7bf36ab9c34b not found: ID does not exist"
Jul 28 13:11:07.312960 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:11:07.312906    2574 scope.go:117] "RemoveContainer" containerID="a06a896ebc22c24414b8413d7218e6e3191cd9a1694320e646e54348ad68f2d8"
Jul 28 13:11:07.313202 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:11:07.313177    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a06a896ebc22c24414b8413d7218e6e3191cd9a1694320e646e54348ad68f2d8\": container with ID starting with a06a896ebc22c24414b8413d7218e6e3191cd9a1694320e646e54348ad68f2d8 not found: ID does not exist" containerID="a06a896ebc22c24414b8413d7218e6e3191cd9a1694320e646e54348ad68f2d8"
Jul 28 13:11:07.313273 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:11:07.313214    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a06a896ebc22c24414b8413d7218e6e3191cd9a1694320e646e54348ad68f2d8"} err="failed to get container status \"a06a896ebc22c24414b8413d7218e6e3191cd9a1694320e646e54348ad68f2d8\": rpc error: code = NotFound desc = could not find container \"a06a896ebc22c24414b8413d7218e6e3191cd9a1694320e646e54348ad68f2d8\": container with ID starting with a06a896ebc22c24414b8413d7218e6e3191cd9a1694320e646e54348ad68f2d8 not found: ID does not exist"
Jul 28 13:11:07.313273 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:11:07.313238    2574 scope.go:117] "RemoveContainer" containerID="7e52c19198b3f170d38ef72e5738aa4190a5138af80770a973184c17143d33d5"
Jul 28 13:11:07.313520 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:11:07.313485    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e52c19198b3f170d38ef72e5738aa4190a5138af80770a973184c17143d33d5\": container with ID starting with 7e52c19198b3f170d38ef72e5738aa4190a5138af80770a973184c17143d33d5 not found: ID does not exist" containerID="7e52c19198b3f170d38ef72e5738aa4190a5138af80770a973184c17143d33d5"
Jul 28 13:11:07.313620 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:11:07.313524    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e52c19198b3f170d38ef72e5738aa4190a5138af80770a973184c17143d33d5"} err="failed to get container status \"7e52c19198b3f170d38ef72e5738aa4190a5138af80770a973184c17143d33d5\": rpc error: code = NotFound desc = could not find container \"7e52c19198b3f170d38ef72e5738aa4190a5138af80770a973184c17143d33d5\": container with ID starting with 7e52c19198b3f170d38ef72e5738aa4190a5138af80770a973184c17143d33d5 not found: ID does not exist"
Jul 28 13:11:07.313620 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:11:07.313540    2574 scope.go:117] "RemoveContainer" containerID="e273ea01f108087c0edbae0b36f750569bd6c0a16f4c420a1012cb50dbbc866b"
Jul 28 13:11:07.314194 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:11:07.314172    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e273ea01f108087c0edbae0b36f750569bd6c0a16f4c420a1012cb50dbbc866b\": container with ID starting with e273ea01f108087c0edbae0b36f750569bd6c0a16f4c420a1012cb50dbbc866b not found: ID does not exist" containerID="e273ea01f108087c0edbae0b36f750569bd6c0a16f4c420a1012cb50dbbc866b"
Jul 28 13:11:07.314272 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:11:07.314199    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e273ea01f108087c0edbae0b36f750569bd6c0a16f4c420a1012cb50dbbc866b"} err="failed to get container status \"e273ea01f108087c0edbae0b36f750569bd6c0a16f4c420a1012cb50dbbc866b\": rpc error: code = NotFound desc = could not find container \"e273ea01f108087c0edbae0b36f750569bd6c0a16f4c420a1012cb50dbbc866b\": container with ID starting with e273ea01f108087c0edbae0b36f750569bd6c0a16f4c420a1012cb50dbbc866b not found: ID does not exist"
Jul 28 13:13:27.625733 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:13:27.625705    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9j57w" event={"ID":"f40f9bce-5596-41b9-9dea-c192431f2fda","Type":"ContainerStarted","Data":"900adff7b9661894038b7d1a47f1a715eb905ac9e718f036927b57501615eb76"}
Jul 28 13:13:28.632367 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:13:28.632324    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9j57w" event={"ID":"f40f9bce-5596-41b9-9dea-c192431f2fda","Type":"ContainerStarted","Data":"aabbfbb754fb2e8b04b1ae00441788ff89e6defb66c31728068d8fbd6819df7d"}
Jul 28 13:13:28.632975 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:13:28.632453    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9j57w"
Jul 28 13:13:29.637383 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:13:29.637349    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9j57w"
Jul 28 13:13:29.638566 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:13:29.638538    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9j57w" podUID="f40f9bce-5596-41b9-9dea-c192431f2fda" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 28 13:13:30.641890 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:13:30.641847    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9j57w" podUID="f40f9bce-5596-41b9-9dea-c192431f2fda" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 28 13:13:35.646278 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:13:35.646251    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9j57w"
Jul 28 13:13:35.646876 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:13:35.646847    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9j57w" podUID="f40f9bce-5596-41b9-9dea-c192431f2fda" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 28 13:13:35.669759 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:13:35.669706    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9j57w" podStartSLOduration=10.343679555 podStartE2EDuration="3m0.669675555s" totalImagesPullingTime="2m22.364888818s" totalInitContainerRuntime="27.961107182s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:10:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:13:28.699155356 +0000 UTC m=+1220.904541956" watchObservedRunningTime="2026-07-28 13:13:35.669675555 +0000 UTC m=+1227.875062157"
Jul 28 13:13:45.648346 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:13:45.648316    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9j57w"
Jul 28 13:14:02.172300 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:02.172219    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9j57w"]
Jul 28 13:14:02.172764 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:02.172513    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9j57w" podUID="f40f9bce-5596-41b9-9dea-c192431f2fda" containerName="kserve-container" containerID="cri-o://900adff7b9661894038b7d1a47f1a715eb905ac9e718f036927b57501615eb76" gracePeriod=30
Jul 28 13:14:02.172764 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:02.172542    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9j57w" podUID="f40f9bce-5596-41b9-9dea-c192431f2fda" containerName="kube-rbac-proxy" containerID="cri-o://aabbfbb754fb2e8b04b1ae00441788ff89e6defb66c31728068d8fbd6819df7d" gracePeriod=30
Jul 28 13:14:02.786037 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:02.786000    2574 generic.go:356] "Generic (PLEG): container finished" podID="f40f9bce-5596-41b9-9dea-c192431f2fda" containerID="aabbfbb754fb2e8b04b1ae00441788ff89e6defb66c31728068d8fbd6819df7d" exitCode=2
Jul 28 13:14:02.786231 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:02.786074    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9j57w" event={"ID":"f40f9bce-5596-41b9-9dea-c192431f2fda","Type":"ContainerDied","Data":"aabbfbb754fb2e8b04b1ae00441788ff89e6defb66c31728068d8fbd6819df7d"}
Jul 28 13:14:03.443955 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:03.443902    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9j57w"
Jul 28 13:14:03.571420 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:03.571397    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f40f9bce-5596-41b9-9dea-c192431f2fda-proxy-tls\") pod \"f40f9bce-5596-41b9-9dea-c192431f2fda\" (UID: \"f40f9bce-5596-41b9-9dea-c192431f2fda\") "
Jul 28 13:14:03.571540 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:03.571475    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f40f9bce-5596-41b9-9dea-c192431f2fda-cabundle-cert\") pod \"f40f9bce-5596-41b9-9dea-c192431f2fda\" (UID: \"f40f9bce-5596-41b9-9dea-c192431f2fda\") "
Jul 28 13:14:03.571540 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:03.571503    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f40f9bce-5596-41b9-9dea-c192431f2fda-kube-rbac-proxy-sar-config\") pod \"f40f9bce-5596-41b9-9dea-c192431f2fda\" (UID: \"f40f9bce-5596-41b9-9dea-c192431f2fda\") "
Jul 28 13:14:03.571540 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:03.571520    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f40f9bce-5596-41b9-9dea-c192431f2fda-kserve-provision-location\") pod \"f40f9bce-5596-41b9-9dea-c192431f2fda\" (UID: \"f40f9bce-5596-41b9-9dea-c192431f2fda\") "
Jul 28 13:14:03.571540 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:03.571539    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bljdh\" (UniqueName: \"kubernetes.io/projected/f40f9bce-5596-41b9-9dea-c192431f2fda-kube-api-access-bljdh\") pod \"f40f9bce-5596-41b9-9dea-c192431f2fda\" (UID: \"f40f9bce-5596-41b9-9dea-c192431f2fda\") "
Jul 28 13:14:03.571901 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:03.571876    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f40f9bce-5596-41b9-9dea-c192431f2fda-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f40f9bce-5596-41b9-9dea-c192431f2fda" (UID: "f40f9bce-5596-41b9-9dea-c192431f2fda"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:14:03.571959 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:03.571874    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f40f9bce-5596-41b9-9dea-c192431f2fda-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f40f9bce-5596-41b9-9dea-c192431f2fda" (UID: "f40f9bce-5596-41b9-9dea-c192431f2fda"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:14:03.573505 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:03.573472    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f40f9bce-5596-41b9-9dea-c192431f2fda-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f40f9bce-5596-41b9-9dea-c192431f2fda" (UID: "f40f9bce-5596-41b9-9dea-c192431f2fda"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:14:03.573673 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:03.573648    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f40f9bce-5596-41b9-9dea-c192431f2fda-kube-api-access-bljdh" (OuterVolumeSpecName: "kube-api-access-bljdh") pod "f40f9bce-5596-41b9-9dea-c192431f2fda" (UID: "f40f9bce-5596-41b9-9dea-c192431f2fda"). InnerVolumeSpecName "kube-api-access-bljdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:14:03.575038 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:03.575016    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f40f9bce-5596-41b9-9dea-c192431f2fda-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f40f9bce-5596-41b9-9dea-c192431f2fda" (UID: "f40f9bce-5596-41b9-9dea-c192431f2fda"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:14:03.672667 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:03.672645    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f40f9bce-5596-41b9-9dea-c192431f2fda-cabundle-cert\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:14:03.672667 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:03.672665    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f40f9bce-5596-41b9-9dea-c192431f2fda-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:14:03.672801 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:03.672676    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f40f9bce-5596-41b9-9dea-c192431f2fda-kserve-provision-location\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:14:03.672801 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:03.672702    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bljdh\" (UniqueName: \"kubernetes.io/projected/f40f9bce-5596-41b9-9dea-c192431f2fda-kube-api-access-bljdh\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:14:03.672801 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:03.672711    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f40f9bce-5596-41b9-9dea-c192431f2fda-proxy-tls\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:14:03.792184 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:03.792129    2574 generic.go:356] "Generic (PLEG): container finished" podID="f40f9bce-5596-41b9-9dea-c192431f2fda" containerID="900adff7b9661894038b7d1a47f1a715eb905ac9e718f036927b57501615eb76" exitCode=0
Jul 28 13:14:03.792263 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:03.792181    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9j57w"
Jul 28 13:14:03.792263 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:03.792207    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9j57w" event={"ID":"f40f9bce-5596-41b9-9dea-c192431f2fda","Type":"ContainerDied","Data":"900adff7b9661894038b7d1a47f1a715eb905ac9e718f036927b57501615eb76"}
Jul 28 13:14:03.792332 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:03.792264    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9j57w" event={"ID":"f40f9bce-5596-41b9-9dea-c192431f2fda","Type":"ContainerDied","Data":"886f79083ee7b8dc0a856d120a11898513b62f32dfb875b49855e156f3e293c8"}
Jul 28 13:14:03.792332 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:03.792291    2574 scope.go:117] "RemoveContainer" containerID="aabbfbb754fb2e8b04b1ae00441788ff89e6defb66c31728068d8fbd6819df7d"
Jul 28 13:14:03.802357 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:03.802338    2574 scope.go:117] "RemoveContainer" containerID="900adff7b9661894038b7d1a47f1a715eb905ac9e718f036927b57501615eb76"
Jul 28 13:14:03.809701 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:03.809672    2574 scope.go:117] "RemoveContainer" containerID="29f9d892733a3a30cafd7ed3d70d8d7e776e86e2c3efb615b558b1f5435edd39"
Jul 28 13:14:03.816906 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:03.816888    2574 scope.go:117] "RemoveContainer" containerID="aabbfbb754fb2e8b04b1ae00441788ff89e6defb66c31728068d8fbd6819df7d"
Jul 28 13:14:03.817157 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:14:03.817136    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aabbfbb754fb2e8b04b1ae00441788ff89e6defb66c31728068d8fbd6819df7d\": container with ID starting with aabbfbb754fb2e8b04b1ae00441788ff89e6defb66c31728068d8fbd6819df7d not found: ID does not exist" containerID="aabbfbb754fb2e8b04b1ae00441788ff89e6defb66c31728068d8fbd6819df7d"
Jul 28 13:14:03.817236 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:03.817163    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aabbfbb754fb2e8b04b1ae00441788ff89e6defb66c31728068d8fbd6819df7d"} err="failed to get container status \"aabbfbb754fb2e8b04b1ae00441788ff89e6defb66c31728068d8fbd6819df7d\": rpc error: code = NotFound desc = could not find container \"aabbfbb754fb2e8b04b1ae00441788ff89e6defb66c31728068d8fbd6819df7d\": container with ID starting with aabbfbb754fb2e8b04b1ae00441788ff89e6defb66c31728068d8fbd6819df7d not found: ID does not exist"
Jul 28 13:14:03.817236 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:03.817180    2574 scope.go:117] "RemoveContainer" containerID="900adff7b9661894038b7d1a47f1a715eb905ac9e718f036927b57501615eb76"
Jul 28 13:14:03.817388 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:14:03.817373    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"900adff7b9661894038b7d1a47f1a715eb905ac9e718f036927b57501615eb76\": container with ID starting with 900adff7b9661894038b7d1a47f1a715eb905ac9e718f036927b57501615eb76 not found: ID does not exist" containerID="900adff7b9661894038b7d1a47f1a715eb905ac9e718f036927b57501615eb76"
Jul 28 13:14:03.817426 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:03.817392    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"900adff7b9661894038b7d1a47f1a715eb905ac9e718f036927b57501615eb76"} err="failed to get container status \"900adff7b9661894038b7d1a47f1a715eb905ac9e718f036927b57501615eb76\": rpc error: code = NotFound desc = could not find container \"900adff7b9661894038b7d1a47f1a715eb905ac9e718f036927b57501615eb76\": container with ID starting with 900adff7b9661894038b7d1a47f1a715eb905ac9e718f036927b57501615eb76 not found: ID does not exist"
Jul 28 13:14:03.817426 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:03.817404    2574 scope.go:117] "RemoveContainer" containerID="29f9d892733a3a30cafd7ed3d70d8d7e776e86e2c3efb615b558b1f5435edd39"
Jul 28 13:14:03.817529 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:03.817504    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9j57w"]
Jul 28 13:14:03.817676 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:14:03.817658    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29f9d892733a3a30cafd7ed3d70d8d7e776e86e2c3efb615b558b1f5435edd39\": container with ID starting with 29f9d892733a3a30cafd7ed3d70d8d7e776e86e2c3efb615b558b1f5435edd39 not found: ID does not exist" containerID="29f9d892733a3a30cafd7ed3d70d8d7e776e86e2c3efb615b558b1f5435edd39"
Jul 28 13:14:03.817748 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:03.817704    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29f9d892733a3a30cafd7ed3d70d8d7e776e86e2c3efb615b558b1f5435edd39"} err="failed to get container status \"29f9d892733a3a30cafd7ed3d70d8d7e776e86e2c3efb615b558b1f5435edd39\": rpc error: code = NotFound desc = could not find container \"29f9d892733a3a30cafd7ed3d70d8d7e776e86e2c3efb615b558b1f5435edd39\": container with ID starting with 29f9d892733a3a30cafd7ed3d70d8d7e776e86e2c3efb615b558b1f5435edd39 not found: ID does not exist"
Jul 28 13:14:03.821035 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:03.821015    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9j57w"]
Jul 28 13:14:04.301838 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:04.301808    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f40f9bce-5596-41b9-9dea-c192431f2fda" path="/var/lib/kubelet/pods/f40f9bce-5596-41b9-9dea-c192431f2fda/volumes"
Jul 28 13:14:07.342030 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:07.341997    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zlgv4"]
Jul 28 13:14:07.342466 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:07.342304    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dac46039-0356-4ad8-abd8-7c5f179a5a48" containerName="agent"
Jul 28 13:14:07.342466 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:07.342315    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="dac46039-0356-4ad8-abd8-7c5f179a5a48" containerName="agent"
Jul 28 13:14:07.342466 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:07.342329    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f40f9bce-5596-41b9-9dea-c192431f2fda" containerName="kube-rbac-proxy"
Jul 28 13:14:07.342466 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:07.342337    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f40f9bce-5596-41b9-9dea-c192431f2fda" containerName="kube-rbac-proxy"
Jul 28 13:14:07.342466 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:07.342349    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63f28749-51c9-4c2a-82c8-58c8476bf1b9" containerName="kserve-container"
Jul 28 13:14:07.342466 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:07.342355    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="63f28749-51c9-4c2a-82c8-58c8476bf1b9" containerName="kserve-container"
Jul 28 13:14:07.342466 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:07.342364    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63f28749-51c9-4c2a-82c8-58c8476bf1b9" containerName="kube-rbac-proxy"
Jul 28 13:14:07.342466 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:07.342369    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="63f28749-51c9-4c2a-82c8-58c8476bf1b9" containerName="kube-rbac-proxy"
Jul 28 13:14:07.342466 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:07.342375    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dac46039-0356-4ad8-abd8-7c5f179a5a48" containerName="kserve-container"
Jul 28 13:14:07.342466 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:07.342380    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="dac46039-0356-4ad8-abd8-7c5f179a5a48" containerName="kserve-container"
Jul 28 13:14:07.342466 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:07.342386    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dac46039-0356-4ad8-abd8-7c5f179a5a48" containerName="kube-rbac-proxy"
Jul 28 13:14:07.342466 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:07.342391    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="dac46039-0356-4ad8-abd8-7c5f179a5a48" containerName="kube-rbac-proxy"
Jul 28 13:14:07.342466 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:07.342399    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f40f9bce-5596-41b9-9dea-c192431f2fda" containerName="storage-initializer"
Jul 28 13:14:07.342466 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:07.342404    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f40f9bce-5596-41b9-9dea-c192431f2fda" containerName="storage-initializer"
Jul 28 13:14:07.342466 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:07.342414    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f40f9bce-5596-41b9-9dea-c192431f2fda" containerName="kserve-container"
Jul 28 13:14:07.342466 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:07.342419    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f40f9bce-5596-41b9-9dea-c192431f2fda" containerName="kserve-container"
Jul 28 13:14:07.342466 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:07.342426    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dac46039-0356-4ad8-abd8-7c5f179a5a48" containerName="storage-initializer"
Jul 28 13:14:07.342466 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:07.342431    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="dac46039-0356-4ad8-abd8-7c5f179a5a48" containerName="storage-initializer"
Jul 28 13:14:07.342466 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:07.342475    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="dac46039-0356-4ad8-abd8-7c5f179a5a48" containerName="agent"
Jul 28 13:14:07.343113 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:07.342483    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="dac46039-0356-4ad8-abd8-7c5f179a5a48" containerName="kube-rbac-proxy"
Jul 28 13:14:07.343113 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:07.342489    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f40f9bce-5596-41b9-9dea-c192431f2fda" containerName="kube-rbac-proxy"
Jul 28 13:14:07.343113 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:07.342496    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="63f28749-51c9-4c2a-82c8-58c8476bf1b9" containerName="kube-rbac-proxy"
Jul 28 13:14:07.343113 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:07.342502    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="dac46039-0356-4ad8-abd8-7c5f179a5a48" containerName="kserve-container"
Jul 28 13:14:07.343113 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:07.342508    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f40f9bce-5596-41b9-9dea-c192431f2fda" containerName="kserve-container"
Jul 28 13:14:07.343113 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:07.342514    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="63f28749-51c9-4c2a-82c8-58c8476bf1b9" containerName="kserve-container"
Jul 28 13:14:07.345752 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:07.345733    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zlgv4"
Jul 28 13:14:07.348519 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:07.348489    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:14:07.348519 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:07.348515    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Jul 28 13:14:07.348706 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:07.348525    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 13:14:07.348706 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:07.348533    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Jul 28 13:14:07.349578 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:07.349559    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rxfwd\""
Jul 28 13:14:07.349578 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:07.349570    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:14:07.349672 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:07.349603    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:14:07.356049 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:07.356029    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zlgv4"]
Jul 28 13:14:07.498921 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:07.498896    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e489fa86-38c0-4961-b6c4-c87d4c24683d-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-zlgv4\" (UID: \"e489fa86-38c0-4961-b6c4-c87d4c24683d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zlgv4"
Jul 28 13:14:07.499046 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:07.498927    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd69m\" (UniqueName: \"kubernetes.io/projected/e489fa86-38c0-4961-b6c4-c87d4c24683d-kube-api-access-vd69m\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-zlgv4\" (UID: \"e489fa86-38c0-4961-b6c4-c87d4c24683d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zlgv4"
Jul 28 13:14:07.499046 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:07.498996    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e489fa86-38c0-4961-b6c4-c87d4c24683d-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-zlgv4\" (UID: \"e489fa86-38c0-4961-b6c4-c87d4c24683d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zlgv4"
Jul 28 13:14:07.499159 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:07.499055    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e489fa86-38c0-4961-b6c4-c87d4c24683d-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-zlgv4\" (UID: \"e489fa86-38c0-4961-b6c4-c87d4c24683d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zlgv4"
Jul 28 13:14:07.499159 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:07.499094    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e489fa86-38c0-4961-b6c4-c87d4c24683d-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-zlgv4\" (UID: \"e489fa86-38c0-4961-b6c4-c87d4c24683d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zlgv4"
Jul 28 13:14:07.600414 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:07.600356    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e489fa86-38c0-4961-b6c4-c87d4c24683d-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-zlgv4\" (UID: \"e489fa86-38c0-4961-b6c4-c87d4c24683d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zlgv4"
Jul 28 13:14:07.600414 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:07.600388    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e489fa86-38c0-4961-b6c4-c87d4c24683d-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-zlgv4\" (UID: \"e489fa86-38c0-4961-b6c4-c87d4c24683d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zlgv4"
Jul 28 13:14:07.600577 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:07.600421    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e489fa86-38c0-4961-b6c4-c87d4c24683d-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-zlgv4\" (UID: \"e489fa86-38c0-4961-b6c4-c87d4c24683d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zlgv4"
Jul 28 13:14:07.600577 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:07.600447    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd69m\" (UniqueName: \"kubernetes.io/projected/e489fa86-38c0-4961-b6c4-c87d4c24683d-kube-api-access-vd69m\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-zlgv4\" (UID: \"e489fa86-38c0-4961-b6c4-c87d4c24683d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zlgv4"
Jul 28 13:14:07.600577 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:07.600495    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e489fa86-38c0-4961-b6c4-c87d4c24683d-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-zlgv4\" (UID: \"e489fa86-38c0-4961-b6c4-c87d4c24683d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zlgv4"
Jul 28 13:14:07.600835 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:07.600817    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e489fa86-38c0-4961-b6c4-c87d4c24683d-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-zlgv4\" (UID: \"e489fa86-38c0-4961-b6c4-c87d4c24683d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zlgv4"
Jul 28 13:14:07.601220 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:07.601198    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e489fa86-38c0-4961-b6c4-c87d4c24683d-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-zlgv4\" (UID: \"e489fa86-38c0-4961-b6c4-c87d4c24683d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zlgv4"
Jul 28 13:14:07.601271 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:07.601204    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e489fa86-38c0-4961-b6c4-c87d4c24683d-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-zlgv4\" (UID: \"e489fa86-38c0-4961-b6c4-c87d4c24683d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zlgv4"
Jul 28 13:14:07.602918 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:07.602897    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e489fa86-38c0-4961-b6c4-c87d4c24683d-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-zlgv4\" (UID: \"e489fa86-38c0-4961-b6c4-c87d4c24683d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zlgv4"
Jul 28 13:14:07.609842 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:07.609820    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd69m\" (UniqueName: \"kubernetes.io/projected/e489fa86-38c0-4961-b6c4-c87d4c24683d-kube-api-access-vd69m\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-zlgv4\" (UID: \"e489fa86-38c0-4961-b6c4-c87d4c24683d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zlgv4"
Jul 28 13:14:07.655763 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:07.655743    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zlgv4"
Jul 28 13:14:07.778081 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:07.778057    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zlgv4"]
Jul 28 13:14:07.779576 ip-10-0-138-158 kubenswrapper[2574]: W0728 13:14:07.779552    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode489fa86_38c0_4961_b6c4_c87d4c24683d.slice/crio-edce6e3064244d8ea3e9c41b416230d5295585119af13bb270f0f1a9a865b3b0 WatchSource:0}: Error finding container edce6e3064244d8ea3e9c41b416230d5295585119af13bb270f0f1a9a865b3b0: Status 404 returned error can't find the container with id edce6e3064244d8ea3e9c41b416230d5295585119af13bb270f0f1a9a865b3b0
Jul 28 13:14:07.813149 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:07.813124    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zlgv4" event={"ID":"e489fa86-38c0-4961-b6c4-c87d4c24683d","Type":"ContainerStarted","Data":"edce6e3064244d8ea3e9c41b416230d5295585119af13bb270f0f1a9a865b3b0"}
Jul 28 13:14:08.818654 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:08.818620    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zlgv4" event={"ID":"e489fa86-38c0-4961-b6c4-c87d4c24683d","Type":"ContainerStarted","Data":"cd2bbd8d8ef2d21d91bcc1880b52a8546d448638b7a763ed13a6be4a258ee1b7"}
Jul 28 13:14:37.952412 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:37.952375    2574 generic.go:356] "Generic (PLEG): container finished" podID="e489fa86-38c0-4961-b6c4-c87d4c24683d" containerID="cd2bbd8d8ef2d21d91bcc1880b52a8546d448638b7a763ed13a6be4a258ee1b7" exitCode=0
Jul 28 13:14:37.952806 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:37.952449    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zlgv4" event={"ID":"e489fa86-38c0-4961-b6c4-c87d4c24683d","Type":"ContainerDied","Data":"cd2bbd8d8ef2d21d91bcc1880b52a8546d448638b7a763ed13a6be4a258ee1b7"}
Jul 28 13:14:38.959183 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:38.959151    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zlgv4" event={"ID":"e489fa86-38c0-4961-b6c4-c87d4c24683d","Type":"ContainerStarted","Data":"619d514d71b85a52745de737d3379980a54e15db19b11950183f939340ca5ce8"}
Jul 28 13:14:38.959789 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:38.959208    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zlgv4" event={"ID":"e489fa86-38c0-4961-b6c4-c87d4c24683d","Type":"ContainerStarted","Data":"c6eb013759f2c50e57914bf035c7f095a53a11def1ec6f6e4dc6e8e4fd43a7c6"}
Jul 28 13:14:38.959789 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:38.959407    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zlgv4"
Jul 28 13:14:38.982290 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:38.982241    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zlgv4" podStartSLOduration=2.620902772 podStartE2EDuration="31.982228953s" totalImagesPullingTime="0s" totalInitContainerRuntime="29.361326181s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:14:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:14:38.980100289 +0000 UTC m=+1291.185486891" watchObservedRunningTime="2026-07-28 13:14:38.982228953 +0000 UTC m=+1291.187615554"
Jul 28 13:14:39.964501 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:39.964462    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zlgv4"
Jul 28 13:14:45.974771 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:14:45.974743    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zlgv4"
Jul 28 13:16:11.300166 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:11.300131    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zlgv4"
Jul 28 13:16:22.338871 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:22.338824    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zlgv4" podUID="e489fa86-38c0-4961-b6c4-c87d4c24683d" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.35:8080/v2/models/isvc-autogluon-v2/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 28 13:16:23.479070 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:23.479034    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zlgv4"]
Jul 28 13:16:23.479531 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:23.479299    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zlgv4" podUID="e489fa86-38c0-4961-b6c4-c87d4c24683d" containerName="kserve-container" containerID="cri-o://c6eb013759f2c50e57914bf035c7f095a53a11def1ec6f6e4dc6e8e4fd43a7c6" gracePeriod=30
Jul 28 13:16:23.479531 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:23.479320    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zlgv4" podUID="e489fa86-38c0-4961-b6c4-c87d4c24683d" containerName="kube-rbac-proxy" containerID="cri-o://619d514d71b85a52745de737d3379980a54e15db19b11950183f939340ca5ce8" gracePeriod=30
Jul 28 13:16:24.419633 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:24.419599    2574 generic.go:356] "Generic (PLEG): container finished" podID="e489fa86-38c0-4961-b6c4-c87d4c24683d" containerID="619d514d71b85a52745de737d3379980a54e15db19b11950183f939340ca5ce8" exitCode=2
Jul 28 13:16:24.419825 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:24.419667    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zlgv4" event={"ID":"e489fa86-38c0-4961-b6c4-c87d4c24683d","Type":"ContainerDied","Data":"619d514d71b85a52745de737d3379980a54e15db19b11950183f939340ca5ce8"}
Jul 28 13:16:24.815156 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:24.815134    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zlgv4"
Jul 28 13:16:24.874122 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:24.874098    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e489fa86-38c0-4961-b6c4-c87d4c24683d-cabundle-cert\") pod \"e489fa86-38c0-4961-b6c4-c87d4c24683d\" (UID: \"e489fa86-38c0-4961-b6c4-c87d4c24683d\") "
Jul 28 13:16:24.874230 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:24.874157    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e489fa86-38c0-4961-b6c4-c87d4c24683d-kserve-provision-location\") pod \"e489fa86-38c0-4961-b6c4-c87d4c24683d\" (UID: \"e489fa86-38c0-4961-b6c4-c87d4c24683d\") "
Jul 28 13:16:24.874230 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:24.874200    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e489fa86-38c0-4961-b6c4-c87d4c24683d-kube-rbac-proxy-sar-config\") pod \"e489fa86-38c0-4961-b6c4-c87d4c24683d\" (UID: \"e489fa86-38c0-4961-b6c4-c87d4c24683d\") "
Jul 28 13:16:24.874308 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:24.874232    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e489fa86-38c0-4961-b6c4-c87d4c24683d-proxy-tls\") pod \"e489fa86-38c0-4961-b6c4-c87d4c24683d\" (UID: \"e489fa86-38c0-4961-b6c4-c87d4c24683d\") "
Jul 28 13:16:24.874308 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:24.874269    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vd69m\" (UniqueName: \"kubernetes.io/projected/e489fa86-38c0-4961-b6c4-c87d4c24683d-kube-api-access-vd69m\") pod \"e489fa86-38c0-4961-b6c4-c87d4c24683d\" (UID: \"e489fa86-38c0-4961-b6c4-c87d4c24683d\") "
Jul 28 13:16:24.874482 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:24.874462    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e489fa86-38c0-4961-b6c4-c87d4c24683d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e489fa86-38c0-4961-b6c4-c87d4c24683d" (UID: "e489fa86-38c0-4961-b6c4-c87d4c24683d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:16:24.874557 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:24.874486    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e489fa86-38c0-4961-b6c4-c87d4c24683d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e489fa86-38c0-4961-b6c4-c87d4c24683d" (UID: "e489fa86-38c0-4961-b6c4-c87d4c24683d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:16:24.876295 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:24.876267    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e489fa86-38c0-4961-b6c4-c87d4c24683d-kube-api-access-vd69m" (OuterVolumeSpecName: "kube-api-access-vd69m") pod "e489fa86-38c0-4961-b6c4-c87d4c24683d" (UID: "e489fa86-38c0-4961-b6c4-c87d4c24683d"). InnerVolumeSpecName "kube-api-access-vd69m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:16:24.876376 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:24.876295    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e489fa86-38c0-4961-b6c4-c87d4c24683d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e489fa86-38c0-4961-b6c4-c87d4c24683d" (UID: "e489fa86-38c0-4961-b6c4-c87d4c24683d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:16:24.880391 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:24.880367    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e489fa86-38c0-4961-b6c4-c87d4c24683d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e489fa86-38c0-4961-b6c4-c87d4c24683d" (UID: "e489fa86-38c0-4961-b6c4-c87d4c24683d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:16:24.974774 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:24.974727    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e489fa86-38c0-4961-b6c4-c87d4c24683d-kserve-provision-location\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:16:24.974774 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:24.974746    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e489fa86-38c0-4961-b6c4-c87d4c24683d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:16:24.974774 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:24.974755    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e489fa86-38c0-4961-b6c4-c87d4c24683d-proxy-tls\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:16:24.974774 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:24.974764    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd69m\" (UniqueName: \"kubernetes.io/projected/e489fa86-38c0-4961-b6c4-c87d4c24683d-kube-api-access-vd69m\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:16:24.974774 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:24.974773    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e489fa86-38c0-4961-b6c4-c87d4c24683d-cabundle-cert\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:16:25.425267 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:25.425188    2574 generic.go:356] "Generic (PLEG): container finished" podID="e489fa86-38c0-4961-b6c4-c87d4c24683d" containerID="c6eb013759f2c50e57914bf035c7f095a53a11def1ec6f6e4dc6e8e4fd43a7c6" exitCode=0
Jul 28 13:16:25.425267 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:25.425243    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zlgv4"
Jul 28 13:16:25.425478 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:25.425247    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zlgv4" event={"ID":"e489fa86-38c0-4961-b6c4-c87d4c24683d","Type":"ContainerDied","Data":"c6eb013759f2c50e57914bf035c7f095a53a11def1ec6f6e4dc6e8e4fd43a7c6"}
Jul 28 13:16:25.425478 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:25.425371    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zlgv4" event={"ID":"e489fa86-38c0-4961-b6c4-c87d4c24683d","Type":"ContainerDied","Data":"edce6e3064244d8ea3e9c41b416230d5295585119af13bb270f0f1a9a865b3b0"}
Jul 28 13:16:25.425478 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:25.425398    2574 scope.go:117] "RemoveContainer" containerID="619d514d71b85a52745de737d3379980a54e15db19b11950183f939340ca5ce8"
Jul 28 13:16:25.434399 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:25.434380    2574 scope.go:117] "RemoveContainer" containerID="c6eb013759f2c50e57914bf035c7f095a53a11def1ec6f6e4dc6e8e4fd43a7c6"
Jul 28 13:16:25.443469 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:25.443453    2574 scope.go:117] "RemoveContainer" containerID="cd2bbd8d8ef2d21d91bcc1880b52a8546d448638b7a763ed13a6be4a258ee1b7"
Jul 28 13:16:25.446888 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:25.446866    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zlgv4"]
Jul 28 13:16:25.450975 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:25.450959    2574 scope.go:117] "RemoveContainer" containerID="619d514d71b85a52745de737d3379980a54e15db19b11950183f939340ca5ce8"
Jul 28 13:16:25.451334 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:16:25.451308    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"619d514d71b85a52745de737d3379980a54e15db19b11950183f939340ca5ce8\": container with ID starting with 619d514d71b85a52745de737d3379980a54e15db19b11950183f939340ca5ce8 not found: ID does not exist" containerID="619d514d71b85a52745de737d3379980a54e15db19b11950183f939340ca5ce8"
Jul 28 13:16:25.451439 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:25.451343    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"619d514d71b85a52745de737d3379980a54e15db19b11950183f939340ca5ce8"} err="failed to get container status \"619d514d71b85a52745de737d3379980a54e15db19b11950183f939340ca5ce8\": rpc error: code = NotFound desc = could not find container \"619d514d71b85a52745de737d3379980a54e15db19b11950183f939340ca5ce8\": container with ID starting with 619d514d71b85a52745de737d3379980a54e15db19b11950183f939340ca5ce8 not found: ID does not exist"
Jul 28 13:16:25.451439 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:25.451365    2574 scope.go:117] "RemoveContainer" containerID="c6eb013759f2c50e57914bf035c7f095a53a11def1ec6f6e4dc6e8e4fd43a7c6"
Jul 28 13:16:25.451630 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:16:25.451614    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6eb013759f2c50e57914bf035c7f095a53a11def1ec6f6e4dc6e8e4fd43a7c6\": container with ID starting with c6eb013759f2c50e57914bf035c7f095a53a11def1ec6f6e4dc6e8e4fd43a7c6 not found: ID does not exist" containerID="c6eb013759f2c50e57914bf035c7f095a53a11def1ec6f6e4dc6e8e4fd43a7c6"
Jul 28 13:16:25.451676 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:25.451637    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6eb013759f2c50e57914bf035c7f095a53a11def1ec6f6e4dc6e8e4fd43a7c6"} err="failed to get container status \"c6eb013759f2c50e57914bf035c7f095a53a11def1ec6f6e4dc6e8e4fd43a7c6\": rpc error: code = NotFound desc = could not find container \"c6eb013759f2c50e57914bf035c7f095a53a11def1ec6f6e4dc6e8e4fd43a7c6\": container with ID starting with c6eb013759f2c50e57914bf035c7f095a53a11def1ec6f6e4dc6e8e4fd43a7c6 not found: ID does not exist"
Jul 28 13:16:25.451676 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:25.451653    2574 scope.go:117] "RemoveContainer" containerID="cd2bbd8d8ef2d21d91bcc1880b52a8546d448638b7a763ed13a6be4a258ee1b7"
Jul 28 13:16:25.451930 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:16:25.451907    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd2bbd8d8ef2d21d91bcc1880b52a8546d448638b7a763ed13a6be4a258ee1b7\": container with ID starting with cd2bbd8d8ef2d21d91bcc1880b52a8546d448638b7a763ed13a6be4a258ee1b7 not found: ID does not exist" containerID="cd2bbd8d8ef2d21d91bcc1880b52a8546d448638b7a763ed13a6be4a258ee1b7"
Jul 28 13:16:25.451975 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:25.451956    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd2bbd8d8ef2d21d91bcc1880b52a8546d448638b7a763ed13a6be4a258ee1b7"} err="failed to get container status \"cd2bbd8d8ef2d21d91bcc1880b52a8546d448638b7a763ed13a6be4a258ee1b7\": rpc error: code = NotFound desc = could not find container \"cd2bbd8d8ef2d21d91bcc1880b52a8546d448638b7a763ed13a6be4a258ee1b7\": container with ID starting with cd2bbd8d8ef2d21d91bcc1880b52a8546d448638b7a763ed13a6be4a258ee1b7 not found: ID does not exist"
Jul 28 13:16:25.452984 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:25.452968    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zlgv4"]
Jul 28 13:16:26.302040 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:26.302009    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e489fa86-38c0-4961-b6c4-c87d4c24683d" path="/var/lib/kubelet/pods/e489fa86-38c0-4961-b6c4-c87d4c24683d/volumes"
Jul 28 13:16:28.644314 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:28.644283    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tjznh"]
Jul 28 13:16:28.644665 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:28.644584    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e489fa86-38c0-4961-b6c4-c87d4c24683d" containerName="storage-initializer"
Jul 28 13:16:28.644665 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:28.644594    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e489fa86-38c0-4961-b6c4-c87d4c24683d" containerName="storage-initializer"
Jul 28 13:16:28.644665 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:28.644611    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e489fa86-38c0-4961-b6c4-c87d4c24683d" containerName="kube-rbac-proxy"
Jul 28 13:16:28.644665 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:28.644616    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e489fa86-38c0-4961-b6c4-c87d4c24683d" containerName="kube-rbac-proxy"
Jul 28 13:16:28.644665 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:28.644631    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e489fa86-38c0-4961-b6c4-c87d4c24683d" containerName="kserve-container"
Jul 28 13:16:28.644665 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:28.644637    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e489fa86-38c0-4961-b6c4-c87d4c24683d" containerName="kserve-container"
Jul 28 13:16:28.644873 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:28.644698    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e489fa86-38c0-4961-b6c4-c87d4c24683d" containerName="kserve-container"
Jul 28 13:16:28.644873 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:28.644710    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e489fa86-38c0-4961-b6c4-c87d4c24683d" containerName="kube-rbac-proxy"
Jul 28 13:16:28.649504 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:28.649487    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tjznh"
Jul 28 13:16:28.652991 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:28.652966    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 13:16:28.653268 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:28.653052    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Jul 28 13:16:28.654194 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:28.654176    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:16:28.654298 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:28.654228    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:16:28.654298 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:28.654250    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Jul 28 13:16:28.654298 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:28.654275    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rxfwd\""
Jul 28 13:16:28.654470 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:28.654232    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:16:28.663378 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:28.663358    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tjznh"]
Jul 28 13:16:28.700512 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:28.700484    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83d95de6-3fba-488a-bfd6-fafcd914507e-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tjznh\" (UID: \"83d95de6-3fba-488a-bfd6-fafcd914507e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tjznh"
Jul 28 13:16:28.700617 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:28.700532    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83d95de6-3fba-488a-bfd6-fafcd914507e-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tjznh\" (UID: \"83d95de6-3fba-488a-bfd6-fafcd914507e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tjznh"
Jul 28 13:16:28.700617 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:28.700572    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5887\" (UniqueName: \"kubernetes.io/projected/83d95de6-3fba-488a-bfd6-fafcd914507e-kube-api-access-l5887\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tjznh\" (UID: \"83d95de6-3fba-488a-bfd6-fafcd914507e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tjznh"
Jul 28 13:16:28.700617 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:28.700597    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83d95de6-3fba-488a-bfd6-fafcd914507e-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tjznh\" (UID: \"83d95de6-3fba-488a-bfd6-fafcd914507e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tjznh"
Jul 28 13:16:28.700802 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:28.700664    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83d95de6-3fba-488a-bfd6-fafcd914507e-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tjznh\" (UID: \"83d95de6-3fba-488a-bfd6-fafcd914507e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tjznh"
Jul 28 13:16:28.802007 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:28.801978    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83d95de6-3fba-488a-bfd6-fafcd914507e-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tjznh\" (UID: \"83d95de6-3fba-488a-bfd6-fafcd914507e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tjznh"
Jul 28 13:16:28.802096 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:28.802060    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83d95de6-3fba-488a-bfd6-fafcd914507e-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tjznh\" (UID: \"83d95de6-3fba-488a-bfd6-fafcd914507e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tjznh"
Jul 28 13:16:28.802141 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:28.802103    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83d95de6-3fba-488a-bfd6-fafcd914507e-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tjznh\" (UID: \"83d95de6-3fba-488a-bfd6-fafcd914507e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tjznh"
Jul 28 13:16:28.802183 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:28.802136    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5887\" (UniqueName: \"kubernetes.io/projected/83d95de6-3fba-488a-bfd6-fafcd914507e-kube-api-access-l5887\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tjznh\" (UID: \"83d95de6-3fba-488a-bfd6-fafcd914507e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tjznh"
Jul 28 13:16:28.802183 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:28.802165    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83d95de6-3fba-488a-bfd6-fafcd914507e-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tjznh\" (UID: \"83d95de6-3fba-488a-bfd6-fafcd914507e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tjznh"
Jul 28 13:16:28.802505 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:28.802483    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83d95de6-3fba-488a-bfd6-fafcd914507e-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tjznh\" (UID: \"83d95de6-3fba-488a-bfd6-fafcd914507e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tjznh"
Jul 28 13:16:28.802776 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:28.802759    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83d95de6-3fba-488a-bfd6-fafcd914507e-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tjznh\" (UID: \"83d95de6-3fba-488a-bfd6-fafcd914507e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tjznh"
Jul 28 13:16:28.802818 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:28.802765    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83d95de6-3fba-488a-bfd6-fafcd914507e-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tjznh\" (UID: \"83d95de6-3fba-488a-bfd6-fafcd914507e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tjznh"
Jul 28 13:16:28.804348 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:28.804333    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83d95de6-3fba-488a-bfd6-fafcd914507e-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tjznh\" (UID: \"83d95de6-3fba-488a-bfd6-fafcd914507e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tjznh"
Jul 28 13:16:28.815215 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:28.815189    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5887\" (UniqueName: \"kubernetes.io/projected/83d95de6-3fba-488a-bfd6-fafcd914507e-kube-api-access-l5887\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tjznh\" (UID: \"83d95de6-3fba-488a-bfd6-fafcd914507e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tjznh"
Jul 28 13:16:28.959335 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:28.959308    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tjznh"
Jul 28 13:16:29.086953 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:29.086930    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tjznh"]
Jul 28 13:16:29.089185 ip-10-0-138-158 kubenswrapper[2574]: W0728 13:16:29.089160    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83d95de6_3fba_488a_bfd6_fafcd914507e.slice/crio-34f7469f7675d9e141cdbf59005dc338e3f966cd806b198e2fb2f6049b11a56b WatchSource:0}: Error finding container 34f7469f7675d9e141cdbf59005dc338e3f966cd806b198e2fb2f6049b11a56b: Status 404 returned error can't find the container with id 34f7469f7675d9e141cdbf59005dc338e3f966cd806b198e2fb2f6049b11a56b
Jul 28 13:16:29.444477 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:29.444444    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tjznh" event={"ID":"83d95de6-3fba-488a-bfd6-fafcd914507e","Type":"ContainerStarted","Data":"a1fff2c3c0247df5cc9a5dd68ed6f35189011349e5913c1247b33d42b7fdf2ae"}
Jul 28 13:16:29.444654 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:29.444499    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tjznh" event={"ID":"83d95de6-3fba-488a-bfd6-fafcd914507e","Type":"ContainerStarted","Data":"34f7469f7675d9e141cdbf59005dc338e3f966cd806b198e2fb2f6049b11a56b"}
Jul 28 13:16:35.472207 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:35.472178    2574 generic.go:356] "Generic (PLEG): container finished" podID="83d95de6-3fba-488a-bfd6-fafcd914507e" containerID="a1fff2c3c0247df5cc9a5dd68ed6f35189011349e5913c1247b33d42b7fdf2ae" exitCode=0
Jul 28 13:16:35.472561 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:35.472235    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tjznh" event={"ID":"83d95de6-3fba-488a-bfd6-fafcd914507e","Type":"ContainerDied","Data":"a1fff2c3c0247df5cc9a5dd68ed6f35189011349e5913c1247b33d42b7fdf2ae"}
Jul 28 13:16:36.479138 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:36.479106    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tjznh" event={"ID":"83d95de6-3fba-488a-bfd6-fafcd914507e","Type":"ContainerStarted","Data":"4f7b42c8072c154998fd73476b8f7d385344673eae00b07c17d543ff7ca99a1f"}
Jul 28 13:16:36.479621 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:36.479169    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tjznh" event={"ID":"83d95de6-3fba-488a-bfd6-fafcd914507e","Type":"ContainerStarted","Data":"f329f56941edd756cea0e589efc17d14abe4987dcdb5aba6eef9bb3e77192df6"}
Jul 28 13:16:36.479621 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:36.479602    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tjznh"
Jul 28 13:16:36.479773 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:36.479661    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tjznh"
Jul 28 13:16:36.501427 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:36.501379    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tjznh" podStartSLOduration=2.874357108 podStartE2EDuration="8.501364021s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.627006913s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:16:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:16:36.499233232 +0000 UTC m=+1408.704619834" watchObservedRunningTime="2026-07-28 13:16:36.501364021 +0000 UTC m=+1408.706750621"
Jul 28 13:16:42.489560 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:16:42.489526    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tjznh"
Jul 28 13:18:09.301203 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:09.301169    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tjznh"
Jul 28 13:18:20.337905 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:20.337856    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tjznh" podUID="83d95de6-3fba-488a-bfd6-fafcd914507e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.36:8080/v2/models/isvc-autogluon-v2-variants/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 28 13:18:25.189657 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:25.189627    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tjznh"]
Jul 28 13:18:25.190069 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:25.189870    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tjznh" podUID="83d95de6-3fba-488a-bfd6-fafcd914507e" containerName="kserve-container" containerID="cri-o://f329f56941edd756cea0e589efc17d14abe4987dcdb5aba6eef9bb3e77192df6" gracePeriod=30
Jul 28 13:18:25.190069 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:25.189924    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tjznh" podUID="83d95de6-3fba-488a-bfd6-fafcd914507e" containerName="kube-rbac-proxy" containerID="cri-o://4f7b42c8072c154998fd73476b8f7d385344673eae00b07c17d543ff7ca99a1f" gracePeriod=30
Jul 28 13:18:25.962929 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:25.962894    2574 generic.go:356] "Generic (PLEG): container finished" podID="83d95de6-3fba-488a-bfd6-fafcd914507e" containerID="4f7b42c8072c154998fd73476b8f7d385344673eae00b07c17d543ff7ca99a1f" exitCode=2
Jul 28 13:18:25.963098 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:25.962959    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tjznh" event={"ID":"83d95de6-3fba-488a-bfd6-fafcd914507e","Type":"ContainerDied","Data":"4f7b42c8072c154998fd73476b8f7d385344673eae00b07c17d543ff7ca99a1f"}
Jul 28 13:18:26.430502 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:26.430483    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tjznh"
Jul 28 13:18:26.565450 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:26.565386    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l5887\" (UniqueName: \"kubernetes.io/projected/83d95de6-3fba-488a-bfd6-fafcd914507e-kube-api-access-l5887\") pod \"83d95de6-3fba-488a-bfd6-fafcd914507e\" (UID: \"83d95de6-3fba-488a-bfd6-fafcd914507e\") "
Jul 28 13:18:26.565450 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:26.565435    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83d95de6-3fba-488a-bfd6-fafcd914507e-cabundle-cert\") pod \"83d95de6-3fba-488a-bfd6-fafcd914507e\" (UID: \"83d95de6-3fba-488a-bfd6-fafcd914507e\") "
Jul 28 13:18:26.565622 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:26.565481    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83d95de6-3fba-488a-bfd6-fafcd914507e-kube-rbac-proxy-sar-config\") pod \"83d95de6-3fba-488a-bfd6-fafcd914507e\" (UID: \"83d95de6-3fba-488a-bfd6-fafcd914507e\") "
Jul 28 13:18:26.565622 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:26.565545    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83d95de6-3fba-488a-bfd6-fafcd914507e-proxy-tls\") pod \"83d95de6-3fba-488a-bfd6-fafcd914507e\" (UID: \"83d95de6-3fba-488a-bfd6-fafcd914507e\") "
Jul 28 13:18:26.565622 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:26.565586    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83d95de6-3fba-488a-bfd6-fafcd914507e-kserve-provision-location\") pod \"83d95de6-3fba-488a-bfd6-fafcd914507e\" (UID: \"83d95de6-3fba-488a-bfd6-fafcd914507e\") "
Jul 28 13:18:26.565810 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:26.565775    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83d95de6-3fba-488a-bfd6-fafcd914507e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "83d95de6-3fba-488a-bfd6-fafcd914507e" (UID: "83d95de6-3fba-488a-bfd6-fafcd914507e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:18:26.565968 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:26.565847    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83d95de6-3fba-488a-bfd6-fafcd914507e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "83d95de6-3fba-488a-bfd6-fafcd914507e" (UID: "83d95de6-3fba-488a-bfd6-fafcd914507e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:18:26.567407 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:26.567388    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83d95de6-3fba-488a-bfd6-fafcd914507e-kube-api-access-l5887" (OuterVolumeSpecName: "kube-api-access-l5887") pod "83d95de6-3fba-488a-bfd6-fafcd914507e" (UID: "83d95de6-3fba-488a-bfd6-fafcd914507e"). InnerVolumeSpecName "kube-api-access-l5887". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:18:26.567498 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:26.567434    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83d95de6-3fba-488a-bfd6-fafcd914507e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "83d95de6-3fba-488a-bfd6-fafcd914507e" (UID: "83d95de6-3fba-488a-bfd6-fafcd914507e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:18:26.569554 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:26.569517    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83d95de6-3fba-488a-bfd6-fafcd914507e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "83d95de6-3fba-488a-bfd6-fafcd914507e" (UID: "83d95de6-3fba-488a-bfd6-fafcd914507e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:18:26.666814 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:26.666790    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83d95de6-3fba-488a-bfd6-fafcd914507e-kserve-provision-location\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:18:26.666814 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:26.666810    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5887\" (UniqueName: \"kubernetes.io/projected/83d95de6-3fba-488a-bfd6-fafcd914507e-kube-api-access-l5887\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:18:26.666967 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:26.666820    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83d95de6-3fba-488a-bfd6-fafcd914507e-cabundle-cert\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:18:26.666967 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:26.666828    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83d95de6-3fba-488a-bfd6-fafcd914507e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:18:26.666967 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:26.666837    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83d95de6-3fba-488a-bfd6-fafcd914507e-proxy-tls\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:18:26.968807 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:26.968780    2574 generic.go:356] "Generic (PLEG): container finished" podID="83d95de6-3fba-488a-bfd6-fafcd914507e" containerID="f329f56941edd756cea0e589efc17d14abe4987dcdb5aba6eef9bb3e77192df6" exitCode=0
Jul 28 13:18:26.968956 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:26.968845    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tjznh"
Jul 28 13:18:26.968956 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:26.968863    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tjznh" event={"ID":"83d95de6-3fba-488a-bfd6-fafcd914507e","Type":"ContainerDied","Data":"f329f56941edd756cea0e589efc17d14abe4987dcdb5aba6eef9bb3e77192df6"}
Jul 28 13:18:26.968956 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:26.968948    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tjznh" event={"ID":"83d95de6-3fba-488a-bfd6-fafcd914507e","Type":"ContainerDied","Data":"34f7469f7675d9e141cdbf59005dc338e3f966cd806b198e2fb2f6049b11a56b"}
Jul 28 13:18:26.969061 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:26.968976    2574 scope.go:117] "RemoveContainer" containerID="4f7b42c8072c154998fd73476b8f7d385344673eae00b07c17d543ff7ca99a1f"
Jul 28 13:18:26.977612 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:26.977595    2574 scope.go:117] "RemoveContainer" containerID="f329f56941edd756cea0e589efc17d14abe4987dcdb5aba6eef9bb3e77192df6"
Jul 28 13:18:26.984914 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:26.984898    2574 scope.go:117] "RemoveContainer" containerID="a1fff2c3c0247df5cc9a5dd68ed6f35189011349e5913c1247b33d42b7fdf2ae"
Jul 28 13:18:26.990551 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:26.990529    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tjznh"]
Jul 28 13:18:26.992558 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:26.992529    2574 scope.go:117] "RemoveContainer" containerID="4f7b42c8072c154998fd73476b8f7d385344673eae00b07c17d543ff7ca99a1f"
Jul 28 13:18:26.992891 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:18:26.992867    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f7b42c8072c154998fd73476b8f7d385344673eae00b07c17d543ff7ca99a1f\": container with ID starting with 4f7b42c8072c154998fd73476b8f7d385344673eae00b07c17d543ff7ca99a1f not found: ID does not exist" containerID="4f7b42c8072c154998fd73476b8f7d385344673eae00b07c17d543ff7ca99a1f"
Jul 28 13:18:26.992958 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:26.992901    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f7b42c8072c154998fd73476b8f7d385344673eae00b07c17d543ff7ca99a1f"} err="failed to get container status \"4f7b42c8072c154998fd73476b8f7d385344673eae00b07c17d543ff7ca99a1f\": rpc error: code = NotFound desc = could not find container \"4f7b42c8072c154998fd73476b8f7d385344673eae00b07c17d543ff7ca99a1f\": container with ID starting with 4f7b42c8072c154998fd73476b8f7d385344673eae00b07c17d543ff7ca99a1f not found: ID does not exist"
Jul 28 13:18:26.992958 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:26.992925    2574 scope.go:117] "RemoveContainer" containerID="f329f56941edd756cea0e589efc17d14abe4987dcdb5aba6eef9bb3e77192df6"
Jul 28 13:18:26.993195 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:18:26.993175    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f329f56941edd756cea0e589efc17d14abe4987dcdb5aba6eef9bb3e77192df6\": container with ID starting with f329f56941edd756cea0e589efc17d14abe4987dcdb5aba6eef9bb3e77192df6 not found: ID does not exist" containerID="f329f56941edd756cea0e589efc17d14abe4987dcdb5aba6eef9bb3e77192df6"
Jul 28 13:18:26.993244 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:26.993202    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f329f56941edd756cea0e589efc17d14abe4987dcdb5aba6eef9bb3e77192df6"} err="failed to get container status \"f329f56941edd756cea0e589efc17d14abe4987dcdb5aba6eef9bb3e77192df6\": rpc error: code = NotFound desc = could not find container \"f329f56941edd756cea0e589efc17d14abe4987dcdb5aba6eef9bb3e77192df6\": container with ID starting with f329f56941edd756cea0e589efc17d14abe4987dcdb5aba6eef9bb3e77192df6 not found: ID does not exist"
Jul 28 13:18:26.993244 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:26.993218    2574 scope.go:117] "RemoveContainer" containerID="a1fff2c3c0247df5cc9a5dd68ed6f35189011349e5913c1247b33d42b7fdf2ae"
Jul 28 13:18:26.993446 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:18:26.993426    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1fff2c3c0247df5cc9a5dd68ed6f35189011349e5913c1247b33d42b7fdf2ae\": container with ID starting with a1fff2c3c0247df5cc9a5dd68ed6f35189011349e5913c1247b33d42b7fdf2ae not found: ID does not exist" containerID="a1fff2c3c0247df5cc9a5dd68ed6f35189011349e5913c1247b33d42b7fdf2ae"
Jul 28 13:18:26.993490 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:26.993453    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1fff2c3c0247df5cc9a5dd68ed6f35189011349e5913c1247b33d42b7fdf2ae"} err="failed to get container status \"a1fff2c3c0247df5cc9a5dd68ed6f35189011349e5913c1247b33d42b7fdf2ae\": rpc error: code = NotFound desc = could not find container \"a1fff2c3c0247df5cc9a5dd68ed6f35189011349e5913c1247b33d42b7fdf2ae\": container with ID starting with a1fff2c3c0247df5cc9a5dd68ed6f35189011349e5913c1247b33d42b7fdf2ae not found: ID does not exist"
Jul 28 13:18:26.994287 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:26.994270    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tjznh"]
Jul 28 13:18:28.301317 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:28.301278    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83d95de6-3fba-488a-bfd6-fafcd914507e" path="/var/lib/kubelet/pods/83d95de6-3fba-488a-bfd6-fafcd914507e/volumes"
Jul 28 13:18:30.529037 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:30.528965    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-cr6qp"]
Jul 28 13:18:30.529376 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:30.529322    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83d95de6-3fba-488a-bfd6-fafcd914507e" containerName="storage-initializer"
Jul 28 13:18:30.529376 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:30.529334    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="83d95de6-3fba-488a-bfd6-fafcd914507e" containerName="storage-initializer"
Jul 28 13:18:30.529376 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:30.529342    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83d95de6-3fba-488a-bfd6-fafcd914507e" containerName="kserve-container"
Jul 28 13:18:30.529376 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:30.529347    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="83d95de6-3fba-488a-bfd6-fafcd914507e" containerName="kserve-container"
Jul 28 13:18:30.529376 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:30.529369    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83d95de6-3fba-488a-bfd6-fafcd914507e" containerName="kube-rbac-proxy"
Jul 28 13:18:30.529376 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:30.529375    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="83d95de6-3fba-488a-bfd6-fafcd914507e" containerName="kube-rbac-proxy"
Jul 28 13:18:30.529574 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:30.529421    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="83d95de6-3fba-488a-bfd6-fafcd914507e" containerName="kserve-container"
Jul 28 13:18:30.529574 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:30.529432    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="83d95de6-3fba-488a-bfd6-fafcd914507e" containerName="kube-rbac-proxy"
Jul 28 13:18:30.534192 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:30.534173    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-cr6qp"
Jul 28 13:18:30.539148 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:30.539131    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:18:30.539366 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:30.539339    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 13:18:30.539458 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:30.539369    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:18:30.539458 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:30.539423    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Jul 28 13:18:30.540274 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:30.540253    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Jul 28 13:18:30.540385 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:30.540297    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:18:30.540447 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:30.540409    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rxfwd\""
Jul 28 13:18:30.562864 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:30.562843    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-cr6qp"]
Jul 28 13:18:30.694190 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:30.694168    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e140448f-f07b-4cd2-8df6-195a25ec5e1c-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-cr6qp\" (UID: \"e140448f-f07b-4cd2-8df6-195a25ec5e1c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-cr6qp"
Jul 28 13:18:30.694317 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:30.694218    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e140448f-f07b-4cd2-8df6-195a25ec5e1c-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-cr6qp\" (UID: \"e140448f-f07b-4cd2-8df6-195a25ec5e1c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-cr6qp"
Jul 28 13:18:30.694317 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:30.694247    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e140448f-f07b-4cd2-8df6-195a25ec5e1c-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-cr6qp\" (UID: \"e140448f-f07b-4cd2-8df6-195a25ec5e1c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-cr6qp"
Jul 28 13:18:30.694429 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:30.694317    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp5tf\" (UniqueName: \"kubernetes.io/projected/e140448f-f07b-4cd2-8df6-195a25ec5e1c-kube-api-access-jp5tf\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-cr6qp\" (UID: \"e140448f-f07b-4cd2-8df6-195a25ec5e1c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-cr6qp"
Jul 28 13:18:30.694429 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:30.694348    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e140448f-f07b-4cd2-8df6-195a25ec5e1c-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-cr6qp\" (UID: \"e140448f-f07b-4cd2-8df6-195a25ec5e1c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-cr6qp"
Jul 28 13:18:30.794750 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:30.794652    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp5tf\" (UniqueName: \"kubernetes.io/projected/e140448f-f07b-4cd2-8df6-195a25ec5e1c-kube-api-access-jp5tf\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-cr6qp\" (UID: \"e140448f-f07b-4cd2-8df6-195a25ec5e1c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-cr6qp"
Jul 28 13:18:30.794750 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:30.794706    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e140448f-f07b-4cd2-8df6-195a25ec5e1c-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-cr6qp\" (UID: \"e140448f-f07b-4cd2-8df6-195a25ec5e1c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-cr6qp"
Jul 28 13:18:30.794750 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:30.794740    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e140448f-f07b-4cd2-8df6-195a25ec5e1c-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-cr6qp\" (UID: \"e140448f-f07b-4cd2-8df6-195a25ec5e1c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-cr6qp"
Jul 28 13:18:30.795002 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:30.794780    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e140448f-f07b-4cd2-8df6-195a25ec5e1c-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-cr6qp\" (UID: \"e140448f-f07b-4cd2-8df6-195a25ec5e1c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-cr6qp"
Jul 28 13:18:30.795002 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:30.794886    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e140448f-f07b-4cd2-8df6-195a25ec5e1c-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-cr6qp\" (UID: \"e140448f-f07b-4cd2-8df6-195a25ec5e1c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-cr6qp"
Jul 28 13:18:30.795111 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:30.795048    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e140448f-f07b-4cd2-8df6-195a25ec5e1c-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-cr6qp\" (UID: \"e140448f-f07b-4cd2-8df6-195a25ec5e1c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-cr6qp"
Jul 28 13:18:30.795383 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:30.795355    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e140448f-f07b-4cd2-8df6-195a25ec5e1c-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-cr6qp\" (UID: \"e140448f-f07b-4cd2-8df6-195a25ec5e1c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-cr6qp"
Jul 28 13:18:30.795480 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:30.795420    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e140448f-f07b-4cd2-8df6-195a25ec5e1c-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-cr6qp\" (UID: \"e140448f-f07b-4cd2-8df6-195a25ec5e1c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-cr6qp"
Jul 28 13:18:30.797224 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:30.797203    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e140448f-f07b-4cd2-8df6-195a25ec5e1c-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-cr6qp\" (UID: \"e140448f-f07b-4cd2-8df6-195a25ec5e1c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-cr6qp"
Jul 28 13:18:30.805945 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:30.805921    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp5tf\" (UniqueName: \"kubernetes.io/projected/e140448f-f07b-4cd2-8df6-195a25ec5e1c-kube-api-access-jp5tf\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-cr6qp\" (UID: \"e140448f-f07b-4cd2-8df6-195a25ec5e1c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-cr6qp"
Jul 28 13:18:30.844753 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:30.844729    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-cr6qp"
Jul 28 13:18:30.978131 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:30.978110    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-cr6qp"]
Jul 28 13:18:30.979940 ip-10-0-138-158 kubenswrapper[2574]: W0728 13:18:30.979909    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode140448f_f07b_4cd2_8df6_195a25ec5e1c.slice/crio-66e1a4e0abd50cf82f40ebea4e6e89f00a0b481a939e6c965026c5663be13bd8 WatchSource:0}: Error finding container 66e1a4e0abd50cf82f40ebea4e6e89f00a0b481a939e6c965026c5663be13bd8: Status 404 returned error can't find the container with id 66e1a4e0abd50cf82f40ebea4e6e89f00a0b481a939e6c965026c5663be13bd8
Jul 28 13:18:30.993072 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:30.993042    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-cr6qp" event={"ID":"e140448f-f07b-4cd2-8df6-195a25ec5e1c","Type":"ContainerStarted","Data":"66e1a4e0abd50cf82f40ebea4e6e89f00a0b481a939e6c965026c5663be13bd8"}
Jul 28 13:18:31.999335 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:31.999303    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-cr6qp" event={"ID":"e140448f-f07b-4cd2-8df6-195a25ec5e1c","Type":"ContainerStarted","Data":"e4edc95c97b4edc48d87d2bcfea5d1facdbb4a4e74f54eadef9dce16160fa98f"}
Jul 28 13:18:52.091132 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:52.091098    2574 generic.go:356] "Generic (PLEG): container finished" podID="e140448f-f07b-4cd2-8df6-195a25ec5e1c" containerID="e4edc95c97b4edc48d87d2bcfea5d1facdbb4a4e74f54eadef9dce16160fa98f" exitCode=0
Jul 28 13:18:52.091516 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:52.091146    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-cr6qp" event={"ID":"e140448f-f07b-4cd2-8df6-195a25ec5e1c","Type":"ContainerDied","Data":"e4edc95c97b4edc48d87d2bcfea5d1facdbb4a4e74f54eadef9dce16160fa98f"}
Jul 28 13:18:53.101592 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:53.101558    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-cr6qp" event={"ID":"e140448f-f07b-4cd2-8df6-195a25ec5e1c","Type":"ContainerStarted","Data":"a9f25efa25a6f3ee52943b5a163aad80064c6521d117136212d4cf6d9950a277"}
Jul 28 13:18:53.101976 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:53.101625    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-cr6qp" event={"ID":"e140448f-f07b-4cd2-8df6-195a25ec5e1c","Type":"ContainerStarted","Data":"edb0183a9dc65129b8d3b1d28933f342b0126e6710b90432c8a38fbecb365fd3"}
Jul 28 13:18:53.101976 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:53.101832    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-cr6qp"
Jul 28 13:18:53.120850 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:53.120794    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-cr6qp" podStartSLOduration=2.727649313 podStartE2EDuration="23.120760785s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.393111472s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:18:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:18:53.119884004 +0000 UTC m=+1545.325270604" watchObservedRunningTime="2026-07-28 13:18:53.120760785 +0000 UTC m=+1545.326147594"
Jul 28 13:18:54.106712 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:18:54.106674    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-cr6qp"
Jul 28 13:19:00.116377 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:19:00.116344    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-cr6qp"
Jul 28 13:20:22.302450 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:22.302380    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-cr6qp"
Jul 28 13:20:33.337894 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:33.337851    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-cr6qp" podUID="e140448f-f07b-4cd2-8df6-195a25ec5e1c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.37:8080/v2/models/isvc-autogluon-v2-noslash/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 28 13:20:36.463965 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:36.463931    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-cr6qp"]
Jul 28 13:20:36.464347 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:36.464168    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-cr6qp" podUID="e140448f-f07b-4cd2-8df6-195a25ec5e1c" containerName="kserve-container" containerID="cri-o://edb0183a9dc65129b8d3b1d28933f342b0126e6710b90432c8a38fbecb365fd3" gracePeriod=30
Jul 28 13:20:36.464347 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:36.464224    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-cr6qp" podUID="e140448f-f07b-4cd2-8df6-195a25ec5e1c" containerName="kube-rbac-proxy" containerID="cri-o://a9f25efa25a6f3ee52943b5a163aad80064c6521d117136212d4cf6d9950a277" gracePeriod=30
Jul 28 13:20:37.553601 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:37.553571    2574 generic.go:356] "Generic (PLEG): container finished" podID="e140448f-f07b-4cd2-8df6-195a25ec5e1c" containerID="a9f25efa25a6f3ee52943b5a163aad80064c6521d117136212d4cf6d9950a277" exitCode=2
Jul 28 13:20:37.553915 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:37.553630    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-cr6qp" event={"ID":"e140448f-f07b-4cd2-8df6-195a25ec5e1c","Type":"ContainerDied","Data":"a9f25efa25a6f3ee52943b5a163aad80064c6521d117136212d4cf6d9950a277"}
Jul 28 13:20:37.703512 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:37.703490    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-cr6qp"
Jul 28 13:20:37.708432 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:37.708415    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e140448f-f07b-4cd2-8df6-195a25ec5e1c-proxy-tls\") pod \"e140448f-f07b-4cd2-8df6-195a25ec5e1c\" (UID: \"e140448f-f07b-4cd2-8df6-195a25ec5e1c\") "
Jul 28 13:20:37.708532 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:37.708446    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e140448f-f07b-4cd2-8df6-195a25ec5e1c-kube-rbac-proxy-sar-config\") pod \"e140448f-f07b-4cd2-8df6-195a25ec5e1c\" (UID: \"e140448f-f07b-4cd2-8df6-195a25ec5e1c\") "
Jul 28 13:20:37.708532 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:37.708482    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e140448f-f07b-4cd2-8df6-195a25ec5e1c-cabundle-cert\") pod \"e140448f-f07b-4cd2-8df6-195a25ec5e1c\" (UID: \"e140448f-f07b-4cd2-8df6-195a25ec5e1c\") "
Jul 28 13:20:37.708532 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:37.708500    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e140448f-f07b-4cd2-8df6-195a25ec5e1c-kserve-provision-location\") pod \"e140448f-f07b-4cd2-8df6-195a25ec5e1c\" (UID: \"e140448f-f07b-4cd2-8df6-195a25ec5e1c\") "
Jul 28 13:20:37.708716 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:37.708538    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jp5tf\" (UniqueName: \"kubernetes.io/projected/e140448f-f07b-4cd2-8df6-195a25ec5e1c-kube-api-access-jp5tf\") pod \"e140448f-f07b-4cd2-8df6-195a25ec5e1c\" (UID: \"e140448f-f07b-4cd2-8df6-195a25ec5e1c\") "
Jul 28 13:20:37.708885 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:37.708854    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e140448f-f07b-4cd2-8df6-195a25ec5e1c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e140448f-f07b-4cd2-8df6-195a25ec5e1c" (UID: "e140448f-f07b-4cd2-8df6-195a25ec5e1c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:20:37.708999 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:37.708859    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e140448f-f07b-4cd2-8df6-195a25ec5e1c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e140448f-f07b-4cd2-8df6-195a25ec5e1c" (UID: "e140448f-f07b-4cd2-8df6-195a25ec5e1c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:20:37.710410 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:37.710385    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e140448f-f07b-4cd2-8df6-195a25ec5e1c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e140448f-f07b-4cd2-8df6-195a25ec5e1c" (UID: "e140448f-f07b-4cd2-8df6-195a25ec5e1c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:20:37.710583 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:37.710561    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e140448f-f07b-4cd2-8df6-195a25ec5e1c-kube-api-access-jp5tf" (OuterVolumeSpecName: "kube-api-access-jp5tf") pod "e140448f-f07b-4cd2-8df6-195a25ec5e1c" (UID: "e140448f-f07b-4cd2-8df6-195a25ec5e1c"). InnerVolumeSpecName "kube-api-access-jp5tf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:20:37.714829 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:37.714804    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e140448f-f07b-4cd2-8df6-195a25ec5e1c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e140448f-f07b-4cd2-8df6-195a25ec5e1c" (UID: "e140448f-f07b-4cd2-8df6-195a25ec5e1c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:20:37.809926 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:37.809867    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e140448f-f07b-4cd2-8df6-195a25ec5e1c-cabundle-cert\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:20:37.809926 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:37.809891    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e140448f-f07b-4cd2-8df6-195a25ec5e1c-kserve-provision-location\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:20:37.809926 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:37.809905    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jp5tf\" (UniqueName: \"kubernetes.io/projected/e140448f-f07b-4cd2-8df6-195a25ec5e1c-kube-api-access-jp5tf\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:20:37.809926 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:37.809918    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e140448f-f07b-4cd2-8df6-195a25ec5e1c-proxy-tls\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:20:37.810116 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:37.809931    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e140448f-f07b-4cd2-8df6-195a25ec5e1c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:20:38.559644 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:38.559562    2574 generic.go:356] "Generic (PLEG): container finished" podID="e140448f-f07b-4cd2-8df6-195a25ec5e1c" containerID="edb0183a9dc65129b8d3b1d28933f342b0126e6710b90432c8a38fbecb365fd3" exitCode=0
Jul 28 13:20:38.559644 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:38.559619    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-cr6qp"
Jul 28 13:20:38.560080 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:38.559615    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-cr6qp" event={"ID":"e140448f-f07b-4cd2-8df6-195a25ec5e1c","Type":"ContainerDied","Data":"edb0183a9dc65129b8d3b1d28933f342b0126e6710b90432c8a38fbecb365fd3"}
Jul 28 13:20:38.560080 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:38.559746    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-cr6qp" event={"ID":"e140448f-f07b-4cd2-8df6-195a25ec5e1c","Type":"ContainerDied","Data":"66e1a4e0abd50cf82f40ebea4e6e89f00a0b481a939e6c965026c5663be13bd8"}
Jul 28 13:20:38.560080 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:38.559773    2574 scope.go:117] "RemoveContainer" containerID="a9f25efa25a6f3ee52943b5a163aad80064c6521d117136212d4cf6d9950a277"
Jul 28 13:20:38.570221 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:38.570198    2574 scope.go:117] "RemoveContainer" containerID="edb0183a9dc65129b8d3b1d28933f342b0126e6710b90432c8a38fbecb365fd3"
Jul 28 13:20:38.578441 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:38.578320    2574 scope.go:117] "RemoveContainer" containerID="e4edc95c97b4edc48d87d2bcfea5d1facdbb4a4e74f54eadef9dce16160fa98f"
Jul 28 13:20:38.578816 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:38.578799    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-cr6qp"]
Jul 28 13:20:38.582245 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:38.582225    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-cr6qp"]
Jul 28 13:20:38.585613 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:38.585598    2574 scope.go:117] "RemoveContainer" containerID="a9f25efa25a6f3ee52943b5a163aad80064c6521d117136212d4cf6d9950a277"
Jul 28 13:20:38.585874 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:20:38.585855    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9f25efa25a6f3ee52943b5a163aad80064c6521d117136212d4cf6d9950a277\": container with ID starting with a9f25efa25a6f3ee52943b5a163aad80064c6521d117136212d4cf6d9950a277 not found: ID does not exist" containerID="a9f25efa25a6f3ee52943b5a163aad80064c6521d117136212d4cf6d9950a277"
Jul 28 13:20:38.585927 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:38.585883    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9f25efa25a6f3ee52943b5a163aad80064c6521d117136212d4cf6d9950a277"} err="failed to get container status \"a9f25efa25a6f3ee52943b5a163aad80064c6521d117136212d4cf6d9950a277\": rpc error: code = NotFound desc = could not find container \"a9f25efa25a6f3ee52943b5a163aad80064c6521d117136212d4cf6d9950a277\": container with ID starting with a9f25efa25a6f3ee52943b5a163aad80064c6521d117136212d4cf6d9950a277 not found: ID does not exist"
Jul 28 13:20:38.585927 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:38.585900    2574 scope.go:117] "RemoveContainer" containerID="edb0183a9dc65129b8d3b1d28933f342b0126e6710b90432c8a38fbecb365fd3"
Jul 28 13:20:38.586134 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:20:38.586117    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edb0183a9dc65129b8d3b1d28933f342b0126e6710b90432c8a38fbecb365fd3\": container with ID starting with edb0183a9dc65129b8d3b1d28933f342b0126e6710b90432c8a38fbecb365fd3 not found: ID does not exist" containerID="edb0183a9dc65129b8d3b1d28933f342b0126e6710b90432c8a38fbecb365fd3"
Jul 28 13:20:38.586197 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:38.586146    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edb0183a9dc65129b8d3b1d28933f342b0126e6710b90432c8a38fbecb365fd3"} err="failed to get container status \"edb0183a9dc65129b8d3b1d28933f342b0126e6710b90432c8a38fbecb365fd3\": rpc error: code = NotFound desc = could not find container \"edb0183a9dc65129b8d3b1d28933f342b0126e6710b90432c8a38fbecb365fd3\": container with ID starting with edb0183a9dc65129b8d3b1d28933f342b0126e6710b90432c8a38fbecb365fd3 not found: ID does not exist"
Jul 28 13:20:38.586197 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:38.586168    2574 scope.go:117] "RemoveContainer" containerID="e4edc95c97b4edc48d87d2bcfea5d1facdbb4a4e74f54eadef9dce16160fa98f"
Jul 28 13:20:38.586399 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:20:38.586379    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4edc95c97b4edc48d87d2bcfea5d1facdbb4a4e74f54eadef9dce16160fa98f\": container with ID starting with e4edc95c97b4edc48d87d2bcfea5d1facdbb4a4e74f54eadef9dce16160fa98f not found: ID does not exist" containerID="e4edc95c97b4edc48d87d2bcfea5d1facdbb4a4e74f54eadef9dce16160fa98f"
Jul 28 13:20:38.586440 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:38.586405    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4edc95c97b4edc48d87d2bcfea5d1facdbb4a4e74f54eadef9dce16160fa98f"} err="failed to get container status \"e4edc95c97b4edc48d87d2bcfea5d1facdbb4a4e74f54eadef9dce16160fa98f\": rpc error: code = NotFound desc = could not find container \"e4edc95c97b4edc48d87d2bcfea5d1facdbb4a4e74f54eadef9dce16160fa98f\": container with ID starting with e4edc95c97b4edc48d87d2bcfea5d1facdbb4a4e74f54eadef9dce16160fa98f not found: ID does not exist"
Jul 28 13:20:40.302230 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:40.302199    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e140448f-f07b-4cd2-8df6-195a25ec5e1c" path="/var/lib/kubelet/pods/e140448f-f07b-4cd2-8df6-195a25ec5e1c/volumes"
Jul 28 13:20:41.694882 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:41.694851    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f8whm"]
Jul 28 13:20:41.695241 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:41.695204    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e140448f-f07b-4cd2-8df6-195a25ec5e1c" containerName="storage-initializer"
Jul 28 13:20:41.695241 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:41.695222    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e140448f-f07b-4cd2-8df6-195a25ec5e1c" containerName="storage-initializer"
Jul 28 13:20:41.695313 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:41.695259    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e140448f-f07b-4cd2-8df6-195a25ec5e1c" containerName="kube-rbac-proxy"
Jul 28 13:20:41.695313 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:41.695270    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e140448f-f07b-4cd2-8df6-195a25ec5e1c" containerName="kube-rbac-proxy"
Jul 28 13:20:41.695313 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:41.695283    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e140448f-f07b-4cd2-8df6-195a25ec5e1c" containerName="kserve-container"
Jul 28 13:20:41.695313 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:41.695292    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e140448f-f07b-4cd2-8df6-195a25ec5e1c" containerName="kserve-container"
Jul 28 13:20:41.695464 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:41.695365    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e140448f-f07b-4cd2-8df6-195a25ec5e1c" containerName="kube-rbac-proxy"
Jul 28 13:20:41.695464 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:41.695375    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e140448f-f07b-4cd2-8df6-195a25ec5e1c" containerName="kserve-container"
Jul 28 13:20:41.698484 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:41.698468    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f8whm"
Jul 28 13:20:41.701039 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:41.701004    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:20:41.701039 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:41.701038    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:20:41.701225 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:41.701182    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 13:20:41.701225 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:41.701191    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:20:41.701336 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:41.701259    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Jul 28 13:20:41.702133 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:41.702110    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rxfwd\""
Jul 28 13:20:41.702252 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:41.702185    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Jul 28 13:20:41.708405 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:41.708386    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f8whm"]
Jul 28 13:20:41.737939 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:41.737916    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/537292f6-3d96-4826-8e3c-73e5186b9aad-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f8whm\" (UID: \"537292f6-3d96-4826-8e3c-73e5186b9aad\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f8whm"
Jul 28 13:20:41.738026 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:41.737947    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpglm\" (UniqueName: \"kubernetes.io/projected/537292f6-3d96-4826-8e3c-73e5186b9aad-kube-api-access-hpglm\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f8whm\" (UID: \"537292f6-3d96-4826-8e3c-73e5186b9aad\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f8whm"
Jul 28 13:20:41.738026 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:41.737994    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/537292f6-3d96-4826-8e3c-73e5186b9aad-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f8whm\" (UID: \"537292f6-3d96-4826-8e3c-73e5186b9aad\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f8whm"
Jul 28 13:20:41.738026 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:41.738015    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/537292f6-3d96-4826-8e3c-73e5186b9aad-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f8whm\" (UID: \"537292f6-3d96-4826-8e3c-73e5186b9aad\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f8whm"
Jul 28 13:20:41.738132 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:41.738038    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/537292f6-3d96-4826-8e3c-73e5186b9aad-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f8whm\" (UID: \"537292f6-3d96-4826-8e3c-73e5186b9aad\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f8whm"
Jul 28 13:20:41.839007 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:41.838979    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/537292f6-3d96-4826-8e3c-73e5186b9aad-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f8whm\" (UID: \"537292f6-3d96-4826-8e3c-73e5186b9aad\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f8whm"
Jul 28 13:20:41.839103 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:41.839019    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpglm\" (UniqueName: \"kubernetes.io/projected/537292f6-3d96-4826-8e3c-73e5186b9aad-kube-api-access-hpglm\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f8whm\" (UID: \"537292f6-3d96-4826-8e3c-73e5186b9aad\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f8whm"
Jul 28 13:20:41.839103 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:41.839072    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/537292f6-3d96-4826-8e3c-73e5186b9aad-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f8whm\" (UID: \"537292f6-3d96-4826-8e3c-73e5186b9aad\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f8whm"
Jul 28 13:20:41.839103 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:41.839095    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/537292f6-3d96-4826-8e3c-73e5186b9aad-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f8whm\" (UID: \"537292f6-3d96-4826-8e3c-73e5186b9aad\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f8whm"
Jul 28 13:20:41.839280 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:41.839136    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/537292f6-3d96-4826-8e3c-73e5186b9aad-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f8whm\" (UID: \"537292f6-3d96-4826-8e3c-73e5186b9aad\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f8whm"
Jul 28 13:20:41.839490 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:41.839471    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/537292f6-3d96-4826-8e3c-73e5186b9aad-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f8whm\" (UID: \"537292f6-3d96-4826-8e3c-73e5186b9aad\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f8whm"
Jul 28 13:20:41.839665 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:41.839647    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/537292f6-3d96-4826-8e3c-73e5186b9aad-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f8whm\" (UID: \"537292f6-3d96-4826-8e3c-73e5186b9aad\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f8whm"
Jul 28 13:20:41.839779 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:41.839763    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/537292f6-3d96-4826-8e3c-73e5186b9aad-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f8whm\" (UID: \"537292f6-3d96-4826-8e3c-73e5186b9aad\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f8whm"
Jul 28 13:20:41.841379 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:41.841361    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/537292f6-3d96-4826-8e3c-73e5186b9aad-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f8whm\" (UID: \"537292f6-3d96-4826-8e3c-73e5186b9aad\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f8whm"
Jul 28 13:20:41.846751 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:41.846731    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpglm\" (UniqueName: \"kubernetes.io/projected/537292f6-3d96-4826-8e3c-73e5186b9aad-kube-api-access-hpglm\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f8whm\" (UID: \"537292f6-3d96-4826-8e3c-73e5186b9aad\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f8whm"
Jul 28 13:20:42.010023 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:42.009972    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f8whm"
Jul 28 13:20:42.135317 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:42.135293    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f8whm"]
Jul 28 13:20:42.137526 ip-10-0-138-158 kubenswrapper[2574]: W0728 13:20:42.137489    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod537292f6_3d96_4826_8e3c_73e5186b9aad.slice/crio-fc139e3036d6f867b16c9ae346c3b48f879c46c4ea0a1f36a860e4bd97e1abfc WatchSource:0}: Error finding container fc139e3036d6f867b16c9ae346c3b48f879c46c4ea0a1f36a860e4bd97e1abfc: Status 404 returned error can't find the container with id fc139e3036d6f867b16c9ae346c3b48f879c46c4ea0a1f36a860e4bd97e1abfc
Jul 28 13:20:42.580232 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:42.580195    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f8whm" event={"ID":"537292f6-3d96-4826-8e3c-73e5186b9aad","Type":"ContainerStarted","Data":"8eff99faea2c046191c23acdf530cd5753cae7ab2caba4f3d2c3cdc21291249b"}
Jul 28 13:20:42.580422 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:42.580262    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f8whm" event={"ID":"537292f6-3d96-4826-8e3c-73e5186b9aad","Type":"ContainerStarted","Data":"fc139e3036d6f867b16c9ae346c3b48f879c46c4ea0a1f36a860e4bd97e1abfc"}
Jul 28 13:20:50.618828 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:50.618799    2574 generic.go:356] "Generic (PLEG): container finished" podID="537292f6-3d96-4826-8e3c-73e5186b9aad" containerID="8eff99faea2c046191c23acdf530cd5753cae7ab2caba4f3d2c3cdc21291249b" exitCode=0
Jul 28 13:20:50.619217 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:50.618875    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f8whm" event={"ID":"537292f6-3d96-4826-8e3c-73e5186b9aad","Type":"ContainerDied","Data":"8eff99faea2c046191c23acdf530cd5753cae7ab2caba4f3d2c3cdc21291249b"}
Jul 28 13:20:51.631387 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:51.631348    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f8whm" event={"ID":"537292f6-3d96-4826-8e3c-73e5186b9aad","Type":"ContainerStarted","Data":"9e70c2675801f75c3f82cf2dfc53722728d39798e190c2c0a69f6ece764fd841"}
Jul 28 13:20:51.631815 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:51.631436    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f8whm" event={"ID":"537292f6-3d96-4826-8e3c-73e5186b9aad","Type":"ContainerStarted","Data":"8a195be650dfc8a2617b30ad584861ba83094d1f4640ee4e10cdaa2cfbb2a76c"}
Jul 28 13:20:51.631815 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:51.631668    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f8whm"
Jul 28 13:20:51.652416 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:51.652362    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f8whm" podStartSLOduration=2.387754038 podStartE2EDuration="10.652348031s" totalImagesPullingTime="0s" totalInitContainerRuntime="8.264593993s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:20:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:20:51.650308825 +0000 UTC m=+1663.855695450" watchObservedRunningTime="2026-07-28 13:20:51.652348031 +0000 UTC m=+1663.857734632"
Jul 28 13:20:52.637447 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:52.637417    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f8whm"
Jul 28 13:20:52.638846 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:52.638812    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f8whm" podUID="537292f6-3d96-4826-8e3c-73e5186b9aad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 28 13:20:53.642091 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:53.642051    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f8whm" podUID="537292f6-3d96-4826-8e3c-73e5186b9aad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 28 13:20:58.646766 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:58.646737    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f8whm"
Jul 28 13:20:58.647338 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:20:58.647307    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f8whm" podUID="537292f6-3d96-4826-8e3c-73e5186b9aad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 28 13:21:08.647838 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:08.647811    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f8whm"
Jul 28 13:21:23.778520 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:23.778489    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f8whm"]
Jul 28 13:21:23.778931 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:23.778742    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f8whm" podUID="537292f6-3d96-4826-8e3c-73e5186b9aad" containerName="kserve-container" containerID="cri-o://8a195be650dfc8a2617b30ad584861ba83094d1f4640ee4e10cdaa2cfbb2a76c" gracePeriod=30
Jul 28 13:21:23.778931 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:23.778827    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f8whm" podUID="537292f6-3d96-4826-8e3c-73e5186b9aad" containerName="kube-rbac-proxy" containerID="cri-o://9e70c2675801f75c3f82cf2dfc53722728d39798e190c2c0a69f6ece764fd841" gracePeriod=30
Jul 28 13:21:24.801800 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:24.801764    2574 generic.go:356] "Generic (PLEG): container finished" podID="537292f6-3d96-4826-8e3c-73e5186b9aad" containerID="9e70c2675801f75c3f82cf2dfc53722728d39798e190c2c0a69f6ece764fd841" exitCode=2
Jul 28 13:21:24.802294 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:24.801842    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f8whm" event={"ID":"537292f6-3d96-4826-8e3c-73e5186b9aad","Type":"ContainerDied","Data":"9e70c2675801f75c3f82cf2dfc53722728d39798e190c2c0a69f6ece764fd841"}
Jul 28 13:21:25.332349 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:25.332324    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f8whm"
Jul 28 13:21:25.481765 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:25.481735    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hpglm\" (UniqueName: \"kubernetes.io/projected/537292f6-3d96-4826-8e3c-73e5186b9aad-kube-api-access-hpglm\") pod \"537292f6-3d96-4826-8e3c-73e5186b9aad\" (UID: \"537292f6-3d96-4826-8e3c-73e5186b9aad\") "
Jul 28 13:21:25.481933 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:25.481791    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/537292f6-3d96-4826-8e3c-73e5186b9aad-proxy-tls\") pod \"537292f6-3d96-4826-8e3c-73e5186b9aad\" (UID: \"537292f6-3d96-4826-8e3c-73e5186b9aad\") "
Jul 28 13:21:25.481933 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:25.481818    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/537292f6-3d96-4826-8e3c-73e5186b9aad-kserve-provision-location\") pod \"537292f6-3d96-4826-8e3c-73e5186b9aad\" (UID: \"537292f6-3d96-4826-8e3c-73e5186b9aad\") "
Jul 28 13:21:25.481933 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:25.481846    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/537292f6-3d96-4826-8e3c-73e5186b9aad-cabundle-cert\") pod \"537292f6-3d96-4826-8e3c-73e5186b9aad\" (UID: \"537292f6-3d96-4826-8e3c-73e5186b9aad\") "
Jul 28 13:21:25.481933 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:25.481871    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/537292f6-3d96-4826-8e3c-73e5186b9aad-kube-rbac-proxy-sar-config\") pod \"537292f6-3d96-4826-8e3c-73e5186b9aad\" (UID: \"537292f6-3d96-4826-8e3c-73e5186b9aad\") "
Jul 28 13:21:25.482237 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:25.482212    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/537292f6-3d96-4826-8e3c-73e5186b9aad-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "537292f6-3d96-4826-8e3c-73e5186b9aad" (UID: "537292f6-3d96-4826-8e3c-73e5186b9aad"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:21:25.482319 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:25.482220    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/537292f6-3d96-4826-8e3c-73e5186b9aad-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "537292f6-3d96-4826-8e3c-73e5186b9aad" (UID: "537292f6-3d96-4826-8e3c-73e5186b9aad"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:21:25.483037 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:25.483012    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/537292f6-3d96-4826-8e3c-73e5186b9aad-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "537292f6-3d96-4826-8e3c-73e5186b9aad" (UID: "537292f6-3d96-4826-8e3c-73e5186b9aad"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:21:25.483855 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:25.483835    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/537292f6-3d96-4826-8e3c-73e5186b9aad-kube-api-access-hpglm" (OuterVolumeSpecName: "kube-api-access-hpglm") pod "537292f6-3d96-4826-8e3c-73e5186b9aad" (UID: "537292f6-3d96-4826-8e3c-73e5186b9aad"). InnerVolumeSpecName "kube-api-access-hpglm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:21:25.484054 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:25.484036    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/537292f6-3d96-4826-8e3c-73e5186b9aad-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "537292f6-3d96-4826-8e3c-73e5186b9aad" (UID: "537292f6-3d96-4826-8e3c-73e5186b9aad"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:21:25.582566 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:25.582528    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpglm\" (UniqueName: \"kubernetes.io/projected/537292f6-3d96-4826-8e3c-73e5186b9aad-kube-api-access-hpglm\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:21:25.582566 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:25.582567    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/537292f6-3d96-4826-8e3c-73e5186b9aad-proxy-tls\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:21:25.582566 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:25.582577    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/537292f6-3d96-4826-8e3c-73e5186b9aad-kserve-provision-location\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:21:25.582789 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:25.582587    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/537292f6-3d96-4826-8e3c-73e5186b9aad-cabundle-cert\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:21:25.582789 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:25.582595    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/537292f6-3d96-4826-8e3c-73e5186b9aad-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:21:25.808469 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:25.808401    2574 generic.go:356] "Generic (PLEG): container finished" podID="537292f6-3d96-4826-8e3c-73e5186b9aad" containerID="8a195be650dfc8a2617b30ad584861ba83094d1f4640ee4e10cdaa2cfbb2a76c" exitCode=0
Jul 28 13:21:25.808469 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:25.808433    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f8whm" event={"ID":"537292f6-3d96-4826-8e3c-73e5186b9aad","Type":"ContainerDied","Data":"8a195be650dfc8a2617b30ad584861ba83094d1f4640ee4e10cdaa2cfbb2a76c"}
Jul 28 13:21:25.808469 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:25.808463    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f8whm"
Jul 28 13:21:25.808921 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:25.808475    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f8whm" event={"ID":"537292f6-3d96-4826-8e3c-73e5186b9aad","Type":"ContainerDied","Data":"fc139e3036d6f867b16c9ae346c3b48f879c46c4ea0a1f36a860e4bd97e1abfc"}
Jul 28 13:21:25.808921 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:25.808500    2574 scope.go:117] "RemoveContainer" containerID="9e70c2675801f75c3f82cf2dfc53722728d39798e190c2c0a69f6ece764fd841"
Jul 28 13:21:25.816872 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:25.816856    2574 scope.go:117] "RemoveContainer" containerID="8a195be650dfc8a2617b30ad584861ba83094d1f4640ee4e10cdaa2cfbb2a76c"
Jul 28 13:21:25.823843 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:25.823826    2574 scope.go:117] "RemoveContainer" containerID="8eff99faea2c046191c23acdf530cd5753cae7ab2caba4f3d2c3cdc21291249b"
Jul 28 13:21:25.831000 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:25.830982    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f8whm"]
Jul 28 13:21:25.831067 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:25.831012    2574 scope.go:117] "RemoveContainer" containerID="9e70c2675801f75c3f82cf2dfc53722728d39798e190c2c0a69f6ece764fd841"
Jul 28 13:21:25.831295 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:21:25.831277    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e70c2675801f75c3f82cf2dfc53722728d39798e190c2c0a69f6ece764fd841\": container with ID starting with 9e70c2675801f75c3f82cf2dfc53722728d39798e190c2c0a69f6ece764fd841 not found: ID does not exist" containerID="9e70c2675801f75c3f82cf2dfc53722728d39798e190c2c0a69f6ece764fd841"
Jul 28 13:21:25.831342 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:25.831303    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e70c2675801f75c3f82cf2dfc53722728d39798e190c2c0a69f6ece764fd841"} err="failed to get container status \"9e70c2675801f75c3f82cf2dfc53722728d39798e190c2c0a69f6ece764fd841\": rpc error: code = NotFound desc = could not find container \"9e70c2675801f75c3f82cf2dfc53722728d39798e190c2c0a69f6ece764fd841\": container with ID starting with 9e70c2675801f75c3f82cf2dfc53722728d39798e190c2c0a69f6ece764fd841 not found: ID does not exist"
Jul 28 13:21:25.831342 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:25.831318    2574 scope.go:117] "RemoveContainer" containerID="8a195be650dfc8a2617b30ad584861ba83094d1f4640ee4e10cdaa2cfbb2a76c"
Jul 28 13:21:25.831525 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:21:25.831506    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a195be650dfc8a2617b30ad584861ba83094d1f4640ee4e10cdaa2cfbb2a76c\": container with ID starting with 8a195be650dfc8a2617b30ad584861ba83094d1f4640ee4e10cdaa2cfbb2a76c not found: ID does not exist" containerID="8a195be650dfc8a2617b30ad584861ba83094d1f4640ee4e10cdaa2cfbb2a76c"
Jul 28 13:21:25.831586 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:25.831533    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a195be650dfc8a2617b30ad584861ba83094d1f4640ee4e10cdaa2cfbb2a76c"} err="failed to get container status \"8a195be650dfc8a2617b30ad584861ba83094d1f4640ee4e10cdaa2cfbb2a76c\": rpc error: code = NotFound desc = could not find container \"8a195be650dfc8a2617b30ad584861ba83094d1f4640ee4e10cdaa2cfbb2a76c\": container with ID starting with 8a195be650dfc8a2617b30ad584861ba83094d1f4640ee4e10cdaa2cfbb2a76c not found: ID does not exist"
Jul 28 13:21:25.831586 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:25.831555    2574 scope.go:117] "RemoveContainer" containerID="8eff99faea2c046191c23acdf530cd5753cae7ab2caba4f3d2c3cdc21291249b"
Jul 28 13:21:25.831825 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:21:25.831808    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8eff99faea2c046191c23acdf530cd5753cae7ab2caba4f3d2c3cdc21291249b\": container with ID starting with 8eff99faea2c046191c23acdf530cd5753cae7ab2caba4f3d2c3cdc21291249b not found: ID does not exist" containerID="8eff99faea2c046191c23acdf530cd5753cae7ab2caba4f3d2c3cdc21291249b"
Jul 28 13:21:25.831876 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:25.831829    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eff99faea2c046191c23acdf530cd5753cae7ab2caba4f3d2c3cdc21291249b"} err="failed to get container status \"8eff99faea2c046191c23acdf530cd5753cae7ab2caba4f3d2c3cdc21291249b\": rpc error: code = NotFound desc = could not find container \"8eff99faea2c046191c23acdf530cd5753cae7ab2caba4f3d2c3cdc21291249b\": container with ID starting with 8eff99faea2c046191c23acdf530cd5753cae7ab2caba4f3d2c3cdc21291249b not found: ID does not exist"
Jul 28 13:21:25.835144 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:25.835128    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f8whm"]
Jul 28 13:21:26.302062 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:26.302036    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="537292f6-3d96-4826-8e3c-73e5186b9aad" path="/var/lib/kubelet/pods/537292f6-3d96-4826-8e3c-73e5186b9aad/volumes"
Jul 28 13:21:28.997341 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:28.997314    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-ztd6b"]
Jul 28 13:21:28.997731 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:28.997624    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="537292f6-3d96-4826-8e3c-73e5186b9aad" containerName="kserve-container"
Jul 28 13:21:28.997731 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:28.997635    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="537292f6-3d96-4826-8e3c-73e5186b9aad" containerName="kserve-container"
Jul 28 13:21:28.997731 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:28.997650    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="537292f6-3d96-4826-8e3c-73e5186b9aad" containerName="storage-initializer"
Jul 28 13:21:28.997731 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:28.997655    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="537292f6-3d96-4826-8e3c-73e5186b9aad" containerName="storage-initializer"
Jul 28 13:21:28.997731 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:28.997664    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="537292f6-3d96-4826-8e3c-73e5186b9aad" containerName="kube-rbac-proxy"
Jul 28 13:21:28.997731 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:28.997670    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="537292f6-3d96-4826-8e3c-73e5186b9aad" containerName="kube-rbac-proxy"
Jul 28 13:21:28.997931 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:28.997760    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="537292f6-3d96-4826-8e3c-73e5186b9aad" containerName="kube-rbac-proxy"
Jul 28 13:21:28.997931 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:28.997767    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="537292f6-3d96-4826-8e3c-73e5186b9aad" containerName="kserve-container"
Jul 28 13:21:29.002763 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:29.002745    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-ztd6b"
Jul 28 13:21:29.005544 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:29.005522    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-predictor-serving-cert\""
Jul 28 13:21:29.005656 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:29.005569    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:21:29.005656 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:29.005609    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 13:21:29.006706 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:29.006672    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rxfwd\""
Jul 28 13:21:29.006815 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:29.006733    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:21:29.006815 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:29.006747    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:21:29.006815 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:29.006763    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-kube-rbac-proxy-sar-config\""
Jul 28 13:21:29.013865 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:29.013844    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-ztd6b"]
Jul 28 13:21:29.105798 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:29.105774    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/088dfe14-683b-48a2-bdf0-0c3b9ad273f9-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-ztd6b\" (UID: \"088dfe14-683b-48a2-bdf0-0c3b9ad273f9\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-ztd6b"
Jul 28 13:21:29.105913 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:29.105807    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/088dfe14-683b-48a2-bdf0-0c3b9ad273f9-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-ztd6b\" (UID: \"088dfe14-683b-48a2-bdf0-0c3b9ad273f9\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-ztd6b"
Jul 28 13:21:29.105913 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:29.105833    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hrtc\" (UniqueName: \"kubernetes.io/projected/088dfe14-683b-48a2-bdf0-0c3b9ad273f9-kube-api-access-6hrtc\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-ztd6b\" (UID: \"088dfe14-683b-48a2-bdf0-0c3b9ad273f9\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-ztd6b"
Jul 28 13:21:29.105913 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:29.105891    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/088dfe14-683b-48a2-bdf0-0c3b9ad273f9-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-ztd6b\" (UID: \"088dfe14-683b-48a2-bdf0-0c3b9ad273f9\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-ztd6b"
Jul 28 13:21:29.105913 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:29.105910    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/088dfe14-683b-48a2-bdf0-0c3b9ad273f9-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-ztd6b\" (UID: \"088dfe14-683b-48a2-bdf0-0c3b9ad273f9\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-ztd6b"
Jul 28 13:21:29.206776 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:29.206748    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/088dfe14-683b-48a2-bdf0-0c3b9ad273f9-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-ztd6b\" (UID: \"088dfe14-683b-48a2-bdf0-0c3b9ad273f9\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-ztd6b"
Jul 28 13:21:29.206929 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:29.206785    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hrtc\" (UniqueName: \"kubernetes.io/projected/088dfe14-683b-48a2-bdf0-0c3b9ad273f9-kube-api-access-6hrtc\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-ztd6b\" (UID: \"088dfe14-683b-48a2-bdf0-0c3b9ad273f9\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-ztd6b"
Jul 28 13:21:29.206929 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:29.206828    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/088dfe14-683b-48a2-bdf0-0c3b9ad273f9-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-ztd6b\" (UID: \"088dfe14-683b-48a2-bdf0-0c3b9ad273f9\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-ztd6b"
Jul 28 13:21:29.206929 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:29.206846    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/088dfe14-683b-48a2-bdf0-0c3b9ad273f9-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-ztd6b\" (UID: \"088dfe14-683b-48a2-bdf0-0c3b9ad273f9\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-ztd6b"
Jul 28 13:21:29.206929 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:29.206881    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/088dfe14-683b-48a2-bdf0-0c3b9ad273f9-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-ztd6b\" (UID: \"088dfe14-683b-48a2-bdf0-0c3b9ad273f9\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-ztd6b"
Jul 28 13:21:29.207200 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:21:29.206968    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-serving-cert: secret "isvc-autogluon-ts-v1-noslash-predictor-serving-cert" not found
Jul 28 13:21:29.207200 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:21:29.207024    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/088dfe14-683b-48a2-bdf0-0c3b9ad273f9-proxy-tls podName:088dfe14-683b-48a2-bdf0-0c3b9ad273f9 nodeName:}" failed. No retries permitted until 2026-07-28 13:21:29.707005291 +0000 UTC m=+1701.912391870 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/088dfe14-683b-48a2-bdf0-0c3b9ad273f9-proxy-tls") pod "isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-ztd6b" (UID: "088dfe14-683b-48a2-bdf0-0c3b9ad273f9") : secret "isvc-autogluon-ts-v1-noslash-predictor-serving-cert" not found
Jul 28 13:21:29.207335 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:29.207272    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/088dfe14-683b-48a2-bdf0-0c3b9ad273f9-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-ztd6b\" (UID: \"088dfe14-683b-48a2-bdf0-0c3b9ad273f9\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-ztd6b"
Jul 28 13:21:29.207457 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:29.207436    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/088dfe14-683b-48a2-bdf0-0c3b9ad273f9-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-ztd6b\" (UID: \"088dfe14-683b-48a2-bdf0-0c3b9ad273f9\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-ztd6b"
Jul 28 13:21:29.207586 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:29.207566    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/088dfe14-683b-48a2-bdf0-0c3b9ad273f9-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-ztd6b\" (UID: \"088dfe14-683b-48a2-bdf0-0c3b9ad273f9\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-ztd6b"
Jul 28 13:21:29.217024 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:29.216999    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hrtc\" (UniqueName: \"kubernetes.io/projected/088dfe14-683b-48a2-bdf0-0c3b9ad273f9-kube-api-access-6hrtc\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-ztd6b\" (UID: \"088dfe14-683b-48a2-bdf0-0c3b9ad273f9\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-ztd6b"
Jul 28 13:21:29.711615 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:29.711580    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/088dfe14-683b-48a2-bdf0-0c3b9ad273f9-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-ztd6b\" (UID: \"088dfe14-683b-48a2-bdf0-0c3b9ad273f9\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-ztd6b"
Jul 28 13:21:29.713849 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:29.713823    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/088dfe14-683b-48a2-bdf0-0c3b9ad273f9-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-ztd6b\" (UID: \"088dfe14-683b-48a2-bdf0-0c3b9ad273f9\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-ztd6b"
Jul 28 13:21:29.913583 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:29.913555    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-ztd6b"
Jul 28 13:21:30.041129 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:30.041107    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-ztd6b"]
Jul 28 13:21:30.042934 ip-10-0-138-158 kubenswrapper[2574]: W0728 13:21:30.042906    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod088dfe14_683b_48a2_bdf0_0c3b9ad273f9.slice/crio-1a6540f03f9e4301b7d1445aca2b33bfe9bd659aad4336c8cf8b1cd0fdcfdb42 WatchSource:0}: Error finding container 1a6540f03f9e4301b7d1445aca2b33bfe9bd659aad4336c8cf8b1cd0fdcfdb42: Status 404 returned error can't find the container with id 1a6540f03f9e4301b7d1445aca2b33bfe9bd659aad4336c8cf8b1cd0fdcfdb42
Jul 28 13:21:30.832247 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:30.832170    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-ztd6b" event={"ID":"088dfe14-683b-48a2-bdf0-0c3b9ad273f9","Type":"ContainerStarted","Data":"8212e316a79ecfec43fc0909937af3512bc760b1264be767f1cf93b6778a2979"}
Jul 28 13:21:30.832403 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:30.832263    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-ztd6b" event={"ID":"088dfe14-683b-48a2-bdf0-0c3b9ad273f9","Type":"ContainerStarted","Data":"1a6540f03f9e4301b7d1445aca2b33bfe9bd659aad4336c8cf8b1cd0fdcfdb42"}
Jul 28 13:21:34.850645 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:34.850579    2574 generic.go:356] "Generic (PLEG): container finished" podID="088dfe14-683b-48a2-bdf0-0c3b9ad273f9" containerID="8212e316a79ecfec43fc0909937af3512bc760b1264be767f1cf93b6778a2979" exitCode=0
Jul 28 13:21:34.850957 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:34.850649    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-ztd6b" event={"ID":"088dfe14-683b-48a2-bdf0-0c3b9ad273f9","Type":"ContainerDied","Data":"8212e316a79ecfec43fc0909937af3512bc760b1264be767f1cf93b6778a2979"}
Jul 28 13:21:35.858098 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:35.858062    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-ztd6b" event={"ID":"088dfe14-683b-48a2-bdf0-0c3b9ad273f9","Type":"ContainerStarted","Data":"6034d1ba3e9ec2ba73b310f930abf88b3a9bbb7d0ecba2f664e28573f826c6bc"}
Jul 28 13:21:35.858567 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:35.858135    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-ztd6b" event={"ID":"088dfe14-683b-48a2-bdf0-0c3b9ad273f9","Type":"ContainerStarted","Data":"2f6651ae8a778f27c78ca7caf4f03f9a5fb1b467d0208f67171c3d9e67281fdd"}
Jul 28 13:21:35.858567 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:35.858356    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-ztd6b"
Jul 28 13:21:35.879659 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:35.879614    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-ztd6b" podStartSLOduration=3.423201139 podStartE2EDuration="7.879599578s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.456398439s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:21:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:21:35.87796985 +0000 UTC m=+1708.083356450" watchObservedRunningTime="2026-07-28 13:21:35.879599578 +0000 UTC m=+1708.084986179"
Jul 28 13:21:36.869066 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:36.869034    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-ztd6b"
Jul 28 13:21:36.870464 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:36.870430    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-ztd6b" podUID="088dfe14-683b-48a2-bdf0-0c3b9ad273f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 28 13:21:37.874380 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:37.874344    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-ztd6b" podUID="088dfe14-683b-48a2-bdf0-0c3b9ad273f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 28 13:21:42.879222 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:42.879197    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-ztd6b"
Jul 28 13:21:42.879769 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:42.879743    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-ztd6b" podUID="088dfe14-683b-48a2-bdf0-0c3b9ad273f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 28 13:21:52.880748 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:21:52.880718    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-ztd6b"
Jul 28 13:22:04.088349 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:04.088317    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-ztd6b"]
Jul 28 13:22:04.088845 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:04.088565    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-ztd6b" podUID="088dfe14-683b-48a2-bdf0-0c3b9ad273f9" containerName="kserve-container" containerID="cri-o://2f6651ae8a778f27c78ca7caf4f03f9a5fb1b467d0208f67171c3d9e67281fdd" gracePeriod=30
Jul 28 13:22:04.088845 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:04.088600    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-ztd6b" podUID="088dfe14-683b-48a2-bdf0-0c3b9ad273f9" containerName="kube-rbac-proxy" containerID="cri-o://6034d1ba3e9ec2ba73b310f930abf88b3a9bbb7d0ecba2f664e28573f826c6bc" gracePeriod=30
Jul 28 13:22:05.002205 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:05.002172    2574 generic.go:356] "Generic (PLEG): container finished" podID="088dfe14-683b-48a2-bdf0-0c3b9ad273f9" containerID="6034d1ba3e9ec2ba73b310f930abf88b3a9bbb7d0ecba2f664e28573f826c6bc" exitCode=2
Jul 28 13:22:05.002371 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:05.002230    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-ztd6b" event={"ID":"088dfe14-683b-48a2-bdf0-0c3b9ad273f9","Type":"ContainerDied","Data":"6034d1ba3e9ec2ba73b310f930abf88b3a9bbb7d0ecba2f664e28573f826c6bc"}
Jul 28 13:22:05.331778 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:05.331750    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-ztd6b"
Jul 28 13:22:05.379248 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:05.379224    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/088dfe14-683b-48a2-bdf0-0c3b9ad273f9-cabundle-cert\") pod \"088dfe14-683b-48a2-bdf0-0c3b9ad273f9\" (UID: \"088dfe14-683b-48a2-bdf0-0c3b9ad273f9\") "
Jul 28 13:22:05.379362 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:05.379342    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6hrtc\" (UniqueName: \"kubernetes.io/projected/088dfe14-683b-48a2-bdf0-0c3b9ad273f9-kube-api-access-6hrtc\") pod \"088dfe14-683b-48a2-bdf0-0c3b9ad273f9\" (UID: \"088dfe14-683b-48a2-bdf0-0c3b9ad273f9\") "
Jul 28 13:22:05.379421 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:05.379403    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/088dfe14-683b-48a2-bdf0-0c3b9ad273f9-kube-rbac-proxy-sar-config\") pod \"088dfe14-683b-48a2-bdf0-0c3b9ad273f9\" (UID: \"088dfe14-683b-48a2-bdf0-0c3b9ad273f9\") "
Jul 28 13:22:05.379486 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:05.379449    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/088dfe14-683b-48a2-bdf0-0c3b9ad273f9-kserve-provision-location\") pod \"088dfe14-683b-48a2-bdf0-0c3b9ad273f9\" (UID: \"088dfe14-683b-48a2-bdf0-0c3b9ad273f9\") "
Jul 28 13:22:05.379542 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:05.379503    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/088dfe14-683b-48a2-bdf0-0c3b9ad273f9-proxy-tls\") pod \"088dfe14-683b-48a2-bdf0-0c3b9ad273f9\" (UID: \"088dfe14-683b-48a2-bdf0-0c3b9ad273f9\") "
Jul 28 13:22:05.379542 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:05.379531    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/088dfe14-683b-48a2-bdf0-0c3b9ad273f9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "088dfe14-683b-48a2-bdf0-0c3b9ad273f9" (UID: "088dfe14-683b-48a2-bdf0-0c3b9ad273f9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:22:05.379784 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:05.379754    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/088dfe14-683b-48a2-bdf0-0c3b9ad273f9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "088dfe14-683b-48a2-bdf0-0c3b9ad273f9" (UID: "088dfe14-683b-48a2-bdf0-0c3b9ad273f9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:22:05.379867 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:05.379771    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/088dfe14-683b-48a2-bdf0-0c3b9ad273f9-cabundle-cert\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:22:05.380403 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:05.380377    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/088dfe14-683b-48a2-bdf0-0c3b9ad273f9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "088dfe14-683b-48a2-bdf0-0c3b9ad273f9" (UID: "088dfe14-683b-48a2-bdf0-0c3b9ad273f9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:22:05.381404 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:05.381380    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/088dfe14-683b-48a2-bdf0-0c3b9ad273f9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "088dfe14-683b-48a2-bdf0-0c3b9ad273f9" (UID: "088dfe14-683b-48a2-bdf0-0c3b9ad273f9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:22:05.381501 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:05.381480    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/088dfe14-683b-48a2-bdf0-0c3b9ad273f9-kube-api-access-6hrtc" (OuterVolumeSpecName: "kube-api-access-6hrtc") pod "088dfe14-683b-48a2-bdf0-0c3b9ad273f9" (UID: "088dfe14-683b-48a2-bdf0-0c3b9ad273f9"). InnerVolumeSpecName "kube-api-access-6hrtc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:22:05.480068 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:05.480044    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/088dfe14-683b-48a2-bdf0-0c3b9ad273f9-proxy-tls\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:22:05.480068 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:05.480064    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hrtc\" (UniqueName: \"kubernetes.io/projected/088dfe14-683b-48a2-bdf0-0c3b9ad273f9-kube-api-access-6hrtc\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:22:05.480203 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:05.480074    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/088dfe14-683b-48a2-bdf0-0c3b9ad273f9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:22:05.480203 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:05.480084    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/088dfe14-683b-48a2-bdf0-0c3b9ad273f9-kserve-provision-location\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:22:06.009342 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:06.009314    2574 generic.go:356] "Generic (PLEG): container finished" podID="088dfe14-683b-48a2-bdf0-0c3b9ad273f9" containerID="2f6651ae8a778f27c78ca7caf4f03f9a5fb1b467d0208f67171c3d9e67281fdd" exitCode=0
Jul 28 13:22:06.009542 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:06.009381    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-ztd6b"
Jul 28 13:22:06.009542 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:06.009400    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-ztd6b" event={"ID":"088dfe14-683b-48a2-bdf0-0c3b9ad273f9","Type":"ContainerDied","Data":"2f6651ae8a778f27c78ca7caf4f03f9a5fb1b467d0208f67171c3d9e67281fdd"}
Jul 28 13:22:06.009542 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:06.009460    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-ztd6b" event={"ID":"088dfe14-683b-48a2-bdf0-0c3b9ad273f9","Type":"ContainerDied","Data":"1a6540f03f9e4301b7d1445aca2b33bfe9bd659aad4336c8cf8b1cd0fdcfdb42"}
Jul 28 13:22:06.009542 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:06.009494    2574 scope.go:117] "RemoveContainer" containerID="6034d1ba3e9ec2ba73b310f930abf88b3a9bbb7d0ecba2f664e28573f826c6bc"
Jul 28 13:22:06.018522 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:06.018504    2574 scope.go:117] "RemoveContainer" containerID="2f6651ae8a778f27c78ca7caf4f03f9a5fb1b467d0208f67171c3d9e67281fdd"
Jul 28 13:22:06.025983 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:06.025964    2574 scope.go:117] "RemoveContainer" containerID="8212e316a79ecfec43fc0909937af3512bc760b1264be767f1cf93b6778a2979"
Jul 28 13:22:06.032801 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:06.032780    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-ztd6b"]
Jul 28 13:22:06.033312 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:06.033294    2574 scope.go:117] "RemoveContainer" containerID="6034d1ba3e9ec2ba73b310f930abf88b3a9bbb7d0ecba2f664e28573f826c6bc"
Jul 28 13:22:06.033587 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:22:06.033568    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6034d1ba3e9ec2ba73b310f930abf88b3a9bbb7d0ecba2f664e28573f826c6bc\": container with ID starting with 6034d1ba3e9ec2ba73b310f930abf88b3a9bbb7d0ecba2f664e28573f826c6bc not found: ID does not exist" containerID="6034d1ba3e9ec2ba73b310f930abf88b3a9bbb7d0ecba2f664e28573f826c6bc"
Jul 28 13:22:06.033653 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:06.033595    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6034d1ba3e9ec2ba73b310f930abf88b3a9bbb7d0ecba2f664e28573f826c6bc"} err="failed to get container status \"6034d1ba3e9ec2ba73b310f930abf88b3a9bbb7d0ecba2f664e28573f826c6bc\": rpc error: code = NotFound desc = could not find container \"6034d1ba3e9ec2ba73b310f930abf88b3a9bbb7d0ecba2f664e28573f826c6bc\": container with ID starting with 6034d1ba3e9ec2ba73b310f930abf88b3a9bbb7d0ecba2f664e28573f826c6bc not found: ID does not exist"
Jul 28 13:22:06.033653 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:06.033611    2574 scope.go:117] "RemoveContainer" containerID="2f6651ae8a778f27c78ca7caf4f03f9a5fb1b467d0208f67171c3d9e67281fdd"
Jul 28 13:22:06.033861 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:22:06.033847    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f6651ae8a778f27c78ca7caf4f03f9a5fb1b467d0208f67171c3d9e67281fdd\": container with ID starting with 2f6651ae8a778f27c78ca7caf4f03f9a5fb1b467d0208f67171c3d9e67281fdd not found: ID does not exist" containerID="2f6651ae8a778f27c78ca7caf4f03f9a5fb1b467d0208f67171c3d9e67281fdd"
Jul 28 13:22:06.033909 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:06.033866    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f6651ae8a778f27c78ca7caf4f03f9a5fb1b467d0208f67171c3d9e67281fdd"} err="failed to get container status \"2f6651ae8a778f27c78ca7caf4f03f9a5fb1b467d0208f67171c3d9e67281fdd\": rpc error: code = NotFound desc = could not find container \"2f6651ae8a778f27c78ca7caf4f03f9a5fb1b467d0208f67171c3d9e67281fdd\": container with ID starting with 2f6651ae8a778f27c78ca7caf4f03f9a5fb1b467d0208f67171c3d9e67281fdd not found: ID does not exist"
Jul 28 13:22:06.033909 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:06.033877    2574 scope.go:117] "RemoveContainer" containerID="8212e316a79ecfec43fc0909937af3512bc760b1264be767f1cf93b6778a2979"
Jul 28 13:22:06.034103 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:22:06.034086    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8212e316a79ecfec43fc0909937af3512bc760b1264be767f1cf93b6778a2979\": container with ID starting with 8212e316a79ecfec43fc0909937af3512bc760b1264be767f1cf93b6778a2979 not found: ID does not exist" containerID="8212e316a79ecfec43fc0909937af3512bc760b1264be767f1cf93b6778a2979"
Jul 28 13:22:06.034165 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:06.034110    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8212e316a79ecfec43fc0909937af3512bc760b1264be767f1cf93b6778a2979"} err="failed to get container status \"8212e316a79ecfec43fc0909937af3512bc760b1264be767f1cf93b6778a2979\": rpc error: code = NotFound desc = could not find container \"8212e316a79ecfec43fc0909937af3512bc760b1264be767f1cf93b6778a2979\": container with ID starting with 8212e316a79ecfec43fc0909937af3512bc760b1264be767f1cf93b6778a2979 not found: ID does not exist"
Jul 28 13:22:06.038261 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:06.038243    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-ztd6b"]
Jul 28 13:22:06.302152 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:06.302096    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="088dfe14-683b-48a2-bdf0-0c3b9ad273f9" path="/var/lib/kubelet/pods/088dfe14-683b-48a2-bdf0-0c3b9ad273f9/volumes"
Jul 28 13:22:09.360094 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:09.360064    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-gmqwb"]
Jul 28 13:22:09.360452 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:09.360415    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="088dfe14-683b-48a2-bdf0-0c3b9ad273f9" containerName="storage-initializer"
Jul 28 13:22:09.360452 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:09.360429    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="088dfe14-683b-48a2-bdf0-0c3b9ad273f9" containerName="storage-initializer"
Jul 28 13:22:09.360452 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:09.360436    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="088dfe14-683b-48a2-bdf0-0c3b9ad273f9" containerName="kserve-container"
Jul 28 13:22:09.360452 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:09.360441    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="088dfe14-683b-48a2-bdf0-0c3b9ad273f9" containerName="kserve-container"
Jul 28 13:22:09.360582 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:09.360456    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="088dfe14-683b-48a2-bdf0-0c3b9ad273f9" containerName="kube-rbac-proxy"
Jul 28 13:22:09.360582 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:09.360462    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="088dfe14-683b-48a2-bdf0-0c3b9ad273f9" containerName="kube-rbac-proxy"
Jul 28 13:22:09.360582 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:09.360508    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="088dfe14-683b-48a2-bdf0-0c3b9ad273f9" containerName="kube-rbac-proxy"
Jul 28 13:22:09.360582 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:09.360516    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="088dfe14-683b-48a2-bdf0-0c3b9ad273f9" containerName="kserve-container"
Jul 28 13:22:09.365202 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:09.365182    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-gmqwb"
Jul 28 13:22:09.367869 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:09.367833    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 13:22:09.368029 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:09.367973    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rxfwd\""
Jul 28 13:22:09.368281 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:09.368261    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Jul 28 13:22:09.368346 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:09.368315    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:22:09.368974 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:09.368941    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Jul 28 13:22:09.368974 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:09.368946    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:22:09.369201 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:09.368946    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:22:09.376468 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:09.376436    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-gmqwb"]
Jul 28 13:22:09.379198 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:09.379177    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq"]
Jul 28 13:22:09.382566 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:09.382547    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq"
Jul 28 13:22:09.384938 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:09.384920    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Jul 28 13:22:09.393170 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:09.393152    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq"]
Jul 28 13:22:09.403737 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:09.403711    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d51af15-6745-417c-b03a-061d477c0246-proxy-tls\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq\" (UID: \"4d51af15-6745-417c-b03a-061d477c0246\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq"
Jul 28 13:22:09.403833 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:09.403744    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d51af15-6745-417c-b03a-061d477c0246-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq\" (UID: \"4d51af15-6745-417c-b03a-061d477c0246\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq"
Jul 28 13:22:09.403833 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:09.403770    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g28nv\" (UniqueName: \"kubernetes.io/projected/6d8a6d15-74e3-45ad-9651-a45f8291cfaa-kube-api-access-g28nv\") pod \"isvc-canary-create-predictor-cb85cf55c-gmqwb\" (UID: \"6d8a6d15-74e3-45ad-9651-a45f8291cfaa\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-gmqwb"
Jul 28 13:22:09.403833 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:09.403813    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvfvf\" (UniqueName: \"kubernetes.io/projected/4d51af15-6745-417c-b03a-061d477c0246-kube-api-access-fvfvf\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq\" (UID: \"4d51af15-6745-417c-b03a-061d477c0246\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq"
Jul 28 13:22:09.403997 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:09.403911    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d8a6d15-74e3-45ad-9651-a45f8291cfaa-proxy-tls\") pod \"isvc-canary-create-predictor-cb85cf55c-gmqwb\" (UID: \"6d8a6d15-74e3-45ad-9651-a45f8291cfaa\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-gmqwb"
Jul 28 13:22:09.403997 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:09.403954    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d8a6d15-74e3-45ad-9651-a45f8291cfaa-kserve-provision-location\") pod \"isvc-canary-create-predictor-cb85cf55c-gmqwb\" (UID: \"6d8a6d15-74e3-45ad-9651-a45f8291cfaa\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-gmqwb"
Jul 28 13:22:09.404086 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:09.403998    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d8a6d15-74e3-45ad-9651-a45f8291cfaa-cabundle-cert\") pod \"isvc-canary-create-predictor-cb85cf55c-gmqwb\" (UID: \"6d8a6d15-74e3-45ad-9651-a45f8291cfaa\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-gmqwb"
Jul 28 13:22:09.404086 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:09.404022    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d51af15-6745-417c-b03a-061d477c0246-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq\" (UID: \"4d51af15-6745-417c-b03a-061d477c0246\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq"
Jul 28 13:22:09.404086 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:09.404044    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d51af15-6745-417c-b03a-061d477c0246-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq\" (UID: \"4d51af15-6745-417c-b03a-061d477c0246\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq"
Jul 28 13:22:09.404086 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:09.404061    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d8a6d15-74e3-45ad-9651-a45f8291cfaa-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-cb85cf55c-gmqwb\" (UID: \"6d8a6d15-74e3-45ad-9651-a45f8291cfaa\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-gmqwb"
Jul 28 13:22:09.504936 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:09.504910    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d8a6d15-74e3-45ad-9651-a45f8291cfaa-cabundle-cert\") pod \"isvc-canary-create-predictor-cb85cf55c-gmqwb\" (UID: \"6d8a6d15-74e3-45ad-9651-a45f8291cfaa\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-gmqwb"
Jul 28 13:22:09.505047 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:09.504954    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d51af15-6745-417c-b03a-061d477c0246-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq\" (UID: \"4d51af15-6745-417c-b03a-061d477c0246\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq"
Jul 28 13:22:09.505047 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:09.504993    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d51af15-6745-417c-b03a-061d477c0246-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq\" (UID: \"4d51af15-6745-417c-b03a-061d477c0246\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq"
Jul 28 13:22:09.505047 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:09.505021    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d8a6d15-74e3-45ad-9651-a45f8291cfaa-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-cb85cf55c-gmqwb\" (UID: \"6d8a6d15-74e3-45ad-9651-a45f8291cfaa\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-gmqwb"
Jul 28 13:22:09.505220 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:09.505052    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d51af15-6745-417c-b03a-061d477c0246-proxy-tls\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq\" (UID: \"4d51af15-6745-417c-b03a-061d477c0246\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq"
Jul 28 13:22:09.505220 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:09.505081    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d51af15-6745-417c-b03a-061d477c0246-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq\" (UID: \"4d51af15-6745-417c-b03a-061d477c0246\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq"
Jul 28 13:22:09.505220 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:09.505128    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g28nv\" (UniqueName: \"kubernetes.io/projected/6d8a6d15-74e3-45ad-9651-a45f8291cfaa-kube-api-access-g28nv\") pod \"isvc-canary-create-predictor-cb85cf55c-gmqwb\" (UID: \"6d8a6d15-74e3-45ad-9651-a45f8291cfaa\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-gmqwb"
Jul 28 13:22:09.505220 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:09.505154    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvfvf\" (UniqueName: \"kubernetes.io/projected/4d51af15-6745-417c-b03a-061d477c0246-kube-api-access-fvfvf\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq\" (UID: \"4d51af15-6745-417c-b03a-061d477c0246\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq"
Jul 28 13:22:09.505220 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:22:09.505176    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-v2-predictor-serving-cert: secret "isvc-canary-create-v2-predictor-serving-cert" not found
Jul 28 13:22:09.505220 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:09.505184    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d8a6d15-74e3-45ad-9651-a45f8291cfaa-proxy-tls\") pod \"isvc-canary-create-predictor-cb85cf55c-gmqwb\" (UID: \"6d8a6d15-74e3-45ad-9651-a45f8291cfaa\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-gmqwb"
Jul 28 13:22:09.505532 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:09.505218    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d8a6d15-74e3-45ad-9651-a45f8291cfaa-kserve-provision-location\") pod \"isvc-canary-create-predictor-cb85cf55c-gmqwb\" (UID: \"6d8a6d15-74e3-45ad-9651-a45f8291cfaa\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-gmqwb"
Jul 28 13:22:09.505532 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:22:09.505240    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d51af15-6745-417c-b03a-061d477c0246-proxy-tls podName:4d51af15-6745-417c-b03a-061d477c0246 nodeName:}" failed. No retries permitted until 2026-07-28 13:22:10.005217842 +0000 UTC m=+1742.210604421 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4d51af15-6745-417c-b03a-061d477c0246-proxy-tls") pod "isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq" (UID: "4d51af15-6745-417c-b03a-061d477c0246") : secret "isvc-canary-create-v2-predictor-serving-cert" not found
Jul 28 13:22:09.505532 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:09.505423    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d51af15-6745-417c-b03a-061d477c0246-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq\" (UID: \"4d51af15-6745-417c-b03a-061d477c0246\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq"
Jul 28 13:22:09.505532 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:22:09.505526    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-predictor-serving-cert: secret "isvc-canary-create-predictor-serving-cert" not found
Jul 28 13:22:09.505738 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:09.505547    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d8a6d15-74e3-45ad-9651-a45f8291cfaa-kserve-provision-location\") pod \"isvc-canary-create-predictor-cb85cf55c-gmqwb\" (UID: \"6d8a6d15-74e3-45ad-9651-a45f8291cfaa\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-gmqwb"
Jul 28 13:22:09.505738 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:22:09.505574    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d8a6d15-74e3-45ad-9651-a45f8291cfaa-proxy-tls podName:6d8a6d15-74e3-45ad-9651-a45f8291cfaa nodeName:}" failed. No retries permitted until 2026-07-28 13:22:10.005557817 +0000 UTC m=+1742.210944408 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6d8a6d15-74e3-45ad-9651-a45f8291cfaa-proxy-tls") pod "isvc-canary-create-predictor-cb85cf55c-gmqwb" (UID: "6d8a6d15-74e3-45ad-9651-a45f8291cfaa") : secret "isvc-canary-create-predictor-serving-cert" not found
Jul 28 13:22:09.505738 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:09.505675    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d8a6d15-74e3-45ad-9651-a45f8291cfaa-cabundle-cert\") pod \"isvc-canary-create-predictor-cb85cf55c-gmqwb\" (UID: \"6d8a6d15-74e3-45ad-9651-a45f8291cfaa\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-gmqwb"
Jul 28 13:22:09.505847 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:09.505782    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d51af15-6745-417c-b03a-061d477c0246-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq\" (UID: \"4d51af15-6745-417c-b03a-061d477c0246\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq"
Jul 28 13:22:09.505847 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:09.505788    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d8a6d15-74e3-45ad-9651-a45f8291cfaa-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-cb85cf55c-gmqwb\" (UID: \"6d8a6d15-74e3-45ad-9651-a45f8291cfaa\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-gmqwb"
Jul 28 13:22:09.505914 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:09.505889    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d51af15-6745-417c-b03a-061d477c0246-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq\" (UID: \"4d51af15-6745-417c-b03a-061d477c0246\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq"
Jul 28 13:22:09.513523 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:09.513507    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g28nv\" (UniqueName: \"kubernetes.io/projected/6d8a6d15-74e3-45ad-9651-a45f8291cfaa-kube-api-access-g28nv\") pod \"isvc-canary-create-predictor-cb85cf55c-gmqwb\" (UID: \"6d8a6d15-74e3-45ad-9651-a45f8291cfaa\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-gmqwb"
Jul 28 13:22:09.513590 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:09.513524    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvfvf\" (UniqueName: \"kubernetes.io/projected/4d51af15-6745-417c-b03a-061d477c0246-kube-api-access-fvfvf\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq\" (UID: \"4d51af15-6745-417c-b03a-061d477c0246\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq"
Jul 28 13:22:10.009231 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:10.009202    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d51af15-6745-417c-b03a-061d477c0246-proxy-tls\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq\" (UID: \"4d51af15-6745-417c-b03a-061d477c0246\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq"
Jul 28 13:22:10.009377 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:10.009247    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d8a6d15-74e3-45ad-9651-a45f8291cfaa-proxy-tls\") pod \"isvc-canary-create-predictor-cb85cf55c-gmqwb\" (UID: \"6d8a6d15-74e3-45ad-9651-a45f8291cfaa\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-gmqwb"
Jul 28 13:22:10.011595 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:10.011574    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d51af15-6745-417c-b03a-061d477c0246-proxy-tls\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq\" (UID: \"4d51af15-6745-417c-b03a-061d477c0246\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq"
Jul 28 13:22:10.011767 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:10.011747    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d8a6d15-74e3-45ad-9651-a45f8291cfaa-proxy-tls\") pod \"isvc-canary-create-predictor-cb85cf55c-gmqwb\" (UID: \"6d8a6d15-74e3-45ad-9651-a45f8291cfaa\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-gmqwb"
Jul 28 13:22:10.275773 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:10.275705    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-gmqwb"
Jul 28 13:22:10.291847 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:10.291816    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq"
Jul 28 13:22:10.417442 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:10.417270    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-gmqwb"]
Jul 28 13:22:10.419666 ip-10-0-138-158 kubenswrapper[2574]: W0728 13:22:10.419642    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d8a6d15_74e3_45ad_9651_a45f8291cfaa.slice/crio-61802bf188fb055efbebfda6d63f8cc5919189ae221c98d813c786fa42980d84 WatchSource:0}: Error finding container 61802bf188fb055efbebfda6d63f8cc5919189ae221c98d813c786fa42980d84: Status 404 returned error can't find the container with id 61802bf188fb055efbebfda6d63f8cc5919189ae221c98d813c786fa42980d84
Jul 28 13:22:10.436054 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:10.436033    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq"]
Jul 28 13:22:10.437537 ip-10-0-138-158 kubenswrapper[2574]: W0728 13:22:10.437513    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d51af15_6745_417c_b03a_061d477c0246.slice/crio-d0d4b6d6f31b6d6ae84ad857e672a897064c11e89272099a8f2eb47192fcd0bf WatchSource:0}: Error finding container d0d4b6d6f31b6d6ae84ad857e672a897064c11e89272099a8f2eb47192fcd0bf: Status 404 returned error can't find the container with id d0d4b6d6f31b6d6ae84ad857e672a897064c11e89272099a8f2eb47192fcd0bf
Jul 28 13:22:11.031940 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:11.031902    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-gmqwb" event={"ID":"6d8a6d15-74e3-45ad-9651-a45f8291cfaa","Type":"ContainerStarted","Data":"18aa19cc11614b6ab2b5773e505c773aa72626e2881a1a6f6dc70c154ff3e613"}
Jul 28 13:22:11.032114 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:11.031970    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-gmqwb" event={"ID":"6d8a6d15-74e3-45ad-9651-a45f8291cfaa","Type":"ContainerStarted","Data":"61802bf188fb055efbebfda6d63f8cc5919189ae221c98d813c786fa42980d84"}
Jul 28 13:22:11.033359 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:11.033334    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq" event={"ID":"4d51af15-6745-417c-b03a-061d477c0246","Type":"ContainerStarted","Data":"3ebb04a927edbffde0c47877a9dfcb8007362df4b053d6a066e800927351b238"}
Jul 28 13:22:11.033461 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:11.033381    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq" event={"ID":"4d51af15-6745-417c-b03a-061d477c0246","Type":"ContainerStarted","Data":"d0d4b6d6f31b6d6ae84ad857e672a897064c11e89272099a8f2eb47192fcd0bf"}
Jul 28 13:22:15.052733 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:15.052699    2574 generic.go:356] "Generic (PLEG): container finished" podID="4d51af15-6745-417c-b03a-061d477c0246" containerID="3ebb04a927edbffde0c47877a9dfcb8007362df4b053d6a066e800927351b238" exitCode=0
Jul 28 13:22:15.053191 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:15.052779    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq" event={"ID":"4d51af15-6745-417c-b03a-061d477c0246","Type":"ContainerDied","Data":"3ebb04a927edbffde0c47877a9dfcb8007362df4b053d6a066e800927351b238"}
Jul 28 13:22:15.054369 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:15.054350    2574 generic.go:356] "Generic (PLEG): container finished" podID="6d8a6d15-74e3-45ad-9651-a45f8291cfaa" containerID="18aa19cc11614b6ab2b5773e505c773aa72626e2881a1a6f6dc70c154ff3e613" exitCode=0
Jul 28 13:22:15.054462 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:15.054419    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-gmqwb" event={"ID":"6d8a6d15-74e3-45ad-9651-a45f8291cfaa","Type":"ContainerDied","Data":"18aa19cc11614b6ab2b5773e505c773aa72626e2881a1a6f6dc70c154ff3e613"}
Jul 28 13:22:16.061618 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:16.061584    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-gmqwb" event={"ID":"6d8a6d15-74e3-45ad-9651-a45f8291cfaa","Type":"ContainerStarted","Data":"f27d4554d02be820a15de8bc4d274a09ed8198952028760e8bf0b6dc8c4954b4"}
Jul 28 13:22:16.062053 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:16.061658    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-gmqwb" event={"ID":"6d8a6d15-74e3-45ad-9651-a45f8291cfaa","Type":"ContainerStarted","Data":"f6aa1acceb1c58cecd870d30852583b6b1f1bd433172b79ec099a398a27821f3"}
Jul 28 13:22:16.062053 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:16.061970    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-gmqwb"
Jul 28 13:22:16.062187 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:16.062132    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-gmqwb"
Jul 28 13:22:16.063576 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:16.063545    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-gmqwb" podUID="6d8a6d15-74e3-45ad-9651-a45f8291cfaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 28 13:22:16.063876 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:16.063859    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq" event={"ID":"4d51af15-6745-417c-b03a-061d477c0246","Type":"ContainerStarted","Data":"885f8363bb17ba2e4c230223954b10f5cd9814c3f1b2abacb927f4799f4ef4e4"}
Jul 28 13:22:16.063927 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:16.063902    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq" event={"ID":"4d51af15-6745-417c-b03a-061d477c0246","Type":"ContainerStarted","Data":"1fb8ea540d3d4c6b144c23abf44014744cfa19b3bc42cc3f390380fb6d6060bf"}
Jul 28 13:22:16.064129 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:16.064114    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq"
Jul 28 13:22:16.084243 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:16.084205    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-gmqwb" podStartSLOduration=3.217741051 podStartE2EDuration="7.084192608s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.866451557s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:22:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:22:16.083087955 +0000 UTC m=+1748.288474576" watchObservedRunningTime="2026-07-28 13:22:16.084192608 +0000 UTC m=+1748.289579208"
Jul 28 13:22:16.101886 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:16.101849    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq" podStartSLOduration=3.33041651 podStartE2EDuration="7.101837865s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.771421355s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:22:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:22:16.100536423 +0000 UTC m=+1748.305923026" watchObservedRunningTime="2026-07-28 13:22:16.101837865 +0000 UTC m=+1748.307224467"
Jul 28 13:22:17.069730 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:17.069696    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq"
Jul 28 13:22:17.069730 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:17.069722    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-gmqwb" podUID="6d8a6d15-74e3-45ad-9651-a45f8291cfaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 28 13:22:17.070609 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:17.070586    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq" podUID="4d51af15-6745-417c-b03a-061d477c0246" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 28 13:22:18.074754 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:18.074723    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq" podUID="4d51af15-6745-417c-b03a-061d477c0246" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 28 13:22:22.073860 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:22.073824    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-gmqwb"
Jul 28 13:22:22.074425 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:22.074398    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-gmqwb" podUID="6d8a6d15-74e3-45ad-9651-a45f8291cfaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 28 13:22:23.080491 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:23.080464    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq"
Jul 28 13:22:23.080999 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:23.080972    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq" podUID="4d51af15-6745-417c-b03a-061d477c0246" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 28 13:22:32.075295 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:32.075255    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-gmqwb" podUID="6d8a6d15-74e3-45ad-9651-a45f8291cfaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 28 13:22:33.081308 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:33.081272    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq" podUID="4d51af15-6745-417c-b03a-061d477c0246" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 28 13:22:42.074754 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:42.074717    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-gmqwb" podUID="6d8a6d15-74e3-45ad-9651-a45f8291cfaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 28 13:22:43.081176 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:43.081140    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq" podUID="4d51af15-6745-417c-b03a-061d477c0246" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 28 13:22:52.075225 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:52.075181    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-gmqwb" podUID="6d8a6d15-74e3-45ad-9651-a45f8291cfaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 28 13:22:53.080915 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:22:53.080878    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq" podUID="4d51af15-6745-417c-b03a-061d477c0246" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 28 13:23:02.074738 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:02.074640    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-gmqwb" podUID="6d8a6d15-74e3-45ad-9651-a45f8291cfaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 28 13:23:03.081083 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:03.081045    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq" podUID="4d51af15-6745-417c-b03a-061d477c0246" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 28 13:23:12.074622 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:12.074580    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-gmqwb" podUID="6d8a6d15-74e3-45ad-9651-a45f8291cfaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 28 13:23:13.081062 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:13.081025    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq" podUID="4d51af15-6745-417c-b03a-061d477c0246" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 28 13:23:22.075352 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:22.075308    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-gmqwb" podUID="6d8a6d15-74e3-45ad-9651-a45f8291cfaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 28 13:23:23.081916 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:23.081877    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq" podUID="4d51af15-6745-417c-b03a-061d477c0246" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 28 13:23:32.075870 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:32.075837    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-gmqwb"
Jul 28 13:23:33.082349 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:33.082322    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq"
Jul 28 13:23:39.436421 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:39.436385    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-gmqwb"]
Jul 28 13:23:39.436855 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:39.436697    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-gmqwb" podUID="6d8a6d15-74e3-45ad-9651-a45f8291cfaa" containerName="kserve-container" containerID="cri-o://f6aa1acceb1c58cecd870d30852583b6b1f1bd433172b79ec099a398a27821f3" gracePeriod=30
Jul 28 13:23:39.436855 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:39.436766    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-gmqwb" podUID="6d8a6d15-74e3-45ad-9651-a45f8291cfaa" containerName="kube-rbac-proxy" containerID="cri-o://f27d4554d02be820a15de8bc4d274a09ed8198952028760e8bf0b6dc8c4954b4" gracePeriod=30
Jul 28 13:23:39.441156 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:39.441132    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq"]
Jul 28 13:23:39.441399 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:39.441375    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq" podUID="4d51af15-6745-417c-b03a-061d477c0246" containerName="kserve-container" containerID="cri-o://1fb8ea540d3d4c6b144c23abf44014744cfa19b3bc42cc3f390380fb6d6060bf" gracePeriod=30
Jul 28 13:23:39.441623 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:39.441401    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq" podUID="4d51af15-6745-417c-b03a-061d477c0246" containerName="kube-rbac-proxy" containerID="cri-o://885f8363bb17ba2e4c230223954b10f5cd9814c3f1b2abacb927f4799f4ef4e4" gracePeriod=30
Jul 28 13:23:40.447241 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:40.447209    2574 generic.go:356] "Generic (PLEG): container finished" podID="6d8a6d15-74e3-45ad-9651-a45f8291cfaa" containerID="f27d4554d02be820a15de8bc4d274a09ed8198952028760e8bf0b6dc8c4954b4" exitCode=2
Jul 28 13:23:40.447667 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:40.447279    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-gmqwb" event={"ID":"6d8a6d15-74e3-45ad-9651-a45f8291cfaa","Type":"ContainerDied","Data":"f27d4554d02be820a15de8bc4d274a09ed8198952028760e8bf0b6dc8c4954b4"}
Jul 28 13:23:40.449462 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:40.449432    2574 generic.go:356] "Generic (PLEG): container finished" podID="4d51af15-6745-417c-b03a-061d477c0246" containerID="885f8363bb17ba2e4c230223954b10f5cd9814c3f1b2abacb927f4799f4ef4e4" exitCode=2
Jul 28 13:23:40.449583 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:40.449486    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq" event={"ID":"4d51af15-6745-417c-b03a-061d477c0246","Type":"ContainerDied","Data":"885f8363bb17ba2e4c230223954b10f5cd9814c3f1b2abacb927f4799f4ef4e4"}
Jul 28 13:23:42.070364 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:42.070323    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-gmqwb" podUID="6d8a6d15-74e3-45ad-9651-a45f8291cfaa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.40:8643/healthz\": dial tcp 10.132.0.40:8643: connect: connection refused"
Jul 28 13:23:42.074587 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:42.074562    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-gmqwb" podUID="6d8a6d15-74e3-45ad-9651-a45f8291cfaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 28 13:23:43.075710 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:43.075641    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq" podUID="4d51af15-6745-417c-b03a-061d477c0246" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.41:8643/healthz\": dial tcp 10.132.0.41:8643: connect: connection refused"
Jul 28 13:23:43.081699 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:43.081662    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq" podUID="4d51af15-6745-417c-b03a-061d477c0246" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 28 13:23:43.812796 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:43.812771    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq"
Jul 28 13:23:43.815463 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:43.815443    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-gmqwb"
Jul 28 13:23:43.921936 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:43.921913    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d51af15-6745-417c-b03a-061d477c0246-proxy-tls\") pod \"4d51af15-6745-417c-b03a-061d477c0246\" (UID: \"4d51af15-6745-417c-b03a-061d477c0246\") "
Jul 28 13:23:43.921936 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:43.921940    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvfvf\" (UniqueName: \"kubernetes.io/projected/4d51af15-6745-417c-b03a-061d477c0246-kube-api-access-fvfvf\") pod \"4d51af15-6745-417c-b03a-061d477c0246\" (UID: \"4d51af15-6745-417c-b03a-061d477c0246\") "
Jul 28 13:23:43.922106 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:43.921959    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d8a6d15-74e3-45ad-9651-a45f8291cfaa-cabundle-cert\") pod \"6d8a6d15-74e3-45ad-9651-a45f8291cfaa\" (UID: \"6d8a6d15-74e3-45ad-9651-a45f8291cfaa\") "
Jul 28 13:23:43.922106 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:43.921990    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g28nv\" (UniqueName: \"kubernetes.io/projected/6d8a6d15-74e3-45ad-9651-a45f8291cfaa-kube-api-access-g28nv\") pod \"6d8a6d15-74e3-45ad-9651-a45f8291cfaa\" (UID: \"6d8a6d15-74e3-45ad-9651-a45f8291cfaa\") "
Jul 28 13:23:43.922106 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:43.922022    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d8a6d15-74e3-45ad-9651-a45f8291cfaa-proxy-tls\") pod \"6d8a6d15-74e3-45ad-9651-a45f8291cfaa\" (UID: \"6d8a6d15-74e3-45ad-9651-a45f8291cfaa\") "
Jul 28 13:23:43.922106 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:43.922043    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d8a6d15-74e3-45ad-9651-a45f8291cfaa-kserve-provision-location\") pod \"6d8a6d15-74e3-45ad-9651-a45f8291cfaa\" (UID: \"6d8a6d15-74e3-45ad-9651-a45f8291cfaa\") "
Jul 28 13:23:43.922106 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:43.922064    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d51af15-6745-417c-b03a-061d477c0246-cabundle-cert\") pod \"4d51af15-6745-417c-b03a-061d477c0246\" (UID: \"4d51af15-6745-417c-b03a-061d477c0246\") "
Jul 28 13:23:43.922106 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:43.922094    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d51af15-6745-417c-b03a-061d477c0246-kube-rbac-proxy-sar-config\") pod \"4d51af15-6745-417c-b03a-061d477c0246\" (UID: \"4d51af15-6745-417c-b03a-061d477c0246\") "
Jul 28 13:23:43.922428 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:43.922132    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d51af15-6745-417c-b03a-061d477c0246-kserve-provision-location\") pod \"4d51af15-6745-417c-b03a-061d477c0246\" (UID: \"4d51af15-6745-417c-b03a-061d477c0246\") "
Jul 28 13:23:43.922428 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:43.922174    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d8a6d15-74e3-45ad-9651-a45f8291cfaa-kube-rbac-proxy-sar-config\") pod \"6d8a6d15-74e3-45ad-9651-a45f8291cfaa\" (UID: \"6d8a6d15-74e3-45ad-9651-a45f8291cfaa\") "
Jul 28 13:23:43.922428 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:43.922390    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d8a6d15-74e3-45ad-9651-a45f8291cfaa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6d8a6d15-74e3-45ad-9651-a45f8291cfaa" (UID: "6d8a6d15-74e3-45ad-9651-a45f8291cfaa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:23:43.922428 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:43.922418    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d8a6d15-74e3-45ad-9651-a45f8291cfaa-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6d8a6d15-74e3-45ad-9651-a45f8291cfaa" (UID: "6d8a6d15-74e3-45ad-9651-a45f8291cfaa"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:23:43.922641 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:43.922433    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d51af15-6745-417c-b03a-061d477c0246-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4d51af15-6745-417c-b03a-061d477c0246" (UID: "4d51af15-6745-417c-b03a-061d477c0246"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:23:43.922798 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:43.922670    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d8a6d15-74e3-45ad-9651-a45f8291cfaa-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6d8a6d15-74e3-45ad-9651-a45f8291cfaa" (UID: "6d8a6d15-74e3-45ad-9651-a45f8291cfaa"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:23:43.922912 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:43.922780    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d51af15-6745-417c-b03a-061d477c0246-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4d51af15-6745-417c-b03a-061d477c0246" (UID: "4d51af15-6745-417c-b03a-061d477c0246"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:23:43.922912 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:43.922813    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d51af15-6745-417c-b03a-061d477c0246-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4d51af15-6745-417c-b03a-061d477c0246" (UID: "4d51af15-6745-417c-b03a-061d477c0246"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:23:43.924280 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:43.924261    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d51af15-6745-417c-b03a-061d477c0246-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4d51af15-6745-417c-b03a-061d477c0246" (UID: "4d51af15-6745-417c-b03a-061d477c0246"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:23:43.924393 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:43.924366    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d8a6d15-74e3-45ad-9651-a45f8291cfaa-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6d8a6d15-74e3-45ad-9651-a45f8291cfaa" (UID: "6d8a6d15-74e3-45ad-9651-a45f8291cfaa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:23:43.924539 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:43.924522    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d51af15-6745-417c-b03a-061d477c0246-kube-api-access-fvfvf" (OuterVolumeSpecName: "kube-api-access-fvfvf") pod "4d51af15-6745-417c-b03a-061d477c0246" (UID: "4d51af15-6745-417c-b03a-061d477c0246"). InnerVolumeSpecName "kube-api-access-fvfvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:23:43.924539 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:43.924527    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d8a6d15-74e3-45ad-9651-a45f8291cfaa-kube-api-access-g28nv" (OuterVolumeSpecName: "kube-api-access-g28nv") pod "6d8a6d15-74e3-45ad-9651-a45f8291cfaa" (UID: "6d8a6d15-74e3-45ad-9651-a45f8291cfaa"). InnerVolumeSpecName "kube-api-access-g28nv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:23:44.022926 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.022904    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d51af15-6745-417c-b03a-061d477c0246-kserve-provision-location\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:23:44.022926 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.022923    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d8a6d15-74e3-45ad-9651-a45f8291cfaa-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:23:44.023046 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.022932    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d51af15-6745-417c-b03a-061d477c0246-proxy-tls\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:23:44.023046 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.022940    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvfvf\" (UniqueName: \"kubernetes.io/projected/4d51af15-6745-417c-b03a-061d477c0246-kube-api-access-fvfvf\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:23:44.023046 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.022948    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d8a6d15-74e3-45ad-9651-a45f8291cfaa-cabundle-cert\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:23:44.023046 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.022957    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g28nv\" (UniqueName: \"kubernetes.io/projected/6d8a6d15-74e3-45ad-9651-a45f8291cfaa-kube-api-access-g28nv\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:23:44.023046 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.022964    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d8a6d15-74e3-45ad-9651-a45f8291cfaa-proxy-tls\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:23:44.023046 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.022972    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d8a6d15-74e3-45ad-9651-a45f8291cfaa-kserve-provision-location\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:23:44.023046 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.022980    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d51af15-6745-417c-b03a-061d477c0246-cabundle-cert\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:23:44.023046 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.022988    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d51af15-6745-417c-b03a-061d477c0246-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:23:44.469970 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.469939    2574 generic.go:356] "Generic (PLEG): container finished" podID="6d8a6d15-74e3-45ad-9651-a45f8291cfaa" containerID="f6aa1acceb1c58cecd870d30852583b6b1f1bd433172b79ec099a398a27821f3" exitCode=0
Jul 28 13:23:44.470418 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.469993    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-gmqwb"
Jul 28 13:23:44.470418 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.470022    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-gmqwb" event={"ID":"6d8a6d15-74e3-45ad-9651-a45f8291cfaa","Type":"ContainerDied","Data":"f6aa1acceb1c58cecd870d30852583b6b1f1bd433172b79ec099a398a27821f3"}
Jul 28 13:23:44.470418 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.470080    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-gmqwb" event={"ID":"6d8a6d15-74e3-45ad-9651-a45f8291cfaa","Type":"ContainerDied","Data":"61802bf188fb055efbebfda6d63f8cc5919189ae221c98d813c786fa42980d84"}
Jul 28 13:23:44.470418 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.470118    2574 scope.go:117] "RemoveContainer" containerID="f27d4554d02be820a15de8bc4d274a09ed8198952028760e8bf0b6dc8c4954b4"
Jul 28 13:23:44.472362 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.472167    2574 generic.go:356] "Generic (PLEG): container finished" podID="4d51af15-6745-417c-b03a-061d477c0246" containerID="1fb8ea540d3d4c6b144c23abf44014744cfa19b3bc42cc3f390380fb6d6060bf" exitCode=0
Jul 28 13:23:44.472362 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.472254    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq" event={"ID":"4d51af15-6745-417c-b03a-061d477c0246","Type":"ContainerDied","Data":"1fb8ea540d3d4c6b144c23abf44014744cfa19b3bc42cc3f390380fb6d6060bf"}
Jul 28 13:23:44.472362 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.472265    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq"
Jul 28 13:23:44.472362 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.472305    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq" event={"ID":"4d51af15-6745-417c-b03a-061d477c0246","Type":"ContainerDied","Data":"d0d4b6d6f31b6d6ae84ad857e672a897064c11e89272099a8f2eb47192fcd0bf"}
Jul 28 13:23:44.478706 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.478672    2574 scope.go:117] "RemoveContainer" containerID="f6aa1acceb1c58cecd870d30852583b6b1f1bd433172b79ec099a398a27821f3"
Jul 28 13:23:44.485876 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.485846    2574 scope.go:117] "RemoveContainer" containerID="18aa19cc11614b6ab2b5773e505c773aa72626e2881a1a6f6dc70c154ff3e613"
Jul 28 13:23:44.489580 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.489552    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-gmqwb"]
Jul 28 13:23:44.493719 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.493700    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-gmqwb"]
Jul 28 13:23:44.494426 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.494408    2574 scope.go:117] "RemoveContainer" containerID="f27d4554d02be820a15de8bc4d274a09ed8198952028760e8bf0b6dc8c4954b4"
Jul 28 13:23:44.494675 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:23:44.494652    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f27d4554d02be820a15de8bc4d274a09ed8198952028760e8bf0b6dc8c4954b4\": container with ID starting with f27d4554d02be820a15de8bc4d274a09ed8198952028760e8bf0b6dc8c4954b4 not found: ID does not exist" containerID="f27d4554d02be820a15de8bc4d274a09ed8198952028760e8bf0b6dc8c4954b4"
Jul 28 13:23:44.494831 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.494704    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f27d4554d02be820a15de8bc4d274a09ed8198952028760e8bf0b6dc8c4954b4"} err="failed to get container status \"f27d4554d02be820a15de8bc4d274a09ed8198952028760e8bf0b6dc8c4954b4\": rpc error: code = NotFound desc = could not find container \"f27d4554d02be820a15de8bc4d274a09ed8198952028760e8bf0b6dc8c4954b4\": container with ID starting with f27d4554d02be820a15de8bc4d274a09ed8198952028760e8bf0b6dc8c4954b4 not found: ID does not exist"
Jul 28 13:23:44.494831 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.494728    2574 scope.go:117] "RemoveContainer" containerID="f6aa1acceb1c58cecd870d30852583b6b1f1bd433172b79ec099a398a27821f3"
Jul 28 13:23:44.494984 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:23:44.494967    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6aa1acceb1c58cecd870d30852583b6b1f1bd433172b79ec099a398a27821f3\": container with ID starting with f6aa1acceb1c58cecd870d30852583b6b1f1bd433172b79ec099a398a27821f3 not found: ID does not exist" containerID="f6aa1acceb1c58cecd870d30852583b6b1f1bd433172b79ec099a398a27821f3"
Jul 28 13:23:44.495050 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.494988    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6aa1acceb1c58cecd870d30852583b6b1f1bd433172b79ec099a398a27821f3"} err="failed to get container status \"f6aa1acceb1c58cecd870d30852583b6b1f1bd433172b79ec099a398a27821f3\": rpc error: code = NotFound desc = could not find container \"f6aa1acceb1c58cecd870d30852583b6b1f1bd433172b79ec099a398a27821f3\": container with ID starting with f6aa1acceb1c58cecd870d30852583b6b1f1bd433172b79ec099a398a27821f3 not found: ID does not exist"
Jul 28 13:23:44.495050 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.495001    2574 scope.go:117] "RemoveContainer" containerID="18aa19cc11614b6ab2b5773e505c773aa72626e2881a1a6f6dc70c154ff3e613"
Jul 28 13:23:44.495233 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:23:44.495216    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18aa19cc11614b6ab2b5773e505c773aa72626e2881a1a6f6dc70c154ff3e613\": container with ID starting with 18aa19cc11614b6ab2b5773e505c773aa72626e2881a1a6f6dc70c154ff3e613 not found: ID does not exist" containerID="18aa19cc11614b6ab2b5773e505c773aa72626e2881a1a6f6dc70c154ff3e613"
Jul 28 13:23:44.495293 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.495240    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18aa19cc11614b6ab2b5773e505c773aa72626e2881a1a6f6dc70c154ff3e613"} err="failed to get container status \"18aa19cc11614b6ab2b5773e505c773aa72626e2881a1a6f6dc70c154ff3e613\": rpc error: code = NotFound desc = could not find container \"18aa19cc11614b6ab2b5773e505c773aa72626e2881a1a6f6dc70c154ff3e613\": container with ID starting with 18aa19cc11614b6ab2b5773e505c773aa72626e2881a1a6f6dc70c154ff3e613 not found: ID does not exist"
Jul 28 13:23:44.495293 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.495258    2574 scope.go:117] "RemoveContainer" containerID="885f8363bb17ba2e4c230223954b10f5cd9814c3f1b2abacb927f4799f4ef4e4"
Jul 28 13:23:44.502198 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.502180    2574 scope.go:117] "RemoveContainer" containerID="1fb8ea540d3d4c6b144c23abf44014744cfa19b3bc42cc3f390380fb6d6060bf"
Jul 28 13:23:44.506093 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.506073    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq"]
Jul 28 13:23:44.510827 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.510806    2574 scope.go:117] "RemoveContainer" containerID="3ebb04a927edbffde0c47877a9dfcb8007362df4b053d6a066e800927351b238"
Jul 28 13:23:44.513806 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.513786    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-l6vvq"]
Jul 28 13:23:44.521096 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.521076    2574 scope.go:117] "RemoveContainer" containerID="885f8363bb17ba2e4c230223954b10f5cd9814c3f1b2abacb927f4799f4ef4e4"
Jul 28 13:23:44.521360 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:23:44.521340    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"885f8363bb17ba2e4c230223954b10f5cd9814c3f1b2abacb927f4799f4ef4e4\": container with ID starting with 885f8363bb17ba2e4c230223954b10f5cd9814c3f1b2abacb927f4799f4ef4e4 not found: ID does not exist" containerID="885f8363bb17ba2e4c230223954b10f5cd9814c3f1b2abacb927f4799f4ef4e4"
Jul 28 13:23:44.521425 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.521368    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"885f8363bb17ba2e4c230223954b10f5cd9814c3f1b2abacb927f4799f4ef4e4"} err="failed to get container status \"885f8363bb17ba2e4c230223954b10f5cd9814c3f1b2abacb927f4799f4ef4e4\": rpc error: code = NotFound desc = could not find container \"885f8363bb17ba2e4c230223954b10f5cd9814c3f1b2abacb927f4799f4ef4e4\": container with ID starting with 885f8363bb17ba2e4c230223954b10f5cd9814c3f1b2abacb927f4799f4ef4e4 not found: ID does not exist"
Jul 28 13:23:44.521425 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.521384    2574 scope.go:117] "RemoveContainer" containerID="1fb8ea540d3d4c6b144c23abf44014744cfa19b3bc42cc3f390380fb6d6060bf"
Jul 28 13:23:44.521645 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:23:44.521631    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fb8ea540d3d4c6b144c23abf44014744cfa19b3bc42cc3f390380fb6d6060bf\": container with ID starting with 1fb8ea540d3d4c6b144c23abf44014744cfa19b3bc42cc3f390380fb6d6060bf not found: ID does not exist" containerID="1fb8ea540d3d4c6b144c23abf44014744cfa19b3bc42cc3f390380fb6d6060bf"
Jul 28 13:23:44.521704 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.521649    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fb8ea540d3d4c6b144c23abf44014744cfa19b3bc42cc3f390380fb6d6060bf"} err="failed to get container status \"1fb8ea540d3d4c6b144c23abf44014744cfa19b3bc42cc3f390380fb6d6060bf\": rpc error: code = NotFound desc = could not find container \"1fb8ea540d3d4c6b144c23abf44014744cfa19b3bc42cc3f390380fb6d6060bf\": container with ID starting with 1fb8ea540d3d4c6b144c23abf44014744cfa19b3bc42cc3f390380fb6d6060bf not found: ID does not exist"
Jul 28 13:23:44.521704 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.521662    2574 scope.go:117] "RemoveContainer" containerID="3ebb04a927edbffde0c47877a9dfcb8007362df4b053d6a066e800927351b238"
Jul 28 13:23:44.521928 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:23:44.521911    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ebb04a927edbffde0c47877a9dfcb8007362df4b053d6a066e800927351b238\": container with ID starting with 3ebb04a927edbffde0c47877a9dfcb8007362df4b053d6a066e800927351b238 not found: ID does not exist" containerID="3ebb04a927edbffde0c47877a9dfcb8007362df4b053d6a066e800927351b238"
Jul 28 13:23:44.521991 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.521937    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ebb04a927edbffde0c47877a9dfcb8007362df4b053d6a066e800927351b238"} err="failed to get container status \"3ebb04a927edbffde0c47877a9dfcb8007362df4b053d6a066e800927351b238\": rpc error: code = NotFound desc = could not find container \"3ebb04a927edbffde0c47877a9dfcb8007362df4b053d6a066e800927351b238\": container with ID starting with 3ebb04a927edbffde0c47877a9dfcb8007362df4b053d6a066e800927351b238 not found: ID does not exist"
Jul 28 13:23:44.613491 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.613428    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7qx4s"]
Jul 28 13:23:44.613784 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.613771    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d51af15-6745-417c-b03a-061d477c0246" containerName="storage-initializer"
Jul 28 13:23:44.613836 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.613787    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d51af15-6745-417c-b03a-061d477c0246" containerName="storage-initializer"
Jul 28 13:23:44.613836 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.613798    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d8a6d15-74e3-45ad-9651-a45f8291cfaa" containerName="kube-rbac-proxy"
Jul 28 13:23:44.613836 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.613804    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d8a6d15-74e3-45ad-9651-a45f8291cfaa" containerName="kube-rbac-proxy"
Jul 28 13:23:44.613836 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.613812    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d8a6d15-74e3-45ad-9651-a45f8291cfaa" containerName="kserve-container"
Jul 28 13:23:44.613836 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.613818    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d8a6d15-74e3-45ad-9651-a45f8291cfaa" containerName="kserve-container"
Jul 28 13:23:44.613836 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.613824    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d8a6d15-74e3-45ad-9651-a45f8291cfaa" containerName="storage-initializer"
Jul 28 13:23:44.613836 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.613830    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d8a6d15-74e3-45ad-9651-a45f8291cfaa" containerName="storage-initializer"
Jul 28 13:23:44.614045 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.613842    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d51af15-6745-417c-b03a-061d477c0246" containerName="kserve-container"
Jul 28 13:23:44.614045 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.613848    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d51af15-6745-417c-b03a-061d477c0246" containerName="kserve-container"
Jul 28 13:23:44.614045 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.613856    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d51af15-6745-417c-b03a-061d477c0246" containerName="kube-rbac-proxy"
Jul 28 13:23:44.614045 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.613862    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d51af15-6745-417c-b03a-061d477c0246" containerName="kube-rbac-proxy"
Jul 28 13:23:44.614045 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.613919    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d8a6d15-74e3-45ad-9651-a45f8291cfaa" containerName="kube-rbac-proxy"
Jul 28 13:23:44.614045 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.613931    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d51af15-6745-417c-b03a-061d477c0246" containerName="kube-rbac-proxy"
Jul 28 13:23:44.614045 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.613944    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d51af15-6745-417c-b03a-061d477c0246" containerName="kserve-container"
Jul 28 13:23:44.614045 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.613956    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d8a6d15-74e3-45ad-9651-a45f8291cfaa" containerName="kserve-container"
Jul 28 13:23:44.618773 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.618755    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7qx4s"
Jul 28 13:23:44.620183 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.620164    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5j6jj"]
Jul 28 13:23:44.621375 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.621248    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Jul 28 13:23:44.621375 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.621298    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Jul 28 13:23:44.621375 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.621369    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:23:44.621564 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.621407    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 13:23:44.621633 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.621616    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rxfwd\""
Jul 28 13:23:44.621706 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.621669    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:23:44.621814 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.621800    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:23:44.623502 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.623484    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5j6jj"
Jul 28 13:23:44.625726 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.625705    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Jul 28 13:23:44.627169 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.627151    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7qx4s"]
Jul 28 13:23:44.633890 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.633866    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5j6jj"]
Jul 28 13:23:44.727078 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.727059    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/67e719ff-8743-4d50-bbb5-f918dbac6adb-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-86766c4958-5j6jj\" (UID: \"67e719ff-8743-4d50-bbb5-f918dbac6adb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5j6jj"
Jul 28 13:23:44.727185 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.727091    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r72rz\" (UniqueName: \"kubernetes.io/projected/67e719ff-8743-4d50-bbb5-f918dbac6adb-kube-api-access-r72rz\") pod \"isvc-canary-promote-v2-predictor-86766c4958-5j6jj\" (UID: \"67e719ff-8743-4d50-bbb5-f918dbac6adb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5j6jj"
Jul 28 13:23:44.727185 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.727110    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b010cf8-fa1f-47a2-a85b-3391e12d0e55-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-d9759ffd5-7qx4s\" (UID: \"5b010cf8-fa1f-47a2-a85b-3391e12d0e55\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7qx4s"
Jul 28 13:23:44.727185 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.727132    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67e719ff-8743-4d50-bbb5-f918dbac6adb-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-86766c4958-5j6jj\" (UID: \"67e719ff-8743-4d50-bbb5-f918dbac6adb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5j6jj"
Jul 28 13:23:44.727294 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.727192    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67e719ff-8743-4d50-bbb5-f918dbac6adb-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-86766c4958-5j6jj\" (UID: \"67e719ff-8743-4d50-bbb5-f918dbac6adb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5j6jj"
Jul 28 13:23:44.727294 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.727241    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b010cf8-fa1f-47a2-a85b-3391e12d0e55-cabundle-cert\") pod \"isvc-canary-promote-predictor-d9759ffd5-7qx4s\" (UID: \"5b010cf8-fa1f-47a2-a85b-3391e12d0e55\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7qx4s"
Jul 28 13:23:44.727294 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.727262    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87j6w\" (UniqueName: \"kubernetes.io/projected/5b010cf8-fa1f-47a2-a85b-3391e12d0e55-kube-api-access-87j6w\") pod \"isvc-canary-promote-predictor-d9759ffd5-7qx4s\" (UID: \"5b010cf8-fa1f-47a2-a85b-3391e12d0e55\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7qx4s"
Jul 28 13:23:44.727294 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.727278    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b010cf8-fa1f-47a2-a85b-3391e12d0e55-proxy-tls\") pod \"isvc-canary-promote-predictor-d9759ffd5-7qx4s\" (UID: \"5b010cf8-fa1f-47a2-a85b-3391e12d0e55\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7qx4s"
Jul 28 13:23:44.727423 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.727296    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b010cf8-fa1f-47a2-a85b-3391e12d0e55-kserve-provision-location\") pod \"isvc-canary-promote-predictor-d9759ffd5-7qx4s\" (UID: \"5b010cf8-fa1f-47a2-a85b-3391e12d0e55\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7qx4s"
Jul 28 13:23:44.727423 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.727353    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67e719ff-8743-4d50-bbb5-f918dbac6adb-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-86766c4958-5j6jj\" (UID: \"67e719ff-8743-4d50-bbb5-f918dbac6adb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5j6jj"
Jul 28 13:23:44.827788 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.827767    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b010cf8-fa1f-47a2-a85b-3391e12d0e55-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-d9759ffd5-7qx4s\" (UID: \"5b010cf8-fa1f-47a2-a85b-3391e12d0e55\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7qx4s"
Jul 28 13:23:44.827882 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.827792    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67e719ff-8743-4d50-bbb5-f918dbac6adb-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-86766c4958-5j6jj\" (UID: \"67e719ff-8743-4d50-bbb5-f918dbac6adb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5j6jj"
Jul 28 13:23:44.827882 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.827814    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67e719ff-8743-4d50-bbb5-f918dbac6adb-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-86766c4958-5j6jj\" (UID: \"67e719ff-8743-4d50-bbb5-f918dbac6adb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5j6jj"
Jul 28 13:23:44.827882 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.827851    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b010cf8-fa1f-47a2-a85b-3391e12d0e55-cabundle-cert\") pod \"isvc-canary-promote-predictor-d9759ffd5-7qx4s\" (UID: \"5b010cf8-fa1f-47a2-a85b-3391e12d0e55\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7qx4s"
Jul 28 13:23:44.828010 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.827881    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87j6w\" (UniqueName: \"kubernetes.io/projected/5b010cf8-fa1f-47a2-a85b-3391e12d0e55-kube-api-access-87j6w\") pod \"isvc-canary-promote-predictor-d9759ffd5-7qx4s\" (UID: \"5b010cf8-fa1f-47a2-a85b-3391e12d0e55\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7qx4s"
Jul 28 13:23:44.828010 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.827912    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b010cf8-fa1f-47a2-a85b-3391e12d0e55-proxy-tls\") pod \"isvc-canary-promote-predictor-d9759ffd5-7qx4s\" (UID: \"5b010cf8-fa1f-47a2-a85b-3391e12d0e55\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7qx4s"
Jul 28 13:23:44.828010 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.827938    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b010cf8-fa1f-47a2-a85b-3391e12d0e55-kserve-provision-location\") pod \"isvc-canary-promote-predictor-d9759ffd5-7qx4s\" (UID: \"5b010cf8-fa1f-47a2-a85b-3391e12d0e55\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7qx4s"
Jul 28 13:23:44.828010 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.827965    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67e719ff-8743-4d50-bbb5-f918dbac6adb-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-86766c4958-5j6jj\" (UID: \"67e719ff-8743-4d50-bbb5-f918dbac6adb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5j6jj"
Jul 28 13:23:44.828010 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.828006    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/67e719ff-8743-4d50-bbb5-f918dbac6adb-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-86766c4958-5j6jj\" (UID: \"67e719ff-8743-4d50-bbb5-f918dbac6adb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5j6jj"
Jul 28 13:23:44.828256 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.828052    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r72rz\" (UniqueName: \"kubernetes.io/projected/67e719ff-8743-4d50-bbb5-f918dbac6adb-kube-api-access-r72rz\") pod \"isvc-canary-promote-v2-predictor-86766c4958-5j6jj\" (UID: \"67e719ff-8743-4d50-bbb5-f918dbac6adb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5j6jj"
Jul 28 13:23:44.828256 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.828206    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67e719ff-8743-4d50-bbb5-f918dbac6adb-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-86766c4958-5j6jj\" (UID: \"67e719ff-8743-4d50-bbb5-f918dbac6adb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5j6jj"
Jul 28 13:23:44.828456 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.828436    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b010cf8-fa1f-47a2-a85b-3391e12d0e55-kserve-provision-location\") pod \"isvc-canary-promote-predictor-d9759ffd5-7qx4s\" (UID: \"5b010cf8-fa1f-47a2-a85b-3391e12d0e55\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7qx4s"
Jul 28 13:23:44.828585 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.828566    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b010cf8-fa1f-47a2-a85b-3391e12d0e55-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-d9759ffd5-7qx4s\" (UID: \"5b010cf8-fa1f-47a2-a85b-3391e12d0e55\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7qx4s"
Jul 28 13:23:44.828635 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.828607    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b010cf8-fa1f-47a2-a85b-3391e12d0e55-cabundle-cert\") pod \"isvc-canary-promote-predictor-d9759ffd5-7qx4s\" (UID: \"5b010cf8-fa1f-47a2-a85b-3391e12d0e55\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7qx4s"
Jul 28 13:23:44.828712 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.828670    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67e719ff-8743-4d50-bbb5-f918dbac6adb-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-86766c4958-5j6jj\" (UID: \"67e719ff-8743-4d50-bbb5-f918dbac6adb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5j6jj"
Jul 28 13:23:44.828834 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.828814    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/67e719ff-8743-4d50-bbb5-f918dbac6adb-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-86766c4958-5j6jj\" (UID: \"67e719ff-8743-4d50-bbb5-f918dbac6adb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5j6jj"
Jul 28 13:23:44.830512 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.830488    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b010cf8-fa1f-47a2-a85b-3391e12d0e55-proxy-tls\") pod \"isvc-canary-promote-predictor-d9759ffd5-7qx4s\" (UID: \"5b010cf8-fa1f-47a2-a85b-3391e12d0e55\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7qx4s"
Jul 28 13:23:44.830756 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.830741    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67e719ff-8743-4d50-bbb5-f918dbac6adb-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-86766c4958-5j6jj\" (UID: \"67e719ff-8743-4d50-bbb5-f918dbac6adb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5j6jj"
Jul 28 13:23:44.836063 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.836040    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87j6w\" (UniqueName: \"kubernetes.io/projected/5b010cf8-fa1f-47a2-a85b-3391e12d0e55-kube-api-access-87j6w\") pod \"isvc-canary-promote-predictor-d9759ffd5-7qx4s\" (UID: \"5b010cf8-fa1f-47a2-a85b-3391e12d0e55\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7qx4s"
Jul 28 13:23:44.836167 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.836149    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r72rz\" (UniqueName: \"kubernetes.io/projected/67e719ff-8743-4d50-bbb5-f918dbac6adb-kube-api-access-r72rz\") pod \"isvc-canary-promote-v2-predictor-86766c4958-5j6jj\" (UID: \"67e719ff-8743-4d50-bbb5-f918dbac6adb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5j6jj"
Jul 28 13:23:44.931276 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.931184    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7qx4s"
Jul 28 13:23:44.937519 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:44.937495    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5j6jj"
Jul 28 13:23:45.071291 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:45.071270    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7qx4s"]
Jul 28 13:23:45.073236 ip-10-0-138-158 kubenswrapper[2574]: W0728 13:23:45.073207    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b010cf8_fa1f_47a2_a85b_3391e12d0e55.slice/crio-0a6f89bed3afd92c5c831d7d7a900c506f2428626c745cd7b0e42cc37faa8ee9 WatchSource:0}: Error finding container 0a6f89bed3afd92c5c831d7d7a900c506f2428626c745cd7b0e42cc37faa8ee9: Status 404 returned error can't find the container with id 0a6f89bed3afd92c5c831d7d7a900c506f2428626c745cd7b0e42cc37faa8ee9
Jul 28 13:23:45.090005 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:45.089986    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5j6jj"]
Jul 28 13:23:45.091832 ip-10-0-138-158 kubenswrapper[2574]: W0728 13:23:45.091811    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67e719ff_8743_4d50_bbb5_f918dbac6adb.slice/crio-b7e76f08fe5b8a266ec2a768a6ca72c656c8a240b4068e302b33d091210ce710 WatchSource:0}: Error finding container b7e76f08fe5b8a266ec2a768a6ca72c656c8a240b4068e302b33d091210ce710: Status 404 returned error can't find the container with id b7e76f08fe5b8a266ec2a768a6ca72c656c8a240b4068e302b33d091210ce710
Jul 28 13:23:45.479442 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:45.479400    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7qx4s" event={"ID":"5b010cf8-fa1f-47a2-a85b-3391e12d0e55","Type":"ContainerStarted","Data":"331a2825401eda0d6c7096d685eb01368b1338306c6d3cb1171b035cb49fc822"}
Jul 28 13:23:45.479914 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:45.479502    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7qx4s" event={"ID":"5b010cf8-fa1f-47a2-a85b-3391e12d0e55","Type":"ContainerStarted","Data":"0a6f89bed3afd92c5c831d7d7a900c506f2428626c745cd7b0e42cc37faa8ee9"}
Jul 28 13:23:45.481250 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:45.481214    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5j6jj" event={"ID":"67e719ff-8743-4d50-bbb5-f918dbac6adb","Type":"ContainerStarted","Data":"ebb08ef3a3af3569daf5bc47afe681defb816981dc1e70788afd1a3a6589810f"}
Jul 28 13:23:45.481396 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:45.481309    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5j6jj" event={"ID":"67e719ff-8743-4d50-bbb5-f918dbac6adb","Type":"ContainerStarted","Data":"b7e76f08fe5b8a266ec2a768a6ca72c656c8a240b4068e302b33d091210ce710"}
Jul 28 13:23:46.301572 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:46.301541    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d51af15-6745-417c-b03a-061d477c0246" path="/var/lib/kubelet/pods/4d51af15-6745-417c-b03a-061d477c0246/volumes"
Jul 28 13:23:46.302034 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:46.302020    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d8a6d15-74e3-45ad-9651-a45f8291cfaa" path="/var/lib/kubelet/pods/6d8a6d15-74e3-45ad-9651-a45f8291cfaa/volumes"
Jul 28 13:23:49.501798 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:49.501766    2574 generic.go:356] "Generic (PLEG): container finished" podID="5b010cf8-fa1f-47a2-a85b-3391e12d0e55" containerID="331a2825401eda0d6c7096d685eb01368b1338306c6d3cb1171b035cb49fc822" exitCode=0
Jul 28 13:23:49.502232 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:49.501839    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7qx4s" event={"ID":"5b010cf8-fa1f-47a2-a85b-3391e12d0e55","Type":"ContainerDied","Data":"331a2825401eda0d6c7096d685eb01368b1338306c6d3cb1171b035cb49fc822"}
Jul 28 13:23:49.503435 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:49.503412    2574 generic.go:356] "Generic (PLEG): container finished" podID="67e719ff-8743-4d50-bbb5-f918dbac6adb" containerID="ebb08ef3a3af3569daf5bc47afe681defb816981dc1e70788afd1a3a6589810f" exitCode=0
Jul 28 13:23:49.503529 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:49.503454    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5j6jj" event={"ID":"67e719ff-8743-4d50-bbb5-f918dbac6adb","Type":"ContainerDied","Data":"ebb08ef3a3af3569daf5bc47afe681defb816981dc1e70788afd1a3a6589810f"}
Jul 28 13:23:50.512356 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:50.512323    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7qx4s" event={"ID":"5b010cf8-fa1f-47a2-a85b-3391e12d0e55","Type":"ContainerStarted","Data":"fd6ebc433c2fe11474f7139a2f6cdf6d596104b573dc17358703c765c2718870"}
Jul 28 13:23:50.512835 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:50.512392    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7qx4s" event={"ID":"5b010cf8-fa1f-47a2-a85b-3391e12d0e55","Type":"ContainerStarted","Data":"55630a349333160564007ac542683643cba1dded5bdba9252120d5df52f7aa0e"}
Jul 28 13:23:50.512835 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:50.512611    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7qx4s"
Jul 28 13:23:50.514311 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:50.514291    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5j6jj" event={"ID":"67e719ff-8743-4d50-bbb5-f918dbac6adb","Type":"ContainerStarted","Data":"0088b2a0210014c021c5a461dcaf2e1f41febb0a0b3468281e1d926990bae442"}
Jul 28 13:23:50.514425 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:50.514334    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5j6jj" event={"ID":"67e719ff-8743-4d50-bbb5-f918dbac6adb","Type":"ContainerStarted","Data":"adcefcef1280119a41f3e7f5643fd76362e12b9ecae2e38e77de6097ec7cbd7a"}
Jul 28 13:23:50.514654 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:50.514637    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5j6jj"
Jul 28 13:23:50.514738 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:50.514712    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5j6jj"
Jul 28 13:23:50.515930 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:50.515907    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5j6jj" podUID="67e719ff-8743-4d50-bbb5-f918dbac6adb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 28 13:23:50.534326 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:50.534279    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7qx4s" podStartSLOduration=2.7853608210000003 podStartE2EDuration="6.534267214s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.748906393s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:23:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:23:50.532628378 +0000 UTC m=+1842.738014990" watchObservedRunningTime="2026-07-28 13:23:50.534267214 +0000 UTC m=+1842.739653814"
Jul 28 13:23:50.552988 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:50.552954    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5j6jj" podStartSLOduration=2.811943011 podStartE2EDuration="6.552942702s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.740999691s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:23:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:23:50.552379189 +0000 UTC m=+1842.757765790" watchObservedRunningTime="2026-07-28 13:23:50.552942702 +0000 UTC m=+1842.758329304"
Jul 28 13:23:51.518997 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:51.518956    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5j6jj" podUID="67e719ff-8743-4d50-bbb5-f918dbac6adb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 28 13:23:51.519353 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:51.519034    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7qx4s"
Jul 28 13:23:51.520130 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:51.520107    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7qx4s" podUID="5b010cf8-fa1f-47a2-a85b-3391e12d0e55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 28 13:23:52.524070 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:52.524027    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5j6jj" podUID="67e719ff-8743-4d50-bbb5-f918dbac6adb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 28 13:23:52.524493 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:52.524155    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7qx4s" podUID="5b010cf8-fa1f-47a2-a85b-3391e12d0e55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 28 13:23:57.528172 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:57.528142    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5j6jj"
Jul 28 13:23:57.528558 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:57.528347    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7qx4s"
Jul 28 13:23:57.528616 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:57.528553    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5j6jj" podUID="67e719ff-8743-4d50-bbb5-f918dbac6adb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 28 13:23:57.529048 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:23:57.529007    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7qx4s" podUID="5b010cf8-fa1f-47a2-a85b-3391e12d0e55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 28 13:24:07.529095 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:24:07.529054    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5j6jj" podUID="67e719ff-8743-4d50-bbb5-f918dbac6adb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 28 13:24:07.529468 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:24:07.529067    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7qx4s" podUID="5b010cf8-fa1f-47a2-a85b-3391e12d0e55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 28 13:24:17.529321 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:24:17.529283    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5j6jj" podUID="67e719ff-8743-4d50-bbb5-f918dbac6adb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 28 13:24:17.529836 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:24:17.529283    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7qx4s" podUID="5b010cf8-fa1f-47a2-a85b-3391e12d0e55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 28 13:24:27.528644 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:24:27.528603    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5j6jj" podUID="67e719ff-8743-4d50-bbb5-f918dbac6adb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 28 13:24:27.529035 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:24:27.528894    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7qx4s" podUID="5b010cf8-fa1f-47a2-a85b-3391e12d0e55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 28 13:24:37.528645 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:24:37.528566    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5j6jj" podUID="67e719ff-8743-4d50-bbb5-f918dbac6adb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 28 13:24:37.529060 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:24:37.528921    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7qx4s" podUID="5b010cf8-fa1f-47a2-a85b-3391e12d0e55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 28 13:24:47.529268 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:24:47.529230    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7qx4s" podUID="5b010cf8-fa1f-47a2-a85b-3391e12d0e55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 28 13:24:47.529647 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:24:47.529230    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5j6jj" podUID="67e719ff-8743-4d50-bbb5-f918dbac6adb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 28 13:24:57.529248 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:24:57.529205    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7qx4s" podUID="5b010cf8-fa1f-47a2-a85b-3391e12d0e55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 28 13:24:57.529759 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:24:57.529247    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5j6jj" podUID="67e719ff-8743-4d50-bbb5-f918dbac6adb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 28 13:25:05.298498 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:25:05.298471    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7qx4s"
Jul 28 13:25:07.529323 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:25:07.529297    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5j6jj"
Jul 28 13:25:14.713353 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:25:14.713318    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7qx4s"]
Jul 28 13:25:14.713818 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:25:14.713591    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7qx4s" podUID="5b010cf8-fa1f-47a2-a85b-3391e12d0e55" containerName="kserve-container" containerID="cri-o://55630a349333160564007ac542683643cba1dded5bdba9252120d5df52f7aa0e" gracePeriod=30
Jul 28 13:25:14.713818 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:25:14.713625    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7qx4s" podUID="5b010cf8-fa1f-47a2-a85b-3391e12d0e55" containerName="kube-rbac-proxy" containerID="cri-o://fd6ebc433c2fe11474f7139a2f6cdf6d596104b573dc17358703c765c2718870" gracePeriod=30
Jul 28 13:25:14.902008 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:25:14.901979    2574 generic.go:356] "Generic (PLEG): container finished" podID="5b010cf8-fa1f-47a2-a85b-3391e12d0e55" containerID="fd6ebc433c2fe11474f7139a2f6cdf6d596104b573dc17358703c765c2718870" exitCode=2
Jul 28 13:25:14.902131 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:25:14.902053    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7qx4s" event={"ID":"5b010cf8-fa1f-47a2-a85b-3391e12d0e55","Type":"ContainerDied","Data":"fd6ebc433c2fe11474f7139a2f6cdf6d596104b573dc17358703c765c2718870"}
Jul 28 13:25:15.298451 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:25:15.298414    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7qx4s" podUID="5b010cf8-fa1f-47a2-a85b-3391e12d0e55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 28 13:25:17.524978 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:25:17.524939    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7qx4s" podUID="5b010cf8-fa1f-47a2-a85b-3391e12d0e55" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
Jul 28 13:25:18.445743 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:25:18.445717    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7qx4s"
Jul 28 13:25:18.524628 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:25:18.524564    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-87j6w\" (UniqueName: \"kubernetes.io/projected/5b010cf8-fa1f-47a2-a85b-3391e12d0e55-kube-api-access-87j6w\") pod \"5b010cf8-fa1f-47a2-a85b-3391e12d0e55\" (UID: \"5b010cf8-fa1f-47a2-a85b-3391e12d0e55\") "
Jul 28 13:25:18.524628 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:25:18.524612    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b010cf8-fa1f-47a2-a85b-3391e12d0e55-cabundle-cert\") pod \"5b010cf8-fa1f-47a2-a85b-3391e12d0e55\" (UID: \"5b010cf8-fa1f-47a2-a85b-3391e12d0e55\") "
Jul 28 13:25:18.524839 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:25:18.524656    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b010cf8-fa1f-47a2-a85b-3391e12d0e55-kserve-provision-location\") pod \"5b010cf8-fa1f-47a2-a85b-3391e12d0e55\" (UID: \"5b010cf8-fa1f-47a2-a85b-3391e12d0e55\") "
Jul 28 13:25:18.524839 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:25:18.524730    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b010cf8-fa1f-47a2-a85b-3391e12d0e55-kube-rbac-proxy-sar-config\") pod \"5b010cf8-fa1f-47a2-a85b-3391e12d0e55\" (UID: \"5b010cf8-fa1f-47a2-a85b-3391e12d0e55\") "
Jul 28 13:25:18.524839 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:25:18.524761    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b010cf8-fa1f-47a2-a85b-3391e12d0e55-proxy-tls\") pod \"5b010cf8-fa1f-47a2-a85b-3391e12d0e55\" (UID: \"5b010cf8-fa1f-47a2-a85b-3391e12d0e55\") "
Jul 28 13:25:18.525075 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:25:18.525045    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b010cf8-fa1f-47a2-a85b-3391e12d0e55-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5b010cf8-fa1f-47a2-a85b-3391e12d0e55" (UID: "5b010cf8-fa1f-47a2-a85b-3391e12d0e55"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:25:18.525338 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:25:18.525081    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b010cf8-fa1f-47a2-a85b-3391e12d0e55-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5b010cf8-fa1f-47a2-a85b-3391e12d0e55" (UID: "5b010cf8-fa1f-47a2-a85b-3391e12d0e55"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:25:18.525338 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:25:18.525186    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b010cf8-fa1f-47a2-a85b-3391e12d0e55-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5b010cf8-fa1f-47a2-a85b-3391e12d0e55" (UID: "5b010cf8-fa1f-47a2-a85b-3391e12d0e55"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:25:18.526728 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:25:18.526700    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b010cf8-fa1f-47a2-a85b-3391e12d0e55-kube-api-access-87j6w" (OuterVolumeSpecName: "kube-api-access-87j6w") pod "5b010cf8-fa1f-47a2-a85b-3391e12d0e55" (UID: "5b010cf8-fa1f-47a2-a85b-3391e12d0e55"). InnerVolumeSpecName "kube-api-access-87j6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:25:18.526728 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:25:18.526710    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b010cf8-fa1f-47a2-a85b-3391e12d0e55-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5b010cf8-fa1f-47a2-a85b-3391e12d0e55" (UID: "5b010cf8-fa1f-47a2-a85b-3391e12d0e55"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:25:18.626259 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:25:18.626235    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87j6w\" (UniqueName: \"kubernetes.io/projected/5b010cf8-fa1f-47a2-a85b-3391e12d0e55-kube-api-access-87j6w\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:25:18.626259 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:25:18.626258    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b010cf8-fa1f-47a2-a85b-3391e12d0e55-cabundle-cert\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:25:18.626386 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:25:18.626267    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b010cf8-fa1f-47a2-a85b-3391e12d0e55-kserve-provision-location\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:25:18.626386 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:25:18.626276    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b010cf8-fa1f-47a2-a85b-3391e12d0e55-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:25:18.626386 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:25:18.626285    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b010cf8-fa1f-47a2-a85b-3391e12d0e55-proxy-tls\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:25:18.921048 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:25:18.920975    2574 generic.go:356] "Generic (PLEG): container finished" podID="5b010cf8-fa1f-47a2-a85b-3391e12d0e55" containerID="55630a349333160564007ac542683643cba1dded5bdba9252120d5df52f7aa0e" exitCode=0
Jul 28 13:25:18.921048 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:25:18.921009    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7qx4s" event={"ID":"5b010cf8-fa1f-47a2-a85b-3391e12d0e55","Type":"ContainerDied","Data":"55630a349333160564007ac542683643cba1dded5bdba9252120d5df52f7aa0e"}
Jul 28 13:25:18.921048 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:25:18.921043    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7qx4s"
Jul 28 13:25:18.921299 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:25:18.921054    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7qx4s" event={"ID":"5b010cf8-fa1f-47a2-a85b-3391e12d0e55","Type":"ContainerDied","Data":"0a6f89bed3afd92c5c831d7d7a900c506f2428626c745cd7b0e42cc37faa8ee9"}
Jul 28 13:25:18.921299 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:25:18.921080    2574 scope.go:117] "RemoveContainer" containerID="fd6ebc433c2fe11474f7139a2f6cdf6d596104b573dc17358703c765c2718870"
Jul 28 13:25:18.929920 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:25:18.929905    2574 scope.go:117] "RemoveContainer" containerID="55630a349333160564007ac542683643cba1dded5bdba9252120d5df52f7aa0e"
Jul 28 13:25:18.937533 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:25:18.937518    2574 scope.go:117] "RemoveContainer" containerID="331a2825401eda0d6c7096d685eb01368b1338306c6d3cb1171b035cb49fc822"
Jul 28 13:25:18.944627 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:25:18.944602    2574 scope.go:117] "RemoveContainer" containerID="fd6ebc433c2fe11474f7139a2f6cdf6d596104b573dc17358703c765c2718870"
Jul 28 13:25:18.944874 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:25:18.944856    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd6ebc433c2fe11474f7139a2f6cdf6d596104b573dc17358703c765c2718870\": container with ID starting with fd6ebc433c2fe11474f7139a2f6cdf6d596104b573dc17358703c765c2718870 not found: ID does not exist" containerID="fd6ebc433c2fe11474f7139a2f6cdf6d596104b573dc17358703c765c2718870"
Jul 28 13:25:18.944961 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:25:18.944888    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd6ebc433c2fe11474f7139a2f6cdf6d596104b573dc17358703c765c2718870"} err="failed to get container status \"fd6ebc433c2fe11474f7139a2f6cdf6d596104b573dc17358703c765c2718870\": rpc error: code = NotFound desc = could not find container \"fd6ebc433c2fe11474f7139a2f6cdf6d596104b573dc17358703c765c2718870\": container with ID starting with fd6ebc433c2fe11474f7139a2f6cdf6d596104b573dc17358703c765c2718870 not found: ID does not exist"
Jul 28 13:25:18.944961 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:25:18.944911    2574 scope.go:117] "RemoveContainer" containerID="55630a349333160564007ac542683643cba1dded5bdba9252120d5df52f7aa0e"
Jul 28 13:25:18.945154 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:25:18.945136    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55630a349333160564007ac542683643cba1dded5bdba9252120d5df52f7aa0e\": container with ID starting with 55630a349333160564007ac542683643cba1dded5bdba9252120d5df52f7aa0e not found: ID does not exist" containerID="55630a349333160564007ac542683643cba1dded5bdba9252120d5df52f7aa0e"
Jul 28 13:25:18.945195 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:25:18.945159    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55630a349333160564007ac542683643cba1dded5bdba9252120d5df52f7aa0e"} err="failed to get container status \"55630a349333160564007ac542683643cba1dded5bdba9252120d5df52f7aa0e\": rpc error: code = NotFound desc = could not find container \"55630a349333160564007ac542683643cba1dded5bdba9252120d5df52f7aa0e\": container with ID starting with 55630a349333160564007ac542683643cba1dded5bdba9252120d5df52f7aa0e not found: ID does not exist"
Jul 28 13:25:18.945195 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:25:18.945174    2574 scope.go:117] "RemoveContainer" containerID="331a2825401eda0d6c7096d685eb01368b1338306c6d3cb1171b035cb49fc822"
Jul 28 13:25:18.945452 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:25:18.945427    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"331a2825401eda0d6c7096d685eb01368b1338306c6d3cb1171b035cb49fc822\": container with ID starting with 331a2825401eda0d6c7096d685eb01368b1338306c6d3cb1171b035cb49fc822 not found: ID does not exist" containerID="331a2825401eda0d6c7096d685eb01368b1338306c6d3cb1171b035cb49fc822"
Jul 28 13:25:18.945504 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:25:18.945454    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"331a2825401eda0d6c7096d685eb01368b1338306c6d3cb1171b035cb49fc822"} err="failed to get container status \"331a2825401eda0d6c7096d685eb01368b1338306c6d3cb1171b035cb49fc822\": rpc error: code = NotFound desc = could not find container \"331a2825401eda0d6c7096d685eb01368b1338306c6d3cb1171b035cb49fc822\": container with ID starting with 331a2825401eda0d6c7096d685eb01368b1338306c6d3cb1171b035cb49fc822 not found: ID does not exist"
Jul 28 13:25:18.964287 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:25:18.964266    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7qx4s"]
Jul 28 13:25:18.967604 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:25:18.967584    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7qx4s"]
Jul 28 13:25:20.302098 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:25:20.302070    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b010cf8-fa1f-47a2-a85b-3391e12d0e55" path="/var/lib/kubelet/pods/5b010cf8-fa1f-47a2-a85b-3391e12d0e55/volumes"
Jul 28 13:27:04.959855 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:04.959825    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5j6jj"]
Jul 28 13:27:04.960360 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:04.960155    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5j6jj" podUID="67e719ff-8743-4d50-bbb5-f918dbac6adb" containerName="kserve-container" containerID="cri-o://adcefcef1280119a41f3e7f5643fd76362e12b9ecae2e38e77de6097ec7cbd7a" gracePeriod=30
Jul 28 13:27:04.960360 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:04.960184    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5j6jj" podUID="67e719ff-8743-4d50-bbb5-f918dbac6adb" containerName="kube-rbac-proxy" containerID="cri-o://0088b2a0210014c021c5a461dcaf2e1f41febb0a0b3468281e1d926990bae442" gracePeriod=30
Jul 28 13:27:05.389740 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:05.389641    2574 generic.go:356] "Generic (PLEG): container finished" podID="67e719ff-8743-4d50-bbb5-f918dbac6adb" containerID="0088b2a0210014c021c5a461dcaf2e1f41febb0a0b3468281e1d926990bae442" exitCode=2
Jul 28 13:27:05.389740 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:05.389715    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5j6jj" event={"ID":"67e719ff-8743-4d50-bbb5-f918dbac6adb","Type":"ContainerDied","Data":"0088b2a0210014c021c5a461dcaf2e1f41febb0a0b3468281e1d926990bae442"}
Jul 28 13:27:07.524193 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:07.524160    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5j6jj" podUID="67e719ff-8743-4d50-bbb5-f918dbac6adb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused"
Jul 28 13:27:07.529366 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:07.529321    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5j6jj" podUID="67e719ff-8743-4d50-bbb5-f918dbac6adb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 28 13:27:08.907156 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:08.907132    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5j6jj"
Jul 28 13:27:09.011388 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:09.011325    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r72rz\" (UniqueName: \"kubernetes.io/projected/67e719ff-8743-4d50-bbb5-f918dbac6adb-kube-api-access-r72rz\") pod \"67e719ff-8743-4d50-bbb5-f918dbac6adb\" (UID: \"67e719ff-8743-4d50-bbb5-f918dbac6adb\") "
Jul 28 13:27:09.011388 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:09.011380    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67e719ff-8743-4d50-bbb5-f918dbac6adb-proxy-tls\") pod \"67e719ff-8743-4d50-bbb5-f918dbac6adb\" (UID: \"67e719ff-8743-4d50-bbb5-f918dbac6adb\") "
Jul 28 13:27:09.011576 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:09.011412    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67e719ff-8743-4d50-bbb5-f918dbac6adb-kube-rbac-proxy-sar-config\") pod \"67e719ff-8743-4d50-bbb5-f918dbac6adb\" (UID: \"67e719ff-8743-4d50-bbb5-f918dbac6adb\") "
Jul 28 13:27:09.011576 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:09.011433    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/67e719ff-8743-4d50-bbb5-f918dbac6adb-cabundle-cert\") pod \"67e719ff-8743-4d50-bbb5-f918dbac6adb\" (UID: \"67e719ff-8743-4d50-bbb5-f918dbac6adb\") "
Jul 28 13:27:09.011576 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:09.011457    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67e719ff-8743-4d50-bbb5-f918dbac6adb-kserve-provision-location\") pod \"67e719ff-8743-4d50-bbb5-f918dbac6adb\" (UID: \"67e719ff-8743-4d50-bbb5-f918dbac6adb\") "
Jul 28 13:27:09.011811 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:09.011788    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67e719ff-8743-4d50-bbb5-f918dbac6adb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "67e719ff-8743-4d50-bbb5-f918dbac6adb" (UID: "67e719ff-8743-4d50-bbb5-f918dbac6adb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:27:09.011811 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:09.011797    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67e719ff-8743-4d50-bbb5-f918dbac6adb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "67e719ff-8743-4d50-bbb5-f918dbac6adb" (UID: "67e719ff-8743-4d50-bbb5-f918dbac6adb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:27:09.011904 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:09.011801    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67e719ff-8743-4d50-bbb5-f918dbac6adb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "67e719ff-8743-4d50-bbb5-f918dbac6adb" (UID: "67e719ff-8743-4d50-bbb5-f918dbac6adb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:27:09.013459 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:09.013442    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67e719ff-8743-4d50-bbb5-f918dbac6adb-kube-api-access-r72rz" (OuterVolumeSpecName: "kube-api-access-r72rz") pod "67e719ff-8743-4d50-bbb5-f918dbac6adb" (UID: "67e719ff-8743-4d50-bbb5-f918dbac6adb"). InnerVolumeSpecName "kube-api-access-r72rz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:27:09.013459 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:09.013449    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67e719ff-8743-4d50-bbb5-f918dbac6adb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "67e719ff-8743-4d50-bbb5-f918dbac6adb" (UID: "67e719ff-8743-4d50-bbb5-f918dbac6adb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:27:09.112827 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:09.112804    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r72rz\" (UniqueName: \"kubernetes.io/projected/67e719ff-8743-4d50-bbb5-f918dbac6adb-kube-api-access-r72rz\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:27:09.112827 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:09.112824    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67e719ff-8743-4d50-bbb5-f918dbac6adb-proxy-tls\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:27:09.112951 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:09.112836    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67e719ff-8743-4d50-bbb5-f918dbac6adb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:27:09.112951 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:09.112844    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/67e719ff-8743-4d50-bbb5-f918dbac6adb-cabundle-cert\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:27:09.112951 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:09.112853    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67e719ff-8743-4d50-bbb5-f918dbac6adb-kserve-provision-location\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:27:09.408319 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:09.408249    2574 generic.go:356] "Generic (PLEG): container finished" podID="67e719ff-8743-4d50-bbb5-f918dbac6adb" containerID="adcefcef1280119a41f3e7f5643fd76362e12b9ecae2e38e77de6097ec7cbd7a" exitCode=0
Jul 28 13:27:09.408319 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:09.408305    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5j6jj"
Jul 28 13:27:09.408500 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:09.408309    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5j6jj" event={"ID":"67e719ff-8743-4d50-bbb5-f918dbac6adb","Type":"ContainerDied","Data":"adcefcef1280119a41f3e7f5643fd76362e12b9ecae2e38e77de6097ec7cbd7a"}
Jul 28 13:27:09.408500 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:09.408435    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5j6jj" event={"ID":"67e719ff-8743-4d50-bbb5-f918dbac6adb","Type":"ContainerDied","Data":"b7e76f08fe5b8a266ec2a768a6ca72c656c8a240b4068e302b33d091210ce710"}
Jul 28 13:27:09.408500 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:09.408464    2574 scope.go:117] "RemoveContainer" containerID="0088b2a0210014c021c5a461dcaf2e1f41febb0a0b3468281e1d926990bae442"
Jul 28 13:27:09.417893 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:09.417876    2574 scope.go:117] "RemoveContainer" containerID="adcefcef1280119a41f3e7f5643fd76362e12b9ecae2e38e77de6097ec7cbd7a"
Jul 28 13:27:09.426616 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:09.426591    2574 scope.go:117] "RemoveContainer" containerID="ebb08ef3a3af3569daf5bc47afe681defb816981dc1e70788afd1a3a6589810f"
Jul 28 13:27:09.434050 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:09.434029    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5j6jj"]
Jul 28 13:27:09.435179 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:09.435166    2574 scope.go:117] "RemoveContainer" containerID="0088b2a0210014c021c5a461dcaf2e1f41febb0a0b3468281e1d926990bae442"
Jul 28 13:27:09.435388 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:27:09.435372    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0088b2a0210014c021c5a461dcaf2e1f41febb0a0b3468281e1d926990bae442\": container with ID starting with 0088b2a0210014c021c5a461dcaf2e1f41febb0a0b3468281e1d926990bae442 not found: ID does not exist" containerID="0088b2a0210014c021c5a461dcaf2e1f41febb0a0b3468281e1d926990bae442"
Jul 28 13:27:09.435425 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:09.435393    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0088b2a0210014c021c5a461dcaf2e1f41febb0a0b3468281e1d926990bae442"} err="failed to get container status \"0088b2a0210014c021c5a461dcaf2e1f41febb0a0b3468281e1d926990bae442\": rpc error: code = NotFound desc = could not find container \"0088b2a0210014c021c5a461dcaf2e1f41febb0a0b3468281e1d926990bae442\": container with ID starting with 0088b2a0210014c021c5a461dcaf2e1f41febb0a0b3468281e1d926990bae442 not found: ID does not exist"
Jul 28 13:27:09.435425 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:09.435409    2574 scope.go:117] "RemoveContainer" containerID="adcefcef1280119a41f3e7f5643fd76362e12b9ecae2e38e77de6097ec7cbd7a"
Jul 28 13:27:09.435616 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:27:09.435598    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adcefcef1280119a41f3e7f5643fd76362e12b9ecae2e38e77de6097ec7cbd7a\": container with ID starting with adcefcef1280119a41f3e7f5643fd76362e12b9ecae2e38e77de6097ec7cbd7a not found: ID does not exist" containerID="adcefcef1280119a41f3e7f5643fd76362e12b9ecae2e38e77de6097ec7cbd7a"
Jul 28 13:27:09.435660 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:09.435624    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adcefcef1280119a41f3e7f5643fd76362e12b9ecae2e38e77de6097ec7cbd7a"} err="failed to get container status \"adcefcef1280119a41f3e7f5643fd76362e12b9ecae2e38e77de6097ec7cbd7a\": rpc error: code = NotFound desc = could not find container \"adcefcef1280119a41f3e7f5643fd76362e12b9ecae2e38e77de6097ec7cbd7a\": container with ID starting with adcefcef1280119a41f3e7f5643fd76362e12b9ecae2e38e77de6097ec7cbd7a not found: ID does not exist"
Jul 28 13:27:09.435660 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:09.435641    2574 scope.go:117] "RemoveContainer" containerID="ebb08ef3a3af3569daf5bc47afe681defb816981dc1e70788afd1a3a6589810f"
Jul 28 13:27:09.435951 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:27:09.435931    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebb08ef3a3af3569daf5bc47afe681defb816981dc1e70788afd1a3a6589810f\": container with ID starting with ebb08ef3a3af3569daf5bc47afe681defb816981dc1e70788afd1a3a6589810f not found: ID does not exist" containerID="ebb08ef3a3af3569daf5bc47afe681defb816981dc1e70788afd1a3a6589810f"
Jul 28 13:27:09.436012 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:09.435959    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebb08ef3a3af3569daf5bc47afe681defb816981dc1e70788afd1a3a6589810f"} err="failed to get container status \"ebb08ef3a3af3569daf5bc47afe681defb816981dc1e70788afd1a3a6589810f\": rpc error: code = NotFound desc = could not find container \"ebb08ef3a3af3569daf5bc47afe681defb816981dc1e70788afd1a3a6589810f\": container with ID starting with ebb08ef3a3af3569daf5bc47afe681defb816981dc1e70788afd1a3a6589810f not found: ID does not exist"
Jul 28 13:27:09.440166 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:09.440148    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5j6jj"]
Jul 28 13:27:10.219216 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:10.219187    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-trls9"]
Jul 28 13:27:10.219657 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:10.219520    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b010cf8-fa1f-47a2-a85b-3391e12d0e55" containerName="storage-initializer"
Jul 28 13:27:10.219657 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:10.219530    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b010cf8-fa1f-47a2-a85b-3391e12d0e55" containerName="storage-initializer"
Jul 28 13:27:10.219657 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:10.219539    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67e719ff-8743-4d50-bbb5-f918dbac6adb" containerName="kserve-container"
Jul 28 13:27:10.219657 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:10.219545    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="67e719ff-8743-4d50-bbb5-f918dbac6adb" containerName="kserve-container"
Jul 28 13:27:10.219657 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:10.219552    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b010cf8-fa1f-47a2-a85b-3391e12d0e55" containerName="kserve-container"
Jul 28 13:27:10.219657 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:10.219558    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b010cf8-fa1f-47a2-a85b-3391e12d0e55" containerName="kserve-container"
Jul 28 13:27:10.219657 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:10.219568    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b010cf8-fa1f-47a2-a85b-3391e12d0e55" containerName="kube-rbac-proxy"
Jul 28 13:27:10.219657 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:10.219573    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b010cf8-fa1f-47a2-a85b-3391e12d0e55" containerName="kube-rbac-proxy"
Jul 28 13:27:10.219657 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:10.219583    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67e719ff-8743-4d50-bbb5-f918dbac6adb" containerName="storage-initializer"
Jul 28 13:27:10.219657 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:10.219588    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="67e719ff-8743-4d50-bbb5-f918dbac6adb" containerName="storage-initializer"
Jul 28 13:27:10.219657 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:10.219594    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67e719ff-8743-4d50-bbb5-f918dbac6adb" containerName="kube-rbac-proxy"
Jul 28 13:27:10.219657 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:10.219599    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="67e719ff-8743-4d50-bbb5-f918dbac6adb" containerName="kube-rbac-proxy"
Jul 28 13:27:10.219657 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:10.219654    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b010cf8-fa1f-47a2-a85b-3391e12d0e55" containerName="kserve-container"
Jul 28 13:27:10.219657 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:10.219660    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="67e719ff-8743-4d50-bbb5-f918dbac6adb" containerName="kserve-container"
Jul 28 13:27:10.220107 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:10.219667    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b010cf8-fa1f-47a2-a85b-3391e12d0e55" containerName="kube-rbac-proxy"
Jul 28 13:27:10.220107 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:10.219674    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="67e719ff-8743-4d50-bbb5-f918dbac6adb" containerName="kube-rbac-proxy"
Jul 28 13:27:10.224063 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:10.224047    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-trls9"
Jul 28 13:27:10.226781 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:10.226757    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Jul 28 13:27:10.226929 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:10.226791    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:27:10.226929 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:10.226828    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:27:10.227063 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:10.227030    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Jul 28 13:27:10.227063 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:10.227037    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:27:10.227063 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:10.227033    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rxfwd\""
Jul 28 13:27:10.227277 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:10.227264    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 13:27:10.238751 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:10.238730    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-trls9"]
Jul 28 13:27:10.302117 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:10.302092    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67e719ff-8743-4d50-bbb5-f918dbac6adb" path="/var/lib/kubelet/pods/67e719ff-8743-4d50-bbb5-f918dbac6adb/volumes"
Jul 28 13:27:10.321507 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:10.321487    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vjgz\" (UniqueName: \"kubernetes.io/projected/d8b4592e-d228-473d-a23e-0d0b0faae458-kube-api-access-5vjgz\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-trls9\" (UID: \"d8b4592e-d228-473d-a23e-0d0b0faae458\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-trls9"
Jul 28 13:27:10.321601 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:10.321515    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d8b4592e-d228-473d-a23e-0d0b0faae458-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-trls9\" (UID: \"d8b4592e-d228-473d-a23e-0d0b0faae458\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-trls9"
Jul 28 13:27:10.321601 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:10.321551    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d8b4592e-d228-473d-a23e-0d0b0faae458-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-trls9\" (UID: \"d8b4592e-d228-473d-a23e-0d0b0faae458\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-trls9"
Jul 28 13:27:10.321752 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:10.321646    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8b4592e-d228-473d-a23e-0d0b0faae458-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-trls9\" (UID: \"d8b4592e-d228-473d-a23e-0d0b0faae458\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-trls9"
Jul 28 13:27:10.321752 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:10.321712    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d8b4592e-d228-473d-a23e-0d0b0faae458-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-trls9\" (UID: \"d8b4592e-d228-473d-a23e-0d0b0faae458\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-trls9"
Jul 28 13:27:10.422966 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:10.422941    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d8b4592e-d228-473d-a23e-0d0b0faae458-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-trls9\" (UID: \"d8b4592e-d228-473d-a23e-0d0b0faae458\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-trls9"
Jul 28 13:27:10.423079 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:10.422994    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8b4592e-d228-473d-a23e-0d0b0faae458-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-trls9\" (UID: \"d8b4592e-d228-473d-a23e-0d0b0faae458\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-trls9"
Jul 28 13:27:10.423079 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:10.423022    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d8b4592e-d228-473d-a23e-0d0b0faae458-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-trls9\" (UID: \"d8b4592e-d228-473d-a23e-0d0b0faae458\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-trls9"
Jul 28 13:27:10.423199 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:10.423077    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vjgz\" (UniqueName: \"kubernetes.io/projected/d8b4592e-d228-473d-a23e-0d0b0faae458-kube-api-access-5vjgz\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-trls9\" (UID: \"d8b4592e-d228-473d-a23e-0d0b0faae458\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-trls9"
Jul 28 13:27:10.423199 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:10.423110    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d8b4592e-d228-473d-a23e-0d0b0faae458-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-trls9\" (UID: \"d8b4592e-d228-473d-a23e-0d0b0faae458\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-trls9"
Jul 28 13:27:10.423443 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:10.423424    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d8b4592e-d228-473d-a23e-0d0b0faae458-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-trls9\" (UID: \"d8b4592e-d228-473d-a23e-0d0b0faae458\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-trls9"
Jul 28 13:27:10.423639 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:10.423620    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d8b4592e-d228-473d-a23e-0d0b0faae458-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-trls9\" (UID: \"d8b4592e-d228-473d-a23e-0d0b0faae458\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-trls9"
Jul 28 13:27:10.423706 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:10.423647    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d8b4592e-d228-473d-a23e-0d0b0faae458-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-trls9\" (UID: \"d8b4592e-d228-473d-a23e-0d0b0faae458\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-trls9"
Jul 28 13:27:10.425522 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:10.425502    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8b4592e-d228-473d-a23e-0d0b0faae458-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-trls9\" (UID: \"d8b4592e-d228-473d-a23e-0d0b0faae458\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-trls9"
Jul 28 13:27:10.430336 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:10.430313    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vjgz\" (UniqueName: \"kubernetes.io/projected/d8b4592e-d228-473d-a23e-0d0b0faae458-kube-api-access-5vjgz\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-trls9\" (UID: \"d8b4592e-d228-473d-a23e-0d0b0faae458\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-trls9"
Jul 28 13:27:10.533967 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:10.533914    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-trls9"
Jul 28 13:27:10.661021 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:10.660999    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-trls9"]
Jul 28 13:27:10.662829 ip-10-0-138-158 kubenswrapper[2574]: W0728 13:27:10.662799    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8b4592e_d228_473d_a23e_0d0b0faae458.slice/crio-f57bd03689a7e60cfa58b646a9f78f6c15d1271e0eca82bb480adfc82b429e25 WatchSource:0}: Error finding container f57bd03689a7e60cfa58b646a9f78f6c15d1271e0eca82bb480adfc82b429e25: Status 404 returned error can't find the container with id f57bd03689a7e60cfa58b646a9f78f6c15d1271e0eca82bb480adfc82b429e25
Jul 28 13:27:11.420654 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:11.420624    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-trls9" event={"ID":"d8b4592e-d228-473d-a23e-0d0b0faae458","Type":"ContainerStarted","Data":"b0206ea8426862e008c89b60b336bdde879b059910d5a75bb33c07453645da84"}
Jul 28 13:27:11.421030 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:11.420678    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-trls9" event={"ID":"d8b4592e-d228-473d-a23e-0d0b0faae458","Type":"ContainerStarted","Data":"f57bd03689a7e60cfa58b646a9f78f6c15d1271e0eca82bb480adfc82b429e25"}
Jul 28 13:27:15.439210 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:15.439178    2574 generic.go:356] "Generic (PLEG): container finished" podID="d8b4592e-d228-473d-a23e-0d0b0faae458" containerID="b0206ea8426862e008c89b60b336bdde879b059910d5a75bb33c07453645da84" exitCode=0
Jul 28 13:27:15.439617 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:15.439249    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-trls9" event={"ID":"d8b4592e-d228-473d-a23e-0d0b0faae458","Type":"ContainerDied","Data":"b0206ea8426862e008c89b60b336bdde879b059910d5a75bb33c07453645da84"}
Jul 28 13:27:16.446197 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:16.446161    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-trls9" event={"ID":"d8b4592e-d228-473d-a23e-0d0b0faae458","Type":"ContainerStarted","Data":"68e93cd137ceb6bdc05a857e42132d9e72cebf1cfab4f2700c764dbc3bfbd947"}
Jul 28 13:27:16.446622 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:16.446218    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-trls9" event={"ID":"d8b4592e-d228-473d-a23e-0d0b0faae458","Type":"ContainerStarted","Data":"0934724a6d8b41ff932292fa2b78d96294ddcc640bf8b8a2b0764ab34a743fd7"}
Jul 28 13:27:16.446622 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:16.446537    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-trls9"
Jul 28 13:27:16.446758 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:16.446667    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-trls9"
Jul 28 13:27:16.447901 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:16.447874    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-trls9" podUID="d8b4592e-d228-473d-a23e-0d0b0faae458" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 28 13:27:16.467431 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:16.467392    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-trls9" podStartSLOduration=2.671833806 podStartE2EDuration="6.467380426s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.79554662s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:27:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:27:16.465405655 +0000 UTC m=+2048.670792257" watchObservedRunningTime="2026-07-28 13:27:16.467380426 +0000 UTC m=+2048.672767026"
Jul 28 13:27:17.451762 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:17.451721    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-trls9" podUID="d8b4592e-d228-473d-a23e-0d0b0faae458" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 28 13:27:18.455717 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:18.455668    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-trls9" podUID="d8b4592e-d228-473d-a23e-0d0b0faae458" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 28 13:27:23.464732 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:23.464705    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-trls9"
Jul 28 13:27:23.465252 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:23.465226    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-trls9" podUID="d8b4592e-d228-473d-a23e-0d0b0faae458" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 28 13:27:33.466221 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:33.466134    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-trls9" podUID="d8b4592e-d228-473d-a23e-0d0b0faae458" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 28 13:27:43.466014 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:43.465978    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-trls9" podUID="d8b4592e-d228-473d-a23e-0d0b0faae458" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 28 13:27:53.465731 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:27:53.465693    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-trls9" podUID="d8b4592e-d228-473d-a23e-0d0b0faae458" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 28 13:28:03.466023 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:03.465987    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-trls9" podUID="d8b4592e-d228-473d-a23e-0d0b0faae458" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 28 13:28:13.466158 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:13.466121    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-trls9" podUID="d8b4592e-d228-473d-a23e-0d0b0faae458" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 28 13:28:23.465966 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:23.465929    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-trls9" podUID="d8b4592e-d228-473d-a23e-0d0b0faae458" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 28 13:28:24.302067 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:24.302040    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-trls9"
Jul 28 13:28:40.127520 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:40.127488    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-trls9"]
Jul 28 13:28:40.127925 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:40.127805    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-trls9" podUID="d8b4592e-d228-473d-a23e-0d0b0faae458" containerName="kserve-container" containerID="cri-o://0934724a6d8b41ff932292fa2b78d96294ddcc640bf8b8a2b0764ab34a743fd7" gracePeriod=30
Jul 28 13:28:40.127925 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:40.127831    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-trls9" podUID="d8b4592e-d228-473d-a23e-0d0b0faae458" containerName="kube-rbac-proxy" containerID="cri-o://68e93cd137ceb6bdc05a857e42132d9e72cebf1cfab4f2700c764dbc3bfbd947" gracePeriod=30
Jul 28 13:28:40.165925 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:40.165895    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db"]
Jul 28 13:28:40.169199 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:40.169183    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db"
Jul 28 13:28:40.171640 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:40.171622    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Jul 28 13:28:40.179522 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:40.179501    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db"]
Jul 28 13:28:40.254495 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:40.254473    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2ldf\" (UniqueName: \"kubernetes.io/projected/f3a51965-6ff6-426e-8314-e83524dc1128-kube-api-access-b2ldf\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db\" (UID: \"f3a51965-6ff6-426e-8314-e83524dc1128\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db"
Jul 28 13:28:40.254574 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:40.254501    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3a51965-6ff6-426e-8314-e83524dc1128-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db\" (UID: \"f3a51965-6ff6-426e-8314-e83524dc1128\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db"
Jul 28 13:28:40.254574 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:40.254529    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f3a51965-6ff6-426e-8314-e83524dc1128-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db\" (UID: \"f3a51965-6ff6-426e-8314-e83524dc1128\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db"
Jul 28 13:28:40.254704 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:40.254651    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3a51965-6ff6-426e-8314-e83524dc1128-proxy-tls\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db\" (UID: \"f3a51965-6ff6-426e-8314-e83524dc1128\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db"
Jul 28 13:28:40.254771 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:40.254719    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f3a51965-6ff6-426e-8314-e83524dc1128-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db\" (UID: \"f3a51965-6ff6-426e-8314-e83524dc1128\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db"
Jul 28 13:28:40.355223 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:40.355192    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3a51965-6ff6-426e-8314-e83524dc1128-proxy-tls\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db\" (UID: \"f3a51965-6ff6-426e-8314-e83524dc1128\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db"
Jul 28 13:28:40.355223 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:40.355228    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f3a51965-6ff6-426e-8314-e83524dc1128-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db\" (UID: \"f3a51965-6ff6-426e-8314-e83524dc1128\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db"
Jul 28 13:28:40.355393 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:40.355265    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2ldf\" (UniqueName: \"kubernetes.io/projected/f3a51965-6ff6-426e-8314-e83524dc1128-kube-api-access-b2ldf\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db\" (UID: \"f3a51965-6ff6-426e-8314-e83524dc1128\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db"
Jul 28 13:28:40.355393 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:40.355281    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3a51965-6ff6-426e-8314-e83524dc1128-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db\" (UID: \"f3a51965-6ff6-426e-8314-e83524dc1128\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db"
Jul 28 13:28:40.355393 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:40.355312    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f3a51965-6ff6-426e-8314-e83524dc1128-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db\" (UID: \"f3a51965-6ff6-426e-8314-e83524dc1128\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db"
Jul 28 13:28:40.355800 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:40.355769    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3a51965-6ff6-426e-8314-e83524dc1128-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db\" (UID: \"f3a51965-6ff6-426e-8314-e83524dc1128\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db"
Jul 28 13:28:40.356016 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:40.355996    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f3a51965-6ff6-426e-8314-e83524dc1128-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db\" (UID: \"f3a51965-6ff6-426e-8314-e83524dc1128\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db"
Jul 28 13:28:40.356089 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:40.356029    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f3a51965-6ff6-426e-8314-e83524dc1128-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db\" (UID: \"f3a51965-6ff6-426e-8314-e83524dc1128\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db"
Jul 28 13:28:40.357526 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:40.357508    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3a51965-6ff6-426e-8314-e83524dc1128-proxy-tls\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db\" (UID: \"f3a51965-6ff6-426e-8314-e83524dc1128\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db"
Jul 28 13:28:40.363917 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:40.363896    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2ldf\" (UniqueName: \"kubernetes.io/projected/f3a51965-6ff6-426e-8314-e83524dc1128-kube-api-access-b2ldf\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db\" (UID: \"f3a51965-6ff6-426e-8314-e83524dc1128\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db"
Jul 28 13:28:40.480060 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:40.480033    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db"
Jul 28 13:28:40.604133 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:40.604107    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db"]
Jul 28 13:28:40.605775 ip-10-0-138-158 kubenswrapper[2574]: W0728 13:28:40.605748    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3a51965_6ff6_426e_8314_e83524dc1128.slice/crio-773eeb22fbc9ff365c3536c0431e31cb9b18d490639ec2d86457afd80b838129 WatchSource:0}: Error finding container 773eeb22fbc9ff365c3536c0431e31cb9b18d490639ec2d86457afd80b838129: Status 404 returned error can't find the container with id 773eeb22fbc9ff365c3536c0431e31cb9b18d490639ec2d86457afd80b838129
Jul 28 13:28:40.821457 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:40.821354    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db" event={"ID":"f3a51965-6ff6-426e-8314-e83524dc1128","Type":"ContainerStarted","Data":"bb3ba48a1eae587204077a191b432bdc605d1e14137b7e88c36de836b51ba10b"}
Jul 28 13:28:40.821457 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:40.821429    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db" event={"ID":"f3a51965-6ff6-426e-8314-e83524dc1128","Type":"ContainerStarted","Data":"773eeb22fbc9ff365c3536c0431e31cb9b18d490639ec2d86457afd80b838129"}
Jul 28 13:28:40.823504 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:40.823476    2574 generic.go:356] "Generic (PLEG): container finished" podID="d8b4592e-d228-473d-a23e-0d0b0faae458" containerID="68e93cd137ceb6bdc05a857e42132d9e72cebf1cfab4f2700c764dbc3bfbd947" exitCode=2
Jul 28 13:28:40.823616 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:40.823528    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-trls9" event={"ID":"d8b4592e-d228-473d-a23e-0d0b0faae458","Type":"ContainerDied","Data":"68e93cd137ceb6bdc05a857e42132d9e72cebf1cfab4f2700c764dbc3bfbd947"}
Jul 28 13:28:43.456708 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:43.456652    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-trls9" podUID="d8b4592e-d228-473d-a23e-0d0b0faae458" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Jul 28 13:28:44.170067 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:44.170046    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-trls9"
Jul 28 13:28:44.284193 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:44.284128    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d8b4592e-d228-473d-a23e-0d0b0faae458-cabundle-cert\") pod \"d8b4592e-d228-473d-a23e-0d0b0faae458\" (UID: \"d8b4592e-d228-473d-a23e-0d0b0faae458\") "
Jul 28 13:28:44.284193 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:44.284164    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d8b4592e-d228-473d-a23e-0d0b0faae458-kserve-provision-location\") pod \"d8b4592e-d228-473d-a23e-0d0b0faae458\" (UID: \"d8b4592e-d228-473d-a23e-0d0b0faae458\") "
Jul 28 13:28:44.284461 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:44.284199    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d8b4592e-d228-473d-a23e-0d0b0faae458-kube-rbac-proxy-sar-config\") pod \"d8b4592e-d228-473d-a23e-0d0b0faae458\" (UID: \"d8b4592e-d228-473d-a23e-0d0b0faae458\") "
Jul 28 13:28:44.284461 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:44.284287    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5vjgz\" (UniqueName: \"kubernetes.io/projected/d8b4592e-d228-473d-a23e-0d0b0faae458-kube-api-access-5vjgz\") pod \"d8b4592e-d228-473d-a23e-0d0b0faae458\" (UID: \"d8b4592e-d228-473d-a23e-0d0b0faae458\") "
Jul 28 13:28:44.284461 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:44.284319    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8b4592e-d228-473d-a23e-0d0b0faae458-proxy-tls\") pod \"d8b4592e-d228-473d-a23e-0d0b0faae458\" (UID: \"d8b4592e-d228-473d-a23e-0d0b0faae458\") "
Jul 28 13:28:44.284630 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:44.284484    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8b4592e-d228-473d-a23e-0d0b0faae458-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d8b4592e-d228-473d-a23e-0d0b0faae458" (UID: "d8b4592e-d228-473d-a23e-0d0b0faae458"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:28:44.284630 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:44.284522    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8b4592e-d228-473d-a23e-0d0b0faae458-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d8b4592e-d228-473d-a23e-0d0b0faae458" (UID: "d8b4592e-d228-473d-a23e-0d0b0faae458"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:28:44.284630 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:44.284555    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8b4592e-d228-473d-a23e-0d0b0faae458-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d8b4592e-d228-473d-a23e-0d0b0faae458" (UID: "d8b4592e-d228-473d-a23e-0d0b0faae458"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:28:44.284630 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:44.284577    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d8b4592e-d228-473d-a23e-0d0b0faae458-cabundle-cert\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:28:44.284630 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:44.284590    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d8b4592e-d228-473d-a23e-0d0b0faae458-kserve-provision-location\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:28:44.286370 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:44.286349    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8b4592e-d228-473d-a23e-0d0b0faae458-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d8b4592e-d228-473d-a23e-0d0b0faae458" (UID: "d8b4592e-d228-473d-a23e-0d0b0faae458"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:28:44.286481 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:44.286461    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8b4592e-d228-473d-a23e-0d0b0faae458-kube-api-access-5vjgz" (OuterVolumeSpecName: "kube-api-access-5vjgz") pod "d8b4592e-d228-473d-a23e-0d0b0faae458" (UID: "d8b4592e-d228-473d-a23e-0d0b0faae458"). InnerVolumeSpecName "kube-api-access-5vjgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:28:44.385456 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:44.385430    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vjgz\" (UniqueName: \"kubernetes.io/projected/d8b4592e-d228-473d-a23e-0d0b0faae458-kube-api-access-5vjgz\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:28:44.385456 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:44.385455    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8b4592e-d228-473d-a23e-0d0b0faae458-proxy-tls\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:28:44.385622 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:44.385466    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d8b4592e-d228-473d-a23e-0d0b0faae458-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:28:44.842131 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:44.842096    2574 generic.go:356] "Generic (PLEG): container finished" podID="f3a51965-6ff6-426e-8314-e83524dc1128" containerID="bb3ba48a1eae587204077a191b432bdc605d1e14137b7e88c36de836b51ba10b" exitCode=0
Jul 28 13:28:44.842594 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:44.842168    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db" event={"ID":"f3a51965-6ff6-426e-8314-e83524dc1128","Type":"ContainerDied","Data":"bb3ba48a1eae587204077a191b432bdc605d1e14137b7e88c36de836b51ba10b"}
Jul 28 13:28:44.844184 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:44.844163    2574 generic.go:356] "Generic (PLEG): container finished" podID="d8b4592e-d228-473d-a23e-0d0b0faae458" containerID="0934724a6d8b41ff932292fa2b78d96294ddcc640bf8b8a2b0764ab34a743fd7" exitCode=0
Jul 28 13:28:44.844299 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:44.844198    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-trls9" event={"ID":"d8b4592e-d228-473d-a23e-0d0b0faae458","Type":"ContainerDied","Data":"0934724a6d8b41ff932292fa2b78d96294ddcc640bf8b8a2b0764ab34a743fd7"}
Jul 28 13:28:44.844299 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:44.844237    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-trls9" event={"ID":"d8b4592e-d228-473d-a23e-0d0b0faae458","Type":"ContainerDied","Data":"f57bd03689a7e60cfa58b646a9f78f6c15d1271e0eca82bb480adfc82b429e25"}
Jul 28 13:28:44.844299 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:44.844241    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-trls9"
Jul 28 13:28:44.844299 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:44.844264    2574 scope.go:117] "RemoveContainer" containerID="68e93cd137ceb6bdc05a857e42132d9e72cebf1cfab4f2700c764dbc3bfbd947"
Jul 28 13:28:44.854441 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:44.854423    2574 scope.go:117] "RemoveContainer" containerID="0934724a6d8b41ff932292fa2b78d96294ddcc640bf8b8a2b0764ab34a743fd7"
Jul 28 13:28:44.862026 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:44.861987    2574 scope.go:117] "RemoveContainer" containerID="b0206ea8426862e008c89b60b336bdde879b059910d5a75bb33c07453645da84"
Jul 28 13:28:44.875618 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:44.875593    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-trls9"]
Jul 28 13:28:44.875821 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:44.875803    2574 scope.go:117] "RemoveContainer" containerID="68e93cd137ceb6bdc05a857e42132d9e72cebf1cfab4f2700c764dbc3bfbd947"
Jul 28 13:28:44.876092 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:28:44.876074    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68e93cd137ceb6bdc05a857e42132d9e72cebf1cfab4f2700c764dbc3bfbd947\": container with ID starting with 68e93cd137ceb6bdc05a857e42132d9e72cebf1cfab4f2700c764dbc3bfbd947 not found: ID does not exist" containerID="68e93cd137ceb6bdc05a857e42132d9e72cebf1cfab4f2700c764dbc3bfbd947"
Jul 28 13:28:44.876163 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:44.876102    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68e93cd137ceb6bdc05a857e42132d9e72cebf1cfab4f2700c764dbc3bfbd947"} err="failed to get container status \"68e93cd137ceb6bdc05a857e42132d9e72cebf1cfab4f2700c764dbc3bfbd947\": rpc error: code = NotFound desc = could not find container \"68e93cd137ceb6bdc05a857e42132d9e72cebf1cfab4f2700c764dbc3bfbd947\": container with ID starting with 68e93cd137ceb6bdc05a857e42132d9e72cebf1cfab4f2700c764dbc3bfbd947 not found: ID does not exist"
Jul 28 13:28:44.876163 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:44.876123    2574 scope.go:117] "RemoveContainer" containerID="0934724a6d8b41ff932292fa2b78d96294ddcc640bf8b8a2b0764ab34a743fd7"
Jul 28 13:28:44.876386 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:28:44.876371    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0934724a6d8b41ff932292fa2b78d96294ddcc640bf8b8a2b0764ab34a743fd7\": container with ID starting with 0934724a6d8b41ff932292fa2b78d96294ddcc640bf8b8a2b0764ab34a743fd7 not found: ID does not exist" containerID="0934724a6d8b41ff932292fa2b78d96294ddcc640bf8b8a2b0764ab34a743fd7"
Jul 28 13:28:44.876452 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:44.876395    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0934724a6d8b41ff932292fa2b78d96294ddcc640bf8b8a2b0764ab34a743fd7"} err="failed to get container status \"0934724a6d8b41ff932292fa2b78d96294ddcc640bf8b8a2b0764ab34a743fd7\": rpc error: code = NotFound desc = could not find container \"0934724a6d8b41ff932292fa2b78d96294ddcc640bf8b8a2b0764ab34a743fd7\": container with ID starting with 0934724a6d8b41ff932292fa2b78d96294ddcc640bf8b8a2b0764ab34a743fd7 not found: ID does not exist"
Jul 28 13:28:44.876452 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:44.876413    2574 scope.go:117] "RemoveContainer" containerID="b0206ea8426862e008c89b60b336bdde879b059910d5a75bb33c07453645da84"
Jul 28 13:28:44.876629 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:28:44.876608    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0206ea8426862e008c89b60b336bdde879b059910d5a75bb33c07453645da84\": container with ID starting with b0206ea8426862e008c89b60b336bdde879b059910d5a75bb33c07453645da84 not found: ID does not exist" containerID="b0206ea8426862e008c89b60b336bdde879b059910d5a75bb33c07453645da84"
Jul 28 13:28:44.876737 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:44.876631    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0206ea8426862e008c89b60b336bdde879b059910d5a75bb33c07453645da84"} err="failed to get container status \"b0206ea8426862e008c89b60b336bdde879b059910d5a75bb33c07453645da84\": rpc error: code = NotFound desc = could not find container \"b0206ea8426862e008c89b60b336bdde879b059910d5a75bb33c07453645da84\": container with ID starting with b0206ea8426862e008c89b60b336bdde879b059910d5a75bb33c07453645da84 not found: ID does not exist"
Jul 28 13:28:44.883407 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:44.883389    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-trls9"]
Jul 28 13:28:45.852241 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:45.852212    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db" event={"ID":"f3a51965-6ff6-426e-8314-e83524dc1128","Type":"ContainerStarted","Data":"5b9e22e367f474ffb5836c4b5b95ea2226cdbffdc3395e662668840cc59bfcc9"}
Jul 28 13:28:45.852722 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:45.852277    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db" event={"ID":"f3a51965-6ff6-426e-8314-e83524dc1128","Type":"ContainerStarted","Data":"f2fc4ebe209ffa4f45da07019f149fabc8b597d5b4c4c4540fad09c58a6b9de1"}
Jul 28 13:28:45.852722 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:45.852491    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db"
Jul 28 13:28:45.873261 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:45.873217    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db" podStartSLOduration=2.145275144 podStartE2EDuration="5.873204083s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.727928939s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:28:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:28:45.871483922 +0000 UTC m=+2138.076870527" watchObservedRunningTime="2026-07-28 13:28:45.873204083 +0000 UTC m=+2138.078590695"
Jul 28 13:28:46.302478 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:46.302448    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8b4592e-d228-473d-a23e-0d0b0faae458" path="/var/lib/kubelet/pods/d8b4592e-d228-473d-a23e-0d0b0faae458/volumes"
Jul 28 13:28:46.859107 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:46.859074    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db"
Jul 28 13:28:46.860420 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:46.860394    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db" podUID="f3a51965-6ff6-426e-8314-e83524dc1128" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 28 13:28:47.864063 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:47.864028    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db" podUID="f3a51965-6ff6-426e-8314-e83524dc1128" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 28 13:28:52.868616 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:52.868585    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db"
Jul 28 13:28:52.869110 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:28:52.869084    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db" podUID="f3a51965-6ff6-426e-8314-e83524dc1128" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 28 13:29:02.869252 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:29:02.869167    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db" podUID="f3a51965-6ff6-426e-8314-e83524dc1128" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 28 13:29:12.869195 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:29:12.869158    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db" podUID="f3a51965-6ff6-426e-8314-e83524dc1128" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 28 13:29:22.869624 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:29:22.869588    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db" podUID="f3a51965-6ff6-426e-8314-e83524dc1128" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 28 13:29:32.869914 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:29:32.869873    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db" podUID="f3a51965-6ff6-426e-8314-e83524dc1128" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 28 13:29:42.869171 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:29:42.869130    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db" podUID="f3a51965-6ff6-426e-8314-e83524dc1128" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 28 13:29:52.869789 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:29:52.869758    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db"
Jul 28 13:30:00.273470 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:00.273438    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db"]
Jul 28 13:30:00.274119 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:00.273742    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db" podUID="f3a51965-6ff6-426e-8314-e83524dc1128" containerName="kserve-container" containerID="cri-o://f2fc4ebe209ffa4f45da07019f149fabc8b597d5b4c4c4540fad09c58a6b9de1" gracePeriod=30
Jul 28 13:30:00.274119 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:00.273793    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db" podUID="f3a51965-6ff6-426e-8314-e83524dc1128" containerName="kube-rbac-proxy" containerID="cri-o://5b9e22e367f474ffb5836c4b5b95ea2226cdbffdc3395e662668840cc59bfcc9" gracePeriod=30
Jul 28 13:30:01.185237 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:01.185200    2574 generic.go:356] "Generic (PLEG): container finished" podID="f3a51965-6ff6-426e-8314-e83524dc1128" containerID="5b9e22e367f474ffb5836c4b5b95ea2226cdbffdc3395e662668840cc59bfcc9" exitCode=2
Jul 28 13:30:01.185388 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:01.185268    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db" event={"ID":"f3a51965-6ff6-426e-8314-e83524dc1128","Type":"ContainerDied","Data":"5b9e22e367f474ffb5836c4b5b95ea2226cdbffdc3395e662668840cc59bfcc9"}
Jul 28 13:30:02.864900 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:02.864864    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db" podUID="f3a51965-6ff6-426e-8314-e83524dc1128" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.45:8643/healthz\": dial tcp 10.132.0.45:8643: connect: connection refused"
Jul 28 13:30:02.869343 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:02.869321    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db" podUID="f3a51965-6ff6-426e-8314-e83524dc1128" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 28 13:30:04.201052 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:04.201017    2574 generic.go:356] "Generic (PLEG): container finished" podID="f3a51965-6ff6-426e-8314-e83524dc1128" containerID="f2fc4ebe209ffa4f45da07019f149fabc8b597d5b4c4c4540fad09c58a6b9de1" exitCode=0
Jul 28 13:30:04.201349 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:04.201086    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db" event={"ID":"f3a51965-6ff6-426e-8314-e83524dc1128","Type":"ContainerDied","Data":"f2fc4ebe209ffa4f45da07019f149fabc8b597d5b4c4c4540fad09c58a6b9de1"}
Jul 28 13:30:04.319946 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:04.319922    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db"
Jul 28 13:30:04.472792 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:04.472767    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b2ldf\" (UniqueName: \"kubernetes.io/projected/f3a51965-6ff6-426e-8314-e83524dc1128-kube-api-access-b2ldf\") pod \"f3a51965-6ff6-426e-8314-e83524dc1128\" (UID: \"f3a51965-6ff6-426e-8314-e83524dc1128\") "
Jul 28 13:30:04.472920 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:04.472820    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f3a51965-6ff6-426e-8314-e83524dc1128-kube-rbac-proxy-sar-config\") pod \"f3a51965-6ff6-426e-8314-e83524dc1128\" (UID: \"f3a51965-6ff6-426e-8314-e83524dc1128\") "
Jul 28 13:30:04.472920 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:04.472847    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3a51965-6ff6-426e-8314-e83524dc1128-proxy-tls\") pod \"f3a51965-6ff6-426e-8314-e83524dc1128\" (UID: \"f3a51965-6ff6-426e-8314-e83524dc1128\") "
Jul 28 13:30:04.472920 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:04.472874    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f3a51965-6ff6-426e-8314-e83524dc1128-cabundle-cert\") pod \"f3a51965-6ff6-426e-8314-e83524dc1128\" (UID: \"f3a51965-6ff6-426e-8314-e83524dc1128\") "
Jul 28 13:30:04.473065 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:04.472923    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3a51965-6ff6-426e-8314-e83524dc1128-kserve-provision-location\") pod \"f3a51965-6ff6-426e-8314-e83524dc1128\" (UID: \"f3a51965-6ff6-426e-8314-e83524dc1128\") "
Jul 28 13:30:04.473247 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:04.473216    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3a51965-6ff6-426e-8314-e83524dc1128-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f3a51965-6ff6-426e-8314-e83524dc1128" (UID: "f3a51965-6ff6-426e-8314-e83524dc1128"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:30:04.473312 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:04.473244    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3a51965-6ff6-426e-8314-e83524dc1128-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f3a51965-6ff6-426e-8314-e83524dc1128" (UID: "f3a51965-6ff6-426e-8314-e83524dc1128"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:30:04.473312 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:04.473220    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3a51965-6ff6-426e-8314-e83524dc1128-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f3a51965-6ff6-426e-8314-e83524dc1128" (UID: "f3a51965-6ff6-426e-8314-e83524dc1128"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:30:04.474975 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:04.474953    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3a51965-6ff6-426e-8314-e83524dc1128-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f3a51965-6ff6-426e-8314-e83524dc1128" (UID: "f3a51965-6ff6-426e-8314-e83524dc1128"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:30:04.475042 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:04.475029    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3a51965-6ff6-426e-8314-e83524dc1128-kube-api-access-b2ldf" (OuterVolumeSpecName: "kube-api-access-b2ldf") pod "f3a51965-6ff6-426e-8314-e83524dc1128" (UID: "f3a51965-6ff6-426e-8314-e83524dc1128"). InnerVolumeSpecName "kube-api-access-b2ldf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:30:04.573901 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:04.573874    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3a51965-6ff6-426e-8314-e83524dc1128-kserve-provision-location\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:30:04.573901 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:04.573897    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2ldf\" (UniqueName: \"kubernetes.io/projected/f3a51965-6ff6-426e-8314-e83524dc1128-kube-api-access-b2ldf\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:30:04.573901 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:04.573907    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f3a51965-6ff6-426e-8314-e83524dc1128-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:30:04.574083 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:04.573916    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3a51965-6ff6-426e-8314-e83524dc1128-proxy-tls\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:30:04.574083 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:04.573925    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f3a51965-6ff6-426e-8314-e83524dc1128-cabundle-cert\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:30:05.207194 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.207159    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db" event={"ID":"f3a51965-6ff6-426e-8314-e83524dc1128","Type":"ContainerDied","Data":"773eeb22fbc9ff365c3536c0431e31cb9b18d490639ec2d86457afd80b838129"}
Jul 28 13:30:05.207586 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.207204    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db"
Jul 28 13:30:05.207586 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.207260    2574 scope.go:117] "RemoveContainer" containerID="5b9e22e367f474ffb5836c4b5b95ea2226cdbffdc3395e662668840cc59bfcc9"
Jul 28 13:30:05.216876 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.216856    2574 scope.go:117] "RemoveContainer" containerID="f2fc4ebe209ffa4f45da07019f149fabc8b597d5b4c4c4540fad09c58a6b9de1"
Jul 28 13:30:05.225754 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.225735    2574 scope.go:117] "RemoveContainer" containerID="bb3ba48a1eae587204077a191b432bdc605d1e14137b7e88c36de836b51ba10b"
Jul 28 13:30:05.230380 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.230360    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db"]
Jul 28 13:30:05.233547 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.233525    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-nb9db"]
Jul 28 13:30:05.475836 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.475772    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-4kd6g"]
Jul 28 13:30:05.476088 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.476076    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3a51965-6ff6-426e-8314-e83524dc1128" containerName="kube-rbac-proxy"
Jul 28 13:30:05.476131 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.476090    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3a51965-6ff6-426e-8314-e83524dc1128" containerName="kube-rbac-proxy"
Jul 28 13:30:05.476131 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.476103    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8b4592e-d228-473d-a23e-0d0b0faae458" containerName="storage-initializer"
Jul 28 13:30:05.476131 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.476108    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8b4592e-d228-473d-a23e-0d0b0faae458" containerName="storage-initializer"
Jul 28 13:30:05.476131 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.476116    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3a51965-6ff6-426e-8314-e83524dc1128" containerName="storage-initializer"
Jul 28 13:30:05.476131 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.476122    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3a51965-6ff6-426e-8314-e83524dc1128" containerName="storage-initializer"
Jul 28 13:30:05.476131 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.476133    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8b4592e-d228-473d-a23e-0d0b0faae458" containerName="kube-rbac-proxy"
Jul 28 13:30:05.476312 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.476138    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8b4592e-d228-473d-a23e-0d0b0faae458" containerName="kube-rbac-proxy"
Jul 28 13:30:05.476312 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.476149    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8b4592e-d228-473d-a23e-0d0b0faae458" containerName="kserve-container"
Jul 28 13:30:05.476312 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.476154    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8b4592e-d228-473d-a23e-0d0b0faae458" containerName="kserve-container"
Jul 28 13:30:05.476312 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.476163    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3a51965-6ff6-426e-8314-e83524dc1128" containerName="kserve-container"
Jul 28 13:30:05.476312 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.476168    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3a51965-6ff6-426e-8314-e83524dc1128" containerName="kserve-container"
Jul 28 13:30:05.476312 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.476214    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8b4592e-d228-473d-a23e-0d0b0faae458" containerName="kube-rbac-proxy"
Jul 28 13:30:05.476312 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.476222    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3a51965-6ff6-426e-8314-e83524dc1128" containerName="kserve-container"
Jul 28 13:30:05.476312 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.476227    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3a51965-6ff6-426e-8314-e83524dc1128" containerName="kube-rbac-proxy"
Jul 28 13:30:05.476312 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.476235    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8b4592e-d228-473d-a23e-0d0b0faae458" containerName="kserve-container"
Jul 28 13:30:05.480863 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.480844    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-4kd6g"
Jul 28 13:30:05.483449 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.483423    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Jul 28 13:30:05.483449 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.483445    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rxfwd\""
Jul 28 13:30:05.483633 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.483431    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 13:30:05.483767 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.483747    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:30:05.483834 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.483796    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:30:05.483945 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.483925    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Jul 28 13:30:05.483992 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.483940    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:30:05.489199 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.489178    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-4kd6g"]
Jul 28 13:30:05.510156 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.510132    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-swgfs"]
Jul 28 13:30:05.513536 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.513520    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-swgfs"
Jul 28 13:30:05.516067 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.516052    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Jul 28 13:30:05.524348 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.524330    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-swgfs"]
Jul 28 13:30:05.581556 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.581536    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d12296bd-99ec-4ab1-9ac1-b43aba975bc5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-86dc9fdb65-4kd6g\" (UID: \"d12296bd-99ec-4ab1-9ac1-b43aba975bc5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-4kd6g"
Jul 28 13:30:05.581670 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.581588    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d12296bd-99ec-4ab1-9ac1-b43aba975bc5-cabundle-cert\") pod \"isvc-canary-stop-predictor-86dc9fdb65-4kd6g\" (UID: \"d12296bd-99ec-4ab1-9ac1-b43aba975bc5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-4kd6g"
Jul 28 13:30:05.581670 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.581652    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d12296bd-99ec-4ab1-9ac1-b43aba975bc5-kserve-provision-location\") pod \"isvc-canary-stop-predictor-86dc9fdb65-4kd6g\" (UID: \"d12296bd-99ec-4ab1-9ac1-b43aba975bc5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-4kd6g"
Jul 28 13:30:05.581854 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.581705    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwvvm\" (UniqueName: \"kubernetes.io/projected/d12296bd-99ec-4ab1-9ac1-b43aba975bc5-kube-api-access-gwvvm\") pod \"isvc-canary-stop-predictor-86dc9fdb65-4kd6g\" (UID: \"d12296bd-99ec-4ab1-9ac1-b43aba975bc5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-4kd6g"
Jul 28 13:30:05.581854 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.581763    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d12296bd-99ec-4ab1-9ac1-b43aba975bc5-proxy-tls\") pod \"isvc-canary-stop-predictor-86dc9fdb65-4kd6g\" (UID: \"d12296bd-99ec-4ab1-9ac1-b43aba975bc5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-4kd6g"
Jul 28 13:30:05.682372 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.682354    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d12296bd-99ec-4ab1-9ac1-b43aba975bc5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-86dc9fdb65-4kd6g\" (UID: \"d12296bd-99ec-4ab1-9ac1-b43aba975bc5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-4kd6g"
Jul 28 13:30:05.682482 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.682401    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d12296bd-99ec-4ab1-9ac1-b43aba975bc5-cabundle-cert\") pod \"isvc-canary-stop-predictor-86dc9fdb65-4kd6g\" (UID: \"d12296bd-99ec-4ab1-9ac1-b43aba975bc5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-4kd6g"
Jul 28 13:30:05.682482 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.682433    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbdf5\" (UniqueName: \"kubernetes.io/projected/96816748-5118-48f8-81fa-bd7459698bc3-kube-api-access-jbdf5\") pod \"isvc-canary-stop-v2-predictor-77f778d864-swgfs\" (UID: \"96816748-5118-48f8-81fa-bd7459698bc3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-swgfs"
Jul 28 13:30:05.682482 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.682459    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96816748-5118-48f8-81fa-bd7459698bc3-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-77f778d864-swgfs\" (UID: \"96816748-5118-48f8-81fa-bd7459698bc3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-swgfs"
Jul 28 13:30:05.682624 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.682499    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d12296bd-99ec-4ab1-9ac1-b43aba975bc5-kserve-provision-location\") pod \"isvc-canary-stop-predictor-86dc9fdb65-4kd6g\" (UID: \"d12296bd-99ec-4ab1-9ac1-b43aba975bc5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-4kd6g"
Jul 28 13:30:05.682624 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.682525    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwvvm\" (UniqueName: \"kubernetes.io/projected/d12296bd-99ec-4ab1-9ac1-b43aba975bc5-kube-api-access-gwvvm\") pod \"isvc-canary-stop-predictor-86dc9fdb65-4kd6g\" (UID: \"d12296bd-99ec-4ab1-9ac1-b43aba975bc5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-4kd6g"
Jul 28 13:30:05.682624 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.682561    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96816748-5118-48f8-81fa-bd7459698bc3-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-77f778d864-swgfs\" (UID: \"96816748-5118-48f8-81fa-bd7459698bc3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-swgfs"
Jul 28 13:30:05.682801 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.682741    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96816748-5118-48f8-81fa-bd7459698bc3-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-77f778d864-swgfs\" (UID: \"96816748-5118-48f8-81fa-bd7459698bc3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-swgfs"
Jul 28 13:30:05.682859 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.682798    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d12296bd-99ec-4ab1-9ac1-b43aba975bc5-proxy-tls\") pod \"isvc-canary-stop-predictor-86dc9fdb65-4kd6g\" (UID: \"d12296bd-99ec-4ab1-9ac1-b43aba975bc5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-4kd6g"
Jul 28 13:30:05.682914 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.682847    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/96816748-5118-48f8-81fa-bd7459698bc3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-77f778d864-swgfs\" (UID: \"96816748-5118-48f8-81fa-bd7459698bc3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-swgfs"
Jul 28 13:30:05.682914 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.682872    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d12296bd-99ec-4ab1-9ac1-b43aba975bc5-kserve-provision-location\") pod \"isvc-canary-stop-predictor-86dc9fdb65-4kd6g\" (UID: \"d12296bd-99ec-4ab1-9ac1-b43aba975bc5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-4kd6g"
Jul 28 13:30:05.683100 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.683081    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d12296bd-99ec-4ab1-9ac1-b43aba975bc5-cabundle-cert\") pod \"isvc-canary-stop-predictor-86dc9fdb65-4kd6g\" (UID: \"d12296bd-99ec-4ab1-9ac1-b43aba975bc5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-4kd6g"
Jul 28 13:30:05.683163 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.683112    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d12296bd-99ec-4ab1-9ac1-b43aba975bc5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-86dc9fdb65-4kd6g\" (UID: \"d12296bd-99ec-4ab1-9ac1-b43aba975bc5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-4kd6g"
Jul 28 13:30:05.685063 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.685043    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d12296bd-99ec-4ab1-9ac1-b43aba975bc5-proxy-tls\") pod \"isvc-canary-stop-predictor-86dc9fdb65-4kd6g\" (UID: \"d12296bd-99ec-4ab1-9ac1-b43aba975bc5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-4kd6g"
Jul 28 13:30:05.691982 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.691965    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwvvm\" (UniqueName: \"kubernetes.io/projected/d12296bd-99ec-4ab1-9ac1-b43aba975bc5-kube-api-access-gwvvm\") pod \"isvc-canary-stop-predictor-86dc9fdb65-4kd6g\" (UID: \"d12296bd-99ec-4ab1-9ac1-b43aba975bc5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-4kd6g"
Jul 28 13:30:05.783564 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.783517    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbdf5\" (UniqueName: \"kubernetes.io/projected/96816748-5118-48f8-81fa-bd7459698bc3-kube-api-access-jbdf5\") pod \"isvc-canary-stop-v2-predictor-77f778d864-swgfs\" (UID: \"96816748-5118-48f8-81fa-bd7459698bc3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-swgfs"
Jul 28 13:30:05.783564 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.783543    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96816748-5118-48f8-81fa-bd7459698bc3-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-77f778d864-swgfs\" (UID: \"96816748-5118-48f8-81fa-bd7459698bc3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-swgfs"
Jul 28 13:30:05.783677 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.783567    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96816748-5118-48f8-81fa-bd7459698bc3-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-77f778d864-swgfs\" (UID: \"96816748-5118-48f8-81fa-bd7459698bc3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-swgfs"
Jul 28 13:30:05.783747 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.783720    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96816748-5118-48f8-81fa-bd7459698bc3-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-77f778d864-swgfs\" (UID: \"96816748-5118-48f8-81fa-bd7459698bc3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-swgfs"
Jul 28 13:30:05.783800 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.783782    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/96816748-5118-48f8-81fa-bd7459698bc3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-77f778d864-swgfs\" (UID: \"96816748-5118-48f8-81fa-bd7459698bc3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-swgfs"
Jul 28 13:30:05.783883 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:30:05.783865    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-serving-cert: secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Jul 28 13:30:05.783921 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.783901    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96816748-5118-48f8-81fa-bd7459698bc3-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-77f778d864-swgfs\" (UID: \"96816748-5118-48f8-81fa-bd7459698bc3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-swgfs"
Jul 28 13:30:05.783962 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:30:05.783937    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96816748-5118-48f8-81fa-bd7459698bc3-proxy-tls podName:96816748-5118-48f8-81fa-bd7459698bc3 nodeName:}" failed. No retries permitted until 2026-07-28 13:30:06.283918273 +0000 UTC m=+2218.489304854 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/96816748-5118-48f8-81fa-bd7459698bc3-proxy-tls") pod "isvc-canary-stop-v2-predictor-77f778d864-swgfs" (UID: "96816748-5118-48f8-81fa-bd7459698bc3") : secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Jul 28 13:30:05.784231 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.784214    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96816748-5118-48f8-81fa-bd7459698bc3-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-77f778d864-swgfs\" (UID: \"96816748-5118-48f8-81fa-bd7459698bc3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-swgfs"
Jul 28 13:30:05.784276 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.784248    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/96816748-5118-48f8-81fa-bd7459698bc3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-77f778d864-swgfs\" (UID: \"96816748-5118-48f8-81fa-bd7459698bc3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-swgfs"
Jul 28 13:30:05.791614 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.791596    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-4kd6g"
Jul 28 13:30:05.793654 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.793638    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbdf5\" (UniqueName: \"kubernetes.io/projected/96816748-5118-48f8-81fa-bd7459698bc3-kube-api-access-jbdf5\") pod \"isvc-canary-stop-v2-predictor-77f778d864-swgfs\" (UID: \"96816748-5118-48f8-81fa-bd7459698bc3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-swgfs"
Jul 28 13:30:05.922582 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:05.922549    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-4kd6g"]
Jul 28 13:30:05.924786 ip-10-0-138-158 kubenswrapper[2574]: W0728 13:30:05.924756    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd12296bd_99ec_4ab1_9ac1_b43aba975bc5.slice/crio-7e88dfa30149acd2681ef2f1d61ffd491ae7a4e265ef21fef6e7a8a0814efd2e WatchSource:0}: Error finding container 7e88dfa30149acd2681ef2f1d61ffd491ae7a4e265ef21fef6e7a8a0814efd2e: Status 404 returned error can't find the container with id 7e88dfa30149acd2681ef2f1d61ffd491ae7a4e265ef21fef6e7a8a0814efd2e
Jul 28 13:30:06.214461 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:06.214424    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-4kd6g" event={"ID":"d12296bd-99ec-4ab1-9ac1-b43aba975bc5","Type":"ContainerStarted","Data":"88224f620f757e335667251426424f3f46af9480f81780ad87c0e4f9256f45b2"}
Jul 28 13:30:06.214920 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:06.214500    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-4kd6g" event={"ID":"d12296bd-99ec-4ab1-9ac1-b43aba975bc5","Type":"ContainerStarted","Data":"7e88dfa30149acd2681ef2f1d61ffd491ae7a4e265ef21fef6e7a8a0814efd2e"}
Jul 28 13:30:06.288331 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:06.288294    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96816748-5118-48f8-81fa-bd7459698bc3-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-77f778d864-swgfs\" (UID: \"96816748-5118-48f8-81fa-bd7459698bc3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-swgfs"
Jul 28 13:30:06.290561 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:06.290544    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96816748-5118-48f8-81fa-bd7459698bc3-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-77f778d864-swgfs\" (UID: \"96816748-5118-48f8-81fa-bd7459698bc3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-swgfs"
Jul 28 13:30:06.302274 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:06.302246    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3a51965-6ff6-426e-8314-e83524dc1128" path="/var/lib/kubelet/pods/f3a51965-6ff6-426e-8314-e83524dc1128/volumes"
Jul 28 13:30:06.422299 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:06.422267    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-swgfs"
Jul 28 13:30:06.555183 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:06.555161    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-swgfs"]
Jul 28 13:30:06.556708 ip-10-0-138-158 kubenswrapper[2574]: W0728 13:30:06.556672    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96816748_5118_48f8_81fa_bd7459698bc3.slice/crio-fab9d114b7d703b5f2f7f5d329f88631986d5d4eace91d596797fe17438dee8d WatchSource:0}: Error finding container fab9d114b7d703b5f2f7f5d329f88631986d5d4eace91d596797fe17438dee8d: Status 404 returned error can't find the container with id fab9d114b7d703b5f2f7f5d329f88631986d5d4eace91d596797fe17438dee8d
Jul 28 13:30:07.221743 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:07.221712    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-swgfs" event={"ID":"96816748-5118-48f8-81fa-bd7459698bc3","Type":"ContainerStarted","Data":"6a052e81ab5786e7df006dddde19851719eca86faa6f0d1948e9de841c1a0454"}
Jul 28 13:30:07.222209 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:07.221769    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-swgfs" event={"ID":"96816748-5118-48f8-81fa-bd7459698bc3","Type":"ContainerStarted","Data":"fab9d114b7d703b5f2f7f5d329f88631986d5d4eace91d596797fe17438dee8d"}
Jul 28 13:30:10.242479 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:10.242445    2574 generic.go:356] "Generic (PLEG): container finished" podID="d12296bd-99ec-4ab1-9ac1-b43aba975bc5" containerID="88224f620f757e335667251426424f3f46af9480f81780ad87c0e4f9256f45b2" exitCode=0
Jul 28 13:30:10.242910 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:10.242515    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-4kd6g" event={"ID":"d12296bd-99ec-4ab1-9ac1-b43aba975bc5","Type":"ContainerDied","Data":"88224f620f757e335667251426424f3f46af9480f81780ad87c0e4f9256f45b2"}
Jul 28 13:30:11.248265 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:11.248231    2574 generic.go:356] "Generic (PLEG): container finished" podID="96816748-5118-48f8-81fa-bd7459698bc3" containerID="6a052e81ab5786e7df006dddde19851719eca86faa6f0d1948e9de841c1a0454" exitCode=0
Jul 28 13:30:11.248716 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:11.248303    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-swgfs" event={"ID":"96816748-5118-48f8-81fa-bd7459698bc3","Type":"ContainerDied","Data":"6a052e81ab5786e7df006dddde19851719eca86faa6f0d1948e9de841c1a0454"}
Jul 28 13:30:11.250721 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:11.250671    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-4kd6g" event={"ID":"d12296bd-99ec-4ab1-9ac1-b43aba975bc5","Type":"ContainerStarted","Data":"dd26a9c09298f2582bea3de73ea1ff1e916328faea4fa418436d71b65d630d97"}
Jul 28 13:30:11.250818 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:11.250759    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-4kd6g" event={"ID":"d12296bd-99ec-4ab1-9ac1-b43aba975bc5","Type":"ContainerStarted","Data":"54dee6b00cc8fe58e0bc8b1b385df1883686daf10cac43a428d968c8bef919fe"}
Jul 28 13:30:11.251032 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:11.251014    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-4kd6g"
Jul 28 13:30:11.285717 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:11.285655    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-4kd6g" podStartSLOduration=2.524809724 podStartE2EDuration="6.28563988s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.760830156s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:30:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:30:11.284338825 +0000 UTC m=+2223.489725428" watchObservedRunningTime="2026-07-28 13:30:11.28563988 +0000 UTC m=+2223.491026551"
Jul 28 13:30:12.267109 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:12.267076    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-swgfs" event={"ID":"96816748-5118-48f8-81fa-bd7459698bc3","Type":"ContainerStarted","Data":"bec50e60cc6f7d8dbe0ca0fea7a7633635457d114ddab2f5143258a072cbfafa"}
Jul 28 13:30:12.267513 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:12.267365    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-4kd6g"
Jul 28 13:30:12.267513 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:12.267409    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-swgfs" event={"ID":"96816748-5118-48f8-81fa-bd7459698bc3","Type":"ContainerStarted","Data":"aaa78d430569c62305bf16378b263b9efc7e997d3282a087f253a6fa75307a41"}
Jul 28 13:30:12.267627 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:12.267609    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-swgfs"
Jul 28 13:30:12.268766 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:12.268740    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-4kd6g" podUID="d12296bd-99ec-4ab1-9ac1-b43aba975bc5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Jul 28 13:30:12.287082 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:12.287042    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-swgfs" podStartSLOduration=3.32099412 podStartE2EDuration="7.287030009s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.966035889s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:30:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:30:12.285617708 +0000 UTC m=+2224.491004308" watchObservedRunningTime="2026-07-28 13:30:12.287030009 +0000 UTC m=+2224.492416668"
Jul 28 13:30:13.272862 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:13.272827    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-4kd6g" podUID="d12296bd-99ec-4ab1-9ac1-b43aba975bc5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Jul 28 13:30:13.273290 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:13.273050    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-swgfs"
Jul 28 13:30:13.274338 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:13.274311    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-swgfs" podUID="96816748-5118-48f8-81fa-bd7459698bc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Jul 28 13:30:14.277289 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:14.277250    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-swgfs" podUID="96816748-5118-48f8-81fa-bd7459698bc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Jul 28 13:30:18.277272 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:18.277244    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-4kd6g"
Jul 28 13:30:18.277785 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:18.277759    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-4kd6g" podUID="d12296bd-99ec-4ab1-9ac1-b43aba975bc5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Jul 28 13:30:19.281850 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:19.281822    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-swgfs"
Jul 28 13:30:19.282374 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:19.282349    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-swgfs" podUID="96816748-5118-48f8-81fa-bd7459698bc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Jul 28 13:30:28.277674 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:28.277634    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-4kd6g" podUID="d12296bd-99ec-4ab1-9ac1-b43aba975bc5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Jul 28 13:30:29.282243 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:29.282207    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-swgfs" podUID="96816748-5118-48f8-81fa-bd7459698bc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Jul 28 13:30:38.278165 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:38.278084    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-4kd6g" podUID="d12296bd-99ec-4ab1-9ac1-b43aba975bc5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Jul 28 13:30:39.282966 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:39.282932    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-swgfs" podUID="96816748-5118-48f8-81fa-bd7459698bc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Jul 28 13:30:48.278079 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:48.278038    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-4kd6g" podUID="d12296bd-99ec-4ab1-9ac1-b43aba975bc5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Jul 28 13:30:49.282965 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:49.282925    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-swgfs" podUID="96816748-5118-48f8-81fa-bd7459698bc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Jul 28 13:30:58.277661 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:58.277619    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-4kd6g" podUID="d12296bd-99ec-4ab1-9ac1-b43aba975bc5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Jul 28 13:30:59.282469 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:30:59.282435    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-swgfs" podUID="96816748-5118-48f8-81fa-bd7459698bc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Jul 28 13:31:08.278383 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:08.278345    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-4kd6g" podUID="d12296bd-99ec-4ab1-9ac1-b43aba975bc5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Jul 28 13:31:09.282911 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:09.282878    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-swgfs" podUID="96816748-5118-48f8-81fa-bd7459698bc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Jul 28 13:31:18.278013 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:18.277977    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-4kd6g" podUID="d12296bd-99ec-4ab1-9ac1-b43aba975bc5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Jul 28 13:31:19.282532 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:19.282489    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-swgfs" podUID="96816748-5118-48f8-81fa-bd7459698bc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Jul 28 13:31:28.279407 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:28.279380    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-4kd6g"
Jul 28 13:31:29.283652 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:29.283627    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-swgfs"
Jul 28 13:31:35.516359 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:35.516323    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-4kd6g"]
Jul 28 13:31:35.516798 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:35.516538    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-4kd6g" podUID="d12296bd-99ec-4ab1-9ac1-b43aba975bc5" containerName="kserve-container" containerID="cri-o://54dee6b00cc8fe58e0bc8b1b385df1883686daf10cac43a428d968c8bef919fe" gracePeriod=30
Jul 28 13:31:35.516798 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:35.516567    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-4kd6g" podUID="d12296bd-99ec-4ab1-9ac1-b43aba975bc5" containerName="kube-rbac-proxy" containerID="cri-o://dd26a9c09298f2582bea3de73ea1ff1e916328faea4fa418436d71b65d630d97" gracePeriod=30
Jul 28 13:31:35.551138 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:35.551113    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-swgfs"]
Jul 28 13:31:35.551382 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:35.551353    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-swgfs" podUID="96816748-5118-48f8-81fa-bd7459698bc3" containerName="kserve-container" containerID="cri-o://aaa78d430569c62305bf16378b263b9efc7e997d3282a087f253a6fa75307a41" gracePeriod=30
Jul 28 13:31:35.551491 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:35.551374    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-swgfs" podUID="96816748-5118-48f8-81fa-bd7459698bc3" containerName="kube-rbac-proxy" containerID="cri-o://bec50e60cc6f7d8dbe0ca0fea7a7633635457d114ddab2f5143258a072cbfafa" gracePeriod=30
Jul 28 13:31:35.637546 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:35.637522    2574 generic.go:356] "Generic (PLEG): container finished" podID="d12296bd-99ec-4ab1-9ac1-b43aba975bc5" containerID="dd26a9c09298f2582bea3de73ea1ff1e916328faea4fa418436d71b65d630d97" exitCode=2
Jul 28 13:31:35.637637 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:35.637593    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-4kd6g" event={"ID":"d12296bd-99ec-4ab1-9ac1-b43aba975bc5","Type":"ContainerDied","Data":"dd26a9c09298f2582bea3de73ea1ff1e916328faea4fa418436d71b65d630d97"}
Jul 28 13:31:36.644901 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:36.644872    2574 generic.go:356] "Generic (PLEG): container finished" podID="96816748-5118-48f8-81fa-bd7459698bc3" containerID="bec50e60cc6f7d8dbe0ca0fea7a7633635457d114ddab2f5143258a072cbfafa" exitCode=2
Jul 28 13:31:36.645270 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:36.644917    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-swgfs" event={"ID":"96816748-5118-48f8-81fa-bd7459698bc3","Type":"ContainerDied","Data":"bec50e60cc6f7d8dbe0ca0fea7a7633635457d114ddab2f5143258a072cbfafa"}
Jul 28 13:31:38.273742 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:38.273702    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-4kd6g" podUID="d12296bd-99ec-4ab1-9ac1-b43aba975bc5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.46:8643/healthz\": dial tcp 10.132.0.46:8643: connect: connection refused"
Jul 28 13:31:38.278159 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:38.278134    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-4kd6g" podUID="d12296bd-99ec-4ab1-9ac1-b43aba975bc5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Jul 28 13:31:39.278349 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:39.278310    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-swgfs" podUID="96816748-5118-48f8-81fa-bd7459698bc3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
Jul 28 13:31:39.282648 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:39.282616    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-swgfs" podUID="96816748-5118-48f8-81fa-bd7459698bc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Jul 28 13:31:39.661994 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:39.661957    2574 generic.go:356] "Generic (PLEG): container finished" podID="d12296bd-99ec-4ab1-9ac1-b43aba975bc5" containerID="54dee6b00cc8fe58e0bc8b1b385df1883686daf10cac43a428d968c8bef919fe" exitCode=0
Jul 28 13:31:39.662108 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:39.662003    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-4kd6g" event={"ID":"d12296bd-99ec-4ab1-9ac1-b43aba975bc5","Type":"ContainerDied","Data":"54dee6b00cc8fe58e0bc8b1b385df1883686daf10cac43a428d968c8bef919fe"}
Jul 28 13:31:39.662108 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:39.662067    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-4kd6g" event={"ID":"d12296bd-99ec-4ab1-9ac1-b43aba975bc5","Type":"ContainerDied","Data":"7e88dfa30149acd2681ef2f1d61ffd491ae7a4e265ef21fef6e7a8a0814efd2e"}
Jul 28 13:31:39.662108 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:39.662099    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e88dfa30149acd2681ef2f1d61ffd491ae7a4e265ef21fef6e7a8a0814efd2e"
Jul 28 13:31:39.664601 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:39.664578    2574 generic.go:356] "Generic (PLEG): container finished" podID="96816748-5118-48f8-81fa-bd7459698bc3" containerID="aaa78d430569c62305bf16378b263b9efc7e997d3282a087f253a6fa75307a41" exitCode=0
Jul 28 13:31:39.664716 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:39.664652    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-swgfs" event={"ID":"96816748-5118-48f8-81fa-bd7459698bc3","Type":"ContainerDied","Data":"aaa78d430569c62305bf16378b263b9efc7e997d3282a087f253a6fa75307a41"}
Jul 28 13:31:39.671347 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:39.671328    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-4kd6g"
Jul 28 13:31:39.696857 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:39.696835    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d12296bd-99ec-4ab1-9ac1-b43aba975bc5-proxy-tls\") pod \"d12296bd-99ec-4ab1-9ac1-b43aba975bc5\" (UID: \"d12296bd-99ec-4ab1-9ac1-b43aba975bc5\") "
Jul 28 13:31:39.696953 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:39.696938    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d12296bd-99ec-4ab1-9ac1-b43aba975bc5-kube-rbac-proxy-sar-config\") pod \"d12296bd-99ec-4ab1-9ac1-b43aba975bc5\" (UID: \"d12296bd-99ec-4ab1-9ac1-b43aba975bc5\") "
Jul 28 13:31:39.697021 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:39.696958    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d12296bd-99ec-4ab1-9ac1-b43aba975bc5-cabundle-cert\") pod \"d12296bd-99ec-4ab1-9ac1-b43aba975bc5\" (UID: \"d12296bd-99ec-4ab1-9ac1-b43aba975bc5\") "
Jul 28 13:31:39.697021 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:39.696978    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d12296bd-99ec-4ab1-9ac1-b43aba975bc5-kserve-provision-location\") pod \"d12296bd-99ec-4ab1-9ac1-b43aba975bc5\" (UID: \"d12296bd-99ec-4ab1-9ac1-b43aba975bc5\") "
Jul 28 13:31:39.697021 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:39.697015    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gwvvm\" (UniqueName: \"kubernetes.io/projected/d12296bd-99ec-4ab1-9ac1-b43aba975bc5-kube-api-access-gwvvm\") pod \"d12296bd-99ec-4ab1-9ac1-b43aba975bc5\" (UID: \"d12296bd-99ec-4ab1-9ac1-b43aba975bc5\") "
Jul 28 13:31:39.697324 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:39.697296    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d12296bd-99ec-4ab1-9ac1-b43aba975bc5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d12296bd-99ec-4ab1-9ac1-b43aba975bc5" (UID: "d12296bd-99ec-4ab1-9ac1-b43aba975bc5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:31:39.697661 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:39.697634    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d12296bd-99ec-4ab1-9ac1-b43aba975bc5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d12296bd-99ec-4ab1-9ac1-b43aba975bc5" (UID: "d12296bd-99ec-4ab1-9ac1-b43aba975bc5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:31:39.697839 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:39.697816    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d12296bd-99ec-4ab1-9ac1-b43aba975bc5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d12296bd-99ec-4ab1-9ac1-b43aba975bc5" (UID: "d12296bd-99ec-4ab1-9ac1-b43aba975bc5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:31:39.699110 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:39.699086    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d12296bd-99ec-4ab1-9ac1-b43aba975bc5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d12296bd-99ec-4ab1-9ac1-b43aba975bc5" (UID: "d12296bd-99ec-4ab1-9ac1-b43aba975bc5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:31:39.699188 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:39.699154    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d12296bd-99ec-4ab1-9ac1-b43aba975bc5-kube-api-access-gwvvm" (OuterVolumeSpecName: "kube-api-access-gwvvm") pod "d12296bd-99ec-4ab1-9ac1-b43aba975bc5" (UID: "d12296bd-99ec-4ab1-9ac1-b43aba975bc5"). InnerVolumeSpecName "kube-api-access-gwvvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:31:39.705379 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:39.705362    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-swgfs"
Jul 28 13:31:39.798242 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:39.798188    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96816748-5118-48f8-81fa-bd7459698bc3-kserve-provision-location\") pod \"96816748-5118-48f8-81fa-bd7459698bc3\" (UID: \"96816748-5118-48f8-81fa-bd7459698bc3\") "
Jul 28 13:31:39.798242 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:39.798222    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jbdf5\" (UniqueName: \"kubernetes.io/projected/96816748-5118-48f8-81fa-bd7459698bc3-kube-api-access-jbdf5\") pod \"96816748-5118-48f8-81fa-bd7459698bc3\" (UID: \"96816748-5118-48f8-81fa-bd7459698bc3\") "
Jul 28 13:31:39.798242 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:39.798240    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/96816748-5118-48f8-81fa-bd7459698bc3-kube-rbac-proxy-sar-config\") pod \"96816748-5118-48f8-81fa-bd7459698bc3\" (UID: \"96816748-5118-48f8-81fa-bd7459698bc3\") "
Jul 28 13:31:39.798447 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:39.798258    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96816748-5118-48f8-81fa-bd7459698bc3-cabundle-cert\") pod \"96816748-5118-48f8-81fa-bd7459698bc3\" (UID: \"96816748-5118-48f8-81fa-bd7459698bc3\") "
Jul 28 13:31:39.798447 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:39.798284    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96816748-5118-48f8-81fa-bd7459698bc3-proxy-tls\") pod \"96816748-5118-48f8-81fa-bd7459698bc3\" (UID: \"96816748-5118-48f8-81fa-bd7459698bc3\") "
Jul 28 13:31:39.798554 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:39.798522    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d12296bd-99ec-4ab1-9ac1-b43aba975bc5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:31:39.798554 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:39.798542    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d12296bd-99ec-4ab1-9ac1-b43aba975bc5-cabundle-cert\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:31:39.798659 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:39.798559    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d12296bd-99ec-4ab1-9ac1-b43aba975bc5-kserve-provision-location\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:31:39.798659 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:39.798576    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwvvm\" (UniqueName: \"kubernetes.io/projected/d12296bd-99ec-4ab1-9ac1-b43aba975bc5-kube-api-access-gwvvm\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:31:39.798659 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:39.798589    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d12296bd-99ec-4ab1-9ac1-b43aba975bc5-proxy-tls\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:31:39.798659 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:39.798536    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96816748-5118-48f8-81fa-bd7459698bc3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "96816748-5118-48f8-81fa-bd7459698bc3" (UID: "96816748-5118-48f8-81fa-bd7459698bc3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:31:39.798659 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:39.798562    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96816748-5118-48f8-81fa-bd7459698bc3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "96816748-5118-48f8-81fa-bd7459698bc3" (UID: "96816748-5118-48f8-81fa-bd7459698bc3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:31:39.798659 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:39.798577    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96816748-5118-48f8-81fa-bd7459698bc3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "96816748-5118-48f8-81fa-bd7459698bc3" (UID: "96816748-5118-48f8-81fa-bd7459698bc3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:31:39.800114 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:39.800096    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96816748-5118-48f8-81fa-bd7459698bc3-kube-api-access-jbdf5" (OuterVolumeSpecName: "kube-api-access-jbdf5") pod "96816748-5118-48f8-81fa-bd7459698bc3" (UID: "96816748-5118-48f8-81fa-bd7459698bc3"). InnerVolumeSpecName "kube-api-access-jbdf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:31:39.800256 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:39.800236    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96816748-5118-48f8-81fa-bd7459698bc3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "96816748-5118-48f8-81fa-bd7459698bc3" (UID: "96816748-5118-48f8-81fa-bd7459698bc3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:31:39.899844 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:39.899824    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96816748-5118-48f8-81fa-bd7459698bc3-cabundle-cert\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:31:39.899844 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:39.899842    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96816748-5118-48f8-81fa-bd7459698bc3-proxy-tls\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:31:39.899962 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:39.899850    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96816748-5118-48f8-81fa-bd7459698bc3-kserve-provision-location\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:31:39.899962 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:39.899860    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbdf5\" (UniqueName: \"kubernetes.io/projected/96816748-5118-48f8-81fa-bd7459698bc3-kube-api-access-jbdf5\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:31:39.899962 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:39.899869    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/96816748-5118-48f8-81fa-bd7459698bc3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:31:40.670413 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:40.670350    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-swgfs"
Jul 28 13:31:40.670413 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:40.670378    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-4kd6g"
Jul 28 13:31:40.670413 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:40.670373    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-swgfs" event={"ID":"96816748-5118-48f8-81fa-bd7459698bc3","Type":"ContainerDied","Data":"fab9d114b7d703b5f2f7f5d329f88631986d5d4eace91d596797fe17438dee8d"}
Jul 28 13:31:40.670964 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:40.670458    2574 scope.go:117] "RemoveContainer" containerID="bec50e60cc6f7d8dbe0ca0fea7a7633635457d114ddab2f5143258a072cbfafa"
Jul 28 13:31:40.678756 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:40.678739    2574 scope.go:117] "RemoveContainer" containerID="aaa78d430569c62305bf16378b263b9efc7e997d3282a087f253a6fa75307a41"
Jul 28 13:31:40.685890 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:40.685874    2574 scope.go:117] "RemoveContainer" containerID="6a052e81ab5786e7df006dddde19851719eca86faa6f0d1948e9de841c1a0454"
Jul 28 13:31:40.689486 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:40.689465    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-swgfs"]
Jul 28 13:31:40.697437 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:40.697418    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-swgfs"]
Jul 28 13:31:40.708423 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:40.708402    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-4kd6g"]
Jul 28 13:31:40.714171 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:40.714154    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-4kd6g"]
Jul 28 13:31:42.301845 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:42.301815    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96816748-5118-48f8-81fa-bd7459698bc3" path="/var/lib/kubelet/pods/96816748-5118-48f8-81fa-bd7459698bc3/volumes"
Jul 28 13:31:42.302274 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:31:42.302261    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d12296bd-99ec-4ab1-9ac1-b43aba975bc5" path="/var/lib/kubelet/pods/d12296bd-99ec-4ab1-9ac1-b43aba975bc5/volumes"
Jul 28 13:33:31.133627 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:33:31.133551    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x"]
Jul 28 13:33:31.134423 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:33:31.134056    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96816748-5118-48f8-81fa-bd7459698bc3" containerName="storage-initializer"
Jul 28 13:33:31.134423 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:33:31.134075    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="96816748-5118-48f8-81fa-bd7459698bc3" containerName="storage-initializer"
Jul 28 13:33:31.134423 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:33:31.134085    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d12296bd-99ec-4ab1-9ac1-b43aba975bc5" containerName="kube-rbac-proxy"
Jul 28 13:33:31.134423 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:33:31.134093    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d12296bd-99ec-4ab1-9ac1-b43aba975bc5" containerName="kube-rbac-proxy"
Jul 28 13:33:31.134423 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:33:31.134103    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96816748-5118-48f8-81fa-bd7459698bc3" containerName="kserve-container"
Jul 28 13:33:31.134423 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:33:31.134111    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="96816748-5118-48f8-81fa-bd7459698bc3" containerName="kserve-container"
Jul 28 13:33:31.134423 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:33:31.134131    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d12296bd-99ec-4ab1-9ac1-b43aba975bc5" containerName="storage-initializer"
Jul 28 13:33:31.134423 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:33:31.134138    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d12296bd-99ec-4ab1-9ac1-b43aba975bc5" containerName="storage-initializer"
Jul 28 13:33:31.134423 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:33:31.134148    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d12296bd-99ec-4ab1-9ac1-b43aba975bc5" containerName="kserve-container"
Jul 28 13:33:31.134423 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:33:31.134155    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d12296bd-99ec-4ab1-9ac1-b43aba975bc5" containerName="kserve-container"
Jul 28 13:33:31.134423 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:33:31.134172    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96816748-5118-48f8-81fa-bd7459698bc3" containerName="kube-rbac-proxy"
Jul 28 13:33:31.134423 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:33:31.134179    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="96816748-5118-48f8-81fa-bd7459698bc3" containerName="kube-rbac-proxy"
Jul 28 13:33:31.134423 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:33:31.134268    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d12296bd-99ec-4ab1-9ac1-b43aba975bc5" containerName="kserve-container"
Jul 28 13:33:31.134423 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:33:31.134279    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="96816748-5118-48f8-81fa-bd7459698bc3" containerName="kserve-container"
Jul 28 13:33:31.134423 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:33:31.134291    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d12296bd-99ec-4ab1-9ac1-b43aba975bc5" containerName="kube-rbac-proxy"
Jul 28 13:33:31.134423 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:33:31.134298    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="96816748-5118-48f8-81fa-bd7459698bc3" containerName="kube-rbac-proxy"
Jul 28 13:33:31.137677 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:33:31.137657    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x"
Jul 28 13:33:31.140474 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:33:31.140453    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\""
Jul 28 13:33:31.140584 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:33:31.140458    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:33:31.140584 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:33:31.140458    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-predictor-serving-cert\""
Jul 28 13:33:31.140810 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:33:31.140792    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:33:31.140882 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:33:31.140851    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rxfwd\""
Jul 28 13:33:31.141561 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:33:31.141545    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 13:33:31.141620 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:33:31.141546    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:33:31.149757 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:33:31.149739    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x"]
Jul 28 13:33:31.168666 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:33:31.168644    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/982af0a9-99e5-459a-8cf4-31883c6eb459-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x\" (UID: \"982af0a9-99e5-459a-8cf4-31883c6eb459\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x"
Jul 28 13:33:31.168769 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:33:31.168707    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sk7m\" (UniqueName: \"kubernetes.io/projected/982af0a9-99e5-459a-8cf4-31883c6eb459-kube-api-access-2sk7m\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x\" (UID: \"982af0a9-99e5-459a-8cf4-31883c6eb459\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x"
Jul 28 13:33:31.168769 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:33:31.168729    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/982af0a9-99e5-459a-8cf4-31883c6eb459-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x\" (UID: \"982af0a9-99e5-459a-8cf4-31883c6eb459\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x"
Jul 28 13:33:31.168769 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:33:31.168756    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/982af0a9-99e5-459a-8cf4-31883c6eb459-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x\" (UID: \"982af0a9-99e5-459a-8cf4-31883c6eb459\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x"
Jul 28 13:33:31.168878 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:33:31.168846    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/982af0a9-99e5-459a-8cf4-31883c6eb459-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x\" (UID: \"982af0a9-99e5-459a-8cf4-31883c6eb459\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x"
Jul 28 13:33:31.269985 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:33:31.269962    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sk7m\" (UniqueName: \"kubernetes.io/projected/982af0a9-99e5-459a-8cf4-31883c6eb459-kube-api-access-2sk7m\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x\" (UID: \"982af0a9-99e5-459a-8cf4-31883c6eb459\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x"
Jul 28 13:33:31.270070 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:33:31.269991    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/982af0a9-99e5-459a-8cf4-31883c6eb459-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x\" (UID: \"982af0a9-99e5-459a-8cf4-31883c6eb459\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x"
Jul 28 13:33:31.270070 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:33:31.270013    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/982af0a9-99e5-459a-8cf4-31883c6eb459-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x\" (UID: \"982af0a9-99e5-459a-8cf4-31883c6eb459\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x"
Jul 28 13:33:31.270157 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:33:31.270135    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/982af0a9-99e5-459a-8cf4-31883c6eb459-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x\" (UID: \"982af0a9-99e5-459a-8cf4-31883c6eb459\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x"
Jul 28 13:33:31.270216 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:33:31.270197    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/982af0a9-99e5-459a-8cf4-31883c6eb459-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x\" (UID: \"982af0a9-99e5-459a-8cf4-31883c6eb459\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x"
Jul 28 13:33:31.270341 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:33:31.270322    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-serving-cert: secret "isvc-lightgbm-runtime-predictor-serving-cert" not found
Jul 28 13:33:31.270403 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:33:31.270392    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/982af0a9-99e5-459a-8cf4-31883c6eb459-proxy-tls podName:982af0a9-99e5-459a-8cf4-31883c6eb459 nodeName:}" failed. No retries permitted until 2026-07-28 13:33:31.770373881 +0000 UTC m=+2423.975760461 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/982af0a9-99e5-459a-8cf4-31883c6eb459-proxy-tls") pod "isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x" (UID: "982af0a9-99e5-459a-8cf4-31883c6eb459") : secret "isvc-lightgbm-runtime-predictor-serving-cert" not found
Jul 28 13:33:31.270510 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:33:31.270493    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/982af0a9-99e5-459a-8cf4-31883c6eb459-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x\" (UID: \"982af0a9-99e5-459a-8cf4-31883c6eb459\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x"
Jul 28 13:33:31.270584 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:33:31.270569    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/982af0a9-99e5-459a-8cf4-31883c6eb459-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x\" (UID: \"982af0a9-99e5-459a-8cf4-31883c6eb459\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x"
Jul 28 13:33:31.270674 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:33:31.270660    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/982af0a9-99e5-459a-8cf4-31883c6eb459-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x\" (UID: \"982af0a9-99e5-459a-8cf4-31883c6eb459\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x"
Jul 28 13:33:31.279729 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:33:31.279711    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sk7m\" (UniqueName: \"kubernetes.io/projected/982af0a9-99e5-459a-8cf4-31883c6eb459-kube-api-access-2sk7m\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x\" (UID: \"982af0a9-99e5-459a-8cf4-31883c6eb459\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x"
Jul 28 13:33:31.773531 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:33:31.773504    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/982af0a9-99e5-459a-8cf4-31883c6eb459-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x\" (UID: \"982af0a9-99e5-459a-8cf4-31883c6eb459\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x"
Jul 28 13:33:31.775909 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:33:31.775889    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/982af0a9-99e5-459a-8cf4-31883c6eb459-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x\" (UID: \"982af0a9-99e5-459a-8cf4-31883c6eb459\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x"
Jul 28 13:33:32.048371 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:33:32.048292    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x"
Jul 28 13:33:32.178314 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:33:32.178291    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x"]
Jul 28 13:33:32.180714 ip-10-0-138-158 kubenswrapper[2574]: W0728 13:33:32.180673    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod982af0a9_99e5_459a_8cf4_31883c6eb459.slice/crio-84f9f3c667cbdee186d1946436e4554f968aff57bbf35c13b062c3de1cfc798f WatchSource:0}: Error finding container 84f9f3c667cbdee186d1946436e4554f968aff57bbf35c13b062c3de1cfc798f: Status 404 returned error can't find the container with id 84f9f3c667cbdee186d1946436e4554f968aff57bbf35c13b062c3de1cfc798f
Jul 28 13:33:33.163989 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:33:33.163952    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x" event={"ID":"982af0a9-99e5-459a-8cf4-31883c6eb459","Type":"ContainerStarted","Data":"7b67ae0b0547a4bf7d1bc7ba3706fde2ab83c960c73411e9a40c2f6b716f42b5"}
Jul 28 13:33:33.164160 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:33:33.164021    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x" event={"ID":"982af0a9-99e5-459a-8cf4-31883c6eb459","Type":"ContainerStarted","Data":"84f9f3c667cbdee186d1946436e4554f968aff57bbf35c13b062c3de1cfc798f"}
Jul 28 13:33:36.178921 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:33:36.178851    2574 generic.go:356] "Generic (PLEG): container finished" podID="982af0a9-99e5-459a-8cf4-31883c6eb459" containerID="7b67ae0b0547a4bf7d1bc7ba3706fde2ab83c960c73411e9a40c2f6b716f42b5" exitCode=0
Jul 28 13:33:36.178921 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:33:36.178910    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x" event={"ID":"982af0a9-99e5-459a-8cf4-31883c6eb459","Type":"ContainerDied","Data":"7b67ae0b0547a4bf7d1bc7ba3706fde2ab83c960c73411e9a40c2f6b716f42b5"}
Jul 28 13:33:36.180044 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:33:36.180025    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 13:33:43.219200 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:33:43.219168    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x" event={"ID":"982af0a9-99e5-459a-8cf4-31883c6eb459","Type":"ContainerStarted","Data":"84b3772cb47e4c70e2ac7fea793dda68fefd612728e55ff2d11e7018c5a1ca69"}
Jul 28 13:33:43.219569 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:33:43.219228    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x" event={"ID":"982af0a9-99e5-459a-8cf4-31883c6eb459","Type":"ContainerStarted","Data":"4b31284d49402e0fd36ac9c0d28d480b071c72571554ee9aa2714eb2b2218ff4"}
Jul 28 13:33:43.219569 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:33:43.219410    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x"
Jul 28 13:33:43.240115 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:33:43.240070    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x" podStartSLOduration=2.459538957 podStartE2EDuration="12.240055796s" totalImagesPullingTime="6.13904985s" totalInitContainerRuntime="3.641466989s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:33:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:33:43.238624748 +0000 UTC m=+2435.444011376" watchObservedRunningTime="2026-07-28 13:33:43.240055796 +0000 UTC m=+2435.445442396"
Jul 28 13:33:44.223931 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:33:44.223902    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x"
Jul 28 13:33:44.225016 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:33:44.224990    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x" podUID="982af0a9-99e5-459a-8cf4-31883c6eb459" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Jul 28 13:33:45.228623 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:33:45.228588    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x" podUID="982af0a9-99e5-459a-8cf4-31883c6eb459" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Jul 28 13:33:50.233567 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:33:50.233530    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x"
Jul 28 13:33:50.234149 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:33:50.234124    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x" podUID="982af0a9-99e5-459a-8cf4-31883c6eb459" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Jul 28 13:34:00.234983 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:34:00.234945    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x" podUID="982af0a9-99e5-459a-8cf4-31883c6eb459" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Jul 28 13:34:10.234530 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:34:10.234493    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x" podUID="982af0a9-99e5-459a-8cf4-31883c6eb459" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Jul 28 13:34:20.234160 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:34:20.234121    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x" podUID="982af0a9-99e5-459a-8cf4-31883c6eb459" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Jul 28 13:34:30.234314 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:34:30.234273    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x" podUID="982af0a9-99e5-459a-8cf4-31883c6eb459" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Jul 28 13:34:40.234964 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:34:40.234926    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x" podUID="982af0a9-99e5-459a-8cf4-31883c6eb459" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Jul 28 13:34:50.234414 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:34:50.234374    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x" podUID="982af0a9-99e5-459a-8cf4-31883c6eb459" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Jul 28 13:35:00.234887 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:35:00.234855    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x"
Jul 28 13:35:01.768238 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:35:01.768207    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x"]
Jul 28 13:35:01.768725 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:35:01.768509    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x" podUID="982af0a9-99e5-459a-8cf4-31883c6eb459" containerName="kserve-container" containerID="cri-o://4b31284d49402e0fd36ac9c0d28d480b071c72571554ee9aa2714eb2b2218ff4" gracePeriod=30
Jul 28 13:35:01.768725 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:35:01.768530    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x" podUID="982af0a9-99e5-459a-8cf4-31883c6eb459" containerName="kube-rbac-proxy" containerID="cri-o://84b3772cb47e4c70e2ac7fea793dda68fefd612728e55ff2d11e7018c5a1ca69" gracePeriod=30
Jul 28 13:35:02.569475 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:35:02.569435    2574 generic.go:356] "Generic (PLEG): container finished" podID="982af0a9-99e5-459a-8cf4-31883c6eb459" containerID="84b3772cb47e4c70e2ac7fea793dda68fefd612728e55ff2d11e7018c5a1ca69" exitCode=2
Jul 28 13:35:02.569645 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:35:02.569496    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x" event={"ID":"982af0a9-99e5-459a-8cf4-31883c6eb459","Type":"ContainerDied","Data":"84b3772cb47e4c70e2ac7fea793dda68fefd612728e55ff2d11e7018c5a1ca69"}
Jul 28 13:35:05.229358 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:35:05.229320    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x" podUID="982af0a9-99e5-459a-8cf4-31883c6eb459" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.48:8643/healthz\": dial tcp 10.132.0.48:8643: connect: connection refused"
Jul 28 13:35:06.107949 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:35:06.107926    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x"
Jul 28 13:35:06.195497 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:35:06.195471    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/982af0a9-99e5-459a-8cf4-31883c6eb459-kube-rbac-proxy-sar-config\") pod \"982af0a9-99e5-459a-8cf4-31883c6eb459\" (UID: \"982af0a9-99e5-459a-8cf4-31883c6eb459\") "
Jul 28 13:35:06.195623 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:35:06.195519    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/982af0a9-99e5-459a-8cf4-31883c6eb459-cabundle-cert\") pod \"982af0a9-99e5-459a-8cf4-31883c6eb459\" (UID: \"982af0a9-99e5-459a-8cf4-31883c6eb459\") "
Jul 28 13:35:06.195623 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:35:06.195562    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/982af0a9-99e5-459a-8cf4-31883c6eb459-kserve-provision-location\") pod \"982af0a9-99e5-459a-8cf4-31883c6eb459\" (UID: \"982af0a9-99e5-459a-8cf4-31883c6eb459\") "
Jul 28 13:35:06.195623 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:35:06.195591    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/982af0a9-99e5-459a-8cf4-31883c6eb459-proxy-tls\") pod \"982af0a9-99e5-459a-8cf4-31883c6eb459\" (UID: \"982af0a9-99e5-459a-8cf4-31883c6eb459\") "
Jul 28 13:35:06.195623 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:35:06.195618    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2sk7m\" (UniqueName: \"kubernetes.io/projected/982af0a9-99e5-459a-8cf4-31883c6eb459-kube-api-access-2sk7m\") pod \"982af0a9-99e5-459a-8cf4-31883c6eb459\" (UID: \"982af0a9-99e5-459a-8cf4-31883c6eb459\") "
Jul 28 13:35:06.195854 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:35:06.195804    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/982af0a9-99e5-459a-8cf4-31883c6eb459-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "982af0a9-99e5-459a-8cf4-31883c6eb459" (UID: "982af0a9-99e5-459a-8cf4-31883c6eb459"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:35:06.195969 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:35:06.195946    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/982af0a9-99e5-459a-8cf4-31883c6eb459-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "982af0a9-99e5-459a-8cf4-31883c6eb459" (UID: "982af0a9-99e5-459a-8cf4-31883c6eb459"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:35:06.196027 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:35:06.195943    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/982af0a9-99e5-459a-8cf4-31883c6eb459-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "982af0a9-99e5-459a-8cf4-31883c6eb459" (UID: "982af0a9-99e5-459a-8cf4-31883c6eb459"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:35:06.197620 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:35:06.197590    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/982af0a9-99e5-459a-8cf4-31883c6eb459-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "982af0a9-99e5-459a-8cf4-31883c6eb459" (UID: "982af0a9-99e5-459a-8cf4-31883c6eb459"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:35:06.197738 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:35:06.197657    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/982af0a9-99e5-459a-8cf4-31883c6eb459-kube-api-access-2sk7m" (OuterVolumeSpecName: "kube-api-access-2sk7m") pod "982af0a9-99e5-459a-8cf4-31883c6eb459" (UID: "982af0a9-99e5-459a-8cf4-31883c6eb459"). InnerVolumeSpecName "kube-api-access-2sk7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:35:06.297191 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:35:06.297128    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/982af0a9-99e5-459a-8cf4-31883c6eb459-kserve-provision-location\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:35:06.297191 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:35:06.297151    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/982af0a9-99e5-459a-8cf4-31883c6eb459-proxy-tls\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:35:06.297191 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:35:06.297163    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sk7m\" (UniqueName: \"kubernetes.io/projected/982af0a9-99e5-459a-8cf4-31883c6eb459-kube-api-access-2sk7m\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:35:06.297191 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:35:06.297172    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/982af0a9-99e5-459a-8cf4-31883c6eb459-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:35:06.297191 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:35:06.297181    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/982af0a9-99e5-459a-8cf4-31883c6eb459-cabundle-cert\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:35:06.590052 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:35:06.589973    2574 generic.go:356] "Generic (PLEG): container finished" podID="982af0a9-99e5-459a-8cf4-31883c6eb459" containerID="4b31284d49402e0fd36ac9c0d28d480b071c72571554ee9aa2714eb2b2218ff4" exitCode=0
Jul 28 13:35:06.590052 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:35:06.590031    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x"
Jul 28 13:35:06.590220 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:35:06.590048    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x" event={"ID":"982af0a9-99e5-459a-8cf4-31883c6eb459","Type":"ContainerDied","Data":"4b31284d49402e0fd36ac9c0d28d480b071c72571554ee9aa2714eb2b2218ff4"}
Jul 28 13:35:06.590220 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:35:06.590108    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x" event={"ID":"982af0a9-99e5-459a-8cf4-31883c6eb459","Type":"ContainerDied","Data":"84f9f3c667cbdee186d1946436e4554f968aff57bbf35c13b062c3de1cfc798f"}
Jul 28 13:35:06.590220 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:35:06.590134    2574 scope.go:117] "RemoveContainer" containerID="84b3772cb47e4c70e2ac7fea793dda68fefd612728e55ff2d11e7018c5a1ca69"
Jul 28 13:35:06.598503 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:35:06.598485    2574 scope.go:117] "RemoveContainer" containerID="4b31284d49402e0fd36ac9c0d28d480b071c72571554ee9aa2714eb2b2218ff4"
Jul 28 13:35:06.608817 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:35:06.608801    2574 scope.go:117] "RemoveContainer" containerID="7b67ae0b0547a4bf7d1bc7ba3706fde2ab83c960c73411e9a40c2f6b716f42b5"
Jul 28 13:35:06.609066 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:35:06.609046    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x"]
Jul 28 13:35:06.612749 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:35:06.612728    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-kzg8x"]
Jul 28 13:35:06.615993 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:35:06.615968    2574 scope.go:117] "RemoveContainer" containerID="84b3772cb47e4c70e2ac7fea793dda68fefd612728e55ff2d11e7018c5a1ca69"
Jul 28 13:35:06.616229 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:35:06.616211    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84b3772cb47e4c70e2ac7fea793dda68fefd612728e55ff2d11e7018c5a1ca69\": container with ID starting with 84b3772cb47e4c70e2ac7fea793dda68fefd612728e55ff2d11e7018c5a1ca69 not found: ID does not exist" containerID="84b3772cb47e4c70e2ac7fea793dda68fefd612728e55ff2d11e7018c5a1ca69"
Jul 28 13:35:06.616299 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:35:06.616240    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84b3772cb47e4c70e2ac7fea793dda68fefd612728e55ff2d11e7018c5a1ca69"} err="failed to get container status \"84b3772cb47e4c70e2ac7fea793dda68fefd612728e55ff2d11e7018c5a1ca69\": rpc error: code = NotFound desc = could not find container \"84b3772cb47e4c70e2ac7fea793dda68fefd612728e55ff2d11e7018c5a1ca69\": container with ID starting with 84b3772cb47e4c70e2ac7fea793dda68fefd612728e55ff2d11e7018c5a1ca69 not found: ID does not exist"
Jul 28 13:35:06.616299 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:35:06.616265    2574 scope.go:117] "RemoveContainer" containerID="4b31284d49402e0fd36ac9c0d28d480b071c72571554ee9aa2714eb2b2218ff4"
Jul 28 13:35:06.616531 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:35:06.616514    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b31284d49402e0fd36ac9c0d28d480b071c72571554ee9aa2714eb2b2218ff4\": container with ID starting with 4b31284d49402e0fd36ac9c0d28d480b071c72571554ee9aa2714eb2b2218ff4 not found: ID does not exist" containerID="4b31284d49402e0fd36ac9c0d28d480b071c72571554ee9aa2714eb2b2218ff4"
Jul 28 13:35:06.616591 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:35:06.616538    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b31284d49402e0fd36ac9c0d28d480b071c72571554ee9aa2714eb2b2218ff4"} err="failed to get container status \"4b31284d49402e0fd36ac9c0d28d480b071c72571554ee9aa2714eb2b2218ff4\": rpc error: code = NotFound desc = could not find container \"4b31284d49402e0fd36ac9c0d28d480b071c72571554ee9aa2714eb2b2218ff4\": container with ID starting with 4b31284d49402e0fd36ac9c0d28d480b071c72571554ee9aa2714eb2b2218ff4 not found: ID does not exist"
Jul 28 13:35:06.616591 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:35:06.616553    2574 scope.go:117] "RemoveContainer" containerID="7b67ae0b0547a4bf7d1bc7ba3706fde2ab83c960c73411e9a40c2f6b716f42b5"
Jul 28 13:35:06.616788 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:35:06.616770    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b67ae0b0547a4bf7d1bc7ba3706fde2ab83c960c73411e9a40c2f6b716f42b5\": container with ID starting with 7b67ae0b0547a4bf7d1bc7ba3706fde2ab83c960c73411e9a40c2f6b716f42b5 not found: ID does not exist" containerID="7b67ae0b0547a4bf7d1bc7ba3706fde2ab83c960c73411e9a40c2f6b716f42b5"
Jul 28 13:35:06.616837 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:35:06.616800    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b67ae0b0547a4bf7d1bc7ba3706fde2ab83c960c73411e9a40c2f6b716f42b5"} err="failed to get container status \"7b67ae0b0547a4bf7d1bc7ba3706fde2ab83c960c73411e9a40c2f6b716f42b5\": rpc error: code = NotFound desc = could not find container \"7b67ae0b0547a4bf7d1bc7ba3706fde2ab83c960c73411e9a40c2f6b716f42b5\": container with ID starting with 7b67ae0b0547a4bf7d1bc7ba3706fde2ab83c960c73411e9a40c2f6b716f42b5 not found: ID does not exist"
Jul 28 13:35:08.301836 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:35:08.301806    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="982af0a9-99e5-459a-8cf4-31883c6eb459" path="/var/lib/kubelet/pods/982af0a9-99e5-459a-8cf4-31883c6eb459/volumes"
Jul 28 13:36:08.542641 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:36:08.542602    2574 scope.go:117] "RemoveContainer" containerID="88224f620f757e335667251426424f3f46af9480f81780ad87c0e4f9256f45b2"
Jul 28 13:37:08.556307 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:37:08.556241    2574 scope.go:117] "RemoveContainer" containerID="54dee6b00cc8fe58e0bc8b1b385df1883686daf10cac43a428d968c8bef919fe"
Jul 28 13:37:08.563982 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:37:08.563964    2574 scope.go:117] "RemoveContainer" containerID="dd26a9c09298f2582bea3de73ea1ff1e916328faea4fa418436d71b65d630d97"
Jul 28 13:40:58.046964 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:40:58.046931    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-r5mx9"]
Jul 28 13:40:58.047419 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:40:58.047274    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="982af0a9-99e5-459a-8cf4-31883c6eb459" containerName="storage-initializer"
Jul 28 13:40:58.047419 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:40:58.047285    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="982af0a9-99e5-459a-8cf4-31883c6eb459" containerName="storage-initializer"
Jul 28 13:40:58.047419 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:40:58.047296    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="982af0a9-99e5-459a-8cf4-31883c6eb459" containerName="kube-rbac-proxy"
Jul 28 13:40:58.047419 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:40:58.047303    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="982af0a9-99e5-459a-8cf4-31883c6eb459" containerName="kube-rbac-proxy"
Jul 28 13:40:58.047419 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:40:58.047316    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="982af0a9-99e5-459a-8cf4-31883c6eb459" containerName="kserve-container"
Jul 28 13:40:58.047419 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:40:58.047322    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="982af0a9-99e5-459a-8cf4-31883c6eb459" containerName="kserve-container"
Jul 28 13:40:58.047419 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:40:58.047378    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="982af0a9-99e5-459a-8cf4-31883c6eb459" containerName="kserve-container"
Jul 28 13:40:58.047419 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:40:58.047386    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="982af0a9-99e5-459a-8cf4-31883c6eb459" containerName="kube-rbac-proxy"
Jul 28 13:40:58.050433 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:40:58.050416    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-r5mx9"
Jul 28 13:40:58.053168 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:40:58.053137    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-kube-rbac-proxy-sar-config\""
Jul 28 13:40:58.053298 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:40:58.053244    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:40:58.053672 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:40:58.053654    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:40:58.054355 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:40:58.054334    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rxfwd\""
Jul 28 13:40:58.054467 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:40:58.054365    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-predictor-serving-cert\""
Jul 28 13:40:58.054467 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:40:58.054388    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:40:58.054467 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:40:58.054336    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 13:40:58.063542 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:40:58.063523    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-r5mx9"]
Jul 28 13:40:58.154960 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:40:58.154937    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4l4z\" (UniqueName: \"kubernetes.io/projected/1d4b06a2-100d-44a8-8410-c2d9c78c2ab5-kube-api-access-c4l4z\") pod \"isvc-paddle-predictor-6d5c9fd4f6-r5mx9\" (UID: \"1d4b06a2-100d-44a8-8410-c2d9c78c2ab5\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-r5mx9"
Jul 28 13:40:58.155108 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:40:58.154978    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1d4b06a2-100d-44a8-8410-c2d9c78c2ab5-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-r5mx9\" (UID: \"1d4b06a2-100d-44a8-8410-c2d9c78c2ab5\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-r5mx9"
Jul 28 13:40:58.155108 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:40:58.155006    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d4b06a2-100d-44a8-8410-c2d9c78c2ab5-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-r5mx9\" (UID: \"1d4b06a2-100d-44a8-8410-c2d9c78c2ab5\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-r5mx9"
Jul 28 13:40:58.155108 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:40:58.155043    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d4b06a2-100d-44a8-8410-c2d9c78c2ab5-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-r5mx9\" (UID: \"1d4b06a2-100d-44a8-8410-c2d9c78c2ab5\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-r5mx9"
Jul 28 13:40:58.155108 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:40:58.155092    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1d4b06a2-100d-44a8-8410-c2d9c78c2ab5-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-r5mx9\" (UID: \"1d4b06a2-100d-44a8-8410-c2d9c78c2ab5\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-r5mx9"
Jul 28 13:40:58.256393 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:40:58.256357    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d4b06a2-100d-44a8-8410-c2d9c78c2ab5-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-r5mx9\" (UID: \"1d4b06a2-100d-44a8-8410-c2d9c78c2ab5\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-r5mx9"
Jul 28 13:40:58.256393 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:40:58.256395    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d4b06a2-100d-44a8-8410-c2d9c78c2ab5-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-r5mx9\" (UID: \"1d4b06a2-100d-44a8-8410-c2d9c78c2ab5\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-r5mx9"
Jul 28 13:40:58.256581 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:40:58.256415    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1d4b06a2-100d-44a8-8410-c2d9c78c2ab5-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-r5mx9\" (UID: \"1d4b06a2-100d-44a8-8410-c2d9c78c2ab5\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-r5mx9"
Jul 28 13:40:58.256581 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:40:58.256522    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-predictor-serving-cert: secret "isvc-paddle-predictor-serving-cert" not found
Jul 28 13:40:58.256581 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:40:58.256532    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4l4z\" (UniqueName: \"kubernetes.io/projected/1d4b06a2-100d-44a8-8410-c2d9c78c2ab5-kube-api-access-c4l4z\") pod \"isvc-paddle-predictor-6d5c9fd4f6-r5mx9\" (UID: \"1d4b06a2-100d-44a8-8410-c2d9c78c2ab5\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-r5mx9"
Jul 28 13:40:58.256787 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:40:58.256597    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d4b06a2-100d-44a8-8410-c2d9c78c2ab5-proxy-tls podName:1d4b06a2-100d-44a8-8410-c2d9c78c2ab5 nodeName:}" failed. No retries permitted until 2026-07-28 13:40:58.756575993 +0000 UTC m=+2870.961962576 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1d4b06a2-100d-44a8-8410-c2d9c78c2ab5-proxy-tls") pod "isvc-paddle-predictor-6d5c9fd4f6-r5mx9" (UID: "1d4b06a2-100d-44a8-8410-c2d9c78c2ab5") : secret "isvc-paddle-predictor-serving-cert" not found
Jul 28 13:40:58.256787 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:40:58.256629    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1d4b06a2-100d-44a8-8410-c2d9c78c2ab5-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-r5mx9\" (UID: \"1d4b06a2-100d-44a8-8410-c2d9c78c2ab5\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-r5mx9"
Jul 28 13:40:58.256935 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:40:58.256915    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1d4b06a2-100d-44a8-8410-c2d9c78c2ab5-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-r5mx9\" (UID: \"1d4b06a2-100d-44a8-8410-c2d9c78c2ab5\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-r5mx9"
Jul 28 13:40:58.257143 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:40:58.257125    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1d4b06a2-100d-44a8-8410-c2d9c78c2ab5-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-r5mx9\" (UID: \"1d4b06a2-100d-44a8-8410-c2d9c78c2ab5\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-r5mx9"
Jul 28 13:40:58.257210 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:40:58.257155    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d4b06a2-100d-44a8-8410-c2d9c78c2ab5-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-r5mx9\" (UID: \"1d4b06a2-100d-44a8-8410-c2d9c78c2ab5\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-r5mx9"
Jul 28 13:40:58.265478 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:40:58.265457    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4l4z\" (UniqueName: \"kubernetes.io/projected/1d4b06a2-100d-44a8-8410-c2d9c78c2ab5-kube-api-access-c4l4z\") pod \"isvc-paddle-predictor-6d5c9fd4f6-r5mx9\" (UID: \"1d4b06a2-100d-44a8-8410-c2d9c78c2ab5\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-r5mx9"
Jul 28 13:40:58.760839 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:40:58.760798    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d4b06a2-100d-44a8-8410-c2d9c78c2ab5-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-r5mx9\" (UID: \"1d4b06a2-100d-44a8-8410-c2d9c78c2ab5\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-r5mx9"
Jul 28 13:40:58.763338 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:40:58.763315    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d4b06a2-100d-44a8-8410-c2d9c78c2ab5-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-r5mx9\" (UID: \"1d4b06a2-100d-44a8-8410-c2d9c78c2ab5\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-r5mx9"
Jul 28 13:40:58.961101 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:40:58.961069    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-r5mx9"
Jul 28 13:40:59.090549 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:40:59.090525    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-r5mx9"]
Jul 28 13:40:59.092728 ip-10-0-138-158 kubenswrapper[2574]: W0728 13:40:59.092699    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d4b06a2_100d_44a8_8410_c2d9c78c2ab5.slice/crio-f83e78769b8310e8d48ebdc8295cbe66ddf56a233d2d24894647e854890a1656 WatchSource:0}: Error finding container f83e78769b8310e8d48ebdc8295cbe66ddf56a233d2d24894647e854890a1656: Status 404 returned error can't find the container with id f83e78769b8310e8d48ebdc8295cbe66ddf56a233d2d24894647e854890a1656
Jul 28 13:40:59.107260 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:40:59.107238    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-r5mx9" event={"ID":"1d4b06a2-100d-44a8-8410-c2d9c78c2ab5","Type":"ContainerStarted","Data":"f83e78769b8310e8d48ebdc8295cbe66ddf56a233d2d24894647e854890a1656"}
Jul 28 13:41:00.116181 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:41:00.116144    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-r5mx9" event={"ID":"1d4b06a2-100d-44a8-8410-c2d9c78c2ab5","Type":"ContainerStarted","Data":"a80ce89b45e25ef7ddcc44bfe8bbc1e0051cb2d8d9ff573d89a6672b98f8f48c"}
Jul 28 13:41:04.133505 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:41:04.133412    2574 generic.go:356] "Generic (PLEG): container finished" podID="1d4b06a2-100d-44a8-8410-c2d9c78c2ab5" containerID="a80ce89b45e25ef7ddcc44bfe8bbc1e0051cb2d8d9ff573d89a6672b98f8f48c" exitCode=0
Jul 28 13:41:04.133505 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:41:04.133466    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-r5mx9" event={"ID":"1d4b06a2-100d-44a8-8410-c2d9c78c2ab5","Type":"ContainerDied","Data":"a80ce89b45e25ef7ddcc44bfe8bbc1e0051cb2d8d9ff573d89a6672b98f8f48c"}
Jul 28 13:41:04.134620 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:41:04.134604    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 13:41:16.194754 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:41:16.194720    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-r5mx9" event={"ID":"1d4b06a2-100d-44a8-8410-c2d9c78c2ab5","Type":"ContainerStarted","Data":"bd06be64930835e5c9722f4246549a0f8e2492980455ac8949d81d5079410aab"}
Jul 28 13:41:16.195173 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:41:16.194781    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-r5mx9" event={"ID":"1d4b06a2-100d-44a8-8410-c2d9c78c2ab5","Type":"ContainerStarted","Data":"76982366f06e4c628f6a38be5e1cc3d69d07f3241ea591e1f3450984842e9c22"}
Jul 28 13:41:16.195173 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:41:16.195089    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-r5mx9"
Jul 28 13:41:16.215035 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:41:16.214991    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-r5mx9" podStartSLOduration=2.345711018 podStartE2EDuration="18.214977147s" totalImagesPullingTime="11.139709957s" totalInitContainerRuntime="4.729556172s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:40:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:41:16.214412671 +0000 UTC m=+2888.419799273" watchObservedRunningTime="2026-07-28 13:41:16.214977147 +0000 UTC m=+2888.420363748"
Jul 28 13:41:17.199738 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:41:17.199712    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-r5mx9"
Jul 28 13:41:17.200789 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:41:17.200753    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-r5mx9" podUID="1d4b06a2-100d-44a8-8410-c2d9c78c2ab5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Jul 28 13:41:18.204495 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:41:18.204456    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-r5mx9" podUID="1d4b06a2-100d-44a8-8410-c2d9c78c2ab5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Jul 28 13:41:23.209213 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:41:23.209184    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-r5mx9"
Jul 28 13:41:23.209669 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:41:23.209640    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-r5mx9" podUID="1d4b06a2-100d-44a8-8410-c2d9c78c2ab5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Jul 28 13:41:33.209759 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:41:33.209724    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-r5mx9" podUID="1d4b06a2-100d-44a8-8410-c2d9c78c2ab5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Jul 28 13:41:43.209638 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:41:43.209602    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-r5mx9" podUID="1d4b06a2-100d-44a8-8410-c2d9c78c2ab5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Jul 28 13:41:53.210532 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:41:53.210495    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-r5mx9" podUID="1d4b06a2-100d-44a8-8410-c2d9c78c2ab5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Jul 28 13:42:03.211255 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:03.211226    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-r5mx9"
Jul 28 13:42:09.740066 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:09.740036    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-r5mx9"]
Jul 28 13:42:09.740484 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:09.740273    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-r5mx9" podUID="1d4b06a2-100d-44a8-8410-c2d9c78c2ab5" containerName="kserve-container" containerID="cri-o://76982366f06e4c628f6a38be5e1cc3d69d07f3241ea591e1f3450984842e9c22" gracePeriod=30
Jul 28 13:42:09.740484 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:09.740318    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-r5mx9" podUID="1d4b06a2-100d-44a8-8410-c2d9c78c2ab5" containerName="kube-rbac-proxy" containerID="cri-o://bd06be64930835e5c9722f4246549a0f8e2492980455ac8949d81d5079410aab" gracePeriod=30
Jul 28 13:42:10.433621 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:10.433586    2574 generic.go:356] "Generic (PLEG): container finished" podID="1d4b06a2-100d-44a8-8410-c2d9c78c2ab5" containerID="bd06be64930835e5c9722f4246549a0f8e2492980455ac8949d81d5079410aab" exitCode=2
Jul 28 13:42:10.433803 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:10.433650    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-r5mx9" event={"ID":"1d4b06a2-100d-44a8-8410-c2d9c78c2ab5","Type":"ContainerDied","Data":"bd06be64930835e5c9722f4246549a0f8e2492980455ac8949d81d5079410aab"}
Jul 28 13:42:11.986653 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:11.986628    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-r5mx9"
Jul 28 13:42:12.117656 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:12.117599    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c4l4z\" (UniqueName: \"kubernetes.io/projected/1d4b06a2-100d-44a8-8410-c2d9c78c2ab5-kube-api-access-c4l4z\") pod \"1d4b06a2-100d-44a8-8410-c2d9c78c2ab5\" (UID: \"1d4b06a2-100d-44a8-8410-c2d9c78c2ab5\") "
Jul 28 13:42:12.117656 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:12.117638    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d4b06a2-100d-44a8-8410-c2d9c78c2ab5-kube-rbac-proxy-sar-config\") pod \"1d4b06a2-100d-44a8-8410-c2d9c78c2ab5\" (UID: \"1d4b06a2-100d-44a8-8410-c2d9c78c2ab5\") "
Jul 28 13:42:12.117817 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:12.117659    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1d4b06a2-100d-44a8-8410-c2d9c78c2ab5-cabundle-cert\") pod \"1d4b06a2-100d-44a8-8410-c2d9c78c2ab5\" (UID: \"1d4b06a2-100d-44a8-8410-c2d9c78c2ab5\") "
Jul 28 13:42:12.117884 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:12.117865    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1d4b06a2-100d-44a8-8410-c2d9c78c2ab5-kserve-provision-location\") pod \"1d4b06a2-100d-44a8-8410-c2d9c78c2ab5\" (UID: \"1d4b06a2-100d-44a8-8410-c2d9c78c2ab5\") "
Jul 28 13:42:12.117945 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:12.117909    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d4b06a2-100d-44a8-8410-c2d9c78c2ab5-proxy-tls\") pod \"1d4b06a2-100d-44a8-8410-c2d9c78c2ab5\" (UID: \"1d4b06a2-100d-44a8-8410-c2d9c78c2ab5\") "
Jul 28 13:42:12.118005 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:12.117979    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d4b06a2-100d-44a8-8410-c2d9c78c2ab5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1d4b06a2-100d-44a8-8410-c2d9c78c2ab5" (UID: "1d4b06a2-100d-44a8-8410-c2d9c78c2ab5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:42:12.118063 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:12.118005    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d4b06a2-100d-44a8-8410-c2d9c78c2ab5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1d4b06a2-100d-44a8-8410-c2d9c78c2ab5" (UID: "1d4b06a2-100d-44a8-8410-c2d9c78c2ab5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:42:12.118249 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:12.118229    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d4b06a2-100d-44a8-8410-c2d9c78c2ab5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:42:12.118335 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:12.118256    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1d4b06a2-100d-44a8-8410-c2d9c78c2ab5-cabundle-cert\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:42:12.119640 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:12.119619    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d4b06a2-100d-44a8-8410-c2d9c78c2ab5-kube-api-access-c4l4z" (OuterVolumeSpecName: "kube-api-access-c4l4z") pod "1d4b06a2-100d-44a8-8410-c2d9c78c2ab5" (UID: "1d4b06a2-100d-44a8-8410-c2d9c78c2ab5"). InnerVolumeSpecName "kube-api-access-c4l4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:42:12.119815 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:12.119796    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d4b06a2-100d-44a8-8410-c2d9c78c2ab5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1d4b06a2-100d-44a8-8410-c2d9c78c2ab5" (UID: "1d4b06a2-100d-44a8-8410-c2d9c78c2ab5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:42:12.127261 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:12.127230    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d4b06a2-100d-44a8-8410-c2d9c78c2ab5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1d4b06a2-100d-44a8-8410-c2d9c78c2ab5" (UID: "1d4b06a2-100d-44a8-8410-c2d9c78c2ab5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:42:12.219528 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:12.219497    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1d4b06a2-100d-44a8-8410-c2d9c78c2ab5-kserve-provision-location\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:42:12.219528 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:12.219529    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d4b06a2-100d-44a8-8410-c2d9c78c2ab5-proxy-tls\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:42:12.219654 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:12.219540    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4l4z\" (UniqueName: \"kubernetes.io/projected/1d4b06a2-100d-44a8-8410-c2d9c78c2ab5-kube-api-access-c4l4z\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:42:12.443020 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:12.442993    2574 generic.go:356] "Generic (PLEG): container finished" podID="1d4b06a2-100d-44a8-8410-c2d9c78c2ab5" containerID="76982366f06e4c628f6a38be5e1cc3d69d07f3241ea591e1f3450984842e9c22" exitCode=0
Jul 28 13:42:12.443120 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:12.443054    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-r5mx9"
Jul 28 13:42:12.443120 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:12.443082    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-r5mx9" event={"ID":"1d4b06a2-100d-44a8-8410-c2d9c78c2ab5","Type":"ContainerDied","Data":"76982366f06e4c628f6a38be5e1cc3d69d07f3241ea591e1f3450984842e9c22"}
Jul 28 13:42:12.443203 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:12.443165    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-r5mx9" event={"ID":"1d4b06a2-100d-44a8-8410-c2d9c78c2ab5","Type":"ContainerDied","Data":"f83e78769b8310e8d48ebdc8295cbe66ddf56a233d2d24894647e854890a1656"}
Jul 28 13:42:12.443203 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:12.443195    2574 scope.go:117] "RemoveContainer" containerID="bd06be64930835e5c9722f4246549a0f8e2492980455ac8949d81d5079410aab"
Jul 28 13:42:12.451566 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:12.451550    2574 scope.go:117] "RemoveContainer" containerID="76982366f06e4c628f6a38be5e1cc3d69d07f3241ea591e1f3450984842e9c22"
Jul 28 13:42:12.460556 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:12.460534    2574 scope.go:117] "RemoveContainer" containerID="a80ce89b45e25ef7ddcc44bfe8bbc1e0051cb2d8d9ff573d89a6672b98f8f48c"
Jul 28 13:42:12.466006 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:12.465982    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-r5mx9"]
Jul 28 13:42:12.467407 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:12.467392    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-r5mx9"]
Jul 28 13:42:12.468872 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:12.468858    2574 scope.go:117] "RemoveContainer" containerID="bd06be64930835e5c9722f4246549a0f8e2492980455ac8949d81d5079410aab"
Jul 28 13:42:12.469134 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:42:12.469113    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd06be64930835e5c9722f4246549a0f8e2492980455ac8949d81d5079410aab\": container with ID starting with bd06be64930835e5c9722f4246549a0f8e2492980455ac8949d81d5079410aab not found: ID does not exist" containerID="bd06be64930835e5c9722f4246549a0f8e2492980455ac8949d81d5079410aab"
Jul 28 13:42:12.469218 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:12.469140    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd06be64930835e5c9722f4246549a0f8e2492980455ac8949d81d5079410aab"} err="failed to get container status \"bd06be64930835e5c9722f4246549a0f8e2492980455ac8949d81d5079410aab\": rpc error: code = NotFound desc = could not find container \"bd06be64930835e5c9722f4246549a0f8e2492980455ac8949d81d5079410aab\": container with ID starting with bd06be64930835e5c9722f4246549a0f8e2492980455ac8949d81d5079410aab not found: ID does not exist"
Jul 28 13:42:12.469218 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:12.469158    2574 scope.go:117] "RemoveContainer" containerID="76982366f06e4c628f6a38be5e1cc3d69d07f3241ea591e1f3450984842e9c22"
Jul 28 13:42:12.469477 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:42:12.469457    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76982366f06e4c628f6a38be5e1cc3d69d07f3241ea591e1f3450984842e9c22\": container with ID starting with 76982366f06e4c628f6a38be5e1cc3d69d07f3241ea591e1f3450984842e9c22 not found: ID does not exist" containerID="76982366f06e4c628f6a38be5e1cc3d69d07f3241ea591e1f3450984842e9c22"
Jul 28 13:42:12.469533 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:12.469484    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76982366f06e4c628f6a38be5e1cc3d69d07f3241ea591e1f3450984842e9c22"} err="failed to get container status \"76982366f06e4c628f6a38be5e1cc3d69d07f3241ea591e1f3450984842e9c22\": rpc error: code = NotFound desc = could not find container \"76982366f06e4c628f6a38be5e1cc3d69d07f3241ea591e1f3450984842e9c22\": container with ID starting with 76982366f06e4c628f6a38be5e1cc3d69d07f3241ea591e1f3450984842e9c22 not found: ID does not exist"
Jul 28 13:42:12.469533 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:12.469511    2574 scope.go:117] "RemoveContainer" containerID="a80ce89b45e25ef7ddcc44bfe8bbc1e0051cb2d8d9ff573d89a6672b98f8f48c"
Jul 28 13:42:12.469826 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:42:12.469808    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a80ce89b45e25ef7ddcc44bfe8bbc1e0051cb2d8d9ff573d89a6672b98f8f48c\": container with ID starting with a80ce89b45e25ef7ddcc44bfe8bbc1e0051cb2d8d9ff573d89a6672b98f8f48c not found: ID does not exist" containerID="a80ce89b45e25ef7ddcc44bfe8bbc1e0051cb2d8d9ff573d89a6672b98f8f48c"
Jul 28 13:42:12.469886 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:12.469830    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a80ce89b45e25ef7ddcc44bfe8bbc1e0051cb2d8d9ff573d89a6672b98f8f48c"} err="failed to get container status \"a80ce89b45e25ef7ddcc44bfe8bbc1e0051cb2d8d9ff573d89a6672b98f8f48c\": rpc error: code = NotFound desc = could not find container \"a80ce89b45e25ef7ddcc44bfe8bbc1e0051cb2d8d9ff573d89a6672b98f8f48c\": container with ID starting with a80ce89b45e25ef7ddcc44bfe8bbc1e0051cb2d8d9ff573d89a6672b98f8f48c not found: ID does not exist"
Jul 28 13:42:14.301461 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:14.301433    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d4b06a2-100d-44a8-8410-c2d9c78c2ab5" path="/var/lib/kubelet/pods/1d4b06a2-100d-44a8-8410-c2d9c78c2ab5/volumes"
Jul 28 13:42:14.889141 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:14.889111    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tp9mn"]
Jul 28 13:42:14.889422 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:14.889410    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d4b06a2-100d-44a8-8410-c2d9c78c2ab5" containerName="storage-initializer"
Jul 28 13:42:14.889466 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:14.889424    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d4b06a2-100d-44a8-8410-c2d9c78c2ab5" containerName="storage-initializer"
Jul 28 13:42:14.889466 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:14.889434    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d4b06a2-100d-44a8-8410-c2d9c78c2ab5" containerName="kube-rbac-proxy"
Jul 28 13:42:14.889466 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:14.889440    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d4b06a2-100d-44a8-8410-c2d9c78c2ab5" containerName="kube-rbac-proxy"
Jul 28 13:42:14.889466 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:14.889461    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d4b06a2-100d-44a8-8410-c2d9c78c2ab5" containerName="kserve-container"
Jul 28 13:42:14.889466 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:14.889467    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d4b06a2-100d-44a8-8410-c2d9c78c2ab5" containerName="kserve-container"
Jul 28 13:42:14.889635 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:14.889526    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d4b06a2-100d-44a8-8410-c2d9c78c2ab5" containerName="kube-rbac-proxy"
Jul 28 13:42:14.889635 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:14.889534    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d4b06a2-100d-44a8-8410-c2d9c78c2ab5" containerName="kserve-container"
Jul 28 13:42:14.893969 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:14.893950    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tp9mn"
Jul 28 13:42:14.896471 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:14.896451    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 13:42:14.896471 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:14.896461    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:42:14.896634 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:14.896546    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:42:14.896821 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:14.896805    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\""
Jul 28 13:42:14.896880 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:14.896833    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:42:14.897615 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:14.897601    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rxfwd\""
Jul 28 13:42:14.897665 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:14.897637    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-predictor-serving-cert\""
Jul 28 13:42:14.906934 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:14.906915    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tp9mn"]
Jul 28 13:42:15.040247 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:15.040217    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-tp9mn\" (UID: \"f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tp9mn"
Jul 28 13:42:15.040393 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:15.040254    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-tp9mn\" (UID: \"f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tp9mn"
Jul 28 13:42:15.040393 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:15.040324    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qhmp\" (UniqueName: \"kubernetes.io/projected/f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9-kube-api-access-6qhmp\") pod \"isvc-paddle-runtime-predictor-58b99fff76-tp9mn\" (UID: \"f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tp9mn"
Jul 28 13:42:15.040393 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:15.040353    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-tp9mn\" (UID: \"f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tp9mn"
Jul 28 13:42:15.040510 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:15.040439    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-tp9mn\" (UID: \"f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tp9mn"
Jul 28 13:42:15.141730 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:15.141655    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-tp9mn\" (UID: \"f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tp9mn"
Jul 28 13:42:15.141730 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:15.141708    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-tp9mn\" (UID: \"f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tp9mn"
Jul 28 13:42:15.141902 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:15.141737    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-tp9mn\" (UID: \"f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tp9mn"
Jul 28 13:42:15.141902 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:15.141771    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qhmp\" (UniqueName: \"kubernetes.io/projected/f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9-kube-api-access-6qhmp\") pod \"isvc-paddle-runtime-predictor-58b99fff76-tp9mn\" (UID: \"f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tp9mn"
Jul 28 13:42:15.141902 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:15.141795    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-tp9mn\" (UID: \"f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tp9mn"
Jul 28 13:42:15.142030 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:42:15.141903    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-runtime-predictor-serving-cert: secret "isvc-paddle-runtime-predictor-serving-cert" not found
Jul 28 13:42:15.142030 ip-10-0-138-158 kubenswrapper[2574]: E0728 13:42:15.141981    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9-proxy-tls podName:f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9 nodeName:}" failed. No retries permitted until 2026-07-28 13:42:15.64194959 +0000 UTC m=+2947.847336176 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9-proxy-tls") pod "isvc-paddle-runtime-predictor-58b99fff76-tp9mn" (UID: "f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9") : secret "isvc-paddle-runtime-predictor-serving-cert" not found
Jul 28 13:42:15.142128 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:15.142110    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-tp9mn\" (UID: \"f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tp9mn"
Jul 28 13:42:15.142299 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:15.142282    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-tp9mn\" (UID: \"f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tp9mn"
Jul 28 13:42:15.142385 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:15.142369    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-tp9mn\" (UID: \"f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tp9mn"
Jul 28 13:42:15.151196 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:15.151172    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qhmp\" (UniqueName: \"kubernetes.io/projected/f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9-kube-api-access-6qhmp\") pod \"isvc-paddle-runtime-predictor-58b99fff76-tp9mn\" (UID: \"f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tp9mn"
Jul 28 13:42:15.644719 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:15.644674    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-tp9mn\" (UID: \"f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tp9mn"
Jul 28 13:42:15.646968 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:15.646949    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-tp9mn\" (UID: \"f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tp9mn"
Jul 28 13:42:15.804227 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:15.804201    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tp9mn"
Jul 28 13:42:15.930873 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:15.930848    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tp9mn"]
Jul 28 13:42:15.932902 ip-10-0-138-158 kubenswrapper[2574]: W0728 13:42:15.932871    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf67b93bf_aaee_4c14_a6b4_ea4a1fd005e9.slice/crio-02107d974f71439f352fd0e2b92b2a5821948ba90ff372cda5ff3ae9c280fe73 WatchSource:0}: Error finding container 02107d974f71439f352fd0e2b92b2a5821948ba90ff372cda5ff3ae9c280fe73: Status 404 returned error can't find the container with id 02107d974f71439f352fd0e2b92b2a5821948ba90ff372cda5ff3ae9c280fe73
Jul 28 13:42:16.462658 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:16.462617    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tp9mn" event={"ID":"f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9","Type":"ContainerStarted","Data":"10463bd0ab94e46c0a844e4d87798c0729e61768dc574ce3c77909d1da9913d6"}
Jul 28 13:42:16.462837 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:16.462710    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tp9mn" event={"ID":"f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9","Type":"ContainerStarted","Data":"02107d974f71439f352fd0e2b92b2a5821948ba90ff372cda5ff3ae9c280fe73"}
Jul 28 13:42:21.486656 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:21.486624    2574 generic.go:356] "Generic (PLEG): container finished" podID="f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9" containerID="10463bd0ab94e46c0a844e4d87798c0729e61768dc574ce3c77909d1da9913d6" exitCode=0
Jul 28 13:42:21.486656 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:21.486660    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tp9mn" event={"ID":"f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9","Type":"ContainerDied","Data":"10463bd0ab94e46c0a844e4d87798c0729e61768dc574ce3c77909d1da9913d6"}
Jul 28 13:42:22.492770 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:22.492735    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tp9mn" event={"ID":"f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9","Type":"ContainerStarted","Data":"0ee42280de76c74bfd273e449bcd76766ca025381f9bec1c6c8ae2e7f18ec9e0"}
Jul 28 13:42:22.493127 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:22.492794    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tp9mn" event={"ID":"f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9","Type":"ContainerStarted","Data":"8d384e1bcbd808d81ff6dd4c7faeb16a025a402b5d665a411f61fb2b184947e3"}
Jul 28 13:42:22.493127 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:22.492970    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tp9mn"
Jul 28 13:42:22.519295 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:22.519253    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tp9mn" podStartSLOduration=3.974536252 podStartE2EDuration="8.519239986s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.544703734s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:42:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:42:22.51683114 +0000 UTC m=+2954.722217741" watchObservedRunningTime="2026-07-28 13:42:22.519239986 +0000 UTC m=+2954.724626587"
Jul 28 13:42:23.497065 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:23.497031    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tp9mn"
Jul 28 13:42:23.498327 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:23.498300    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tp9mn" podUID="f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Jul 28 13:42:24.502312 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:24.502256    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tp9mn" podUID="f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Jul 28 13:42:29.507192 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:29.507160    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tp9mn"
Jul 28 13:42:29.507815 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:29.507790    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tp9mn" podUID="f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Jul 28 13:42:39.508433 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:39.508341    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tp9mn" podUID="f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Jul 28 13:42:49.508378 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:49.508340    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tp9mn" podUID="f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Jul 28 13:42:59.507671 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:42:59.507632    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tp9mn" podUID="f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Jul 28 13:43:09.508837 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:43:09.508808    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tp9mn"
Jul 28 13:43:16.445875 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:43:16.445846    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tp9mn"]
Jul 28 13:43:16.446247 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:43:16.446060    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tp9mn" podUID="f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9" containerName="kserve-container" containerID="cri-o://8d384e1bcbd808d81ff6dd4c7faeb16a025a402b5d665a411f61fb2b184947e3" gracePeriod=30
Jul 28 13:43:16.446247 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:43:16.446078    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tp9mn" podUID="f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9" containerName="kube-rbac-proxy" containerID="cri-o://0ee42280de76c74bfd273e449bcd76766ca025381f9bec1c6c8ae2e7f18ec9e0" gracePeriod=30
Jul 28 13:43:16.738775 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:43:16.738707    2574 generic.go:356] "Generic (PLEG): container finished" podID="f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9" containerID="0ee42280de76c74bfd273e449bcd76766ca025381f9bec1c6c8ae2e7f18ec9e0" exitCode=2
Jul 28 13:43:16.738898 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:43:16.738774    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tp9mn" event={"ID":"f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9","Type":"ContainerDied","Data":"0ee42280de76c74bfd273e449bcd76766ca025381f9bec1c6c8ae2e7f18ec9e0"}
Jul 28 13:43:18.751637 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:43:18.751609    2574 generic.go:356] "Generic (PLEG): container finished" podID="f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9" containerID="8d384e1bcbd808d81ff6dd4c7faeb16a025a402b5d665a411f61fb2b184947e3" exitCode=0
Jul 28 13:43:18.751973 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:43:18.751644    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tp9mn" event={"ID":"f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9","Type":"ContainerDied","Data":"8d384e1bcbd808d81ff6dd4c7faeb16a025a402b5d665a411f61fb2b184947e3"}
Jul 28 13:43:18.790481 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:43:18.790461    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tp9mn"
Jul 28 13:43:18.919103 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:43:18.919046    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9-cabundle-cert\") pod \"f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9\" (UID: \"f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9\") "
Jul 28 13:43:18.919103 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:43:18.919086    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6qhmp\" (UniqueName: \"kubernetes.io/projected/f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9-kube-api-access-6qhmp\") pod \"f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9\" (UID: \"f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9\") "
Jul 28 13:43:18.919246 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:43:18.919107    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9-kserve-provision-location\") pod \"f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9\" (UID: \"f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9\") "
Jul 28 13:43:18.919327 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:43:18.919305    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9-kube-rbac-proxy-sar-config\") pod \"f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9\" (UID: \"f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9\") "
Jul 28 13:43:18.919369 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:43:18.919358    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9-proxy-tls\") pod \"f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9\" (UID: \"f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9\") "
Jul 28 13:43:18.919430 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:43:18.919416    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9" (UID: "f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:43:18.919595 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:43:18.919578    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9-cabundle-cert\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:43:18.919704 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:43:18.919658    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9" (UID: "f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:43:18.921162 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:43:18.921138    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9-kube-api-access-6qhmp" (OuterVolumeSpecName: "kube-api-access-6qhmp") pod "f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9" (UID: "f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9"). InnerVolumeSpecName "kube-api-access-6qhmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:43:18.921273 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:43:18.921227    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9" (UID: "f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:43:18.928878 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:43:18.928856    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9" (UID: "f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:43:19.019948 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:43:19.019919    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:43:19.020031 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:43:19.019948    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9-proxy-tls\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:43:19.020031 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:43:19.019962    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qhmp\" (UniqueName: \"kubernetes.io/projected/f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9-kube-api-access-6qhmp\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:43:19.020031 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:43:19.019973    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9-kserve-provision-location\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 13:43:19.758424 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:43:19.758393    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tp9mn"
Jul 28 13:43:19.758424 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:43:19.758414    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tp9mn" event={"ID":"f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9","Type":"ContainerDied","Data":"02107d974f71439f352fd0e2b92b2a5821948ba90ff372cda5ff3ae9c280fe73"}
Jul 28 13:43:19.758900 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:43:19.758486    2574 scope.go:117] "RemoveContainer" containerID="0ee42280de76c74bfd273e449bcd76766ca025381f9bec1c6c8ae2e7f18ec9e0"
Jul 28 13:43:19.767206 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:43:19.767185    2574 scope.go:117] "RemoveContainer" containerID="8d384e1bcbd808d81ff6dd4c7faeb16a025a402b5d665a411f61fb2b184947e3"
Jul 28 13:43:19.774369 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:43:19.774354    2574 scope.go:117] "RemoveContainer" containerID="10463bd0ab94e46c0a844e4d87798c0729e61768dc574ce3c77909d1da9913d6"
Jul 28 13:43:19.785353 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:43:19.785331    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tp9mn"]
Jul 28 13:43:19.789430 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:43:19.789406    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tp9mn"]
Jul 28 13:43:20.302244 ip-10-0-138-158 kubenswrapper[2574]: I0728 13:43:20.302203    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9" path="/var/lib/kubelet/pods/f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9/volumes"
Jul 28 14:04:45.145738 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:04:45.145706    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-kjqj2"]
Jul 28 14:04:45.146187 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:04:45.146028    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9" containerName="storage-initializer"
Jul 28 14:04:45.146187 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:04:45.146040    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9" containerName="storage-initializer"
Jul 28 14:04:45.146187 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:04:45.146058    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9" containerName="kube-rbac-proxy"
Jul 28 14:04:45.146187 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:04:45.146063    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9" containerName="kube-rbac-proxy"
Jul 28 14:04:45.146187 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:04:45.146072    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9" containerName="kserve-container"
Jul 28 14:04:45.146187 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:04:45.146079    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9" containerName="kserve-container"
Jul 28 14:04:45.146187 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:04:45.146129    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9" containerName="kserve-container"
Jul 28 14:04:45.146187 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:04:45.146139    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f67b93bf-aaee-4c14-a6b4-ea4a1fd005e9" containerName="kube-rbac-proxy"
Jul 28 14:04:45.149186 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:04:45.149172    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-kjqj2"
Jul 28 14:04:45.153567 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:04:45.153543    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-predictor-serving-cert\""
Jul 28 14:04:45.153863 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:04:45.153847    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 14:04:45.153926 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:04:45.153847    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\""
Jul 28 14:04:45.153926 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:04:45.153847    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 14:04:45.154047 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:04:45.153959    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 14:04:45.154720 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:04:45.154698    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rxfwd\""
Jul 28 14:04:45.154824 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:04:45.154748    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 14:04:45.164864 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:04:45.164845    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-kjqj2"]
Jul 28 14:04:45.233065 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:04:45.233036    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df78202e-f013-4541-b6af-06b47fc27fa1-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-kjqj2\" (UID: \"df78202e-f013-4541-b6af-06b47fc27fa1\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-kjqj2"
Jul 28 14:04:45.233223 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:04:45.233078    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df78202e-f013-4541-b6af-06b47fc27fa1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-kjqj2\" (UID: \"df78202e-f013-4541-b6af-06b47fc27fa1\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-kjqj2"
Jul 28 14:04:45.233223 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:04:45.233105    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df78202e-f013-4541-b6af-06b47fc27fa1-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-kjqj2\" (UID: \"df78202e-f013-4541-b6af-06b47fc27fa1\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-kjqj2"
Jul 28 14:04:45.233223 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:04:45.233172    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7wrh\" (UniqueName: \"kubernetes.io/projected/df78202e-f013-4541-b6af-06b47fc27fa1-kube-api-access-v7wrh\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-kjqj2\" (UID: \"df78202e-f013-4541-b6af-06b47fc27fa1\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-kjqj2"
Jul 28 14:04:45.233223 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:04:45.233222    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df78202e-f013-4541-b6af-06b47fc27fa1-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-kjqj2\" (UID: \"df78202e-f013-4541-b6af-06b47fc27fa1\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-kjqj2"
Jul 28 14:04:45.334105 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:04:45.334075    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df78202e-f013-4541-b6af-06b47fc27fa1-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-kjqj2\" (UID: \"df78202e-f013-4541-b6af-06b47fc27fa1\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-kjqj2"
Jul 28 14:04:45.334260 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:04:45.334114    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df78202e-f013-4541-b6af-06b47fc27fa1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-kjqj2\" (UID: \"df78202e-f013-4541-b6af-06b47fc27fa1\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-kjqj2"
Jul 28 14:04:45.334260 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:04:45.334132    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df78202e-f013-4541-b6af-06b47fc27fa1-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-kjqj2\" (UID: \"df78202e-f013-4541-b6af-06b47fc27fa1\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-kjqj2"
Jul 28 14:04:45.334260 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:04:45.334183    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7wrh\" (UniqueName: \"kubernetes.io/projected/df78202e-f013-4541-b6af-06b47fc27fa1-kube-api-access-v7wrh\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-kjqj2\" (UID: \"df78202e-f013-4541-b6af-06b47fc27fa1\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-kjqj2"
Jul 28 14:04:45.334260 ip-10-0-138-158 kubenswrapper[2574]: E0728 14:04:45.334210    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-serving-cert: secret "isvc-sklearn-runtime-predictor-serving-cert" not found
Jul 28 14:04:45.334260 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:04:45.334213    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df78202e-f013-4541-b6af-06b47fc27fa1-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-kjqj2\" (UID: \"df78202e-f013-4541-b6af-06b47fc27fa1\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-kjqj2"
Jul 28 14:04:45.334472 ip-10-0-138-158 kubenswrapper[2574]: E0728 14:04:45.334276    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df78202e-f013-4541-b6af-06b47fc27fa1-proxy-tls podName:df78202e-f013-4541-b6af-06b47fc27fa1 nodeName:}" failed. No retries permitted until 2026-07-28 14:04:45.834257851 +0000 UTC m=+4298.039644434 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/df78202e-f013-4541-b6af-06b47fc27fa1-proxy-tls") pod "isvc-sklearn-runtime-predictor-599b58cf6-kjqj2" (UID: "df78202e-f013-4541-b6af-06b47fc27fa1") : secret "isvc-sklearn-runtime-predictor-serving-cert" not found
Jul 28 14:04:45.334602 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:04:45.334585    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df78202e-f013-4541-b6af-06b47fc27fa1-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-kjqj2\" (UID: \"df78202e-f013-4541-b6af-06b47fc27fa1\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-kjqj2"
Jul 28 14:04:45.334775 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:04:45.334760    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df78202e-f013-4541-b6af-06b47fc27fa1-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-kjqj2\" (UID: \"df78202e-f013-4541-b6af-06b47fc27fa1\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-kjqj2"
Jul 28 14:04:45.334841 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:04:45.334824    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df78202e-f013-4541-b6af-06b47fc27fa1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-kjqj2\" (UID: \"df78202e-f013-4541-b6af-06b47fc27fa1\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-kjqj2"
Jul 28 14:04:45.350729 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:04:45.350680    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7wrh\" (UniqueName: \"kubernetes.io/projected/df78202e-f013-4541-b6af-06b47fc27fa1-kube-api-access-v7wrh\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-kjqj2\" (UID: \"df78202e-f013-4541-b6af-06b47fc27fa1\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-kjqj2"
Jul 28 14:04:45.838336 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:04:45.838299    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df78202e-f013-4541-b6af-06b47fc27fa1-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-kjqj2\" (UID: \"df78202e-f013-4541-b6af-06b47fc27fa1\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-kjqj2"
Jul 28 14:04:45.840653 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:04:45.840625    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df78202e-f013-4541-b6af-06b47fc27fa1-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-kjqj2\" (UID: \"df78202e-f013-4541-b6af-06b47fc27fa1\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-kjqj2"
Jul 28 14:04:46.059403 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:04:46.059377    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-kjqj2"
Jul 28 14:04:46.187836 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:04:46.187813    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-kjqj2"]
Jul 28 14:04:46.189758 ip-10-0-138-158 kubenswrapper[2574]: W0728 14:04:46.189729    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf78202e_f013_4541_b6af_06b47fc27fa1.slice/crio-c6726b60643e6bb4ca02c15c5ad1ce625b412b4da31837464e44308cc4129376 WatchSource:0}: Error finding container c6726b60643e6bb4ca02c15c5ad1ce625b412b4da31837464e44308cc4129376: Status 404 returned error can't find the container with id c6726b60643e6bb4ca02c15c5ad1ce625b412b4da31837464e44308cc4129376
Jul 28 14:04:46.233478 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:04:46.233441    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-kjqj2" event={"ID":"df78202e-f013-4541-b6af-06b47fc27fa1","Type":"ContainerStarted","Data":"c6726b60643e6bb4ca02c15c5ad1ce625b412b4da31837464e44308cc4129376"}
Jul 28 14:04:47.238936 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:04:47.238901    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-kjqj2" event={"ID":"df78202e-f013-4541-b6af-06b47fc27fa1","Type":"ContainerStarted","Data":"c741ebce7474b26b77b3a086f47c7d25ebdf7b5b001deee000c3994b0c589b48"}
Jul 28 14:04:52.262537 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:04:52.262508    2574 generic.go:356] "Generic (PLEG): container finished" podID="df78202e-f013-4541-b6af-06b47fc27fa1" containerID="c741ebce7474b26b77b3a086f47c7d25ebdf7b5b001deee000c3994b0c589b48" exitCode=0
Jul 28 14:04:52.262912 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:04:52.262582    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-kjqj2" event={"ID":"df78202e-f013-4541-b6af-06b47fc27fa1","Type":"ContainerDied","Data":"c741ebce7474b26b77b3a086f47c7d25ebdf7b5b001deee000c3994b0c589b48"}
Jul 28 14:04:53.269332 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:04:53.269298    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-kjqj2" event={"ID":"df78202e-f013-4541-b6af-06b47fc27fa1","Type":"ContainerStarted","Data":"41d5209b18233f64ddab6f9347c68b9450548889086eeca49cce32e8b12bbcdd"}
Jul 28 14:04:53.269841 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:04:53.269360    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-kjqj2" event={"ID":"df78202e-f013-4541-b6af-06b47fc27fa1","Type":"ContainerStarted","Data":"a4a0179e338b9b104a2ffebb19b533bf546a0be9a91211cfddd235da6145738d"}
Jul 28 14:04:53.269841 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:04:53.269627    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-kjqj2"
Jul 28 14:04:53.269841 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:04:53.269790    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-kjqj2"
Jul 28 14:04:53.271211 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:04:53.271184    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-kjqj2" podUID="df78202e-f013-4541-b6af-06b47fc27fa1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Jul 28 14:04:53.300741 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:04:53.300672    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-kjqj2" podStartSLOduration=2.526420838 podStartE2EDuration="8.300658579s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.774237741s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:04:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:04:53.299916142 +0000 UTC m=+4305.505302743" watchObservedRunningTime="2026-07-28 14:04:53.300658579 +0000 UTC m=+4305.506045180"
Jul 28 14:04:54.274902 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:04:54.274857    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-kjqj2" podUID="df78202e-f013-4541-b6af-06b47fc27fa1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Jul 28 14:04:59.279571 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:04:59.279537    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-kjqj2"
Jul 28 14:04:59.280981 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:04:59.280933    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-kjqj2" podUID="df78202e-f013-4541-b6af-06b47fc27fa1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Jul 28 14:05:09.281738 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:05:09.281653    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-kjqj2"
Jul 28 14:05:16.813763 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:05:16.813732    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-599b58cf6-kjqj2_df78202e-f013-4541-b6af-06b47fc27fa1/kserve-container/0.log"
Jul 28 14:05:16.977615 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:05:16.977579    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-kjqj2"]
Jul 28 14:05:16.977885 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:05:16.977862    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-kjqj2" podUID="df78202e-f013-4541-b6af-06b47fc27fa1" containerName="kserve-container" containerID="cri-o://a4a0179e338b9b104a2ffebb19b533bf546a0be9a91211cfddd235da6145738d" gracePeriod=30
Jul 28 14:05:16.977986 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:05:16.977880    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-kjqj2" podUID="df78202e-f013-4541-b6af-06b47fc27fa1" containerName="kube-rbac-proxy" containerID="cri-o://41d5209b18233f64ddab6f9347c68b9450548889086eeca49cce32e8b12bbcdd" gracePeriod=30
Jul 28 14:05:17.389829 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:05:17.389745    2574 generic.go:356] "Generic (PLEG): container finished" podID="df78202e-f013-4541-b6af-06b47fc27fa1" containerID="41d5209b18233f64ddab6f9347c68b9450548889086eeca49cce32e8b12bbcdd" exitCode=2
Jul 28 14:05:17.389956 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:05:17.389825    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-kjqj2" event={"ID":"df78202e-f013-4541-b6af-06b47fc27fa1","Type":"ContainerDied","Data":"41d5209b18233f64ddab6f9347c68b9450548889086eeca49cce32e8b12bbcdd"}
Jul 28 14:05:17.915185 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:05:17.915163    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-kjqj2"
Jul 28 14:05:17.999973 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:05:17.999909    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v7wrh\" (UniqueName: \"kubernetes.io/projected/df78202e-f013-4541-b6af-06b47fc27fa1-kube-api-access-v7wrh\") pod \"df78202e-f013-4541-b6af-06b47fc27fa1\" (UID: \"df78202e-f013-4541-b6af-06b47fc27fa1\") "
Jul 28 14:05:17.999973 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:05:17.999950    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df78202e-f013-4541-b6af-06b47fc27fa1-proxy-tls\") pod \"df78202e-f013-4541-b6af-06b47fc27fa1\" (UID: \"df78202e-f013-4541-b6af-06b47fc27fa1\") "
Jul 28 14:05:18.000151 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:05:17.999992    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df78202e-f013-4541-b6af-06b47fc27fa1-cabundle-cert\") pod \"df78202e-f013-4541-b6af-06b47fc27fa1\" (UID: \"df78202e-f013-4541-b6af-06b47fc27fa1\") "
Jul 28 14:05:18.000151 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:05:18.000035    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df78202e-f013-4541-b6af-06b47fc27fa1-kserve-provision-location\") pod \"df78202e-f013-4541-b6af-06b47fc27fa1\" (UID: \"df78202e-f013-4541-b6af-06b47fc27fa1\") "
Jul 28 14:05:18.000151 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:05:18.000121    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df78202e-f013-4541-b6af-06b47fc27fa1-kube-rbac-proxy-sar-config\") pod \"df78202e-f013-4541-b6af-06b47fc27fa1\" (UID: \"df78202e-f013-4541-b6af-06b47fc27fa1\") "
Jul 28 14:05:18.000491 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:05:18.000463    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df78202e-f013-4541-b6af-06b47fc27fa1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "df78202e-f013-4541-b6af-06b47fc27fa1" (UID: "df78202e-f013-4541-b6af-06b47fc27fa1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:05:18.000612 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:05:18.000531    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df78202e-f013-4541-b6af-06b47fc27fa1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "df78202e-f013-4541-b6af-06b47fc27fa1" (UID: "df78202e-f013-4541-b6af-06b47fc27fa1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:05:18.002016 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:05:18.001990    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df78202e-f013-4541-b6af-06b47fc27fa1-kube-api-access-v7wrh" (OuterVolumeSpecName: "kube-api-access-v7wrh") pod "df78202e-f013-4541-b6af-06b47fc27fa1" (UID: "df78202e-f013-4541-b6af-06b47fc27fa1"). InnerVolumeSpecName "kube-api-access-v7wrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:05:18.002125 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:05:18.002059    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df78202e-f013-4541-b6af-06b47fc27fa1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "df78202e-f013-4541-b6af-06b47fc27fa1" (UID: "df78202e-f013-4541-b6af-06b47fc27fa1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 14:05:18.024983 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:05:18.024952    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df78202e-f013-4541-b6af-06b47fc27fa1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "df78202e-f013-4541-b6af-06b47fc27fa1" (UID: "df78202e-f013-4541-b6af-06b47fc27fa1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:05:18.101479 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:05:18.101454    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df78202e-f013-4541-b6af-06b47fc27fa1-cabundle-cert\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 14:05:18.101479 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:05:18.101478    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df78202e-f013-4541-b6af-06b47fc27fa1-kserve-provision-location\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 14:05:18.101615 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:05:18.101489    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df78202e-f013-4541-b6af-06b47fc27fa1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 14:05:18.101615 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:05:18.101499    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7wrh\" (UniqueName: \"kubernetes.io/projected/df78202e-f013-4541-b6af-06b47fc27fa1-kube-api-access-v7wrh\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 14:05:18.101615 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:05:18.101509    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df78202e-f013-4541-b6af-06b47fc27fa1-proxy-tls\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 14:05:18.395719 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:05:18.395676    2574 generic.go:356] "Generic (PLEG): container finished" podID="df78202e-f013-4541-b6af-06b47fc27fa1" containerID="a4a0179e338b9b104a2ffebb19b533bf546a0be9a91211cfddd235da6145738d" exitCode=0
Jul 28 14:05:18.395811 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:05:18.395742    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-kjqj2" event={"ID":"df78202e-f013-4541-b6af-06b47fc27fa1","Type":"ContainerDied","Data":"a4a0179e338b9b104a2ffebb19b533bf546a0be9a91211cfddd235da6145738d"}
Jul 28 14:05:18.395811 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:05:18.395753    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-kjqj2"
Jul 28 14:05:18.395811 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:05:18.395794    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-kjqj2" event={"ID":"df78202e-f013-4541-b6af-06b47fc27fa1","Type":"ContainerDied","Data":"c6726b60643e6bb4ca02c15c5ad1ce625b412b4da31837464e44308cc4129376"}
Jul 28 14:05:18.395914 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:05:18.395820    2574 scope.go:117] "RemoveContainer" containerID="41d5209b18233f64ddab6f9347c68b9450548889086eeca49cce32e8b12bbcdd"
Jul 28 14:05:18.404092 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:05:18.404075    2574 scope.go:117] "RemoveContainer" containerID="a4a0179e338b9b104a2ffebb19b533bf546a0be9a91211cfddd235da6145738d"
Jul 28 14:05:18.411087 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:05:18.411071    2574 scope.go:117] "RemoveContainer" containerID="c741ebce7474b26b77b3a086f47c7d25ebdf7b5b001deee000c3994b0c589b48"
Jul 28 14:05:18.416963 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:05:18.416758    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-kjqj2"]
Jul 28 14:05:18.420721 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:05:18.420676    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-kjqj2"]
Jul 28 14:05:18.423856 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:05:18.423837    2574 scope.go:117] "RemoveContainer" containerID="41d5209b18233f64ddab6f9347c68b9450548889086eeca49cce32e8b12bbcdd"
Jul 28 14:05:18.424109 ip-10-0-138-158 kubenswrapper[2574]: E0728 14:05:18.424091    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41d5209b18233f64ddab6f9347c68b9450548889086eeca49cce32e8b12bbcdd\": container with ID starting with 41d5209b18233f64ddab6f9347c68b9450548889086eeca49cce32e8b12bbcdd not found: ID does not exist" containerID="41d5209b18233f64ddab6f9347c68b9450548889086eeca49cce32e8b12bbcdd"
Jul 28 14:05:18.424171 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:05:18.424114    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41d5209b18233f64ddab6f9347c68b9450548889086eeca49cce32e8b12bbcdd"} err="failed to get container status \"41d5209b18233f64ddab6f9347c68b9450548889086eeca49cce32e8b12bbcdd\": rpc error: code = NotFound desc = could not find container \"41d5209b18233f64ddab6f9347c68b9450548889086eeca49cce32e8b12bbcdd\": container with ID starting with 41d5209b18233f64ddab6f9347c68b9450548889086eeca49cce32e8b12bbcdd not found: ID does not exist"
Jul 28 14:05:18.424171 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:05:18.424130    2574 scope.go:117] "RemoveContainer" containerID="a4a0179e338b9b104a2ffebb19b533bf546a0be9a91211cfddd235da6145738d"
Jul 28 14:05:18.424360 ip-10-0-138-158 kubenswrapper[2574]: E0728 14:05:18.424344    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4a0179e338b9b104a2ffebb19b533bf546a0be9a91211cfddd235da6145738d\": container with ID starting with a4a0179e338b9b104a2ffebb19b533bf546a0be9a91211cfddd235da6145738d not found: ID does not exist" containerID="a4a0179e338b9b104a2ffebb19b533bf546a0be9a91211cfddd235da6145738d"
Jul 28 14:05:18.424405 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:05:18.424363    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4a0179e338b9b104a2ffebb19b533bf546a0be9a91211cfddd235da6145738d"} err="failed to get container status \"a4a0179e338b9b104a2ffebb19b533bf546a0be9a91211cfddd235da6145738d\": rpc error: code = NotFound desc = could not find container \"a4a0179e338b9b104a2ffebb19b533bf546a0be9a91211cfddd235da6145738d\": container with ID starting with a4a0179e338b9b104a2ffebb19b533bf546a0be9a91211cfddd235da6145738d not found: ID does not exist"
Jul 28 14:05:18.424405 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:05:18.424382    2574 scope.go:117] "RemoveContainer" containerID="c741ebce7474b26b77b3a086f47c7d25ebdf7b5b001deee000c3994b0c589b48"
Jul 28 14:05:18.424590 ip-10-0-138-158 kubenswrapper[2574]: E0728 14:05:18.424574    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c741ebce7474b26b77b3a086f47c7d25ebdf7b5b001deee000c3994b0c589b48\": container with ID starting with c741ebce7474b26b77b3a086f47c7d25ebdf7b5b001deee000c3994b0c589b48 not found: ID does not exist" containerID="c741ebce7474b26b77b3a086f47c7d25ebdf7b5b001deee000c3994b0c589b48"
Jul 28 14:05:18.424630 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:05:18.424595    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c741ebce7474b26b77b3a086f47c7d25ebdf7b5b001deee000c3994b0c589b48"} err="failed to get container status \"c741ebce7474b26b77b3a086f47c7d25ebdf7b5b001deee000c3994b0c589b48\": rpc error: code = NotFound desc = could not find container \"c741ebce7474b26b77b3a086f47c7d25ebdf7b5b001deee000c3994b0c589b48\": container with ID starting with c741ebce7474b26b77b3a086f47c7d25ebdf7b5b001deee000c3994b0c589b48 not found: ID does not exist"
Jul 28 14:05:20.301867 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:05:20.301835    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df78202e-f013-4541-b6af-06b47fc27fa1" path="/var/lib/kubelet/pods/df78202e-f013-4541-b6af-06b47fc27fa1/volumes"
Jul 28 14:10:33.923670 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:10:33.923639    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9"]
Jul 28 14:10:33.924415 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:10:33.923996    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df78202e-f013-4541-b6af-06b47fc27fa1" containerName="storage-initializer"
Jul 28 14:10:33.924415 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:10:33.924011    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="df78202e-f013-4541-b6af-06b47fc27fa1" containerName="storage-initializer"
Jul 28 14:10:33.924415 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:10:33.924028    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df78202e-f013-4541-b6af-06b47fc27fa1" containerName="kube-rbac-proxy"
Jul 28 14:10:33.924415 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:10:33.924034    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="df78202e-f013-4541-b6af-06b47fc27fa1" containerName="kube-rbac-proxy"
Jul 28 14:10:33.924415 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:10:33.924044    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df78202e-f013-4541-b6af-06b47fc27fa1" containerName="kserve-container"
Jul 28 14:10:33.924415 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:10:33.924050    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="df78202e-f013-4541-b6af-06b47fc27fa1" containerName="kserve-container"
Jul 28 14:10:33.924415 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:10:33.924113    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="df78202e-f013-4541-b6af-06b47fc27fa1" containerName="kube-rbac-proxy"
Jul 28 14:10:33.924415 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:10:33.924121    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="df78202e-f013-4541-b6af-06b47fc27fa1" containerName="kserve-container"
Jul 28 14:10:33.927224 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:10:33.927209    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9"
Jul 28 14:10:33.932529 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:10:33.932509    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 14:10:33.932799 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:10:33.932783    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 14:10:33.933211 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:10:33.933181    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 14:10:33.933211 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:10:33.933194    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rxfwd\""
Jul 28 14:10:33.933211 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:10:33.933183    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 14:10:33.933858 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:10:33.933674    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-predictor-serving-cert\""
Jul 28 14:10:33.933858 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:10:33.933754    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\""
Jul 28 14:10:33.946052 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:10:33.946029    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9"]
Jul 28 14:10:33.998290 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:10:33.998269    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8314a768-d3c1-48cd-aa85-953de1d97400-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9\" (UID: \"8314a768-d3c1-48cd-aa85-953de1d97400\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9"
Jul 28 14:10:33.998410 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:10:33.998302    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bbsc\" (UniqueName: \"kubernetes.io/projected/8314a768-d3c1-48cd-aa85-953de1d97400-kube-api-access-8bbsc\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9\" (UID: \"8314a768-d3c1-48cd-aa85-953de1d97400\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9"
Jul 28 14:10:33.998410 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:10:33.998342    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8314a768-d3c1-48cd-aa85-953de1d97400-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9\" (UID: \"8314a768-d3c1-48cd-aa85-953de1d97400\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9"
Jul 28 14:10:33.998410 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:10:33.998363    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8314a768-d3c1-48cd-aa85-953de1d97400-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9\" (UID: \"8314a768-d3c1-48cd-aa85-953de1d97400\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9"
Jul 28 14:10:33.998410 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:10:33.998384    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8314a768-d3c1-48cd-aa85-953de1d97400-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9\" (UID: \"8314a768-d3c1-48cd-aa85-953de1d97400\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9"
Jul 28 14:10:34.099028 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:10:34.099005    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8314a768-d3c1-48cd-aa85-953de1d97400-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9\" (UID: \"8314a768-d3c1-48cd-aa85-953de1d97400\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9"
Jul 28 14:10:34.099117 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:10:34.099034    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8314a768-d3c1-48cd-aa85-953de1d97400-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9\" (UID: \"8314a768-d3c1-48cd-aa85-953de1d97400\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9"
Jul 28 14:10:34.099117 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:10:34.099050    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8314a768-d3c1-48cd-aa85-953de1d97400-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9\" (UID: \"8314a768-d3c1-48cd-aa85-953de1d97400\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9"
Jul 28 14:10:34.099117 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:10:34.099088    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8314a768-d3c1-48cd-aa85-953de1d97400-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9\" (UID: \"8314a768-d3c1-48cd-aa85-953de1d97400\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9"
Jul 28 14:10:34.099241 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:10:34.099116    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bbsc\" (UniqueName: \"kubernetes.io/projected/8314a768-d3c1-48cd-aa85-953de1d97400-kube-api-access-8bbsc\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9\" (UID: \"8314a768-d3c1-48cd-aa85-953de1d97400\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9"
Jul 28 14:10:34.099372 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:10:34.099351    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8314a768-d3c1-48cd-aa85-953de1d97400-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9\" (UID: \"8314a768-d3c1-48cd-aa85-953de1d97400\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9"
Jul 28 14:10:34.099667 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:10:34.099648    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8314a768-d3c1-48cd-aa85-953de1d97400-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9\" (UID: \"8314a768-d3c1-48cd-aa85-953de1d97400\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9"
Jul 28 14:10:34.099761 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:10:34.099747    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8314a768-d3c1-48cd-aa85-953de1d97400-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9\" (UID: \"8314a768-d3c1-48cd-aa85-953de1d97400\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9"
Jul 28 14:10:34.101507 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:10:34.101482    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8314a768-d3c1-48cd-aa85-953de1d97400-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9\" (UID: \"8314a768-d3c1-48cd-aa85-953de1d97400\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9"
Jul 28 14:10:34.109086 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:10:34.109065    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bbsc\" (UniqueName: \"kubernetes.io/projected/8314a768-d3c1-48cd-aa85-953de1d97400-kube-api-access-8bbsc\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9\" (UID: \"8314a768-d3c1-48cd-aa85-953de1d97400\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9"
Jul 28 14:10:34.237460 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:10:34.237430    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9"
Jul 28 14:10:34.372779 ip-10-0-138-158 kubenswrapper[2574]: W0728 14:10:34.369259    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8314a768_d3c1_48cd_aa85_953de1d97400.slice/crio-593a68e2c24d83565e2422fa0743205bd8c5f93f0dd12b88535dfb1eee668aef WatchSource:0}: Error finding container 593a68e2c24d83565e2422fa0743205bd8c5f93f0dd12b88535dfb1eee668aef: Status 404 returned error can't find the container with id 593a68e2c24d83565e2422fa0743205bd8c5f93f0dd12b88535dfb1eee668aef
Jul 28 14:10:34.372779 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:10:34.372463    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9"]
Jul 28 14:10:34.748011 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:10:34.747980    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9" event={"ID":"8314a768-d3c1-48cd-aa85-953de1d97400","Type":"ContainerStarted","Data":"e222904a5f1da97ee07b43387b8e4e10a13f79b19ad2ff3142b2b5f8f8ecaa98"}
Jul 28 14:10:34.748181 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:10:34.748038    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9" event={"ID":"8314a768-d3c1-48cd-aa85-953de1d97400","Type":"ContainerStarted","Data":"593a68e2c24d83565e2422fa0743205bd8c5f93f0dd12b88535dfb1eee668aef"}
Jul 28 14:10:39.769963 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:10:39.769930    2574 generic.go:356] "Generic (PLEG): container finished" podID="8314a768-d3c1-48cd-aa85-953de1d97400" containerID="e222904a5f1da97ee07b43387b8e4e10a13f79b19ad2ff3142b2b5f8f8ecaa98" exitCode=0
Jul 28 14:10:39.770416 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:10:39.770004    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9" event={"ID":"8314a768-d3c1-48cd-aa85-953de1d97400","Type":"ContainerDied","Data":"e222904a5f1da97ee07b43387b8e4e10a13f79b19ad2ff3142b2b5f8f8ecaa98"}
Jul 28 14:10:39.771134 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:10:39.771121    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 14:10:43.791563 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:10:43.791527    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9" event={"ID":"8314a768-d3c1-48cd-aa85-953de1d97400","Type":"ContainerStarted","Data":"9f8b8a4f4b13536b333bc1a54d7c9fa01f8b6e1d10359e49064fa8b10fb1e6e2"}
Jul 28 14:10:43.791922 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:10:43.791591    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9" event={"ID":"8314a768-d3c1-48cd-aa85-953de1d97400","Type":"ContainerStarted","Data":"2c21d1c614ba5fe95596ae4ecc551e883f610dd339a90d7365ee9aac1f78fda7"}
Jul 28 14:10:43.791922 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:10:43.791797    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9"
Jul 28 14:10:43.817479 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:10:43.817401    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9" podStartSLOduration=2.409714455 podStartE2EDuration="10.817386858s" totalImagesPullingTime="3.778146109s" totalInitContainerRuntime="4.629526294s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:10:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:10:43.814996259 +0000 UTC m=+4656.020382870" watchObservedRunningTime="2026-07-28 14:10:43.817386858 +0000 UTC m=+4656.022773507"
Jul 28 14:10:44.796616 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:10:44.796588    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9"
Jul 28 14:10:44.797742 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:10:44.797721    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9" podUID="8314a768-d3c1-48cd-aa85-953de1d97400" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Jul 28 14:10:45.801095 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:10:45.801056    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9" podUID="8314a768-d3c1-48cd-aa85-953de1d97400" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Jul 28 14:10:50.805488 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:10:50.805460    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9"
Jul 28 14:10:50.805986 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:10:50.805961    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9" podUID="8314a768-d3c1-48cd-aa85-953de1d97400" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Jul 28 14:11:00.806544 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:11:00.806466    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9"
Jul 28 14:11:13.321106 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:11:13.321077    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9"]
Jul 28 14:11:13.321639 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:11:13.321282    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9" podUID="8314a768-d3c1-48cd-aa85-953de1d97400" containerName="kserve-container" containerID="cri-o://2c21d1c614ba5fe95596ae4ecc551e883f610dd339a90d7365ee9aac1f78fda7" gracePeriod=30
Jul 28 14:11:13.321639 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:11:13.321318    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9" podUID="8314a768-d3c1-48cd-aa85-953de1d97400" containerName="kube-rbac-proxy" containerID="cri-o://9f8b8a4f4b13536b333bc1a54d7c9fa01f8b6e1d10359e49064fa8b10fb1e6e2" gracePeriod=30
Jul 28 14:11:13.925255 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:11:13.925225    2574 generic.go:356] "Generic (PLEG): container finished" podID="8314a768-d3c1-48cd-aa85-953de1d97400" containerID="9f8b8a4f4b13536b333bc1a54d7c9fa01f8b6e1d10359e49064fa8b10fb1e6e2" exitCode=2
Jul 28 14:11:13.925414 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:11:13.925292    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9" event={"ID":"8314a768-d3c1-48cd-aa85-953de1d97400","Type":"ContainerDied","Data":"9f8b8a4f4b13536b333bc1a54d7c9fa01f8b6e1d10359e49064fa8b10fb1e6e2"}
Jul 28 14:11:15.802135 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:11:15.802100    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9" podUID="8314a768-d3c1-48cd-aa85-953de1d97400" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.52:8643/healthz\": dial tcp 10.132.0.52:8643: connect: connection refused"
Jul 28 14:11:20.801538 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:11:20.801495    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9" podUID="8314a768-d3c1-48cd-aa85-953de1d97400" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.52:8643/healthz\": dial tcp 10.132.0.52:8643: connect: connection refused"
Jul 28 14:11:25.801969 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:11:25.801926    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9" podUID="8314a768-d3c1-48cd-aa85-953de1d97400" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.52:8643/healthz\": dial tcp 10.132.0.52:8643: connect: connection refused"
Jul 28 14:11:25.802333 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:11:25.802036    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9"
Jul 28 14:11:30.801458 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:11:30.801418    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9" podUID="8314a768-d3c1-48cd-aa85-953de1d97400" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.52:8643/healthz\": dial tcp 10.132.0.52:8643: connect: connection refused"
Jul 28 14:11:35.801814 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:11:35.801777    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9" podUID="8314a768-d3c1-48cd-aa85-953de1d97400" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.52:8643/healthz\": dial tcp 10.132.0.52:8643: connect: connection refused"
Jul 28 14:11:40.802283 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:11:40.802234    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9" podUID="8314a768-d3c1-48cd-aa85-953de1d97400" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.52:8643/healthz\": dial tcp 10.132.0.52:8643: connect: connection refused"
Jul 28 14:11:43.977593 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:11:43.977564    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9"
Jul 28 14:11:44.055471 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:11:44.055442    2574 generic.go:356] "Generic (PLEG): container finished" podID="8314a768-d3c1-48cd-aa85-953de1d97400" containerID="2c21d1c614ba5fe95596ae4ecc551e883f610dd339a90d7365ee9aac1f78fda7" exitCode=137
Jul 28 14:11:44.055658 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:11:44.055502    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9"
Jul 28 14:11:44.055658 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:11:44.055522    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9" event={"ID":"8314a768-d3c1-48cd-aa85-953de1d97400","Type":"ContainerDied","Data":"2c21d1c614ba5fe95596ae4ecc551e883f610dd339a90d7365ee9aac1f78fda7"}
Jul 28 14:11:44.055658 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:11:44.055587    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9" event={"ID":"8314a768-d3c1-48cd-aa85-953de1d97400","Type":"ContainerDied","Data":"593a68e2c24d83565e2422fa0743205bd8c5f93f0dd12b88535dfb1eee668aef"}
Jul 28 14:11:44.055658 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:11:44.055630    2574 scope.go:117] "RemoveContainer" containerID="9f8b8a4f4b13536b333bc1a54d7c9fa01f8b6e1d10359e49064fa8b10fb1e6e2"
Jul 28 14:11:44.063976 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:11:44.063957    2574 scope.go:117] "RemoveContainer" containerID="2c21d1c614ba5fe95596ae4ecc551e883f610dd339a90d7365ee9aac1f78fda7"
Jul 28 14:11:44.072108 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:11:44.072083    2574 scope.go:117] "RemoveContainer" containerID="e222904a5f1da97ee07b43387b8e4e10a13f79b19ad2ff3142b2b5f8f8ecaa98"
Jul 28 14:11:44.079728 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:11:44.079699    2574 scope.go:117] "RemoveContainer" containerID="9f8b8a4f4b13536b333bc1a54d7c9fa01f8b6e1d10359e49064fa8b10fb1e6e2"
Jul 28 14:11:44.080053 ip-10-0-138-158 kubenswrapper[2574]: E0728 14:11:44.080034    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f8b8a4f4b13536b333bc1a54d7c9fa01f8b6e1d10359e49064fa8b10fb1e6e2\": container with ID starting with 9f8b8a4f4b13536b333bc1a54d7c9fa01f8b6e1d10359e49064fa8b10fb1e6e2 not found: ID does not exist" containerID="9f8b8a4f4b13536b333bc1a54d7c9fa01f8b6e1d10359e49064fa8b10fb1e6e2"
Jul 28 14:11:44.080118 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:11:44.080064    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f8b8a4f4b13536b333bc1a54d7c9fa01f8b6e1d10359e49064fa8b10fb1e6e2"} err="failed to get container status \"9f8b8a4f4b13536b333bc1a54d7c9fa01f8b6e1d10359e49064fa8b10fb1e6e2\": rpc error: code = NotFound desc = could not find container \"9f8b8a4f4b13536b333bc1a54d7c9fa01f8b6e1d10359e49064fa8b10fb1e6e2\": container with ID starting with 9f8b8a4f4b13536b333bc1a54d7c9fa01f8b6e1d10359e49064fa8b10fb1e6e2 not found: ID does not exist"
Jul 28 14:11:44.080118 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:11:44.080084    2574 scope.go:117] "RemoveContainer" containerID="2c21d1c614ba5fe95596ae4ecc551e883f610dd339a90d7365ee9aac1f78fda7"
Jul 28 14:11:44.080362 ip-10-0-138-158 kubenswrapper[2574]: E0728 14:11:44.080341    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c21d1c614ba5fe95596ae4ecc551e883f610dd339a90d7365ee9aac1f78fda7\": container with ID starting with 2c21d1c614ba5fe95596ae4ecc551e883f610dd339a90d7365ee9aac1f78fda7 not found: ID does not exist" containerID="2c21d1c614ba5fe95596ae4ecc551e883f610dd339a90d7365ee9aac1f78fda7"
Jul 28 14:11:44.080404 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:11:44.080369    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c21d1c614ba5fe95596ae4ecc551e883f610dd339a90d7365ee9aac1f78fda7"} err="failed to get container status \"2c21d1c614ba5fe95596ae4ecc551e883f610dd339a90d7365ee9aac1f78fda7\": rpc error: code = NotFound desc = could not find container \"2c21d1c614ba5fe95596ae4ecc551e883f610dd339a90d7365ee9aac1f78fda7\": container with ID starting with 2c21d1c614ba5fe95596ae4ecc551e883f610dd339a90d7365ee9aac1f78fda7 not found: ID does not exist"
Jul 28 14:11:44.080404 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:11:44.080387    2574 scope.go:117] "RemoveContainer" containerID="e222904a5f1da97ee07b43387b8e4e10a13f79b19ad2ff3142b2b5f8f8ecaa98"
Jul 28 14:11:44.080608 ip-10-0-138-158 kubenswrapper[2574]: E0728 14:11:44.080592    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e222904a5f1da97ee07b43387b8e4e10a13f79b19ad2ff3142b2b5f8f8ecaa98\": container with ID starting with e222904a5f1da97ee07b43387b8e4e10a13f79b19ad2ff3142b2b5f8f8ecaa98 not found: ID does not exist" containerID="e222904a5f1da97ee07b43387b8e4e10a13f79b19ad2ff3142b2b5f8f8ecaa98"
Jul 28 14:11:44.080664 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:11:44.080613    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e222904a5f1da97ee07b43387b8e4e10a13f79b19ad2ff3142b2b5f8f8ecaa98"} err="failed to get container status \"e222904a5f1da97ee07b43387b8e4e10a13f79b19ad2ff3142b2b5f8f8ecaa98\": rpc error: code = NotFound desc = could not find container \"e222904a5f1da97ee07b43387b8e4e10a13f79b19ad2ff3142b2b5f8f8ecaa98\": container with ID starting with e222904a5f1da97ee07b43387b8e4e10a13f79b19ad2ff3142b2b5f8f8ecaa98 not found: ID does not exist"
Jul 28 14:11:44.133104 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:11:44.133014    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8314a768-d3c1-48cd-aa85-953de1d97400-kube-rbac-proxy-sar-config\") pod \"8314a768-d3c1-48cd-aa85-953de1d97400\" (UID: \"8314a768-d3c1-48cd-aa85-953de1d97400\") "
Jul 28 14:11:44.133104 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:11:44.133062    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8314a768-d3c1-48cd-aa85-953de1d97400-kserve-provision-location\") pod \"8314a768-d3c1-48cd-aa85-953de1d97400\" (UID: \"8314a768-d3c1-48cd-aa85-953de1d97400\") "
Jul 28 14:11:44.133104 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:11:44.133089    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8314a768-d3c1-48cd-aa85-953de1d97400-cabundle-cert\") pod \"8314a768-d3c1-48cd-aa85-953de1d97400\" (UID: \"8314a768-d3c1-48cd-aa85-953de1d97400\") "
Jul 28 14:11:44.133389 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:11:44.133176    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8314a768-d3c1-48cd-aa85-953de1d97400-proxy-tls\") pod \"8314a768-d3c1-48cd-aa85-953de1d97400\" (UID: \"8314a768-d3c1-48cd-aa85-953de1d97400\") "
Jul 28 14:11:44.133389 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:11:44.133205    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8bbsc\" (UniqueName: \"kubernetes.io/projected/8314a768-d3c1-48cd-aa85-953de1d97400-kube-api-access-8bbsc\") pod \"8314a768-d3c1-48cd-aa85-953de1d97400\" (UID: \"8314a768-d3c1-48cd-aa85-953de1d97400\") "
Jul 28 14:11:44.133521 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:11:44.133431    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8314a768-d3c1-48cd-aa85-953de1d97400-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8314a768-d3c1-48cd-aa85-953de1d97400" (UID: "8314a768-d3c1-48cd-aa85-953de1d97400"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:11:44.133578 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:11:44.133534    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8314a768-d3c1-48cd-aa85-953de1d97400-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8314a768-d3c1-48cd-aa85-953de1d97400" (UID: "8314a768-d3c1-48cd-aa85-953de1d97400"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:11:44.135311 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:11:44.135287    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8314a768-d3c1-48cd-aa85-953de1d97400-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8314a768-d3c1-48cd-aa85-953de1d97400" (UID: "8314a768-d3c1-48cd-aa85-953de1d97400"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 14:11:44.135311 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:11:44.135293    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8314a768-d3c1-48cd-aa85-953de1d97400-kube-api-access-8bbsc" (OuterVolumeSpecName: "kube-api-access-8bbsc") pod "8314a768-d3c1-48cd-aa85-953de1d97400" (UID: "8314a768-d3c1-48cd-aa85-953de1d97400"). InnerVolumeSpecName "kube-api-access-8bbsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:11:44.144347 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:11:44.144314    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8314a768-d3c1-48cd-aa85-953de1d97400-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8314a768-d3c1-48cd-aa85-953de1d97400" (UID: "8314a768-d3c1-48cd-aa85-953de1d97400"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:11:44.234143 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:11:44.234088    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8314a768-d3c1-48cd-aa85-953de1d97400-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 14:11:44.234143 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:11:44.234137    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8314a768-d3c1-48cd-aa85-953de1d97400-kserve-provision-location\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 14:11:44.234143 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:11:44.234152    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8314a768-d3c1-48cd-aa85-953de1d97400-cabundle-cert\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 14:11:44.234475 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:11:44.234169    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8314a768-d3c1-48cd-aa85-953de1d97400-proxy-tls\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 14:11:44.234475 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:11:44.234181    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bbsc\" (UniqueName: \"kubernetes.io/projected/8314a768-d3c1-48cd-aa85-953de1d97400-kube-api-access-8bbsc\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 14:11:44.376403 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:11:44.376365    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9"]
Jul 28 14:11:44.381838 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:11:44.381808    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vc5g9"]
Jul 28 14:11:46.301911 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:11:46.301866    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8314a768-d3c1-48cd-aa85-953de1d97400" path="/var/lib/kubelet/pods/8314a768-d3c1-48cd-aa85-953de1d97400/volumes"
Jul 28 14:19:32.800609 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:19:32.800576    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w4mvw"]
Jul 28 14:19:32.801075 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:19:32.800925    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8314a768-d3c1-48cd-aa85-953de1d97400" containerName="storage-initializer"
Jul 28 14:19:32.801075 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:19:32.800937    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8314a768-d3c1-48cd-aa85-953de1d97400" containerName="storage-initializer"
Jul 28 14:19:32.801075 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:19:32.800945    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8314a768-d3c1-48cd-aa85-953de1d97400" containerName="kserve-container"
Jul 28 14:19:32.801075 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:19:32.800950    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8314a768-d3c1-48cd-aa85-953de1d97400" containerName="kserve-container"
Jul 28 14:19:32.801075 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:19:32.800956    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8314a768-d3c1-48cd-aa85-953de1d97400" containerName="kube-rbac-proxy"
Jul 28 14:19:32.801075 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:19:32.800961    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8314a768-d3c1-48cd-aa85-953de1d97400" containerName="kube-rbac-proxy"
Jul 28 14:19:32.801075 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:19:32.801020    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8314a768-d3c1-48cd-aa85-953de1d97400" containerName="kserve-container"
Jul 28 14:19:32.801075 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:19:32.801027    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8314a768-d3c1-48cd-aa85-953de1d97400" containerName="kube-rbac-proxy"
Jul 28 14:19:32.804104 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:19:32.804090    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w4mvw"
Jul 28 14:19:32.807114 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:19:32.807086    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 14:19:32.807266 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:19:32.807221    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-predictor-serving-cert\""
Jul 28 14:19:32.808164 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:19:32.808144    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 14:19:32.808311 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:19:32.808195    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 14:19:32.808311 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:19:32.808197    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rxfwd\""
Jul 28 14:19:32.808311 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:19:32.808251    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-kube-rbac-proxy-sar-config\""
Jul 28 14:19:32.808311 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:19:32.808208    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 14:19:32.823508 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:19:32.823484    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w4mvw"]
Jul 28 14:19:32.833985 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:19:32.833964    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-w4mvw\" (UID: \"1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w4mvw"
Jul 28 14:19:32.834095 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:19:32.833998    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-w4mvw\" (UID: \"1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w4mvw"
Jul 28 14:19:32.834095 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:19:32.834031    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-w4mvw\" (UID: \"1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w4mvw"
Jul 28 14:19:32.834181 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:19:32.834092    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-w4mvw\" (UID: \"1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w4mvw"
Jul 28 14:19:32.834181 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:19:32.834118    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfz9g\" (UniqueName: \"kubernetes.io/projected/1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98-kube-api-access-zfz9g\") pod \"isvc-xgboost-v2-predictor-fcc668bc-w4mvw\" (UID: \"1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w4mvw"
Jul 28 14:19:32.935123 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:19:32.935096    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-w4mvw\" (UID: \"1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w4mvw"
Jul 28 14:19:32.935259 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:19:32.935137    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-w4mvw\" (UID: \"1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w4mvw"
Jul 28 14:19:32.935259 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:19:32.935166    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfz9g\" (UniqueName: \"kubernetes.io/projected/1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98-kube-api-access-zfz9g\") pod \"isvc-xgboost-v2-predictor-fcc668bc-w4mvw\" (UID: \"1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w4mvw"
Jul 28 14:19:32.935259 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:19:32.935243    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-w4mvw\" (UID: \"1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w4mvw"
Jul 28 14:19:32.935401 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:19:32.935278    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-w4mvw\" (UID: \"1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w4mvw"
Jul 28 14:19:32.935401 ip-10-0-138-158 kubenswrapper[2574]: E0728 14:19:32.935360    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-predictor-serving-cert: secret "isvc-xgboost-v2-predictor-serving-cert" not found
Jul 28 14:19:32.935473 ip-10-0-138-158 kubenswrapper[2574]: E0728 14:19:32.935441    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98-proxy-tls podName:1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98 nodeName:}" failed. No retries permitted until 2026-07-28 14:19:33.435421204 +0000 UTC m=+5185.640807806 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98-proxy-tls") pod "isvc-xgboost-v2-predictor-fcc668bc-w4mvw" (UID: "1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98") : secret "isvc-xgboost-v2-predictor-serving-cert" not found
Jul 28 14:19:32.935522 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:19:32.935498    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-w4mvw\" (UID: \"1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w4mvw"
Jul 28 14:19:32.935925 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:19:32.935906    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-w4mvw\" (UID: \"1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w4mvw"
Jul 28 14:19:32.935970 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:19:32.935931    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-w4mvw\" (UID: \"1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w4mvw"
Jul 28 14:19:32.946214 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:19:32.946193    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfz9g\" (UniqueName: \"kubernetes.io/projected/1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98-kube-api-access-zfz9g\") pod \"isvc-xgboost-v2-predictor-fcc668bc-w4mvw\" (UID: \"1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w4mvw"
Jul 28 14:19:33.439701 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:19:33.439658    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-w4mvw\" (UID: \"1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w4mvw"
Jul 28 14:19:33.442072 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:19:33.442047    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-w4mvw\" (UID: \"1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w4mvw"
Jul 28 14:19:33.713454 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:19:33.713368    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w4mvw"
Jul 28 14:19:33.841646 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:19:33.841613    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w4mvw"]
Jul 28 14:19:33.844628 ip-10-0-138-158 kubenswrapper[2574]: W0728 14:19:33.844601    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a88ea51_d7e5_4e81_a74c_4cc5faf9ad98.slice/crio-810561f41445e3dea626ccab00241562da59a44cb99810bf25342b783613b5dd WatchSource:0}: Error finding container 810561f41445e3dea626ccab00241562da59a44cb99810bf25342b783613b5dd: Status 404 returned error can't find the container with id 810561f41445e3dea626ccab00241562da59a44cb99810bf25342b783613b5dd
Jul 28 14:19:34.065020 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:19:34.064918    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w4mvw" event={"ID":"1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98","Type":"ContainerStarted","Data":"f5b23c4c4c388f2888c222f697c5d7bb8caf4c2e38021a113f51b8e7b481b272"}
Jul 28 14:19:34.065020 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:19:34.064995    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w4mvw" event={"ID":"1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98","Type":"ContainerStarted","Data":"810561f41445e3dea626ccab00241562da59a44cb99810bf25342b783613b5dd"}
Jul 28 14:19:38.084019 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:19:38.083988    2574 generic.go:356] "Generic (PLEG): container finished" podID="1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98" containerID="f5b23c4c4c388f2888c222f697c5d7bb8caf4c2e38021a113f51b8e7b481b272" exitCode=0
Jul 28 14:19:38.084366 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:19:38.084059    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w4mvw" event={"ID":"1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98","Type":"ContainerDied","Data":"f5b23c4c4c388f2888c222f697c5d7bb8caf4c2e38021a113f51b8e7b481b272"}
Jul 28 14:19:38.085139 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:19:38.085123    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 14:19:58.190416 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:19:58.190377    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w4mvw" event={"ID":"1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98","Type":"ContainerStarted","Data":"c129e8e171de0efabc7bf0b1c42e57ed35af4e8aa0a444cdf9f7d53ab385f52a"}
Jul 28 14:19:58.190912 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:19:58.190452    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w4mvw" event={"ID":"1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98","Type":"ContainerStarted","Data":"f448d27f249c18fed3faa09c7f0d7024f10779ac28ce680ee64b25e9e8587fc6"}
Jul 28 14:19:58.190912 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:19:58.190697    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w4mvw"
Jul 28 14:19:58.210527 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:19:58.210434    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w4mvw" podStartSLOduration=2.311147901 podStartE2EDuration="26.210420804s" totalImagesPullingTime="19.872570456s" totalInitContainerRuntime="4.026702447s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:19:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:19:58.208844987 +0000 UTC m=+5210.414231588" watchObservedRunningTime="2026-07-28 14:19:58.210420804 +0000 UTC m=+5210.415807404"
Jul 28 14:19:59.195736 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:19:59.195675    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w4mvw"
Jul 28 14:19:59.197000 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:19:59.196972    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w4mvw" podUID="1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Jul 28 14:20:00.200410 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:20:00.200369    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w4mvw" podUID="1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Jul 28 14:20:05.205242 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:20:05.205211    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w4mvw"
Jul 28 14:20:05.205770 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:20:05.205726    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w4mvw" podUID="1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Jul 28 14:20:15.206375 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:20:15.206335    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w4mvw" podUID="1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Jul 28 14:20:25.206235 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:20:25.206198    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w4mvw" podUID="1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Jul 28 14:20:35.206645 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:20:35.206606    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w4mvw" podUID="1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Jul 28 14:20:45.206748 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:20:45.206673    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w4mvw" podUID="1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Jul 28 14:20:55.206836 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:20:55.206807    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w4mvw"
Jul 28 14:21:02.942098 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:21:02.942061    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w4mvw"]
Jul 28 14:21:02.942732 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:21:02.942312    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w4mvw" podUID="1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98" containerName="kserve-container" containerID="cri-o://f448d27f249c18fed3faa09c7f0d7024f10779ac28ce680ee64b25e9e8587fc6" gracePeriod=30
Jul 28 14:21:02.942732 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:21:02.942368    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w4mvw" podUID="1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98" containerName="kube-rbac-proxy" containerID="cri-o://c129e8e171de0efabc7bf0b1c42e57ed35af4e8aa0a444cdf9f7d53ab385f52a" gracePeriod=30
Jul 28 14:21:03.487775 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:21:03.487745    2574 generic.go:356] "Generic (PLEG): container finished" podID="1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98" containerID="c129e8e171de0efabc7bf0b1c42e57ed35af4e8aa0a444cdf9f7d53ab385f52a" exitCode=2
Jul 28 14:21:03.487957 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:21:03.487796    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w4mvw" event={"ID":"1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98","Type":"ContainerDied","Data":"c129e8e171de0efabc7bf0b1c42e57ed35af4e8aa0a444cdf9f7d53ab385f52a"}
Jul 28 14:21:05.200773 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:21:05.200732    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w4mvw" podUID="1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.53:8643/healthz\": dial tcp 10.132.0.53:8643: connect: connection refused"
Jul 28 14:21:05.206260 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:21:05.206227    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w4mvw" podUID="1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Jul 28 14:21:09.690990 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:21:09.690963    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w4mvw"
Jul 28 14:21:09.833975 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:21:09.833898    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98-proxy-tls\") pod \"1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98\" (UID: \"1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98\") "
Jul 28 14:21:09.833975 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:21:09.833933    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zfz9g\" (UniqueName: \"kubernetes.io/projected/1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98-kube-api-access-zfz9g\") pod \"1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98\" (UID: \"1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98\") "
Jul 28 14:21:09.833975 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:21:09.833956    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98-kube-rbac-proxy-sar-config\") pod \"1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98\" (UID: \"1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98\") "
Jul 28 14:21:09.834212 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:21:09.833999    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98-kserve-provision-location\") pod \"1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98\" (UID: \"1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98\") "
Jul 28 14:21:09.834212 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:21:09.834024    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98-cabundle-cert\") pod \"1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98\" (UID: \"1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98\") "
Jul 28 14:21:09.834379 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:21:09.834353    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98" (UID: "1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:21:09.834445 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:21:09.834363    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98" (UID: "1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:21:09.834495 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:21:09.834442    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98" (UID: "1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:21:09.836046 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:21:09.836026    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98" (UID: "1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 14:21:09.836126 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:21:09.836108    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98-kube-api-access-zfz9g" (OuterVolumeSpecName: "kube-api-access-zfz9g") pod "1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98" (UID: "1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98"). InnerVolumeSpecName "kube-api-access-zfz9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:21:09.934621 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:21:09.934592    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98-cabundle-cert\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 14:21:09.934621 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:21:09.934618    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98-proxy-tls\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 14:21:09.934780 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:21:09.934628    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfz9g\" (UniqueName: \"kubernetes.io/projected/1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98-kube-api-access-zfz9g\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 14:21:09.934780 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:21:09.934637    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 14:21:09.934780 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:21:09.934645    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98-kserve-provision-location\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 14:21:10.519109 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:21:10.519079    2574 generic.go:356] "Generic (PLEG): container finished" podID="1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98" containerID="f448d27f249c18fed3faa09c7f0d7024f10779ac28ce680ee64b25e9e8587fc6" exitCode=0
Jul 28 14:21:10.519269 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:21:10.519144    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w4mvw"
Jul 28 14:21:10.519269 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:21:10.519163    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w4mvw" event={"ID":"1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98","Type":"ContainerDied","Data":"f448d27f249c18fed3faa09c7f0d7024f10779ac28ce680ee64b25e9e8587fc6"}
Jul 28 14:21:10.519269 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:21:10.519218    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w4mvw" event={"ID":"1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98","Type":"ContainerDied","Data":"810561f41445e3dea626ccab00241562da59a44cb99810bf25342b783613b5dd"}
Jul 28 14:21:10.519269 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:21:10.519247    2574 scope.go:117] "RemoveContainer" containerID="c129e8e171de0efabc7bf0b1c42e57ed35af4e8aa0a444cdf9f7d53ab385f52a"
Jul 28 14:21:10.532631 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:21:10.532610    2574 scope.go:117] "RemoveContainer" containerID="f448d27f249c18fed3faa09c7f0d7024f10779ac28ce680ee64b25e9e8587fc6"
Jul 28 14:21:10.540222 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:21:10.540201    2574 scope.go:117] "RemoveContainer" containerID="f5b23c4c4c388f2888c222f697c5d7bb8caf4c2e38021a113f51b8e7b481b272"
Jul 28 14:21:10.543474 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:21:10.543455    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w4mvw"]
Jul 28 14:21:10.547656 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:21:10.547631    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w4mvw"]
Jul 28 14:21:10.547976 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:21:10.547963    2574 scope.go:117] "RemoveContainer" containerID="c129e8e171de0efabc7bf0b1c42e57ed35af4e8aa0a444cdf9f7d53ab385f52a"
Jul 28 14:21:10.548228 ip-10-0-138-158 kubenswrapper[2574]: E0728 14:21:10.548210    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c129e8e171de0efabc7bf0b1c42e57ed35af4e8aa0a444cdf9f7d53ab385f52a\": container with ID starting with c129e8e171de0efabc7bf0b1c42e57ed35af4e8aa0a444cdf9f7d53ab385f52a not found: ID does not exist" containerID="c129e8e171de0efabc7bf0b1c42e57ed35af4e8aa0a444cdf9f7d53ab385f52a"
Jul 28 14:21:10.548306 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:21:10.548238    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c129e8e171de0efabc7bf0b1c42e57ed35af4e8aa0a444cdf9f7d53ab385f52a"} err="failed to get container status \"c129e8e171de0efabc7bf0b1c42e57ed35af4e8aa0a444cdf9f7d53ab385f52a\": rpc error: code = NotFound desc = could not find container \"c129e8e171de0efabc7bf0b1c42e57ed35af4e8aa0a444cdf9f7d53ab385f52a\": container with ID starting with c129e8e171de0efabc7bf0b1c42e57ed35af4e8aa0a444cdf9f7d53ab385f52a not found: ID does not exist"
Jul 28 14:21:10.548306 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:21:10.548261    2574 scope.go:117] "RemoveContainer" containerID="f448d27f249c18fed3faa09c7f0d7024f10779ac28ce680ee64b25e9e8587fc6"
Jul 28 14:21:10.548524 ip-10-0-138-158 kubenswrapper[2574]: E0728 14:21:10.548506    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f448d27f249c18fed3faa09c7f0d7024f10779ac28ce680ee64b25e9e8587fc6\": container with ID starting with f448d27f249c18fed3faa09c7f0d7024f10779ac28ce680ee64b25e9e8587fc6 not found: ID does not exist" containerID="f448d27f249c18fed3faa09c7f0d7024f10779ac28ce680ee64b25e9e8587fc6"
Jul 28 14:21:10.548564 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:21:10.548530    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f448d27f249c18fed3faa09c7f0d7024f10779ac28ce680ee64b25e9e8587fc6"} err="failed to get container status \"f448d27f249c18fed3faa09c7f0d7024f10779ac28ce680ee64b25e9e8587fc6\": rpc error: code = NotFound desc = could not find container \"f448d27f249c18fed3faa09c7f0d7024f10779ac28ce680ee64b25e9e8587fc6\": container with ID starting with f448d27f249c18fed3faa09c7f0d7024f10779ac28ce680ee64b25e9e8587fc6 not found: ID does not exist"
Jul 28 14:21:10.548564 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:21:10.548545    2574 scope.go:117] "RemoveContainer" containerID="f5b23c4c4c388f2888c222f697c5d7bb8caf4c2e38021a113f51b8e7b481b272"
Jul 28 14:21:10.548779 ip-10-0-138-158 kubenswrapper[2574]: E0728 14:21:10.548762    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5b23c4c4c388f2888c222f697c5d7bb8caf4c2e38021a113f51b8e7b481b272\": container with ID starting with f5b23c4c4c388f2888c222f697c5d7bb8caf4c2e38021a113f51b8e7b481b272 not found: ID does not exist" containerID="f5b23c4c4c388f2888c222f697c5d7bb8caf4c2e38021a113f51b8e7b481b272"
Jul 28 14:21:10.548821 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:21:10.548786    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5b23c4c4c388f2888c222f697c5d7bb8caf4c2e38021a113f51b8e7b481b272"} err="failed to get container status \"f5b23c4c4c388f2888c222f697c5d7bb8caf4c2e38021a113f51b8e7b481b272\": rpc error: code = NotFound desc = could not find container \"f5b23c4c4c388f2888c222f697c5d7bb8caf4c2e38021a113f51b8e7b481b272\": container with ID starting with f5b23c4c4c388f2888c222f697c5d7bb8caf4c2e38021a113f51b8e7b481b272 not found: ID does not exist"
Jul 28 14:21:12.302307 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:21:12.302273    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98" path="/var/lib/kubelet/pods/1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98/volumes"
Jul 28 14:27:26.185382 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:27:26.185349    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-gfscw/must-gather-2gzqg"]
Jul 28 14:27:26.185857 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:27:26.185694    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98" containerName="kube-rbac-proxy"
Jul 28 14:27:26.185857 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:27:26.185705    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98" containerName="kube-rbac-proxy"
Jul 28 14:27:26.185857 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:27:26.185716    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98" containerName="storage-initializer"
Jul 28 14:27:26.185857 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:27:26.185722    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98" containerName="storage-initializer"
Jul 28 14:27:26.185857 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:27:26.185731    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98" containerName="kserve-container"
Jul 28 14:27:26.185857 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:27:26.185739    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98" containerName="kserve-container"
Jul 28 14:27:26.185857 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:27:26.185796    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98" containerName="kube-rbac-proxy"
Jul 28 14:27:26.185857 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:27:26.185804    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a88ea51-d7e5-4e81-a74c-4cc5faf9ad98" containerName="kserve-container"
Jul 28 14:27:26.188628 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:27:26.188613    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gfscw/must-gather-2gzqg"
Jul 28 14:27:26.191081 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:27:26.191062    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gfscw\"/\"kube-root-ca.crt\""
Jul 28 14:27:26.191643 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:27:26.191628    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gfscw\"/\"openshift-service-ca.crt\""
Jul 28 14:27:26.199826 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:27:26.199804    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gfscw/must-gather-2gzqg"]
Jul 28 14:27:26.236265 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:27:26.236237    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/177288ff-e880-4a9a-ba2c-3b445be7e15f-must-gather-output\") pod \"must-gather-2gzqg\" (UID: \"177288ff-e880-4a9a-ba2c-3b445be7e15f\") " pod="openshift-must-gather-gfscw/must-gather-2gzqg"
Jul 28 14:27:26.236372 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:27:26.236278    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4zqg\" (UniqueName: \"kubernetes.io/projected/177288ff-e880-4a9a-ba2c-3b445be7e15f-kube-api-access-v4zqg\") pod \"must-gather-2gzqg\" (UID: \"177288ff-e880-4a9a-ba2c-3b445be7e15f\") " pod="openshift-must-gather-gfscw/must-gather-2gzqg"
Jul 28 14:27:26.337143 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:27:26.337121    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4zqg\" (UniqueName: \"kubernetes.io/projected/177288ff-e880-4a9a-ba2c-3b445be7e15f-kube-api-access-v4zqg\") pod \"must-gather-2gzqg\" (UID: \"177288ff-e880-4a9a-ba2c-3b445be7e15f\") " pod="openshift-must-gather-gfscw/must-gather-2gzqg"
Jul 28 14:27:26.337237 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:27:26.337178    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/177288ff-e880-4a9a-ba2c-3b445be7e15f-must-gather-output\") pod \"must-gather-2gzqg\" (UID: \"177288ff-e880-4a9a-ba2c-3b445be7e15f\") " pod="openshift-must-gather-gfscw/must-gather-2gzqg"
Jul 28 14:27:26.337454 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:27:26.337440    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/177288ff-e880-4a9a-ba2c-3b445be7e15f-must-gather-output\") pod \"must-gather-2gzqg\" (UID: \"177288ff-e880-4a9a-ba2c-3b445be7e15f\") " pod="openshift-must-gather-gfscw/must-gather-2gzqg"
Jul 28 14:27:26.347004 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:27:26.346978    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4zqg\" (UniqueName: \"kubernetes.io/projected/177288ff-e880-4a9a-ba2c-3b445be7e15f-kube-api-access-v4zqg\") pod \"must-gather-2gzqg\" (UID: \"177288ff-e880-4a9a-ba2c-3b445be7e15f\") " pod="openshift-must-gather-gfscw/must-gather-2gzqg"
Jul 28 14:27:26.507281 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:27:26.507208    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gfscw/must-gather-2gzqg"
Jul 28 14:27:26.629716 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:27:26.629650    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gfscw/must-gather-2gzqg"]
Jul 28 14:27:26.631868 ip-10-0-138-158 kubenswrapper[2574]: W0728 14:27:26.631840    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod177288ff_e880_4a9a_ba2c_3b445be7e15f.slice/crio-2c02be30d911d0fb9460edd2c98e1863cec2ddd509d63834d93f7e7d48762c23 WatchSource:0}: Error finding container 2c02be30d911d0fb9460edd2c98e1863cec2ddd509d63834d93f7e7d48762c23: Status 404 returned error can't find the container with id 2c02be30d911d0fb9460edd2c98e1863cec2ddd509d63834d93f7e7d48762c23
Jul 28 14:27:26.633578 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:27:26.633558    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 14:27:27.146634 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:27:27.146601    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gfscw/must-gather-2gzqg" event={"ID":"177288ff-e880-4a9a-ba2c-3b445be7e15f","Type":"ContainerStarted","Data":"2c02be30d911d0fb9460edd2c98e1863cec2ddd509d63834d93f7e7d48762c23"}
Jul 28 14:27:34.186798 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:27:34.186762    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gfscw/must-gather-2gzqg" event={"ID":"177288ff-e880-4a9a-ba2c-3b445be7e15f","Type":"ContainerStarted","Data":"d32fb86e5a1fbd4a1992cfe078b9e3ae7f14adb11c2f8339ac0487adfc262706"}
Jul 28 14:27:34.186798 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:27:34.186811    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gfscw/must-gather-2gzqg" event={"ID":"177288ff-e880-4a9a-ba2c-3b445be7e15f","Type":"ContainerStarted","Data":"7ceff89fd74b1b55c73770a14c0a019f7a6bd481a4d900f32e3f3650d1901fa1"}
Jul 28 14:27:34.212159 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:27:34.212105    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-gfscw/must-gather-2gzqg" podStartSLOduration=1.690785902 podStartE2EDuration="8.212089323s" totalImagesPullingTime="6.521303421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:27:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:27:34.210776346 +0000 UTC m=+5666.416162949" watchObservedRunningTime="2026-07-28 14:27:34.212089323 +0000 UTC m=+5666.417475923"
Jul 28 14:27:59.306079 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:27:59.306050    2574 generic.go:356] "Generic (PLEG): container finished" podID="177288ff-e880-4a9a-ba2c-3b445be7e15f" containerID="7ceff89fd74b1b55c73770a14c0a019f7a6bd481a4d900f32e3f3650d1901fa1" exitCode=0
Jul 28 14:27:59.306591 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:27:59.306129    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gfscw/must-gather-2gzqg" event={"ID":"177288ff-e880-4a9a-ba2c-3b445be7e15f","Type":"ContainerDied","Data":"7ceff89fd74b1b55c73770a14c0a019f7a6bd481a4d900f32e3f3650d1901fa1"}
Jul 28 14:27:59.306591 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:27:59.306453    2574 scope.go:117] "RemoveContainer" containerID="7ceff89fd74b1b55c73770a14c0a019f7a6bd481a4d900f32e3f3650d1901fa1"
Jul 28 14:28:00.130989 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:00.130954    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-gfscw_must-gather-2gzqg_177288ff-e880-4a9a-ba2c-3b445be7e15f/gather/0.log"
Jul 28 14:28:04.131785 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:04.131752    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-bdgzk_da8b58f9-2528-4906-b7d2-aadea159b4f6/global-pull-secret-syncer/0.log"
Jul 28 14:28:04.352636 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:04.352606    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-cxz2b_3943de15-60a3-4e53-af9f-125cb8bae2ab/konnectivity-agent/0.log"
Jul 28 14:28:04.440193 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:04.440166    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-158.ec2.internal_6f72deeae9606046a5729c2df486eb15/haproxy/0.log"
Jul 28 14:28:05.697310 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:05.697278    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-gfscw/must-gather-2gzqg"]
Jul 28 14:28:05.697736 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:05.697441    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-gfscw/must-gather-2gzqg" podUID="177288ff-e880-4a9a-ba2c-3b445be7e15f" containerName="copy" containerID="cri-o://d32fb86e5a1fbd4a1992cfe078b9e3ae7f14adb11c2f8339ac0487adfc262706" gracePeriod=2
Jul 28 14:28:05.703026 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:05.703002    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-gfscw/must-gather-2gzqg"]
Jul 28 14:28:05.922538 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:05.922520    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-gfscw_must-gather-2gzqg_177288ff-e880-4a9a-ba2c-3b445be7e15f/copy/0.log"
Jul 28 14:28:05.925133 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:05.925117    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gfscw/must-gather-2gzqg"
Jul 28 14:28:05.927409 ip-10-0-138-158 kubenswrapper[2574]: E0728 14:28:05.927388    2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-2gzqg\" is forbidden: User \"system:node:ip-10-0-138-158.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-gfscw\": no relationship found between node 'ip-10-0-138-158.ec2.internal' and this object" podUID="177288ff-e880-4a9a-ba2c-3b445be7e15f" pod="openshift-must-gather-gfscw/must-gather-2gzqg"
Jul 28 14:28:06.079533 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:06.079477    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/177288ff-e880-4a9a-ba2c-3b445be7e15f-must-gather-output\") pod \"177288ff-e880-4a9a-ba2c-3b445be7e15f\" (UID: \"177288ff-e880-4a9a-ba2c-3b445be7e15f\") "
Jul 28 14:28:06.079670 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:06.079595    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v4zqg\" (UniqueName: \"kubernetes.io/projected/177288ff-e880-4a9a-ba2c-3b445be7e15f-kube-api-access-v4zqg\") pod \"177288ff-e880-4a9a-ba2c-3b445be7e15f\" (UID: \"177288ff-e880-4a9a-ba2c-3b445be7e15f\") "
Jul 28 14:28:06.080637 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:06.080613    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/177288ff-e880-4a9a-ba2c-3b445be7e15f-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "177288ff-e880-4a9a-ba2c-3b445be7e15f" (UID: "177288ff-e880-4a9a-ba2c-3b445be7e15f"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:28:06.081867 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:06.081840    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/177288ff-e880-4a9a-ba2c-3b445be7e15f-kube-api-access-v4zqg" (OuterVolumeSpecName: "kube-api-access-v4zqg") pod "177288ff-e880-4a9a-ba2c-3b445be7e15f" (UID: "177288ff-e880-4a9a-ba2c-3b445be7e15f"). InnerVolumeSpecName "kube-api-access-v4zqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:28:06.180265 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:06.180244    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4zqg\" (UniqueName: \"kubernetes.io/projected/177288ff-e880-4a9a-ba2c-3b445be7e15f-kube-api-access-v4zqg\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 14:28:06.180265 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:06.180263    2574 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/177288ff-e880-4a9a-ba2c-3b445be7e15f-must-gather-output\") on node \"ip-10-0-138-158.ec2.internal\" DevicePath \"\""
Jul 28 14:28:06.301692 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:06.301658    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="177288ff-e880-4a9a-ba2c-3b445be7e15f" path="/var/lib/kubelet/pods/177288ff-e880-4a9a-ba2c-3b445be7e15f/volumes"
Jul 28 14:28:06.338517 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:06.338470    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-gfscw_must-gather-2gzqg_177288ff-e880-4a9a-ba2c-3b445be7e15f/copy/0.log"
Jul 28 14:28:06.338870 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:06.338851    2574 generic.go:356] "Generic (PLEG): container finished" podID="177288ff-e880-4a9a-ba2c-3b445be7e15f" containerID="d32fb86e5a1fbd4a1992cfe078b9e3ae7f14adb11c2f8339ac0487adfc262706" exitCode=143
Jul 28 14:28:06.338926 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:06.338888    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gfscw/must-gather-2gzqg"
Jul 28 14:28:06.338964 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:06.338930    2574 scope.go:117] "RemoveContainer" containerID="d32fb86e5a1fbd4a1992cfe078b9e3ae7f14adb11c2f8339ac0487adfc262706"
Jul 28 14:28:06.346546 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:06.346522    2574 scope.go:117] "RemoveContainer" containerID="7ceff89fd74b1b55c73770a14c0a019f7a6bd481a4d900f32e3f3650d1901fa1"
Jul 28 14:28:06.358650 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:06.358636    2574 scope.go:117] "RemoveContainer" containerID="d32fb86e5a1fbd4a1992cfe078b9e3ae7f14adb11c2f8339ac0487adfc262706"
Jul 28 14:28:06.358901 ip-10-0-138-158 kubenswrapper[2574]: E0728 14:28:06.358883    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d32fb86e5a1fbd4a1992cfe078b9e3ae7f14adb11c2f8339ac0487adfc262706\": container with ID starting with d32fb86e5a1fbd4a1992cfe078b9e3ae7f14adb11c2f8339ac0487adfc262706 not found: ID does not exist" containerID="d32fb86e5a1fbd4a1992cfe078b9e3ae7f14adb11c2f8339ac0487adfc262706"
Jul 28 14:28:06.358980 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:06.358912    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d32fb86e5a1fbd4a1992cfe078b9e3ae7f14adb11c2f8339ac0487adfc262706"} err="failed to get container status \"d32fb86e5a1fbd4a1992cfe078b9e3ae7f14adb11c2f8339ac0487adfc262706\": rpc error: code = NotFound desc = could not find container \"d32fb86e5a1fbd4a1992cfe078b9e3ae7f14adb11c2f8339ac0487adfc262706\": container with ID starting with d32fb86e5a1fbd4a1992cfe078b9e3ae7f14adb11c2f8339ac0487adfc262706 not found: ID does not exist"
Jul 28 14:28:06.358980 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:06.358938    2574 scope.go:117] "RemoveContainer" containerID="7ceff89fd74b1b55c73770a14c0a019f7a6bd481a4d900f32e3f3650d1901fa1"
Jul 28 14:28:06.359172 ip-10-0-138-158 kubenswrapper[2574]: E0728 14:28:06.359152    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ceff89fd74b1b55c73770a14c0a019f7a6bd481a4d900f32e3f3650d1901fa1\": container with ID starting with 7ceff89fd74b1b55c73770a14c0a019f7a6bd481a4d900f32e3f3650d1901fa1 not found: ID does not exist" containerID="7ceff89fd74b1b55c73770a14c0a019f7a6bd481a4d900f32e3f3650d1901fa1"
Jul 28 14:28:06.359210 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:06.359178    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ceff89fd74b1b55c73770a14c0a019f7a6bd481a4d900f32e3f3650d1901fa1"} err="failed to get container status \"7ceff89fd74b1b55c73770a14c0a019f7a6bd481a4d900f32e3f3650d1901fa1\": rpc error: code = NotFound desc = could not find container \"7ceff89fd74b1b55c73770a14c0a019f7a6bd481a4d900f32e3f3650d1901fa1\": container with ID starting with 7ceff89fd74b1b55c73770a14c0a019f7a6bd481a4d900f32e3f3650d1901fa1 not found: ID does not exist"
Jul 28 14:28:07.778119 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:07.778090    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4d592775-753a-47d3-bc13-ff297afd9f51/alertmanager/0.log"
Jul 28 14:28:07.808883 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:07.808861    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4d592775-753a-47d3-bc13-ff297afd9f51/config-reloader/0.log"
Jul 28 14:28:07.844965 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:07.844943    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4d592775-753a-47d3-bc13-ff297afd9f51/kube-rbac-proxy-web/0.log"
Jul 28 14:28:07.879539 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:07.879517    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4d592775-753a-47d3-bc13-ff297afd9f51/kube-rbac-proxy/0.log"
Jul 28 14:28:07.911158 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:07.911142    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4d592775-753a-47d3-bc13-ff297afd9f51/kube-rbac-proxy-metric/0.log"
Jul 28 14:28:07.946118 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:07.946097    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4d592775-753a-47d3-bc13-ff297afd9f51/prom-label-proxy/0.log"
Jul 28 14:28:07.980476 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:07.980461    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4d592775-753a-47d3-bc13-ff297afd9f51/init-config-reloader/0.log"
Jul 28 14:28:08.084744 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:08.084665    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-h585r_61f6e666-8451-4f07-bcc9-cbe18de924f1/kube-state-metrics/0.log"
Jul 28 14:28:08.114243 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:08.114218    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-h585r_61f6e666-8451-4f07-bcc9-cbe18de924f1/kube-rbac-proxy-main/0.log"
Jul 28 14:28:08.140410 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:08.140390    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-h585r_61f6e666-8451-4f07-bcc9-cbe18de924f1/kube-rbac-proxy-self/0.log"
Jul 28 14:28:08.219353 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:08.219330    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-8qxp6_cd34484f-2793-4e67-89eb-586d70b97eb4/monitoring-plugin/0.log"
Jul 28 14:28:08.539297 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:08.539258    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nlkwm_1099cd90-ebb3-4cc5-b316-17a0bb480a75/node-exporter/0.log"
Jul 28 14:28:08.585164 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:08.585143    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nlkwm_1099cd90-ebb3-4cc5-b316-17a0bb480a75/kube-rbac-proxy/0.log"
Jul 28 14:28:08.639381 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:08.639362    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nlkwm_1099cd90-ebb3-4cc5-b316-17a0bb480a75/init-textfile/0.log"
Jul 28 14:28:08.709243 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:08.709223    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-h47vh_9fefb33c-bdac-47fb-9d68-bbc531bc37cc/kube-rbac-proxy-main/0.log"
Jul 28 14:28:08.760283 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:08.760261    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-h47vh_9fefb33c-bdac-47fb-9d68-bbc531bc37cc/kube-rbac-proxy-self/0.log"
Jul 28 14:28:08.806325 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:08.806265    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-h47vh_9fefb33c-bdac-47fb-9d68-bbc531bc37cc/openshift-state-metrics/0.log"
Jul 28 14:28:09.196421 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:09.196398    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-pqmvv_13c3f2a0-fa93-4793-808d-10ce77dafd69/prometheus-operator/0.log"
Jul 28 14:28:09.220404 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:09.220378    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-pqmvv_13c3f2a0-fa93-4793-808d-10ce77dafd69/kube-rbac-proxy/0.log"
Jul 28 14:28:10.994862 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:10.994833    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-v6fhs_e686c7fc-a026-4f10-b033-38955eecea40/networking-console-plugin/0.log"
Jul 28 14:28:12.034632 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:12.034599    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-vj7z5_c626bd54-efbd-42bf-b1d5-0e761f15c5dd/download-server/0.log"
Jul 28 14:28:12.070581 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:12.070554    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-j8v88/perf-node-gather-daemonset-s6gtn"]
Jul 28 14:28:12.070948 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:12.070932    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="177288ff-e880-4a9a-ba2c-3b445be7e15f" containerName="copy"
Jul 28 14:28:12.071006 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:12.070950    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="177288ff-e880-4a9a-ba2c-3b445be7e15f" containerName="copy"
Jul 28 14:28:12.071006 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:12.070975    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="177288ff-e880-4a9a-ba2c-3b445be7e15f" containerName="gather"
Jul 28 14:28:12.071006 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:12.070981    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="177288ff-e880-4a9a-ba2c-3b445be7e15f" containerName="gather"
Jul 28 14:28:12.071102 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:12.071037    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="177288ff-e880-4a9a-ba2c-3b445be7e15f" containerName="gather"
Jul 28 14:28:12.071102 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:12.071044    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="177288ff-e880-4a9a-ba2c-3b445be7e15f" containerName="copy"
Jul 28 14:28:12.075176 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:12.075162    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j8v88/perf-node-gather-daemonset-s6gtn"
Jul 28 14:28:12.077784 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:12.077750    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j8v88\"/\"openshift-service-ca.crt\""
Jul 28 14:28:12.077908 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:12.077791    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-j8v88\"/\"default-dockercfg-hswfh\""
Jul 28 14:28:12.078934 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:12.078910    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j8v88\"/\"kube-root-ca.crt\""
Jul 28 14:28:12.080760 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:12.080742    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j8v88/perf-node-gather-daemonset-s6gtn"]
Jul 28 14:28:12.227436 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:12.227407    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5a377f7-37d6-4d92-a01a-777ea5bf61b4-sys\") pod \"perf-node-gather-daemonset-s6gtn\" (UID: \"d5a377f7-37d6-4d92-a01a-777ea5bf61b4\") " pod="openshift-must-gather-j8v88/perf-node-gather-daemonset-s6gtn"
Jul 28 14:28:12.227436 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:12.227437    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d5a377f7-37d6-4d92-a01a-777ea5bf61b4-lib-modules\") pod \"perf-node-gather-daemonset-s6gtn\" (UID: \"d5a377f7-37d6-4d92-a01a-777ea5bf61b4\") " pod="openshift-must-gather-j8v88/perf-node-gather-daemonset-s6gtn"
Jul 28 14:28:12.227622 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:12.227463    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d5a377f7-37d6-4d92-a01a-777ea5bf61b4-proc\") pod \"perf-node-gather-daemonset-s6gtn\" (UID: \"d5a377f7-37d6-4d92-a01a-777ea5bf61b4\") " pod="openshift-must-gather-j8v88/perf-node-gather-daemonset-s6gtn"
Jul 28 14:28:12.227622 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:12.227541    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sndbs\" (UniqueName: \"kubernetes.io/projected/d5a377f7-37d6-4d92-a01a-777ea5bf61b4-kube-api-access-sndbs\") pod \"perf-node-gather-daemonset-s6gtn\" (UID: \"d5a377f7-37d6-4d92-a01a-777ea5bf61b4\") " pod="openshift-must-gather-j8v88/perf-node-gather-daemonset-s6gtn"
Jul 28 14:28:12.227761 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:12.227626    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d5a377f7-37d6-4d92-a01a-777ea5bf61b4-podres\") pod \"perf-node-gather-daemonset-s6gtn\" (UID: \"d5a377f7-37d6-4d92-a01a-777ea5bf61b4\") " pod="openshift-must-gather-j8v88/perf-node-gather-daemonset-s6gtn"
Jul 28 14:28:12.328177 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:12.328109    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d5a377f7-37d6-4d92-a01a-777ea5bf61b4-proc\") pod \"perf-node-gather-daemonset-s6gtn\" (UID: \"d5a377f7-37d6-4d92-a01a-777ea5bf61b4\") " pod="openshift-must-gather-j8v88/perf-node-gather-daemonset-s6gtn"
Jul 28 14:28:12.328177 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:12.328153    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sndbs\" (UniqueName: \"kubernetes.io/projected/d5a377f7-37d6-4d92-a01a-777ea5bf61b4-kube-api-access-sndbs\") pod \"perf-node-gather-daemonset-s6gtn\" (UID: \"d5a377f7-37d6-4d92-a01a-777ea5bf61b4\") " pod="openshift-must-gather-j8v88/perf-node-gather-daemonset-s6gtn"
Jul 28 14:28:12.328351 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:12.328197    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d5a377f7-37d6-4d92-a01a-777ea5bf61b4-podres\") pod \"perf-node-gather-daemonset-s6gtn\" (UID: \"d5a377f7-37d6-4d92-a01a-777ea5bf61b4\") " pod="openshift-must-gather-j8v88/perf-node-gather-daemonset-s6gtn"
Jul 28 14:28:12.328351 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:12.328223    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5a377f7-37d6-4d92-a01a-777ea5bf61b4-sys\") pod \"perf-node-gather-daemonset-s6gtn\" (UID: \"d5a377f7-37d6-4d92-a01a-777ea5bf61b4\") " pod="openshift-must-gather-j8v88/perf-node-gather-daemonset-s6gtn"
Jul 28 14:28:12.328351 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:12.328236    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d5a377f7-37d6-4d92-a01a-777ea5bf61b4-proc\") pod \"perf-node-gather-daemonset-s6gtn\" (UID: \"d5a377f7-37d6-4d92-a01a-777ea5bf61b4\") " pod="openshift-must-gather-j8v88/perf-node-gather-daemonset-s6gtn"
Jul 28 14:28:12.328351 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:12.328239    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d5a377f7-37d6-4d92-a01a-777ea5bf61b4-lib-modules\") pod \"perf-node-gather-daemonset-s6gtn\" (UID: \"d5a377f7-37d6-4d92-a01a-777ea5bf61b4\") " pod="openshift-must-gather-j8v88/perf-node-gather-daemonset-s6gtn"
Jul 28 14:28:12.328351 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:12.328292    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5a377f7-37d6-4d92-a01a-777ea5bf61b4-sys\") pod \"perf-node-gather-daemonset-s6gtn\" (UID: \"d5a377f7-37d6-4d92-a01a-777ea5bf61b4\") " pod="openshift-must-gather-j8v88/perf-node-gather-daemonset-s6gtn"
Jul 28 14:28:12.328351 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:12.328337    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d5a377f7-37d6-4d92-a01a-777ea5bf61b4-podres\") pod \"perf-node-gather-daemonset-s6gtn\" (UID: \"d5a377f7-37d6-4d92-a01a-777ea5bf61b4\") " pod="openshift-must-gather-j8v88/perf-node-gather-daemonset-s6gtn"
Jul 28 14:28:12.328351 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:12.328344    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d5a377f7-37d6-4d92-a01a-777ea5bf61b4-lib-modules\") pod \"perf-node-gather-daemonset-s6gtn\" (UID: \"d5a377f7-37d6-4d92-a01a-777ea5bf61b4\") " pod="openshift-must-gather-j8v88/perf-node-gather-daemonset-s6gtn"
Jul 28 14:28:12.336467 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:12.336447    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sndbs\" (UniqueName: \"kubernetes.io/projected/d5a377f7-37d6-4d92-a01a-777ea5bf61b4-kube-api-access-sndbs\") pod \"perf-node-gather-daemonset-s6gtn\" (UID: \"d5a377f7-37d6-4d92-a01a-777ea5bf61b4\") " pod="openshift-must-gather-j8v88/perf-node-gather-daemonset-s6gtn"
Jul 28 14:28:12.385000 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:12.384981    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j8v88/perf-node-gather-daemonset-s6gtn"
Jul 28 14:28:12.507883 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:12.507857    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j8v88/perf-node-gather-daemonset-s6gtn"]
Jul 28 14:28:12.509456 ip-10-0-138-158 kubenswrapper[2574]: W0728 14:28:12.509428    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd5a377f7_37d6_4d92_a01a_777ea5bf61b4.slice/crio-9f66c4d4516862f72776af95cf15849b8315840b50a37d03fb2fceb79b1a5c74 WatchSource:0}: Error finding container 9f66c4d4516862f72776af95cf15849b8315840b50a37d03fb2fceb79b1a5c74: Status 404 returned error can't find the container with id 9f66c4d4516862f72776af95cf15849b8315840b50a37d03fb2fceb79b1a5c74
Jul 28 14:28:13.371044 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:13.371012    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j8v88/perf-node-gather-daemonset-s6gtn" event={"ID":"d5a377f7-37d6-4d92-a01a-777ea5bf61b4","Type":"ContainerStarted","Data":"66b88abdae78b18554e585e7f6035b60414e3109b5231a5ea80241468f0de0cf"}
Jul 28 14:28:13.371426 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:13.371065    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j8v88/perf-node-gather-daemonset-s6gtn" event={"ID":"d5a377f7-37d6-4d92-a01a-777ea5bf61b4","Type":"ContainerStarted","Data":"9f66c4d4516862f72776af95cf15849b8315840b50a37d03fb2fceb79b1a5c74"}
Jul 28 14:28:13.371426 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:13.371148    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-j8v88/perf-node-gather-daemonset-s6gtn"
Jul 28 14:28:13.389409 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:13.389357    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-j8v88/perf-node-gather-daemonset-s6gtn" podStartSLOduration=1.389342596 podStartE2EDuration="1.389342596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 14:28:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:28:13.387036984 +0000 UTC m=+5705.592423585" watchObservedRunningTime="2026-07-28 14:28:13.389342596 +0000 UTC m=+5705.594729196"
Jul 28 14:28:13.531850 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:13.531824    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-r8l5q_f8bc5bca-7c04-47ec-b3dc-f6ce86345325/dns/0.log"
Jul 28 14:28:13.554092 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:13.554073    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-r8l5q_f8bc5bca-7c04-47ec-b3dc-f6ce86345325/kube-rbac-proxy/0.log"
Jul 28 14:28:13.757607 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:13.757574    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-66b94_a89e422d-76c2-4e62-b9fb-4283d5fa3de9/dns-node-resolver/0.log"
Jul 28 14:28:14.369909 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:14.369880    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7n245_46ca2ea9-2097-4729-8b59-52882a7ea6bf/node-ca/0.log"
Jul 28 14:28:15.507054 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:15.507017    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-b9htm_8b530194-44d2-4993-9d2e-5aae3188a18d/serve-healthcheck-canary/0.log"
Jul 28 14:28:15.906196 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:15.906132    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-27vf2_f80dfaf5-fb64-4f4e-ab75-e1164768f586/kube-rbac-proxy/0.log"
Jul 28 14:28:15.929070 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:15.929043    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-27vf2_f80dfaf5-fb64-4f4e-ab75-e1164768f586/exporter/0.log"
Jul 28 14:28:15.951438 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:15.951409    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-27vf2_f80dfaf5-fb64-4f4e-ab75-e1164768f586/extractor/0.log"
Jul 28 14:28:18.430580 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:18.430546    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-84fc959947-hwnfs_264cd878-2814-4ff8-9a3e-3d8f40e01535/manager/0.log"
Jul 28 14:28:18.783328 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:18.783306    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-shspn_87e4fd7e-2d34-470a-a8a1-01634e2db6a2/s3-init/0.log"
Jul 28 14:28:18.813087 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:18.813064    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-shspn_87e4fd7e-2d34-470a-a8a1-01634e2db6a2/download-hf-model/0.log"
Jul 28 14:28:18.846362 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:18.846337    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-v6w9j_04e39b3c-cec6-4dd2-bc7e-e82734cae114/s3-tls-init-custom/0.log"
Jul 28 14:28:18.892967 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:18.892939    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-v6w9j_04e39b3c-cec6-4dd2-bc7e-e82734cae114/download-hf-model/0.log"
Jul 28 14:28:18.922872 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:18.922845    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-hgsct_657bad82-d976-492f-9230-b6a889dcfade/s3-tls-init-serving/0.log"
Jul 28 14:28:18.943493 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:18.943474    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-hgsct_657bad82-d976-492f-9230-b6a889dcfade/download-hf-model/0.log"
Jul 28 14:28:19.384698 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:19.384663    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-j8v88/perf-node-gather-daemonset-s6gtn"
Jul 28 14:28:23.519753 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:23.519733    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-mxcxh_53b9048b-bb55-4753-b5a3-96b9aae769d9/migrator/0.log"
Jul 28 14:28:23.540106 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:23.540086    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-mxcxh_53b9048b-bb55-4753-b5a3-96b9aae769d9/graceful-termination/0.log"
Jul 28 14:28:25.166240 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:25.166215    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zf5gt_03193c25-6f01-4389-b250-6eae144b3298/kube-multus-additional-cni-plugins/0.log"
Jul 28 14:28:25.189983 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:25.189956    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zf5gt_03193c25-6f01-4389-b250-6eae144b3298/egress-router-binary-copy/0.log"
Jul 28 14:28:25.215534 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:25.215510    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zf5gt_03193c25-6f01-4389-b250-6eae144b3298/cni-plugins/0.log"
Jul 28 14:28:25.265197 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:25.265176    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zf5gt_03193c25-6f01-4389-b250-6eae144b3298/bond-cni-plugin/0.log"
Jul 28 14:28:25.283226 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:25.283209    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zf5gt_03193c25-6f01-4389-b250-6eae144b3298/routeoverride-cni/0.log"
Jul 28 14:28:25.304568 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:25.304548    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zf5gt_03193c25-6f01-4389-b250-6eae144b3298/whereabouts-cni-bincopy/0.log"
Jul 28 14:28:25.327200 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:25.327183    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zf5gt_03193c25-6f01-4389-b250-6eae144b3298/whereabouts-cni/0.log"
Jul 28 14:28:25.399750 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:25.399728    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-xtlx4_e03e76bd-a14d-4bab-8fea-35b382ecabd7/kube-multus/0.log"
Jul 28 14:28:25.476147 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:25.476111    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ps58v_a418b22c-fa2d-4d69-8a21-7e2209a892de/network-metrics-daemon/0.log"
Jul 28 14:28:25.493293 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:25.493270    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ps58v_a418b22c-fa2d-4d69-8a21-7e2209a892de/kube-rbac-proxy/0.log"
Jul 28 14:28:26.526817 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:26.526782    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nd4vq_a874f52e-baf3-4227-b2ef-f66115db2ffb/ovn-controller/0.log"
Jul 28 14:28:26.543149 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:26.543128    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nd4vq_a874f52e-baf3-4227-b2ef-f66115db2ffb/ovn-acl-logging/0.log"
Jul 28 14:28:26.567582 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:26.567562    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nd4vq_a874f52e-baf3-4227-b2ef-f66115db2ffb/kube-rbac-proxy-node/0.log"
Jul 28 14:28:26.587417 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:26.587399    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nd4vq_a874f52e-baf3-4227-b2ef-f66115db2ffb/kube-rbac-proxy-ovn-metrics/0.log"
Jul 28 14:28:26.606465 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:26.606448    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nd4vq_a874f52e-baf3-4227-b2ef-f66115db2ffb/northd/0.log"
Jul 28 14:28:26.629206 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:26.629184    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nd4vq_a874f52e-baf3-4227-b2ef-f66115db2ffb/nbdb/0.log"
Jul 28 14:28:26.651714 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:26.651679    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nd4vq_a874f52e-baf3-4227-b2ef-f66115db2ffb/sbdb/0.log"
Jul 28 14:28:26.725772 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:26.725748    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nd4vq_a874f52e-baf3-4227-b2ef-f66115db2ffb/ovnkube-controller/0.log"
Jul 28 14:28:27.995579 ip-10-0-138-158 kubenswrapper[2574]: I0728 14:28:27.995549    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-wk6vn_4807337c-67b7-4b80-bf38-58ecdd8d972c/network-check-target-container/0.log"