Sep 18 19:13:58.636591 ip-10-0-131-221 systemd[1]: Starting Kubernetes Kubelet...
Sep 18 19:13:59.018900 ip-10-0-131-221 kubenswrapper[2522]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 19:13:59.018900 ip-10-0-131-221 kubenswrapper[2522]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 18 19:13:59.018900 ip-10-0-131-221 kubenswrapper[2522]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 19:13:59.018900 ip-10-0-131-221 kubenswrapper[2522]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 18 19:13:59.018900 ip-10-0-131-221 kubenswrapper[2522]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 19:13:59.021375 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.021271    2522 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 18 19:13:59.024631 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024615    2522 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:13:59.024631 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024631    2522 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:13:59.024703 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024634    2522 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:13:59.024703 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024638    2522 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:13:59.024703 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024641    2522 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:13:59.024703 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024645    2522 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:13:59.024703 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024647    2522 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:13:59.024703 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024651    2522 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:13:59.024703 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024653    2522 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:13:59.024703 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024656    2522 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:13:59.024703 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024660    2522 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:13:59.024703 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024664    2522 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:13:59.024703 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024667    2522 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:13:59.024703 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024670    2522 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:13:59.024703 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024673    2522 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:13:59.024703 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024676    2522 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:13:59.024703 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024678    2522 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:13:59.024703 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024681    2522 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:13:59.024703 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024684    2522 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:13:59.024703 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024686    2522 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:13:59.024703 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024689    2522 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:13:59.025144 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024692    2522 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:13:59.025144 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024696    2522 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:13:59.025144 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024700    2522 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:13:59.025144 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024702    2522 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:13:59.025144 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024705    2522 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:13:59.025144 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024708    2522 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:13:59.025144 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024711    2522 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:13:59.025144 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024714    2522 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:13:59.025144 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024717    2522 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:13:59.025144 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024720    2522 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:13:59.025144 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024723    2522 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:13:59.025144 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024725    2522 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:13:59.025144 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024728    2522 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:13:59.025144 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024730    2522 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:13:59.025144 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024733    2522 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:13:59.025144 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024735    2522 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:13:59.025144 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024738    2522 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:13:59.025144 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024741    2522 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:13:59.025144 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024744    2522 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:13:59.025144 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024747    2522 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:13:59.025651 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024750    2522 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:13:59.025651 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024753    2522 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:13:59.025651 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024755    2522 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:13:59.025651 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024758    2522 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:13:59.025651 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024760    2522 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:13:59.025651 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024763    2522 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:13:59.025651 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024765    2522 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:13:59.025651 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024768    2522 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:13:59.025651 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024770    2522 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:13:59.025651 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024773    2522 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:13:59.025651 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024775    2522 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:13:59.025651 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024778    2522 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:13:59.025651 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024780    2522 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:13:59.025651 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024784    2522 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:13:59.025651 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024787    2522 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:13:59.025651 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024789    2522 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:13:59.025651 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024792    2522 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:13:59.025651 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024794    2522 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:13:59.025651 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024797    2522 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:13:59.025651 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024800    2522 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:13:59.026133 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024802    2522 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:13:59.026133 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024804    2522 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:13:59.026133 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024807    2522 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:13:59.026133 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024809    2522 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:13:59.026133 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024812    2522 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:13:59.026133 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024814    2522 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:13:59.026133 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024817    2522 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:13:59.026133 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024819    2522 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:13:59.026133 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024822    2522 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:13:59.026133 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024824    2522 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:13:59.026133 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024828    2522 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:13:59.026133 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024830    2522 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:13:59.026133 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024833    2522 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:13:59.026133 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024837    2522 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:13:59.026133 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024839    2522 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:13:59.026133 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024842    2522 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:13:59.026133 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024845    2522 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:13:59.026133 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024847    2522 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:13:59.026133 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024849    2522 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:13:59.026133 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024852    2522 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:13:59.026631 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024855    2522 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:13:59.026631 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024857    2522 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:13:59.026631 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024861    2522 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:13:59.026631 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024864    2522 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:13:59.026631 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024868    2522 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:13:59.026631 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024871    2522 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:13:59.026631 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024874    2522 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:13:59.026631 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024877    2522 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:13:59.026631 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024880    2522 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:13:59.026631 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024883    2522 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:13:59.026631 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024886    2522 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:13:59.026631 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024889    2522 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:13:59.026631 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024891    2522 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:13:59.026631 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024894    2522 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:13:59.026631 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024896    2522 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:13:59.026631 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024899    2522 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:13:59.026631 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024901    2522 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:13:59.026631 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.024904    2522 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:13:59.026631 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025639    2522 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:13:59.027113 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025647    2522 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:13:59.027113 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025649    2522 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:13:59.027113 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025652    2522 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:13:59.027113 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025656    2522 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:13:59.027113 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025659    2522 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:13:59.027113 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025662    2522 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:13:59.027113 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025665    2522 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:13:59.027113 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025668    2522 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:13:59.027113 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025671    2522 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:13:59.027113 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025674    2522 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:13:59.027113 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025677    2522 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:13:59.027113 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025679    2522 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:13:59.027113 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025682    2522 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:13:59.027113 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025685    2522 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:13:59.027113 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025687    2522 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:13:59.027113 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025690    2522 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:13:59.027113 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025693    2522 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:13:59.027113 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025696    2522 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:13:59.027113 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025700    2522 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:13:59.027113 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025703    2522 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:13:59.027661 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025706    2522 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:13:59.027661 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025709    2522 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:13:59.027661 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025711    2522 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:13:59.027661 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025714    2522 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:13:59.027661 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025717    2522 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:13:59.027661 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025719    2522 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:13:59.027661 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025722    2522 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:13:59.027661 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025724    2522 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:13:59.027661 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025727    2522 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:13:59.027661 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025729    2522 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:13:59.027661 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025732    2522 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:13:59.027661 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025735    2522 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:13:59.027661 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025737    2522 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:13:59.027661 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025740    2522 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:13:59.027661 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025742    2522 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:13:59.027661 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025745    2522 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:13:59.027661 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025747    2522 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:13:59.027661 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025750    2522 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:13:59.027661 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025753    2522 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:13:59.027661 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025755    2522 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:13:59.028140 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025758    2522 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:13:59.028140 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025763    2522 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:13:59.028140 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025767    2522 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:13:59.028140 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025770    2522 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:13:59.028140 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025773    2522 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:13:59.028140 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025776    2522 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:13:59.028140 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025778    2522 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:13:59.028140 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025781    2522 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:13:59.028140 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025784    2522 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:13:59.028140 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025787    2522 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:13:59.028140 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025791    2522 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:13:59.028140 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025793    2522 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:13:59.028140 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025796    2522 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:13:59.028140 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025799    2522 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:13:59.028140 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025802    2522 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:13:59.028140 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025804    2522 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:13:59.028140 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025807    2522 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:13:59.028140 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025809    2522 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:13:59.028140 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025812    2522 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:13:59.028616 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025815    2522 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:13:59.028616 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025817    2522 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:13:59.028616 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025820    2522 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:13:59.028616 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025822    2522 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:13:59.028616 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025825    2522 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:13:59.028616 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025827    2522 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:13:59.028616 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025830    2522 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:13:59.028616 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025832    2522 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:13:59.028616 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025835    2522 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:13:59.028616 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025837    2522 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:13:59.028616 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025839    2522 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:13:59.028616 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025843    2522 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:13:59.028616 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025845    2522 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:13:59.028616 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025848    2522 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:13:59.028616 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025850    2522 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:13:59.028616 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025853    2522 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:13:59.028616 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025855    2522 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:13:59.028616 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025858    2522 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:13:59.028616 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025862    2522 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:13:59.029065 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025865    2522 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:13:59.029065 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025868    2522 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:13:59.029065 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025871    2522 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:13:59.029065 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025874    2522 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:13:59.029065 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025877    2522 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:13:59.029065 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025880    2522 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:13:59.029065 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025882    2522 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:13:59.029065 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025885    2522 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:13:59.029065 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025888    2522 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:13:59.029065 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025891    2522 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:13:59.029065 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025894    2522 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:13:59.029065 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025898    2522 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:13:59.029065 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025901    2522 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:13:59.029065 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025904    2522 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:13:59.029065 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025908    2522 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:13:59.029065 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025910    2522 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:13:59.029065 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025913    2522 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:13:59.029065 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025916    2522 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:13:59.029065 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025918    2522 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:13:59.029556 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.025921    2522 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:13:59.029556 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026374    2522 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:13:59.029556 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026379    2522 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:13:59.029556 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026382    2522 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:13:59.029556 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026385    2522 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:13:59.029556 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026393    2522 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:13:59.029556 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026396    2522 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:13:59.029556 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026399    2522 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:13:59.029556 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026402    2522 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:13:59.029556 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026406    2522 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:13:59.029556 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026410    2522 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:13:59.029556 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026413    2522 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:13:59.029556 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026416    2522 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:13:59.029556 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026418    2522 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:13:59.029556 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026422    2522 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:13:59.029556 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026426    2522 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:13:59.029556 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026429    2522 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:13:59.029556 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026432    2522 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:13:59.029556 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026435    2522 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:13:59.030008 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026438    2522 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:13:59.030008 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026440    2522 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:13:59.030008 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026443    2522 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:13:59.030008 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026446    2522 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:13:59.030008 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026448    2522 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:13:59.030008 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026451    2522 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:13:59.030008 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026454    2522 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:13:59.030008 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026456    2522 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:13:59.030008 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026459    2522 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:13:59.030008 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026462    2522 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:13:59.030008 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026465    2522 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:13:59.030008 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026467    2522 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:13:59.030008 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026470    2522 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:13:59.030008 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026472    2522 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:13:59.030008 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026475    2522 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:13:59.030008 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026477    2522 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:13:59.030008 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026480    2522 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:13:59.030008 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026482    2522 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:13:59.030008 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026485    2522 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:13:59.030008 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026498    2522 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:13:59.030508 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026501    2522 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:13:59.030508 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026503    2522 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:13:59.030508 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026506    2522 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:13:59.030508 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026509    2522 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:13:59.030508 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026511    2522 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:13:59.030508 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026513    2522 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:13:59.030508 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026516    2522 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:13:59.030508 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026518    2522 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:13:59.030508 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026521    2522 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:13:59.030508 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026526    2522 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:13:59.030508 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026529    2522 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:13:59.030508 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026532    2522 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:13:59.030508 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026534    2522 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:13:59.030508 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026537    2522 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:13:59.030508 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026540    2522 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:13:59.030508 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026542    2522 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:13:59.030508 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026545    2522 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:13:59.030508 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026548    2522 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:13:59.030508 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026550    2522 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:13:59.030508 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026553    2522 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:13:59.030987 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026555    2522 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:13:59.030987 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026558    2522 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:13:59.030987 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026560    2522 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:13:59.030987 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026563    2522 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:13:59.030987 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026565    2522 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:13:59.030987 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026568    2522 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:13:59.030987 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026570    2522 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:13:59.030987 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026573    2522 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:13:59.030987 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026576    2522 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:13:59.030987 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026578    2522 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:13:59.030987 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026581    2522 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:13:59.030987 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026583    2522 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:13:59.030987 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026591    2522 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:13:59.030987 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026594    2522 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:13:59.030987 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026596    2522 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:13:59.030987 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026598    2522 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:13:59.030987 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026601    2522 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:13:59.030987 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026604    2522 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:13:59.030987 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026606    2522 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:13:59.030987 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026608    2522 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:13:59.031473 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026611    2522 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:13:59.031473 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026614    2522 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:13:59.031473 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026618    2522 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:13:59.031473 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026620    2522 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:13:59.031473 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026623    2522 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:13:59.031473 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026626    2522 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:13:59.031473 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026628    2522 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:13:59.031473 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026631    2522 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:13:59.031473 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026634    2522 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:13:59.031473 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026636    2522 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:13:59.031473 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026639    2522 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:13:59.031473 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026642    2522 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:13:59.031473 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026644    2522 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:13:59.031473 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026647    2522 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:13:59.031473 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026649    2522 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:13:59.031473 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026652    2522 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:13:59.031473 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026654    2522 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:13:59.031473 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026657    2522 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:13:59.031473 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026661    2522 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:13:59.031941 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026663    2522 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:13:59.031941 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.026667    2522 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:13:59.031941 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.026758    2522 flags.go:64] FLAG: --address="0.0.0.0"
Sep 18 19:13:59.031941 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.026798    2522 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 18 19:13:59.031941 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.026807    2522 flags.go:64] FLAG: --anonymous-auth="true"
Sep 18 19:13:59.031941 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.026812    2522 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 18 19:13:59.031941 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.026825    2522 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 18 19:13:59.031941 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.026828    2522 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 18 19:13:59.031941 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.026833    2522 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 18 19:13:59.031941 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.026838    2522 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 18 19:13:59.031941 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.026841    2522 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 18 19:13:59.031941 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.026844    2522 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 18 19:13:59.031941 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.026848    2522 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 18 19:13:59.031941 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.026851    2522 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 18 19:13:59.031941 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.026855    2522 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 18 19:13:59.031941 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.026859    2522 flags.go:64] FLAG: --cgroup-root=""
Sep 18 19:13:59.031941 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.026861    2522 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 18 19:13:59.031941 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.026868    2522 flags.go:64] FLAG: --client-ca-file=""
Sep 18 19:13:59.031941 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.026871    2522 flags.go:64] FLAG: --cloud-provider="external"
Sep 18 19:13:59.031941 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.026874    2522 flags.go:64] FLAG: --cluster-dns="[]"
Sep 18 19:13:59.031941 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.026880    2522 flags.go:64] FLAG: --cluster-domain=""
Sep 18 19:13:59.031941 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.026883    2522 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 18 19:13:59.031941 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.026886    2522 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 18 19:13:59.032539 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.026889    2522 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 18 19:13:59.032539 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.026893    2522 flags.go:64] FLAG: --container-log-max-files="5"
Sep 18 19:13:59.032539 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.026896    2522 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 18 19:13:59.032539 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.026899    2522 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 18 19:13:59.032539 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.026902    2522 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 18 19:13:59.032539 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.026905    2522 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 18 19:13:59.032539 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.026908    2522 flags.go:64] FLAG: --contention-profiling="false"
Sep 18 19:13:59.032539 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.026911    2522 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 18 19:13:59.032539 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.026914    2522 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 18 19:13:59.032539 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.026917    2522 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 18 19:13:59.032539 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.026920    2522 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 18 19:13:59.032539 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.026924    2522 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 18 19:13:59.032539 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.026927    2522 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 18 19:13:59.032539 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.026931    2522 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 18 19:13:59.032539 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.026934    2522 flags.go:64] FLAG: --enable-load-reader="false"
Sep 18 19:13:59.032539 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.026937    2522 flags.go:64] FLAG: --enable-server="true"
Sep 18 19:13:59.032539 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.026946    2522 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 18 19:13:59.032539 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.026977    2522 flags.go:64] FLAG: --event-burst="100"
Sep 18 19:13:59.032539 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.026981    2522 flags.go:64] FLAG: --event-qps="50"
Sep 18 19:13:59.032539 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.026984    2522 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 18 19:13:59.032539 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.026987    2522 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 18 19:13:59.032539 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.026991    2522 flags.go:64] FLAG: --eviction-hard=""
Sep 18 19:13:59.032539 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.026998    2522 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 18 19:13:59.032539 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027002    2522 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 18 19:13:59.033117 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027005    2522 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 18 19:13:59.033117 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027008    2522 flags.go:64] FLAG: --eviction-soft=""
Sep 18 19:13:59.033117 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027011    2522 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 18 19:13:59.033117 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027014    2522 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 18 19:13:59.033117 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027017    2522 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 18 19:13:59.033117 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027020    2522 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 18 19:13:59.033117 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027022    2522 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 18 19:13:59.033117 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027025    2522 flags.go:64] FLAG: --fail-swap-on="true"
Sep 18 19:13:59.033117 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027028    2522 flags.go:64] FLAG: --feature-gates=""
Sep 18 19:13:59.033117 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027032    2522 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 18 19:13:59.033117 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027035    2522 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 18 19:13:59.033117 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027038    2522 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 18 19:13:59.033117 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027041    2522 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 18 19:13:59.033117 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027045    2522 flags.go:64] FLAG: --healthz-port="10248"
Sep 18 19:13:59.033117 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027047    2522 flags.go:64] FLAG: --help="false"
Sep 18 19:13:59.033117 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027050    2522 flags.go:64] FLAG: --hostname-override="ip-10-0-131-221.ec2.internal"
Sep 18 19:13:59.033117 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027053    2522 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 18 19:13:59.033117 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027056    2522 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 18 19:13:59.033117 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027059    2522 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 18 19:13:59.033117 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027062    2522 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 18 19:13:59.033117 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027065    2522 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 18 19:13:59.033117 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027070    2522 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 18 19:13:59.033117 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027072    2522 flags.go:64] FLAG: --image-service-endpoint=""
Sep 18 19:13:59.033725 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027075    2522 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 18 19:13:59.033725 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027078    2522 flags.go:64] FLAG: --kube-api-burst="100"
Sep 18 19:13:59.033725 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027086    2522 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 18 19:13:59.033725 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027090    2522 flags.go:64] FLAG: --kube-api-qps="50"
Sep 18 19:13:59.033725 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027093    2522 flags.go:64] FLAG: --kube-reserved=""
Sep 18 19:13:59.033725 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027096    2522 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 18 19:13:59.033725 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027099    2522 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 18 19:13:59.033725 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027102    2522 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 18 19:13:59.033725 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027106    2522 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 18 19:13:59.033725 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027109    2522 flags.go:64] FLAG: --lock-file=""
Sep 18 19:13:59.033725 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027111    2522 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 18 19:13:59.033725 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027114    2522 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 18 19:13:59.033725 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027118    2522 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 18 19:13:59.033725 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027123    2522 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 18 19:13:59.033725 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027126    2522 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 18 19:13:59.033725 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027129    2522 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 18 19:13:59.033725 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027132    2522 flags.go:64] FLAG: --logging-format="text"
Sep 18 19:13:59.033725 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027135    2522 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 18 19:13:59.033725 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027139    2522 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 18 19:13:59.033725 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027141    2522 flags.go:64] FLAG: --manifest-url=""
Sep 18 19:13:59.033725 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027144    2522 flags.go:64] FLAG: --manifest-url-header=""
Sep 18 19:13:59.033725 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027149    2522 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 18 19:13:59.033725 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027152    2522 flags.go:64] FLAG: --max-open-files="1000000"
Sep 18 19:13:59.033725 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027156    2522 flags.go:64] FLAG: --max-pods="110"
Sep 18 19:13:59.033725 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027160    2522 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 18 19:13:59.034355 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027163    2522 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 18 19:13:59.034355 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027166    2522 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 18 19:13:59.034355 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027169    2522 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 18 19:13:59.034355 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027172    2522 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 18 19:13:59.034355 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027174    2522 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 18 19:13:59.034355 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027179    2522 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 18 19:13:59.034355 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027186    2522 flags.go:64] FLAG: --node-status-max-images="50"
Sep 18 19:13:59.034355 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027189    2522 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 18 19:13:59.034355 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027192    2522 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 18 19:13:59.034355 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027195    2522 flags.go:64] FLAG: --pod-cidr=""
Sep 18 19:13:59.034355 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027204    2522 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 18 19:13:59.034355 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027210    2522 flags.go:64] FLAG: --pod-manifest-path=""
Sep 18 19:13:59.034355 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027213    2522 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 18 19:13:59.034355 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027216    2522 flags.go:64] FLAG: --pods-per-core="0"
Sep 18 19:13:59.034355 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027219    2522 flags.go:64] FLAG: --port="10250"
Sep 18 19:13:59.034355 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027222    2522 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 18 19:13:59.034355 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027225    2522 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e68d6d26aad28ccc"
Sep 18 19:13:59.034355 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027228    2522 flags.go:64] FLAG: --qos-reserved=""
Sep 18 19:13:59.034355 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027231    2522 flags.go:64] FLAG: --read-only-port="10255"
Sep 18 19:13:59.034355 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027234    2522 flags.go:64] FLAG: --register-node="true"
Sep 18 19:13:59.034355 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027237    2522 flags.go:64] FLAG: --register-with-taints=""
Sep 18 19:13:59.034355 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027241    2522 flags.go:64] FLAG: --registry-burst="10"
Sep 18 19:13:59.034355 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027257    2522 flags.go:64] FLAG: --registry-qps="5"
Sep 18 19:13:59.034355 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027261    2522 flags.go:64] FLAG: --reserved-cpus=""
Sep 18 19:13:59.034942 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027264    2522 flags.go:64] FLAG: --reserved-memory=""
Sep 18 19:13:59.034942 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027267    2522 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 18 19:13:59.034942 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027270    2522 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 18 19:13:59.034942 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027273    2522 flags.go:64] FLAG: --rotate-certificates="false"
Sep 18 19:13:59.034942 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027276    2522 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 18 19:13:59.034942 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027279    2522 flags.go:64] FLAG: --runonce="false"
Sep 18 19:13:59.034942 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027282    2522 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 18 19:13:59.034942 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027285    2522 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 18 19:13:59.034942 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027288    2522 flags.go:64] FLAG: --seccomp-default="false"
Sep 18 19:13:59.034942 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027291    2522 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 18 19:13:59.034942 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027294    2522 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 18 19:13:59.034942 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027297    2522 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 18 19:13:59.034942 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027300    2522 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 18 19:13:59.034942 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027304    2522 flags.go:64] FLAG: --storage-driver-password="root"
Sep 18 19:13:59.034942 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027307    2522 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 18 19:13:59.034942 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027311    2522 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 18 19:13:59.034942 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027314    2522 flags.go:64] FLAG: --storage-driver-user="root"
Sep 18 19:13:59.034942 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027316    2522 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 18 19:13:59.034942 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027319    2522 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 18 19:13:59.034942 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027328    2522 flags.go:64] FLAG: --system-cgroups=""
Sep 18 19:13:59.034942 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027331    2522 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 18 19:13:59.034942 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027336    2522 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 18 19:13:59.034942 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027339    2522 flags.go:64] FLAG: --tls-cert-file=""
Sep 18 19:13:59.034942 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027343    2522 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 18 19:13:59.034942 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027349    2522 flags.go:64] FLAG: --tls-min-version=""
Sep 18 19:13:59.035575 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027351    2522 flags.go:64] FLAG: --tls-private-key-file=""
Sep 18 19:13:59.035575 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027354    2522 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 18 19:13:59.035575 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027357    2522 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 18 19:13:59.035575 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027360    2522 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 18 19:13:59.035575 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027363    2522 flags.go:64] FLAG: --v="2"
Sep 18 19:13:59.035575 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027367    2522 flags.go:64] FLAG: --version="false"
Sep 18 19:13:59.035575 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027371    2522 flags.go:64] FLAG: --vmodule=""
Sep 18 19:13:59.035575 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027375    2522 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 18 19:13:59.035575 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.027378    2522 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 18 19:13:59.035575 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027517    2522 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:13:59.035575 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027521    2522 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:13:59.035575 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027526    2522 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:13:59.035575 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027530    2522 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:13:59.035575 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027533    2522 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:13:59.035575 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027536    2522 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:13:59.035575 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027538    2522 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:13:59.035575 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027541    2522 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:13:59.035575 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027544    2522 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:13:59.035575 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027546    2522 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:13:59.035575 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027549    2522 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:13:59.035575 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027552    2522 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:13:59.036116 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027556    2522 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:13:59.036116 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027558    2522 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:13:59.036116 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027561    2522 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:13:59.036116 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027564    2522 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:13:59.036116 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027567    2522 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:13:59.036116 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027569    2522 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:13:59.036116 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027578    2522 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:13:59.036116 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027581    2522 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:13:59.036116 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027583    2522 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:13:59.036116 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027586    2522 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:13:59.036116 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027589    2522 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:13:59.036116 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027592    2522 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:13:59.036116 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027594    2522 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:13:59.036116 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027597    2522 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:13:59.036116 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027599    2522 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:13:59.036116 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027601    2522 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:13:59.036116 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027604    2522 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:13:59.036116 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027607    2522 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:13:59.036116 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027609    2522 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:13:59.036116 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027612    2522 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:13:59.036636 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027614    2522 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:13:59.036636 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027617    2522 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:13:59.036636 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027619    2522 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:13:59.036636 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027622    2522 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:13:59.036636 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027625    2522 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:13:59.036636 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027628    2522 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:13:59.036636 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027630    2522 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:13:59.036636 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027633    2522 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:13:59.036636 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027635    2522 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:13:59.036636 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027638    2522 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:13:59.036636 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027640    2522 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:13:59.036636 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027643    2522 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:13:59.036636 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027648    2522 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:13:59.036636 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027651    2522 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:13:59.036636 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027653    2522 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:13:59.036636 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027656    2522 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:13:59.036636 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027659    2522 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:13:59.036636 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027661    2522 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:13:59.036636 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027664    2522 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:13:59.036636 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027672    2522 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:13:59.037118 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027675    2522 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:13:59.037118 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027678    2522 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:13:59.037118 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027680    2522 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:13:59.037118 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027683    2522 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:13:59.037118 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027686    2522 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:13:59.037118 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027688    2522 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:13:59.037118 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027691    2522 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:13:59.037118 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027694    2522 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:13:59.037118 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027698    2522 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:13:59.037118 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027701    2522 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:13:59.037118 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027704    2522 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:13:59.037118 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027706    2522 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:13:59.037118 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027709    2522 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:13:59.037118 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027712    2522 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:13:59.037118 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027715    2522 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:13:59.037118 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027717    2522 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:13:59.037118 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027720    2522 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:13:59.037118 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027723    2522 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:13:59.037118 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027725    2522 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:13:59.037601 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027728    2522 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:13:59.037601 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027730    2522 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:13:59.037601 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027733    2522 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:13:59.037601 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027736    2522 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:13:59.037601 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027738    2522 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:13:59.037601 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027743    2522 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:13:59.037601 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027747    2522 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:13:59.037601 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027750    2522 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:13:59.037601 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027753    2522 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:13:59.037601 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027755    2522 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:13:59.037601 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027758    2522 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:13:59.037601 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027760    2522 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:13:59.037601 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027763    2522 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:13:59.037601 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027773    2522 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:13:59.037601 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027776    2522 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:13:59.037601 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027779    2522 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:13:59.037601 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027781    2522 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:13:59.037601 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027784    2522 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:13:59.037601 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027786    2522 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:13:59.037601 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027789    2522 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:13:59.038094 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027791    2522 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:13:59.038094 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027794    2522 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:13:59.038094 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027796    2522 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:13:59.038094 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027799    2522 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:13:59.038094 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027801    2522 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:13:59.038094 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027804    2522 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:13:59.038094 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027806    2522 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:13:59.038094 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.027809    2522 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:13:59.038094 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.028447    2522 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 19:13:59.038488 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.038466    2522 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 18 19:13:59.038488 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.038488    2522 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 18 19:13:59.039407 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.039398    2522 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 18 19:13:59.039445 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.039407    2522 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 18 19:13:59.039486 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039478    2522 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:13:59.039486 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039485    2522 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:13:59.039536 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039488    2522 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:13:59.039536 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039491    2522 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:13:59.039536 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039495    2522 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:13:59.039536 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039498    2522 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:13:59.039536 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039502    2522 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:13:59.039536 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039505    2522 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:13:59.039536 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039508    2522 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:13:59.039536 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039511    2522 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:13:59.039536 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039515    2522 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:13:59.039536 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039520    2522 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:13:59.039536 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039523    2522 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:13:59.039536 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039525    2522 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:13:59.039536 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039528    2522 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:13:59.039536 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039531    2522 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:13:59.039536 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039533    2522 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:13:59.039536 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039536    2522 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:13:59.039536 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039539    2522 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:13:59.039536 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039542    2522 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:13:59.039536 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039545    2522 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:13:59.040068 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039548    2522 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:13:59.040068 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039550    2522 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:13:59.040068 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039553    2522 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:13:59.040068 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039556    2522 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:13:59.040068 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039558    2522 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:13:59.040068 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039561    2522 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:13:59.040068 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039564    2522 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:13:59.040068 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039566    2522 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:13:59.040068 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039574    2522 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:13:59.040068 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039577    2522 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:13:59.040068 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039580    2522 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:13:59.040068 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039582    2522 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:13:59.040068 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039585    2522 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:13:59.040068 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039587    2522 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:13:59.040068 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039590    2522 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:13:59.040068 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039592    2522 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:13:59.040068 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039595    2522 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:13:59.040068 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039598    2522 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:13:59.040068 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039600    2522 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:13:59.040760 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039603    2522 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:13:59.040760 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039605    2522 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:13:59.040760 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039608    2522 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:13:59.040760 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039610    2522 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:13:59.040760 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039614    2522 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:13:59.040760 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039618    2522 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:13:59.040760 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039621    2522 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:13:59.040760 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039624    2522 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:13:59.040760 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039627    2522 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:13:59.040760 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039629    2522 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:13:59.040760 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039632    2522 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:13:59.040760 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039635    2522 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:13:59.040760 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039638    2522 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:13:59.040760 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039641    2522 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:13:59.040760 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039644    2522 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:13:59.040760 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039647    2522 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:13:59.040760 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039650    2522 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:13:59.040760 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039652    2522 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:13:59.040760 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039655    2522 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:13:59.041529 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039657    2522 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:13:59.041529 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039660    2522 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:13:59.041529 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039662    2522 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:13:59.041529 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039670    2522 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:13:59.041529 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039673    2522 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:13:59.041529 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039675    2522 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:13:59.041529 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039678    2522 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:13:59.041529 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039680    2522 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:13:59.041529 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039683    2522 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:13:59.041529 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039686    2522 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:13:59.041529 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039688    2522 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:13:59.041529 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039691    2522 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:13:59.041529 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039693    2522 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:13:59.041529 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039696    2522 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:13:59.041529 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039699    2522 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:13:59.041529 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039701    2522 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:13:59.041529 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039703    2522 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:13:59.041529 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039706    2522 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:13:59.041529 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039708    2522 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:13:59.041529 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039711    2522 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:13:59.042099 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039719    2522 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:13:59.042099 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039721    2522 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:13:59.042099 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039724    2522 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:13:59.042099 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039727    2522 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:13:59.042099 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039729    2522 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:13:59.042099 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039732    2522 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:13:59.042099 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039734    2522 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:13:59.042099 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039738    2522 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:13:59.042099 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039741    2522 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:13:59.042099 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039743    2522 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:13:59.042099 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039746    2522 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:13:59.042099 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039748    2522 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:13:59.042099 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039752    2522 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:13:59.042099 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039756    2522 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:13:59.042099 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039759    2522 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:13:59.042099 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039762    2522 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:13:59.042099 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039770    2522 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:13:59.042099 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039774    2522 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:13:59.042099 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039777    2522 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:13:59.042099 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039780    2522 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:13:59.042716 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.039785    2522 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 19:13:59.042716 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039946    2522 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:13:59.042716 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039951    2522 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:13:59.042716 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039954    2522 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:13:59.042716 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039958    2522 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:13:59.042716 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039961    2522 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:13:59.042716 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039964    2522 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:13:59.042716 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039967    2522 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:13:59.042716 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039970    2522 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:13:59.042716 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039972    2522 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:13:59.042716 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039975    2522 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:13:59.042716 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039978    2522 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:13:59.042716 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039980    2522 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:13:59.042716 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039983    2522 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:13:59.042716 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039985    2522 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:13:59.043091 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039988    2522 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:13:59.043091 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039991    2522 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:13:59.043091 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039993    2522 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:13:59.043091 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039995    2522 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:13:59.043091 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.039998    2522 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:13:59.043091 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040001    2522 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:13:59.043091 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040004    2522 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:13:59.043091 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040006    2522 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:13:59.043091 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040009    2522 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:13:59.043091 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040012    2522 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:13:59.043091 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040016    2522 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:13:59.043091 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040019    2522 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:13:59.043091 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040022    2522 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:13:59.043091 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040024    2522 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:13:59.043091 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040032    2522 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:13:59.043091 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040035    2522 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:13:59.043091 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040038    2522 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:13:59.043091 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040040    2522 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:13:59.043091 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040043    2522 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:13:59.043091 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040045    2522 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:13:59.043690 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040048    2522 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:13:59.043690 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040050    2522 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:13:59.043690 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040053    2522 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:13:59.043690 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040055    2522 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:13:59.043690 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040058    2522 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:13:59.043690 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040060    2522 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:13:59.043690 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040063    2522 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:13:59.043690 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040065    2522 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:13:59.043690 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040068    2522 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:13:59.043690 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040070    2522 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:13:59.043690 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040073    2522 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:13:59.043690 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040076    2522 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:13:59.043690 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040079    2522 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:13:59.043690 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040081    2522 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:13:59.043690 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040084    2522 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:13:59.043690 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040086    2522 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:13:59.043690 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040090    2522 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:13:59.043690 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040094    2522 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:13:59.043690 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040097    2522 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:13:59.044142 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040100    2522 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:13:59.044142 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040103    2522 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:13:59.044142 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040106    2522 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:13:59.044142 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040109    2522 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:13:59.044142 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040111    2522 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:13:59.044142 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040114    2522 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:13:59.044142 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040116    2522 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:13:59.044142 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040118    2522 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:13:59.044142 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040127    2522 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:13:59.044142 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040130    2522 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:13:59.044142 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040133    2522 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:13:59.044142 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040136    2522 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:13:59.044142 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040138    2522 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:13:59.044142 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040141    2522 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:13:59.044142 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040144    2522 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:13:59.044142 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040146    2522 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:13:59.044142 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040149    2522 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:13:59.044142 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040151    2522 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:13:59.044142 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040154    2522 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:13:59.044142 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040156    2522 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:13:59.044645 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040159    2522 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:13:59.044645 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040161    2522 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:13:59.044645 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040164    2522 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:13:59.044645 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040167    2522 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:13:59.044645 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040169    2522 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:13:59.044645 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040171    2522 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:13:59.044645 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040174    2522 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:13:59.044645 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040176    2522 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:13:59.044645 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040179    2522 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:13:59.044645 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040182    2522 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:13:59.044645 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040184    2522 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:13:59.044645 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040187    2522 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:13:59.044645 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040190    2522 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:13:59.044645 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040192    2522 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:13:59.044645 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040195    2522 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:13:59.044645 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040197    2522 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:13:59.044645 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040200    2522 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:13:59.044645 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040202    2522 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:13:59.044645 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040205    2522 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:13:59.044645 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040207    2522 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:13:59.045121 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040210    2522 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:13:59.045121 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040217    2522 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:13:59.045121 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040221    2522 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:13:59.045121 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040224    2522 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:13:59.045121 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040227    2522 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:13:59.045121 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:13:59.040230    2522 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:13:59.045121 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.040235    2522 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 19:13:59.045121 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.040802    2522 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 18 19:13:59.045494 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.045478    2522 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 18 19:13:59.046641 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.046629    2522 server.go:1019] "Starting client certificate rotation"
Sep 18 19:13:59.046747 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.046729    2522 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 19:13:59.046791 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.046768    2522 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 19:13:59.049702 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.049689    2522 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 19:13:59.050856 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.050838    2522 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 19:13:59.066033 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.066013    2522 log.go:25] "Validated CRI v1 runtime API"
Sep 18 19:13:59.068090 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.068075    2522 log.go:25] "Validated CRI v1 image API"
Sep 18 19:13:59.069676 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.069658    2522 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 18 19:13:59.072340 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.072322    2522 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 95218e38-ab48-4403-acf5-d98168d064ec:/dev/nvme0n1p3 c46fb0d0-7c2c-4536-8939-3d25b3ef5c6a:/dev/nvme0n1p4]
Sep 18 19:13:59.072387 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.072340    2522 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 18 19:13:59.078323 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.078191    2522 manager.go:216] Machine: {Timestamp:2026-09-18 19:13:59.076818498 +0000 UTC m=+0.345659742 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100521 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22a7ea579bbe8f32cf6e0fafd36b47 SystemUUID:ec22a7ea-579b-be8f-32cf-6e0fafd36b47 BootID:fac0e6f7-1b9b-47e1-bc58-40f1f20602e0 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ec:15:9a:99 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ec:15:9a:99 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ba:9c:d0:5a:23:0d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 18 19:13:59.078323 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.078321    2522 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 18 19:13:59.078434 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.078412    2522 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 18 19:13:59.081125 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.081100    2522 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 18 19:13:59.081293 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.081126    2522 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-221.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 18 19:13:59.081986 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.081975    2522 topology_manager.go:138] "Creating topology manager with none policy"
Sep 18 19:13:59.082039 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.081988    2522 container_manager_linux.go:309] "Creating device plugin manager"
Sep 18 19:13:59.082039 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.081996    2522 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 19:13:59.082947 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.082937    2522 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 19:13:59.083025 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.083016    2522 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 18 19:13:59.089188 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.089166    2522 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 19:13:59.089399 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.089384    2522 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 19:13:59.089518 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.089508    2522 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 18 19:13:59.089558 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.089538    2522 kubelet.go:486] "Attempting to sync node with API server"
Sep 18 19:13:59.089558 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.089549    2522 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 18 19:13:59.089630 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.089562    2522 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 18 19:13:59.089630 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.089572    2522 kubelet.go:398] "Adding apiserver pod source"
Sep 18 19:13:59.089630 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.089581    2522 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 18 19:13:59.089959 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.089945    2522 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 19:13:59.089959 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.089963    2522 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 19:13:59.092633 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.092615    2522 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 18 19:13:59.095574 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.095557    2522 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 18 19:13:59.095647 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.095588    2522 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 18 19:13:59.097005 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.096990    2522 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 18 19:13:59.097050 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.097009    2522 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 18 19:13:59.097050 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.097015    2522 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 18 19:13:59.097050 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.097021    2522 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 18 19:13:59.097050 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.097029    2522 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 18 19:13:59.097050 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.097034    2522 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 18 19:13:59.097050 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.097048    2522 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 18 19:13:59.097050 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.097053    2522 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 18 19:13:59.097274 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.097059    2522 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 18 19:13:59.097274 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.097065    2522 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 18 19:13:59.097274 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.097073    2522 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 18 19:13:59.097274 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.097079    2522 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 18 19:13:59.097274 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.097100    2522 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 18 19:13:59.097274 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.097105    2522 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 18 19:13:59.100761 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.100747    2522 server.go:1268] "Started kubelet"
Sep 18 19:13:59.100892 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.100836    2522 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 18 19:13:59.100892 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.100876    2522 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 18 19:13:59.101086 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.100915    2522 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 18 19:13:59.101678 ip-10-0-131-221 systemd[1]: Started Kubernetes Kubelet.
Sep 18 19:13:59.102500 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.102482    2522 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 18 19:13:59.102732 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.102720    2522 server.go:310] "Adding debug handlers to kubelet server"
Sep 18 19:13:59.107789 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.107747    2522 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-221.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 19:13:59.107789 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:13:59.107770    2522 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 19:13:59.107789 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:13:59.107776    2522 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-221.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 19:13:59.107934 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:13:59.107799    2522 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 18 19:13:59.111765 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.111747    2522 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 18 19:13:59.112482 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.112465    2522 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 18 19:13:59.112634 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:13:59.112618    2522 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-221.ec2.internal\" not found"
Sep 18 19:13:59.112863 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:13:59.109663    2522 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-221.ec2.internal.18d6802e7e36c8ee  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-221.ec2.internal,UID:ip-10-0-131-221.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-221.ec2.internal,},FirstTimestamp:2026-09-18 19:13:59.10072139 +0000 UTC m=+0.369562635,LastTimestamp:2026-09-18 19:13:59.10072139 +0000 UTC m=+0.369562635,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-221.ec2.internal,}"
Sep 18 19:13:59.113294 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.113271    2522 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 18 19:13:59.113294 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.113272    2522 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 18 19:13:59.113393 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.113304    2522 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 18 19:13:59.113393 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.113307    2522 factory.go:55] Registering systemd factory
Sep 18 19:13:59.113393 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.113323    2522 factory.go:223] Registration of the systemd container factory successfully
Sep 18 19:13:59.113519 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.113398    2522 reconstruct.go:97] "Volume reconstruction finished"
Sep 18 19:13:59.113519 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.113408    2522 reconciler.go:29] "Reconciler: start to sync state"
Sep 18 19:13:59.113578 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.113545    2522 factory.go:153] Registering CRI-O factory
Sep 18 19:13:59.113578 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.113555    2522 factory.go:223] Registration of the crio container factory successfully
Sep 18 19:13:59.113628 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.113599    2522 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 18 19:13:59.113628 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.113613    2522 factory.go:103] Registering Raw factory
Sep 18 19:13:59.113628 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.113624    2522 manager.go:1196] Started watching for new ooms in manager
Sep 18 19:13:59.114701 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.114684    2522 manager.go:349] Starting recovery of all containers
Sep 18 19:13:59.118638 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:13:59.118618    2522 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 19:13:59.118769 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:13:59.118728    2522 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-221.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 18 19:13:59.128118 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.127995    2522 manager.go:354] Recovery completed
Sep 18 19:13:59.134087 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.134073    2522 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:13:59.137409 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.137373    2522 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-221.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:13:59.137476 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.137419    2522 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-221.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:13:59.137476 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.137432    2522 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-221.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:13:59.138067 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.138039    2522 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 18 19:13:59.138067 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.138056    2522 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 18 19:13:59.138184 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.138074    2522 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 19:13:59.138184 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.138046    2522 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:13:59.139370 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.139352    2522 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-221.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:13:59.139443 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.139385    2522 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-221.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:13:59.139443 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.139400    2522 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-221.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:13:59.140332 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.140320    2522 policy_none.go:49] "None policy: Start"
Sep 18 19:13:59.140375 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.140336    2522 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 18 19:13:59.140375 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.140346    2522 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 18 19:13:59.140936 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:13:59.140872    2522 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-221.ec2.internal.18d6802e80667ee4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-221.ec2.internal,UID:ip-10-0-131-221.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-221.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-221.ec2.internal,},FirstTimestamp:2026-09-18 19:13:59.137402596 +0000 UTC m=+0.406243841,LastTimestamp:2026-09-18 19:13:59.137402596 +0000 UTC m=+0.406243841,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-221.ec2.internal,}"
Sep 18 19:13:59.142324 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.142312    2522 policy_none.go:47] "Start"
Sep 18 19:13:59.150569 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:13:59.150505    2522 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-221.ec2.internal.18d6802e8066d4fe  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-221.ec2.internal,UID:ip-10-0-131-221.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-221.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-221.ec2.internal,},FirstTimestamp:2026-09-18 19:13:59.137424638 +0000 UTC m=+0.406265881,LastTimestamp:2026-09-18 19:13:59.137424638 +0000 UTC m=+0.406265881,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-221.ec2.internal,}"
Sep 18 19:13:59.164670 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:13:59.164607    2522 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-221.ec2.internal.18d6802e80670947  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-221.ec2.internal,UID:ip-10-0-131-221.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-131-221.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-131-221.ec2.internal,},FirstTimestamp:2026-09-18 19:13:59.137438023 +0000 UTC m=+0.406279269,LastTimestamp:2026-09-18 19:13:59.137438023 +0000 UTC m=+0.406279269,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-221.ec2.internal,}"
Sep 18 19:13:59.177151 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:13:59.177083    2522 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-221.ec2.internal.18d6802e80667ee4\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-221.ec2.internal.18d6802e80667ee4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-221.ec2.internal,UID:ip-10-0-131-221.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-221.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-221.ec2.internal,},FirstTimestamp:2026-09-18 19:13:59.137402596 +0000 UTC m=+0.406243841,LastTimestamp:2026-09-18 19:13:59.139370279 +0000 UTC m=+0.408211525,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-221.ec2.internal,}"
Sep 18 19:13:59.180849 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.180836    2522 manager.go:337] "Starting Device Plugin manager"
Sep 18 19:13:59.180956 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:13:59.180944    2522 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 18 19:13:59.181013 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.180959    2522 server.go:87] "Starting device plugin registration server"
Sep 18 19:13:59.181396 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.181381    2522 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 18 19:13:59.181471 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.181406    2522 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 18 19:13:59.181663 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.181565    2522 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 18 19:13:59.181750 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.181673    2522 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 18 19:13:59.181750 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.181684    2522 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 18 19:13:59.182342 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:13:59.182316    2522 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 18 19:13:59.182401 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:13:59.182376    2522 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-221.ec2.internal\" not found"
Sep 18 19:13:59.188208 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:13:59.188140    2522 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-221.ec2.internal.18d6802e8066d4fe\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-221.ec2.internal.18d6802e8066d4fe  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-221.ec2.internal,UID:ip-10-0-131-221.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-221.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-221.ec2.internal,},FirstTimestamp:2026-09-18 19:13:59.137424638 +0000 UTC m=+0.406265881,LastTimestamp:2026-09-18 19:13:59.139391562 +0000 UTC m=+0.408232808,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-221.ec2.internal,}"
Sep 18 19:13:59.197560 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.197538    2522 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fsn2f"
Sep 18 19:13:59.199138 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.199123    2522 container_manager_linux.go:676] "Failed to ensure state" containerName="/system.slice" err="failed to move PID 2555 into the system container \"/system.slice\": <nil>"
Sep 18 19:13:59.204166 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:13:59.204092    2522 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-221.ec2.internal.18d6802e80670947\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-221.ec2.internal.18d6802e80670947  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-221.ec2.internal,UID:ip-10-0-131-221.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-131-221.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-131-221.ec2.internal,},FirstTimestamp:2026-09-18 19:13:59.137438023 +0000 UTC m=+0.406279269,LastTimestamp:2026-09-18 19:13:59.139404822 +0000 UTC m=+0.408246068,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-221.ec2.internal,}"
Sep 18 19:13:59.215326 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.215303    2522 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fsn2f"
Sep 18 19:13:59.216618 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:13:59.216549    2522 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-221.ec2.internal.18d6802e832544e3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-221.ec2.internal,UID:ip-10-0-131-221.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-131-221.ec2.internal,},FirstTimestamp:2026-09-18 19:13:59.183459555 +0000 UTC m=+0.452300789,LastTimestamp:2026-09-18 19:13:59.183459555 +0000 UTC m=+0.452300789,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-221.ec2.internal,}"
Sep 18 19:13:59.282409 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.282383    2522 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:13:59.301046 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.283417    2522 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-221.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:13:59.301046 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.283456    2522 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-221.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:13:59.301046 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.283472    2522 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-221.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:13:59.301046 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.283512    2522 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-221.ec2.internal"
Sep 18 19:13:59.311221 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.311200    2522 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-221.ec2.internal"
Sep 18 19:13:59.311334 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:13:59.311226    2522 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-221.ec2.internal\": node \"ip-10-0-131-221.ec2.internal\" not found"
Sep 18 19:13:59.328805 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.328763    2522 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 18 19:13:59.330082 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.330055    2522 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 18 19:13:59.330175 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.330117    2522 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 18 19:13:59.330175 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.330147    2522 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 18 19:13:59.330293 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:13:59.330263    2522 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 18 19:13:59.332320 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.332301    2522 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:13:59.381989 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:13:59.381957    2522 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-221.ec2.internal\" not found"
Sep 18 19:13:59.431402 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.431326    2522 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-221.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-221.ec2.internal"]
Sep 18 19:13:59.431548 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.431448    2522 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:13:59.433475 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.433459    2522 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-221.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:13:59.433547 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.433489    2522 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-221.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:13:59.433547 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.433499    2522 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-221.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:13:59.436199 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.436187    2522 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:13:59.436325 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.436310    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-221.ec2.internal"
Sep 18 19:13:59.436385 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.436344    2522 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:13:59.436934 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.436920    2522 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-221.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:13:59.437011 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.436942    2522 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-221.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:13:59.437011 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.436957    2522 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-221.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:13:59.437011 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.436983    2522 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-221.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:13:59.437011 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.437000    2522 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-221.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:13:59.437011 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.437009    2522 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-221.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:13:59.439182 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.439168    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-221.ec2.internal"
Sep 18 19:13:59.439270 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.439191    2522 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:13:59.440398 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.440382    2522 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-221.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:13:59.440459 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.440412    2522 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-221.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:13:59.440459 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.440426    2522 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-221.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:13:59.466039 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:13:59.466010    2522 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-221.ec2.internal\" not found" node="ip-10-0-131-221.ec2.internal"
Sep 18 19:13:59.470286 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:13:59.470270    2522 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-221.ec2.internal\" not found" node="ip-10-0-131-221.ec2.internal"
Sep 18 19:13:59.482772 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:13:59.482746    2522 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-221.ec2.internal\" not found"
Sep 18 19:13:59.516110 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.516070    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/31c0bde74e3407e1ddada35fdb2a70e7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-221.ec2.internal\" (UID: \"31c0bde74e3407e1ddada35fdb2a70e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-221.ec2.internal"
Sep 18 19:13:59.516290 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.516112    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/31c0bde74e3407e1ddada35fdb2a70e7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-221.ec2.internal\" (UID: \"31c0bde74e3407e1ddada35fdb2a70e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-221.ec2.internal"
Sep 18 19:13:59.516290 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.516141    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/523614729ae9b688582b641ea2fafde6-config\") pod \"kube-apiserver-proxy-ip-10-0-131-221.ec2.internal\" (UID: \"523614729ae9b688582b641ea2fafde6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-221.ec2.internal"
Sep 18 19:13:59.583362 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:13:59.583272    2522 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-221.ec2.internal\" not found"
Sep 18 19:13:59.617234 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.617192    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/31c0bde74e3407e1ddada35fdb2a70e7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-221.ec2.internal\" (UID: \"31c0bde74e3407e1ddada35fdb2a70e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-221.ec2.internal"
Sep 18 19:13:59.617406 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.617265    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/31c0bde74e3407e1ddada35fdb2a70e7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-221.ec2.internal\" (UID: \"31c0bde74e3407e1ddada35fdb2a70e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-221.ec2.internal"
Sep 18 19:13:59.617406 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.617199    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/31c0bde74e3407e1ddada35fdb2a70e7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-221.ec2.internal\" (UID: \"31c0bde74e3407e1ddada35fdb2a70e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-221.ec2.internal"
Sep 18 19:13:59.617406 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.617294    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/523614729ae9b688582b641ea2fafde6-config\") pod \"kube-apiserver-proxy-ip-10-0-131-221.ec2.internal\" (UID: \"523614729ae9b688582b641ea2fafde6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-221.ec2.internal"
Sep 18 19:13:59.617406 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.617341    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/31c0bde74e3407e1ddada35fdb2a70e7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-221.ec2.internal\" (UID: \"31c0bde74e3407e1ddada35fdb2a70e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-221.ec2.internal"
Sep 18 19:13:59.617406 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.617351    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/523614729ae9b688582b641ea2fafde6-config\") pod \"kube-apiserver-proxy-ip-10-0-131-221.ec2.internal\" (UID: \"523614729ae9b688582b641ea2fafde6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-221.ec2.internal"
Sep 18 19:13:59.683597 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:13:59.683570    2522 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-221.ec2.internal\" not found"
Sep 18 19:13:59.769583 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.769541    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-221.ec2.internal"
Sep 18 19:13:59.773779 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:13:59.773757    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-221.ec2.internal"
Sep 18 19:13:59.784368 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:13:59.784351    2522 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-221.ec2.internal\" not found"
Sep 18 19:13:59.884893 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:13:59.884817    2522 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-221.ec2.internal\" not found"
Sep 18 19:13:59.985425 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:13:59.985395    2522 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-221.ec2.internal\" not found"
Sep 18 19:14:00.046854 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:00.046827    2522 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 18 19:14:00.047292 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:00.046960    2522 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 19:14:00.086135 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:00.086111    2522 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-221.ec2.internal\" not found"
Sep 18 19:14:01.555948 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:01.555894    2522 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:14:01.556843 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:01.556829    2522 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-221.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:14:01.556903 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:01.556857    2522 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-221.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:14:01.556903 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:01.556870    2522 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-221.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:14:01.571818 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:01.571800    2522 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-221.ec2.internal\" not found"
Sep 18 19:14:01.606756 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:01.606731    2522 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-17 19:09:00 +0000 UTC" deadline="2028-04-17 11:37:51.449741945 +0000 UTC"
Sep 18 19:14:01.606756 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:01.606753    2522 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13840h23m49.842991975s"
Sep 18 19:14:01.672919 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:01.672893    2522 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-221.ec2.internal\" not found"
Sep 18 19:14:01.710239 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:01.710221    2522 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:14:01.798781 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:01.798755    2522 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-221.ec2.internal"
Sep 18 19:14:01.804808 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:01.804786    2522 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:14:01.824460 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:01.824409    2522 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 19:14:01.825216 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:01.825203    2522 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-221.ec2.internal"
Sep 18 19:14:01.844315 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:01.844294    2522 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 19:14:01.869939 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:14:01.869912    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31c0bde74e3407e1ddada35fdb2a70e7.slice/crio-2c3e2a798798b126ab56c253da80704f92df72862f7f525fb61339ca39e96790 WatchSource:0}: Error finding container 2c3e2a798798b126ab56c253da80704f92df72862f7f525fb61339ca39e96790: Status 404 returned error can't find the container with id 2c3e2a798798b126ab56c253da80704f92df72862f7f525fb61339ca39e96790
Sep 18 19:14:01.870402 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:14:01.870384    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod523614729ae9b688582b641ea2fafde6.slice/crio-2444e6710fada79f35297c75a6776359456cffc6f5a44b91abdcdced828250ca WatchSource:0}: Error finding container 2444e6710fada79f35297c75a6776359456cffc6f5a44b91abdcdced828250ca: Status 404 returned error can't find the container with id 2444e6710fada79f35297c75a6776359456cffc6f5a44b91abdcdced828250ca
Sep 18 19:14:01.874850 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:01.874833    2522 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:14:01.976483 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:01.976457    2522 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:14:02.475727 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.475698    2522 apiserver.go:52] "Watching apiserver"
Sep 18 19:14:02.483882 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.483861    2522 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 18 19:14:02.485535 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.485506    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-dk2hl","openshift-network-operator/iptables-alerter-844d4","openshift-ovn-kubernetes/ovnkube-node-hbhqv","kube-system/konnectivity-agent-s7ckz","kube-system/kube-apiserver-proxy-ip-10-0-131-221.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cql5","openshift-cluster-node-tuning-operator/tuned-zp9rb","openshift-image-registry/node-ca-hmfqw","openshift-multus/multus-additional-cni-plugins-jrg4l","openshift-multus/network-metrics-daemon-5j97v","kube-system/global-pull-secret-syncer-g679p","openshift-dns/node-resolver-w4tjm","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-221.ec2.internal","openshift-multus/multus-d8dg5"]
Sep 18 19:14:02.487986 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.487967    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zp9rb"
Sep 18 19:14:02.490077 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.490053    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dk2hl"
Sep 18 19:14:02.490182 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:02.490140    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dk2hl" podUID="3f0563df-380a-41eb-bb44-902bf8ceb4f6"
Sep 18 19:14:02.492051 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.492026    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:14:02.492051 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.492026    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 18 19:14:02.492210 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.492174    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-d8dg5"
Sep 18 19:14:02.492339 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.492322    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-l5nsb\""
Sep 18 19:14:02.494345 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.494327    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-s7ckz"
Sep 18 19:14:02.496478 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.496460    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hmfqw"
Sep 18 19:14:02.497751 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.497735    2522 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 18 19:14:02.497900 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.497880    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 18 19:14:02.498002 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.497900    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 18 19:14:02.498186 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.498168    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-mdzzp\""
Sep 18 19:14:02.498770 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.498752    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5j97v"
Sep 18 19:14:02.498861 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:02.498819    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5j97v" podUID="f516e39e-b125-4617-94ce-7f0dccfae7bb"
Sep 18 19:14:02.501599 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.501298    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-844d4"
Sep 18 19:14:02.504312 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.503951    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jrg4l"
Sep 18 19:14:02.506736 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.506717    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cql5"
Sep 18 19:14:02.509100 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.509078    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7-host-run-k8s-cni-cncf-io\") pod \"multus-d8dg5\" (UID: \"f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7\") " pod="openshift-multus/multus-d8dg5"
Sep 18 19:14:02.509193 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.509113    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/72118d02-4cb7-4dd4-bec1-1de6ba1614b1-etc-modprobe-d\") pod \"tuned-zp9rb\" (UID: \"72118d02-4cb7-4dd4-bec1-1de6ba1614b1\") " pod="openshift-cluster-node-tuning-operator/tuned-zp9rb"
Sep 18 19:14:02.509193 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.509136    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/72118d02-4cb7-4dd4-bec1-1de6ba1614b1-var-lib-kubelet\") pod \"tuned-zp9rb\" (UID: \"72118d02-4cb7-4dd4-bec1-1de6ba1614b1\") " pod="openshift-cluster-node-tuning-operator/tuned-zp9rb"
Sep 18 19:14:02.509193 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.509158    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7-system-cni-dir\") pod \"multus-d8dg5\" (UID: \"f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7\") " pod="openshift-multus/multus-d8dg5"
Sep 18 19:14:02.509193 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.509141    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g679p"
Sep 18 19:14:02.509353 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.509210    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhf56\" (UniqueName: \"kubernetes.io/projected/f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7-kube-api-access-nhf56\") pod \"multus-d8dg5\" (UID: \"f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7\") " pod="openshift-multus/multus-d8dg5"
Sep 18 19:14:02.509353 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:02.509228    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g679p" podUID="24ec319c-fec4-4b8b-bcb6-6331ec8574fa"
Sep 18 19:14:02.509353 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.509248    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/72118d02-4cb7-4dd4-bec1-1de6ba1614b1-lib-modules\") pod \"tuned-zp9rb\" (UID: \"72118d02-4cb7-4dd4-bec1-1de6ba1614b1\") " pod="openshift-cluster-node-tuning-operator/tuned-zp9rb"
Sep 18 19:14:02.509353 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.509307    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/72118d02-4cb7-4dd4-bec1-1de6ba1614b1-host\") pod \"tuned-zp9rb\" (UID: \"72118d02-4cb7-4dd4-bec1-1de6ba1614b1\") " pod="openshift-cluster-node-tuning-operator/tuned-zp9rb"
Sep 18 19:14:02.509353 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.509326    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e31f28a0-45fc-45ae-a307-578e281565a9-konnectivity-ca\") pod \"konnectivity-agent-s7ckz\" (UID: \"e31f28a0-45fc-45ae-a307-578e281565a9\") " pod="kube-system/konnectivity-agent-s7ckz"
Sep 18 19:14:02.509353 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.509348    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7-host-var-lib-cni-bin\") pod \"multus-d8dg5\" (UID: \"f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7\") " pod="openshift-multus/multus-d8dg5"
Sep 18 19:14:02.509662 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.509371    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7-host-run-netns\") pod \"multus-d8dg5\" (UID: \"f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7\") " pod="openshift-multus/multus-d8dg5"
Sep 18 19:14:02.509662 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.509393    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/72118d02-4cb7-4dd4-bec1-1de6ba1614b1-run\") pod \"tuned-zp9rb\" (UID: \"72118d02-4cb7-4dd4-bec1-1de6ba1614b1\") " pod="openshift-cluster-node-tuning-operator/tuned-zp9rb"
Sep 18 19:14:02.509662 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.509432    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/72118d02-4cb7-4dd4-bec1-1de6ba1614b1-etc-sysconfig\") pod \"tuned-zp9rb\" (UID: \"72118d02-4cb7-4dd4-bec1-1de6ba1614b1\") " pod="openshift-cluster-node-tuning-operator/tuned-zp9rb"
Sep 18 19:14:02.509662 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.509473    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bfd5ec3a-8ede-49e3-b8e2-00cc159b65aa-iptables-alerter-script\") pod \"iptables-alerter-844d4\" (UID: \"bfd5ec3a-8ede-49e3-b8e2-00cc159b65aa\") " pod="openshift-network-operator/iptables-alerter-844d4"
Sep 18 19:14:02.509662 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.509507    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7-multus-cni-dir\") pod \"multus-d8dg5\" (UID: \"f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7\") " pod="openshift-multus/multus-d8dg5"
Sep 18 19:14:02.509662 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.509533    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7-cnibin\") pod \"multus-d8dg5\" (UID: \"f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7\") " pod="openshift-multus/multus-d8dg5"
Sep 18 19:14:02.509662 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.509556    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7-multus-conf-dir\") pod \"multus-d8dg5\" (UID: \"f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7\") " pod="openshift-multus/multus-d8dg5"
Sep 18 19:14:02.509662 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.509606    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7-host-run-multus-certs\") pod \"multus-d8dg5\" (UID: \"f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7\") " pod="openshift-multus/multus-d8dg5"
Sep 18 19:14:02.509662 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.509638    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72118d02-4cb7-4dd4-bec1-1de6ba1614b1-etc-kubernetes\") pod \"tuned-zp9rb\" (UID: \"72118d02-4cb7-4dd4-bec1-1de6ba1614b1\") " pod="openshift-cluster-node-tuning-operator/tuned-zp9rb"
Sep 18 19:14:02.509662 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.509660    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/72118d02-4cb7-4dd4-bec1-1de6ba1614b1-etc-systemd\") pod \"tuned-zp9rb\" (UID: \"72118d02-4cb7-4dd4-bec1-1de6ba1614b1\") " pod="openshift-cluster-node-tuning-operator/tuned-zp9rb"
Sep 18 19:14:02.510106 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.509682    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7f72\" (UniqueName: \"kubernetes.io/projected/3f0563df-380a-41eb-bb44-902bf8ceb4f6-kube-api-access-t7f72\") pod \"network-check-target-dk2hl\" (UID: \"3f0563df-380a-41eb-bb44-902bf8ceb4f6\") " pod="openshift-network-diagnostics/network-check-target-dk2hl"
Sep 18 19:14:02.510106 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.509705    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7-hostroot\") pod \"multus-d8dg5\" (UID: \"f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7\") " pod="openshift-multus/multus-d8dg5"
Sep 18 19:14:02.510106 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.509727    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/438a5295-6f51-49ed-99d1-96a8fe037805-serviceca\") pod \"node-ca-hmfqw\" (UID: \"438a5295-6f51-49ed-99d1-96a8fe037805\") " pod="openshift-image-registry/node-ca-hmfqw"
Sep 18 19:14:02.510106 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.509749    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bfd5ec3a-8ede-49e3-b8e2-00cc159b65aa-host-slash\") pod \"iptables-alerter-844d4\" (UID: \"bfd5ec3a-8ede-49e3-b8e2-00cc159b65aa\") " pod="openshift-network-operator/iptables-alerter-844d4"
Sep 18 19:14:02.510106 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.509771    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/72118d02-4cb7-4dd4-bec1-1de6ba1614b1-sys\") pod \"tuned-zp9rb\" (UID: \"72118d02-4cb7-4dd4-bec1-1de6ba1614b1\") " pod="openshift-cluster-node-tuning-operator/tuned-zp9rb"
Sep 18 19:14:02.510106 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.509824    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/72118d02-4cb7-4dd4-bec1-1de6ba1614b1-etc-tuned\") pod \"tuned-zp9rb\" (UID: \"72118d02-4cb7-4dd4-bec1-1de6ba1614b1\") " pod="openshift-cluster-node-tuning-operator/tuned-zp9rb"
Sep 18 19:14:02.510106 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.509894    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e31f28a0-45fc-45ae-a307-578e281565a9-agent-certs\") pod \"konnectivity-agent-s7ckz\" (UID: \"e31f28a0-45fc-45ae-a307-578e281565a9\") " pod="kube-system/konnectivity-agent-s7ckz"
Sep 18 19:14:02.510106 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.509962    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7-host-var-lib-cni-multus\") pod \"multus-d8dg5\" (UID: \"f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7\") " pod="openshift-multus/multus-d8dg5"
Sep 18 19:14:02.510106 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.510000    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7-host-var-lib-kubelet\") pod \"multus-d8dg5\" (UID: \"f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7\") " pod="openshift-multus/multus-d8dg5"
Sep 18 19:14:02.510106 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.510049    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/438a5295-6f51-49ed-99d1-96a8fe037805-host\") pod \"node-ca-hmfqw\" (UID: \"438a5295-6f51-49ed-99d1-96a8fe037805\") " pod="openshift-image-registry/node-ca-hmfqw"
Sep 18 19:14:02.510106 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.510080    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rddgw\" (UniqueName: \"kubernetes.io/projected/438a5295-6f51-49ed-99d1-96a8fe037805-kube-api-access-rddgw\") pod \"node-ca-hmfqw\" (UID: \"438a5295-6f51-49ed-99d1-96a8fe037805\") " pod="openshift-image-registry/node-ca-hmfqw"
Sep 18 19:14:02.510106 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.510108    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f516e39e-b125-4617-94ce-7f0dccfae7bb-metrics-certs\") pod \"network-metrics-daemon-5j97v\" (UID: \"f516e39e-b125-4617-94ce-7f0dccfae7bb\") " pod="openshift-multus/network-metrics-daemon-5j97v"
Sep 18 19:14:02.510672 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.510135    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps7b2\" (UniqueName: \"kubernetes.io/projected/bfd5ec3a-8ede-49e3-b8e2-00cc159b65aa-kube-api-access-ps7b2\") pod \"iptables-alerter-844d4\" (UID: \"bfd5ec3a-8ede-49e3-b8e2-00cc159b65aa\") " pod="openshift-network-operator/iptables-alerter-844d4"
Sep 18 19:14:02.510672 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.510156    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/72118d02-4cb7-4dd4-bec1-1de6ba1614b1-tmp\") pod \"tuned-zp9rb\" (UID: \"72118d02-4cb7-4dd4-bec1-1de6ba1614b1\") " pod="openshift-cluster-node-tuning-operator/tuned-zp9rb"
Sep 18 19:14:02.510672 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.510186    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7-multus-daemon-config\") pod \"multus-d8dg5\" (UID: \"f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7\") " pod="openshift-multus/multus-d8dg5"
Sep 18 19:14:02.510672 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.510209    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7-etc-kubernetes\") pod \"multus-d8dg5\" (UID: \"f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7\") " pod="openshift-multus/multus-d8dg5"
Sep 18 19:14:02.510672 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.510243    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94lwf\" (UniqueName: \"kubernetes.io/projected/f516e39e-b125-4617-94ce-7f0dccfae7bb-kube-api-access-94lwf\") pod \"network-metrics-daemon-5j97v\" (UID: \"f516e39e-b125-4617-94ce-7f0dccfae7bb\") " pod="openshift-multus/network-metrics-daemon-5j97v"
Sep 18 19:14:02.510672 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.510270    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/72118d02-4cb7-4dd4-bec1-1de6ba1614b1-etc-sysctl-d\") pod \"tuned-zp9rb\" (UID: \"72118d02-4cb7-4dd4-bec1-1de6ba1614b1\") " pod="openshift-cluster-node-tuning-operator/tuned-zp9rb"
Sep 18 19:14:02.510672 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.510293    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/72118d02-4cb7-4dd4-bec1-1de6ba1614b1-etc-sysctl-conf\") pod \"tuned-zp9rb\" (UID: \"72118d02-4cb7-4dd4-bec1-1de6ba1614b1\") " pod="openshift-cluster-node-tuning-operator/tuned-zp9rb"
Sep 18 19:14:02.510672 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.510323    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmlh2\" (UniqueName: \"kubernetes.io/projected/72118d02-4cb7-4dd4-bec1-1de6ba1614b1-kube-api-access-qmlh2\") pod \"tuned-zp9rb\" (UID: \"72118d02-4cb7-4dd4-bec1-1de6ba1614b1\") " pod="openshift-cluster-node-tuning-operator/tuned-zp9rb"
Sep 18 19:14:02.510672 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.510358    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7-cni-binary-copy\") pod \"multus-d8dg5\" (UID: \"f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7\") " pod="openshift-multus/multus-d8dg5"
Sep 18 19:14:02.510672 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.510395    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7-multus-socket-dir-parent\") pod \"multus-d8dg5\" (UID: \"f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7\") " pod="openshift-multus/multus-d8dg5"
Sep 18 19:14:02.512652 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.512634    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-w4tjm"
Sep 18 19:14:02.515113 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.515088    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:02.523917 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.523896    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 18 19:14:02.524234 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.524211    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 18 19:14:02.524536 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.524521    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 18 19:14:02.524918 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.524687    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 18 19:14:02.524918 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.524735    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 18 19:14:02.525027 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.524925    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 18 19:14:02.525027 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.524969    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:14:02.525342 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.525312    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 18 19:14:02.525480 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.525464    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 18 19:14:02.525549 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.525507    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 18 19:14:02.525723 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.525708    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 18 19:14:02.531903 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.531879    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 18 19:14:02.531989 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.531930    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 18 19:14:02.532185 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.532168    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 18 19:14:02.532367 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.532351    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 18 19:14:02.532426 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.532387    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 18 19:14:02.532652 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.532634    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 18 19:14:02.533402 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.533385    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 18 19:14:02.533486 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.533401    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 18 19:14:02.533486 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.533449    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 18 19:14:02.533608 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.533547    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 18 19:14:02.533664 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.533656    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-w8tt6\""
Sep 18 19:14:02.533719 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.533454    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8s6sd\""
Sep 18 19:14:02.533719 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.533704    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-b7s89\""
Sep 18 19:14:02.533813 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.533771    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9prhf\""
Sep 18 19:14:02.533863 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.533814    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 18 19:14:02.533946 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.533924    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kfc8q\""
Sep 18 19:14:02.533946 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.533945    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fm96q\""
Sep 18 19:14:02.534050 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.533952    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-qmdbr\""
Sep 18 19:14:02.534050 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.534023    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 18 19:14:02.549154 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.549134    2522 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 19:14:02.599199 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.599182    2522 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 18 19:14:02.611122 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.611100    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/72118d02-4cb7-4dd4-bec1-1de6ba1614b1-etc-sysconfig\") pod \"tuned-zp9rb\" (UID: \"72118d02-4cb7-4dd4-bec1-1de6ba1614b1\") " pod="openshift-cluster-node-tuning-operator/tuned-zp9rb"
Sep 18 19:14:02.611236 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.611128    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e8cd8a19-7cbb-4585-b5b3-6d940ebe41c6-etc-selinux\") pod \"aws-ebs-csi-driver-node-2cql5\" (UID: \"e8cd8a19-7cbb-4585-b5b3-6d940ebe41c6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cql5"
Sep 18 19:14:02.611236 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.611143    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e8cd8a19-7cbb-4585-b5b3-6d940ebe41c6-sys-fs\") pod \"aws-ebs-csi-driver-node-2cql5\" (UID: \"e8cd8a19-7cbb-4585-b5b3-6d940ebe41c6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cql5"
Sep 18 19:14:02.611236 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.611158    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e31f28a0-45fc-45ae-a307-578e281565a9-agent-certs\") pod \"konnectivity-agent-s7ckz\" (UID: \"e31f28a0-45fc-45ae-a307-578e281565a9\") " pod="kube-system/konnectivity-agent-s7ckz"
Sep 18 19:14:02.611236 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.611173    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5dhq\" (UniqueName: \"kubernetes.io/projected/5b796f3a-2d43-47a8-8d73-a34a2af2d62b-kube-api-access-j5dhq\") pod \"node-resolver-w4tjm\" (UID: \"5b796f3a-2d43-47a8-8d73-a34a2af2d62b\") " pod="openshift-dns/node-resolver-w4tjm"
Sep 18 19:14:02.611236 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.611198    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7-cnibin\") pod \"multus-d8dg5\" (UID: \"f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7\") " pod="openshift-multus/multus-d8dg5"
Sep 18 19:14:02.611236 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.611220    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7-host-run-multus-certs\") pod \"multus-d8dg5\" (UID: \"f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7\") " pod="openshift-multus/multus-d8dg5"
Sep 18 19:14:02.611464 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.611244    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/72118d02-4cb7-4dd4-bec1-1de6ba1614b1-etc-sysconfig\") pod \"tuned-zp9rb\" (UID: \"72118d02-4cb7-4dd4-bec1-1de6ba1614b1\") " pod="openshift-cluster-node-tuning-operator/tuned-zp9rb"
Sep 18 19:14:02.611464 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.611243    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72118d02-4cb7-4dd4-bec1-1de6ba1614b1-etc-kubernetes\") pod \"tuned-zp9rb\" (UID: \"72118d02-4cb7-4dd4-bec1-1de6ba1614b1\") " pod="openshift-cluster-node-tuning-operator/tuned-zp9rb"
Sep 18 19:14:02.611464 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.611272    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7-cnibin\") pod \"multus-d8dg5\" (UID: \"f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7\") " pod="openshift-multus/multus-d8dg5"
Sep 18 19:14:02.611464 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.611286    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eda05f97-2fcc-4f43-af17-9f5d514486bf-etc-openvswitch\") pod \"ovnkube-node-hbhqv\" (UID: \"eda05f97-2fcc-4f43-af17-9f5d514486bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:02.611464 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.611308    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/72118d02-4cb7-4dd4-bec1-1de6ba1614b1-sys\") pod \"tuned-zp9rb\" (UID: \"72118d02-4cb7-4dd4-bec1-1de6ba1614b1\") " pod="openshift-cluster-node-tuning-operator/tuned-zp9rb"
Sep 18 19:14:02.611464 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.611328    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eda05f97-2fcc-4f43-af17-9f5d514486bf-ovnkube-script-lib\") pod \"ovnkube-node-hbhqv\" (UID: \"eda05f97-2fcc-4f43-af17-9f5d514486bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:02.611464 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.611348    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72118d02-4cb7-4dd4-bec1-1de6ba1614b1-etc-kubernetes\") pod \"tuned-zp9rb\" (UID: \"72118d02-4cb7-4dd4-bec1-1de6ba1614b1\") " pod="openshift-cluster-node-tuning-operator/tuned-zp9rb"
Sep 18 19:14:02.611464 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.611354    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e8cd8a19-7cbb-4585-b5b3-6d940ebe41c6-registration-dir\") pod \"aws-ebs-csi-driver-node-2cql5\" (UID: \"e8cd8a19-7cbb-4585-b5b3-6d940ebe41c6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cql5"
Sep 18 19:14:02.611464 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.611379    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7-host-var-lib-kubelet\") pod \"multus-d8dg5\" (UID: \"f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7\") " pod="openshift-multus/multus-d8dg5"
Sep 18 19:14:02.611464 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.611394    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7-host-run-multus-certs\") pod \"multus-d8dg5\" (UID: \"f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7\") " pod="openshift-multus/multus-d8dg5"
Sep 18 19:14:02.611464 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.611405    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rddgw\" (UniqueName: \"kubernetes.io/projected/438a5295-6f51-49ed-99d1-96a8fe037805-kube-api-access-rddgw\") pod \"node-ca-hmfqw\" (UID: \"438a5295-6f51-49ed-99d1-96a8fe037805\") " pod="openshift-image-registry/node-ca-hmfqw"
Sep 18 19:14:02.611464 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.611428    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eda05f97-2fcc-4f43-af17-9f5d514486bf-host-slash\") pod \"ovnkube-node-hbhqv\" (UID: \"eda05f97-2fcc-4f43-af17-9f5d514486bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:02.611464 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.611454    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/72118d02-4cb7-4dd4-bec1-1de6ba1614b1-sys\") pod \"tuned-zp9rb\" (UID: \"72118d02-4cb7-4dd4-bec1-1de6ba1614b1\") " pod="openshift-cluster-node-tuning-operator/tuned-zp9rb"
Sep 18 19:14:02.611464 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.611453    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rkvw\" (UniqueName: \"kubernetes.io/projected/eda05f97-2fcc-4f43-af17-9f5d514486bf-kube-api-access-9rkvw\") pod \"ovnkube-node-hbhqv\" (UID: \"eda05f97-2fcc-4f43-af17-9f5d514486bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:02.612083 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.611493    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bw4n\" (UniqueName: \"kubernetes.io/projected/e8cd8a19-7cbb-4585-b5b3-6d940ebe41c6-kube-api-access-9bw4n\") pod \"aws-ebs-csi-driver-node-2cql5\" (UID: \"e8cd8a19-7cbb-4585-b5b3-6d940ebe41c6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cql5"
Sep 18 19:14:02.612083 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.611519    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7-host-var-lib-kubelet\") pod \"multus-d8dg5\" (UID: \"f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7\") " pod="openshift-multus/multus-d8dg5"
Sep 18 19:14:02.612083 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.611578    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7-multus-daemon-config\") pod \"multus-d8dg5\" (UID: \"f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7\") " pod="openshift-multus/multus-d8dg5"
Sep 18 19:14:02.612083 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.611604    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/72118d02-4cb7-4dd4-bec1-1de6ba1614b1-etc-sysctl-conf\") pod \"tuned-zp9rb\" (UID: \"72118d02-4cb7-4dd4-bec1-1de6ba1614b1\") " pod="openshift-cluster-node-tuning-operator/tuned-zp9rb"
Sep 18 19:14:02.612083 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.611622    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eda05f97-2fcc-4f43-af17-9f5d514486bf-ovn-node-metrics-cert\") pod \"ovnkube-node-hbhqv\" (UID: \"eda05f97-2fcc-4f43-af17-9f5d514486bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:02.612083 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.611673    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e8cd8a19-7cbb-4585-b5b3-6d940ebe41c6-device-dir\") pod \"aws-ebs-csi-driver-node-2cql5\" (UID: \"e8cd8a19-7cbb-4585-b5b3-6d940ebe41c6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cql5"
Sep 18 19:14:02.612083 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.611696    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7-multus-socket-dir-parent\") pod \"multus-d8dg5\" (UID: \"f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7\") " pod="openshift-multus/multus-d8dg5"
Sep 18 19:14:02.612083 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.611716    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/72118d02-4cb7-4dd4-bec1-1de6ba1614b1-var-lib-kubelet\") pod \"tuned-zp9rb\" (UID: \"72118d02-4cb7-4dd4-bec1-1de6ba1614b1\") " pod="openshift-cluster-node-tuning-operator/tuned-zp9rb"
Sep 18 19:14:02.612083 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.611737    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eda05f97-2fcc-4f43-af17-9f5d514486bf-systemd-units\") pod \"ovnkube-node-hbhqv\" (UID: \"eda05f97-2fcc-4f43-af17-9f5d514486bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:02.612083 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.611760    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eda05f97-2fcc-4f43-af17-9f5d514486bf-host-run-ovn-kubernetes\") pod \"ovnkube-node-hbhqv\" (UID: \"eda05f97-2fcc-4f43-af17-9f5d514486bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:02.612083 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.611782    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/72118d02-4cb7-4dd4-bec1-1de6ba1614b1-etc-sysctl-conf\") pod \"tuned-zp9rb\" (UID: \"72118d02-4cb7-4dd4-bec1-1de6ba1614b1\") " pod="openshift-cluster-node-tuning-operator/tuned-zp9rb"
Sep 18 19:14:02.612083 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.611797    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7-multus-socket-dir-parent\") pod \"multus-d8dg5\" (UID: \"f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7\") " pod="openshift-multus/multus-d8dg5"
Sep 18 19:14:02.612083 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.611807    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7-system-cni-dir\") pod \"multus-d8dg5\" (UID: \"f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7\") " pod="openshift-multus/multus-d8dg5"
Sep 18 19:14:02.612083 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.611831    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/72118d02-4cb7-4dd4-bec1-1de6ba1614b1-var-lib-kubelet\") pod \"tuned-zp9rb\" (UID: \"72118d02-4cb7-4dd4-bec1-1de6ba1614b1\") " pod="openshift-cluster-node-tuning-operator/tuned-zp9rb"
Sep 18 19:14:02.612083 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.611839    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhf56\" (UniqueName: \"kubernetes.io/projected/f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7-kube-api-access-nhf56\") pod \"multus-d8dg5\" (UID: \"f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7\") " pod="openshift-multus/multus-d8dg5"
Sep 18 19:14:02.612083 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.611839    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7-system-cni-dir\") pod \"multus-d8dg5\" (UID: \"f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7\") " pod="openshift-multus/multus-d8dg5"
Sep 18 19:14:02.612083 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.611883    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/72118d02-4cb7-4dd4-bec1-1de6ba1614b1-lib-modules\") pod \"tuned-zp9rb\" (UID: \"72118d02-4cb7-4dd4-bec1-1de6ba1614b1\") " pod="openshift-cluster-node-tuning-operator/tuned-zp9rb"
Sep 18 19:14:02.612918 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.611884    2522 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 18 19:14:02.612918 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.611927    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/72118d02-4cb7-4dd4-bec1-1de6ba1614b1-host\") pod \"tuned-zp9rb\" (UID: \"72118d02-4cb7-4dd4-bec1-1de6ba1614b1\") " pod="openshift-cluster-node-tuning-operator/tuned-zp9rb"
Sep 18 19:14:02.612918 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.611957    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eda05f97-2fcc-4f43-af17-9f5d514486bf-log-socket\") pod \"ovnkube-node-hbhqv\" (UID: \"eda05f97-2fcc-4f43-af17-9f5d514486bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:02.612918 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.611985    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/72118d02-4cb7-4dd4-bec1-1de6ba1614b1-run\") pod \"tuned-zp9rb\" (UID: \"72118d02-4cb7-4dd4-bec1-1de6ba1614b1\") " pod="openshift-cluster-node-tuning-operator/tuned-zp9rb"
Sep 18 19:14:02.612918 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.612010    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eda05f97-2fcc-4f43-af17-9f5d514486bf-env-overrides\") pod \"ovnkube-node-hbhqv\" (UID: \"eda05f97-2fcc-4f43-af17-9f5d514486bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:02.612918 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.612034    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6c2a6c5f-990b-4ceb-9308-f7a6e29e22a2-system-cni-dir\") pod \"multus-additional-cni-plugins-jrg4l\" (UID: \"6c2a6c5f-990b-4ceb-9308-f7a6e29e22a2\") " pod="openshift-multus/multus-additional-cni-plugins-jrg4l"
Sep 18 19:14:02.612918 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.612055    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/72118d02-4cb7-4dd4-bec1-1de6ba1614b1-lib-modules\") pod \"tuned-zp9rb\" (UID: \"72118d02-4cb7-4dd4-bec1-1de6ba1614b1\") " pod="openshift-cluster-node-tuning-operator/tuned-zp9rb"
Sep 18 19:14:02.612918 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.612060    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc5xq\" (UniqueName: \"kubernetes.io/projected/6c2a6c5f-990b-4ceb-9308-f7a6e29e22a2-kube-api-access-lc5xq\") pod \"multus-additional-cni-plugins-jrg4l\" (UID: \"6c2a6c5f-990b-4ceb-9308-f7a6e29e22a2\") " pod="openshift-multus/multus-additional-cni-plugins-jrg4l"
Sep 18 19:14:02.612918 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.612071    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/72118d02-4cb7-4dd4-bec1-1de6ba1614b1-host\") pod \"tuned-zp9rb\" (UID: \"72118d02-4cb7-4dd4-bec1-1de6ba1614b1\") " pod="openshift-cluster-node-tuning-operator/tuned-zp9rb"
Sep 18 19:14:02.612918 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.612084    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eda05f97-2fcc-4f43-af17-9f5d514486bf-host-run-netns\") pod \"ovnkube-node-hbhqv\" (UID: \"eda05f97-2fcc-4f43-af17-9f5d514486bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:02.612918 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.612101    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/72118d02-4cb7-4dd4-bec1-1de6ba1614b1-run\") pod \"tuned-zp9rb\" (UID: \"72118d02-4cb7-4dd4-bec1-1de6ba1614b1\") " pod="openshift-cluster-node-tuning-operator/tuned-zp9rb"
Sep 18 19:14:02.612918 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.612127    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7-multus-daemon-config\") pod \"multus-d8dg5\" (UID: \"f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7\") " pod="openshift-multus/multus-d8dg5"
Sep 18 19:14:02.612918 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.612130    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eda05f97-2fcc-4f43-af17-9f5d514486bf-run-openvswitch\") pod \"ovnkube-node-hbhqv\" (UID: \"eda05f97-2fcc-4f43-af17-9f5d514486bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:02.612918 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.612618    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eda05f97-2fcc-4f43-af17-9f5d514486bf-run-ovn\") pod \"ovnkube-node-hbhqv\" (UID: \"eda05f97-2fcc-4f43-af17-9f5d514486bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:02.612918 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.612656    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eda05f97-2fcc-4f43-af17-9f5d514486bf-node-log\") pod \"ovnkube-node-hbhqv\" (UID: \"eda05f97-2fcc-4f43-af17-9f5d514486bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:02.612918 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.612683    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6c2a6c5f-990b-4ceb-9308-f7a6e29e22a2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jrg4l\" (UID: \"6c2a6c5f-990b-4ceb-9308-f7a6e29e22a2\") " pod="openshift-multus/multus-additional-cni-plugins-jrg4l"
Sep 18 19:14:02.612918 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.612713    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e8cd8a19-7cbb-4585-b5b3-6d940ebe41c6-socket-dir\") pod \"aws-ebs-csi-driver-node-2cql5\" (UID: \"e8cd8a19-7cbb-4585-b5b3-6d940ebe41c6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cql5"
Sep 18 19:14:02.613732 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.612740    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/72118d02-4cb7-4dd4-bec1-1de6ba1614b1-etc-tuned\") pod \"tuned-zp9rb\" (UID: \"72118d02-4cb7-4dd4-bec1-1de6ba1614b1\") " pod="openshift-cluster-node-tuning-operator/tuned-zp9rb"
Sep 18 19:14:02.613732 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.612774    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/24ec319c-fec4-4b8b-bcb6-6331ec8574fa-dbus\") pod \"global-pull-secret-syncer-g679p\" (UID: \"24ec319c-fec4-4b8b-bcb6-6331ec8574fa\") " pod="kube-system/global-pull-secret-syncer-g679p"
Sep 18 19:14:02.613732 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.612804    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eda05f97-2fcc-4f43-af17-9f5d514486bf-var-lib-openvswitch\") pod \"ovnkube-node-hbhqv\" (UID: \"eda05f97-2fcc-4f43-af17-9f5d514486bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:02.613732 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.612835    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bfd5ec3a-8ede-49e3-b8e2-00cc159b65aa-iptables-alerter-script\") pod \"iptables-alerter-844d4\" (UID: \"bfd5ec3a-8ede-49e3-b8e2-00cc159b65aa\") " pod="openshift-network-operator/iptables-alerter-844d4"
Sep 18 19:14:02.613732 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.612860    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7-multus-cni-dir\") pod \"multus-d8dg5\" (UID: \"f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7\") " pod="openshift-multus/multus-d8dg5"
Sep 18 19:14:02.613732 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.612904    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7-multus-conf-dir\") pod \"multus-d8dg5\" (UID: \"f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7\") " pod="openshift-multus/multus-d8dg5"
Sep 18 19:14:02.613732 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.612926    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/72118d02-4cb7-4dd4-bec1-1de6ba1614b1-etc-systemd\") pod \"tuned-zp9rb\" (UID: \"72118d02-4cb7-4dd4-bec1-1de6ba1614b1\") " pod="openshift-cluster-node-tuning-operator/tuned-zp9rb"
Sep 18 19:14:02.613732 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.612956    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eda05f97-2fcc-4f43-af17-9f5d514486bf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hbhqv\" (UID: \"eda05f97-2fcc-4f43-af17-9f5d514486bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:02.613732 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.612982    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7f72\" (UniqueName: \"kubernetes.io/projected/3f0563df-380a-41eb-bb44-902bf8ceb4f6-kube-api-access-t7f72\") pod \"network-check-target-dk2hl\" (UID: \"3f0563df-380a-41eb-bb44-902bf8ceb4f6\") " pod="openshift-network-diagnostics/network-check-target-dk2hl"
Sep 18 19:14:02.613732 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.613007    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7-hostroot\") pod \"multus-d8dg5\" (UID: \"f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7\") " pod="openshift-multus/multus-d8dg5"
Sep 18 19:14:02.613732 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.613029    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/438a5295-6f51-49ed-99d1-96a8fe037805-serviceca\") pod \"node-ca-hmfqw\" (UID: \"438a5295-6f51-49ed-99d1-96a8fe037805\") " pod="openshift-image-registry/node-ca-hmfqw"
Sep 18 19:14:02.613732 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.613052    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bfd5ec3a-8ede-49e3-b8e2-00cc159b65aa-host-slash\") pod \"iptables-alerter-844d4\" (UID: \"bfd5ec3a-8ede-49e3-b8e2-00cc159b65aa\") " pod="openshift-network-operator/iptables-alerter-844d4"
Sep 18 19:14:02.613732 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.613090    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7-multus-conf-dir\") pod \"multus-d8dg5\" (UID: \"f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7\") " pod="openshift-multus/multus-d8dg5"
Sep 18 19:14:02.613732 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.613158    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7-multus-cni-dir\") pod \"multus-d8dg5\" (UID: \"f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7\") " pod="openshift-multus/multus-d8dg5"
Sep 18 19:14:02.613732 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.613487    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/438a5295-6f51-49ed-99d1-96a8fe037805-serviceca\") pod \"node-ca-hmfqw\" (UID: \"438a5295-6f51-49ed-99d1-96a8fe037805\") " pod="openshift-image-registry/node-ca-hmfqw"
Sep 18 19:14:02.613732 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.613555    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/72118d02-4cb7-4dd4-bec1-1de6ba1614b1-etc-systemd\") pod \"tuned-zp9rb\" (UID: \"72118d02-4cb7-4dd4-bec1-1de6ba1614b1\") " pod="openshift-cluster-node-tuning-operator/tuned-zp9rb"
Sep 18 19:14:02.613732 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.613606    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b796f3a-2d43-47a8-8d73-a34a2af2d62b-tmp-dir\") pod \"node-resolver-w4tjm\" (UID: \"5b796f3a-2d43-47a8-8d73-a34a2af2d62b\") " pod="openshift-dns/node-resolver-w4tjm"
Sep 18 19:14:02.614585 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.613630    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eda05f97-2fcc-4f43-af17-9f5d514486bf-run-systemd\") pod \"ovnkube-node-hbhqv\" (UID: \"eda05f97-2fcc-4f43-af17-9f5d514486bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:02.614585 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.613654    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eda05f97-2fcc-4f43-af17-9f5d514486bf-host-cni-bin\") pod \"ovnkube-node-hbhqv\" (UID: \"eda05f97-2fcc-4f43-af17-9f5d514486bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:02.614585 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.613677    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6c2a6c5f-990b-4ceb-9308-f7a6e29e22a2-cnibin\") pod \"multus-additional-cni-plugins-jrg4l\" (UID: \"6c2a6c5f-990b-4ceb-9308-f7a6e29e22a2\") " pod="openshift-multus/multus-additional-cni-plugins-jrg4l"
Sep 18 19:14:02.614585 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.613704    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7-host-var-lib-cni-multus\") pod \"multus-d8dg5\" (UID: \"f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7\") " pod="openshift-multus/multus-d8dg5"
Sep 18 19:14:02.614585 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.613704    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bfd5ec3a-8ede-49e3-b8e2-00cc159b65aa-iptables-alerter-script\") pod \"iptables-alerter-844d4\" (UID: \"bfd5ec3a-8ede-49e3-b8e2-00cc159b65aa\") " pod="openshift-network-operator/iptables-alerter-844d4"
Sep 18 19:14:02.614585 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.613765    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7-host-var-lib-cni-multus\") pod \"multus-d8dg5\" (UID: \"f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7\") " pod="openshift-multus/multus-d8dg5"
Sep 18 19:14:02.614585 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.613811    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bfd5ec3a-8ede-49e3-b8e2-00cc159b65aa-host-slash\") pod \"iptables-alerter-844d4\" (UID: \"bfd5ec3a-8ede-49e3-b8e2-00cc159b65aa\") " pod="openshift-network-operator/iptables-alerter-844d4"
Sep 18 19:14:02.614585 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.613845    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7-hostroot\") pod \"multus-d8dg5\" (UID: \"f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7\") " pod="openshift-multus/multus-d8dg5"
Sep 18 19:14:02.614585 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.613877    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/438a5295-6f51-49ed-99d1-96a8fe037805-host\") pod \"node-ca-hmfqw\" (UID: \"438a5295-6f51-49ed-99d1-96a8fe037805\") " pod="openshift-image-registry/node-ca-hmfqw"
Sep 18 19:14:02.614585 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.613903    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f516e39e-b125-4617-94ce-7f0dccfae7bb-metrics-certs\") pod \"network-metrics-daemon-5j97v\" (UID: \"f516e39e-b125-4617-94ce-7f0dccfae7bb\") " pod="openshift-multus/network-metrics-daemon-5j97v"
Sep 18 19:14:02.614585 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.613938    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps7b2\" (UniqueName: \"kubernetes.io/projected/bfd5ec3a-8ede-49e3-b8e2-00cc159b65aa-kube-api-access-ps7b2\") pod \"iptables-alerter-844d4\" (UID: \"bfd5ec3a-8ede-49e3-b8e2-00cc159b65aa\") " pod="openshift-network-operator/iptables-alerter-844d4"
Sep 18 19:14:02.614585 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:02.613978    2522 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:02.614585 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.614009    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/72118d02-4cb7-4dd4-bec1-1de6ba1614b1-tmp\") pod \"tuned-zp9rb\" (UID: \"72118d02-4cb7-4dd4-bec1-1de6ba1614b1\") " pod="openshift-cluster-node-tuning-operator/tuned-zp9rb"
Sep 18 19:14:02.614585 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.614035    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/24ec319c-fec4-4b8b-bcb6-6331ec8574fa-kubelet-config\") pod \"global-pull-secret-syncer-g679p\" (UID: \"24ec319c-fec4-4b8b-bcb6-6331ec8574fa\") " pod="kube-system/global-pull-secret-syncer-g679p"
Sep 18 19:14:02.614585 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.614063    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/438a5295-6f51-49ed-99d1-96a8fe037805-host\") pod \"node-ca-hmfqw\" (UID: \"438a5295-6f51-49ed-99d1-96a8fe037805\") " pod="openshift-image-registry/node-ca-hmfqw"
Sep 18 19:14:02.614585 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.614063    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6c2a6c5f-990b-4ceb-9308-f7a6e29e22a2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jrg4l\" (UID: \"6c2a6c5f-990b-4ceb-9308-f7a6e29e22a2\") " pod="openshift-multus/multus-additional-cni-plugins-jrg4l"
Sep 18 19:14:02.614585 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.614091    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6c2a6c5f-990b-4ceb-9308-f7a6e29e22a2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jrg4l\" (UID: \"6c2a6c5f-990b-4ceb-9308-f7a6e29e22a2\") " pod="openshift-multus/multus-additional-cni-plugins-jrg4l"
Sep 18 19:14:02.614585 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.614115    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7-etc-kubernetes\") pod \"multus-d8dg5\" (UID: \"f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7\") " pod="openshift-multus/multus-d8dg5"
Sep 18 19:14:02.615427 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:02.614139    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f516e39e-b125-4617-94ce-7f0dccfae7bb-metrics-certs podName:f516e39e-b125-4617-94ce-7f0dccfae7bb nodeName:}" failed. No retries permitted until 2026-09-18 19:14:03.114107082 +0000 UTC m=+2.997633522 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f516e39e-b125-4617-94ce-7f0dccfae7bb-metrics-certs") pod "network-metrics-daemon-5j97v" (UID: "f516e39e-b125-4617-94ce-7f0dccfae7bb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:02.615427 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.614162    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7-etc-kubernetes\") pod \"multus-d8dg5\" (UID: \"f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7\") " pod="openshift-multus/multus-d8dg5"
Sep 18 19:14:02.615427 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.614312    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94lwf\" (UniqueName: \"kubernetes.io/projected/f516e39e-b125-4617-94ce-7f0dccfae7bb-kube-api-access-94lwf\") pod \"network-metrics-daemon-5j97v\" (UID: \"f516e39e-b125-4617-94ce-7f0dccfae7bb\") " pod="openshift-multus/network-metrics-daemon-5j97v"
Sep 18 19:14:02.615427 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.614356    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/72118d02-4cb7-4dd4-bec1-1de6ba1614b1-etc-sysctl-d\") pod \"tuned-zp9rb\" (UID: \"72118d02-4cb7-4dd4-bec1-1de6ba1614b1\") " pod="openshift-cluster-node-tuning-operator/tuned-zp9rb"
Sep 18 19:14:02.615427 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.614382    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmlh2\" (UniqueName: \"kubernetes.io/projected/72118d02-4cb7-4dd4-bec1-1de6ba1614b1-kube-api-access-qmlh2\") pod \"tuned-zp9rb\" (UID: \"72118d02-4cb7-4dd4-bec1-1de6ba1614b1\") " pod="openshift-cluster-node-tuning-operator/tuned-zp9rb"
Sep 18 19:14:02.615427 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.614430    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e8cd8a19-7cbb-4585-b5b3-6d940ebe41c6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2cql5\" (UID: \"e8cd8a19-7cbb-4585-b5b3-6d940ebe41c6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cql5"
Sep 18 19:14:02.615427 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.614456    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7-cni-binary-copy\") pod \"multus-d8dg5\" (UID: \"f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7\") " pod="openshift-multus/multus-d8dg5"
Sep 18 19:14:02.615427 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.614482    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7-host-run-k8s-cni-cncf-io\") pod \"multus-d8dg5\" (UID: \"f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7\") " pod="openshift-multus/multus-d8dg5"
Sep 18 19:14:02.615427 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.614506    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/72118d02-4cb7-4dd4-bec1-1de6ba1614b1-etc-modprobe-d\") pod \"tuned-zp9rb\" (UID: \"72118d02-4cb7-4dd4-bec1-1de6ba1614b1\") " pod="openshift-cluster-node-tuning-operator/tuned-zp9rb"
Sep 18 19:14:02.615427 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.614582    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eda05f97-2fcc-4f43-af17-9f5d514486bf-host-kubelet\") pod \"ovnkube-node-hbhqv\" (UID: \"eda05f97-2fcc-4f43-af17-9f5d514486bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:02.615427 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.614613    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eda05f97-2fcc-4f43-af17-9f5d514486bf-host-cni-netd\") pod \"ovnkube-node-hbhqv\" (UID: \"eda05f97-2fcc-4f43-af17-9f5d514486bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:02.615427 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.614644    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eda05f97-2fcc-4f43-af17-9f5d514486bf-ovnkube-config\") pod \"ovnkube-node-hbhqv\" (UID: \"eda05f97-2fcc-4f43-af17-9f5d514486bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:02.615427 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.614657    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7-host-run-k8s-cni-cncf-io\") pod \"multus-d8dg5\" (UID: \"f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7\") " pod="openshift-multus/multus-d8dg5"
Sep 18 19:14:02.615427 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.614799    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e31f28a0-45fc-45ae-a307-578e281565a9-konnectivity-ca\") pod \"konnectivity-agent-s7ckz\" (UID: \"e31f28a0-45fc-45ae-a307-578e281565a9\") " pod="kube-system/konnectivity-agent-s7ckz"
Sep 18 19:14:02.615427 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.614827    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24ec319c-fec4-4b8b-bcb6-6331ec8574fa-original-pull-secret\") pod \"global-pull-secret-syncer-g679p\" (UID: \"24ec319c-fec4-4b8b-bcb6-6331ec8574fa\") " pod="kube-system/global-pull-secret-syncer-g679p"
Sep 18 19:14:02.615427 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.614829    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/72118d02-4cb7-4dd4-bec1-1de6ba1614b1-etc-sysctl-d\") pod \"tuned-zp9rb\" (UID: \"72118d02-4cb7-4dd4-bec1-1de6ba1614b1\") " pod="openshift-cluster-node-tuning-operator/tuned-zp9rb"
Sep 18 19:14:02.616215 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.614863    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7-host-var-lib-cni-bin\") pod \"multus-d8dg5\" (UID: \"f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7\") " pod="openshift-multus/multus-d8dg5"
Sep 18 19:14:02.616215 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.614868    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/72118d02-4cb7-4dd4-bec1-1de6ba1614b1-etc-modprobe-d\") pod \"tuned-zp9rb\" (UID: \"72118d02-4cb7-4dd4-bec1-1de6ba1614b1\") " pod="openshift-cluster-node-tuning-operator/tuned-zp9rb"
Sep 18 19:14:02.616215 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.614900    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7-host-run-netns\") pod \"multus-d8dg5\" (UID: \"f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7\") " pod="openshift-multus/multus-d8dg5"
Sep 18 19:14:02.616215 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.614938    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5b796f3a-2d43-47a8-8d73-a34a2af2d62b-hosts-file\") pod \"node-resolver-w4tjm\" (UID: \"5b796f3a-2d43-47a8-8d73-a34a2af2d62b\") " pod="openshift-dns/node-resolver-w4tjm"
Sep 18 19:14:02.616215 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.614953    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7-host-var-lib-cni-bin\") pod \"multus-d8dg5\" (UID: \"f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7\") " pod="openshift-multus/multus-d8dg5"
Sep 18 19:14:02.616215 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.614964    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6c2a6c5f-990b-4ceb-9308-f7a6e29e22a2-cni-binary-copy\") pod \"multus-additional-cni-plugins-jrg4l\" (UID: \"6c2a6c5f-990b-4ceb-9308-f7a6e29e22a2\") " pod="openshift-multus/multus-additional-cni-plugins-jrg4l"
Sep 18 19:14:02.616215 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.614991    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7-host-run-netns\") pod \"multus-d8dg5\" (UID: \"f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7\") " pod="openshift-multus/multus-d8dg5"
Sep 18 19:14:02.616215 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.615145    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7-cni-binary-copy\") pod \"multus-d8dg5\" (UID: \"f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7\") " pod="openshift-multus/multus-d8dg5"
Sep 18 19:14:02.616215 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.615298    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e31f28a0-45fc-45ae-a307-578e281565a9-konnectivity-ca\") pod \"konnectivity-agent-s7ckz\" (UID: \"e31f28a0-45fc-45ae-a307-578e281565a9\") " pod="kube-system/konnectivity-agent-s7ckz"
Sep 18 19:14:02.616215 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.616113    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/72118d02-4cb7-4dd4-bec1-1de6ba1614b1-etc-tuned\") pod \"tuned-zp9rb\" (UID: \"72118d02-4cb7-4dd4-bec1-1de6ba1614b1\") " pod="openshift-cluster-node-tuning-operator/tuned-zp9rb"
Sep 18 19:14:02.616215 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.616189    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e31f28a0-45fc-45ae-a307-578e281565a9-agent-certs\") pod \"konnectivity-agent-s7ckz\" (UID: \"e31f28a0-45fc-45ae-a307-578e281565a9\") " pod="kube-system/konnectivity-agent-s7ckz"
Sep 18 19:14:02.617007 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.616991    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/72118d02-4cb7-4dd4-bec1-1de6ba1614b1-tmp\") pod \"tuned-zp9rb\" (UID: \"72118d02-4cb7-4dd4-bec1-1de6ba1614b1\") " pod="openshift-cluster-node-tuning-operator/tuned-zp9rb"
Sep 18 19:14:02.620615 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.620595    2522 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-6wf2f"
Sep 18 19:14:02.630124 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:02.630100    2522 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:14:02.630218 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:02.630128    2522 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:14:02.630218 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:02.630138    2522 projected.go:196] Error preparing data for projected volume kube-api-access-t7f72 for pod openshift-network-diagnostics/network-check-target-dk2hl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:02.630218 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:02.630188    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f0563df-380a-41eb-bb44-902bf8ceb4f6-kube-api-access-t7f72 podName:3f0563df-380a-41eb-bb44-902bf8ceb4f6 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:03.130175203 +0000 UTC m=+3.013701643 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-t7f72" (UniqueName: "kubernetes.io/projected/3f0563df-380a-41eb-bb44-902bf8ceb4f6-kube-api-access-t7f72") pod "network-check-target-dk2hl" (UID: "3f0563df-380a-41eb-bb44-902bf8ceb4f6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:02.632833 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.632782    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhf56\" (UniqueName: \"kubernetes.io/projected/f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7-kube-api-access-nhf56\") pod \"multus-d8dg5\" (UID: \"f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7\") " pod="openshift-multus/multus-d8dg5"
Sep 18 19:14:02.632833 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.632816    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps7b2\" (UniqueName: \"kubernetes.io/projected/bfd5ec3a-8ede-49e3-b8e2-00cc159b65aa-kube-api-access-ps7b2\") pod \"iptables-alerter-844d4\" (UID: \"bfd5ec3a-8ede-49e3-b8e2-00cc159b65aa\") " pod="openshift-network-operator/iptables-alerter-844d4"
Sep 18 19:14:02.632981 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.632846    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94lwf\" (UniqueName: \"kubernetes.io/projected/f516e39e-b125-4617-94ce-7f0dccfae7bb-kube-api-access-94lwf\") pod \"network-metrics-daemon-5j97v\" (UID: \"f516e39e-b125-4617-94ce-7f0dccfae7bb\") " pod="openshift-multus/network-metrics-daemon-5j97v"
Sep 18 19:14:02.632981 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.632944    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmlh2\" (UniqueName: \"kubernetes.io/projected/72118d02-4cb7-4dd4-bec1-1de6ba1614b1-kube-api-access-qmlh2\") pod \"tuned-zp9rb\" (UID: \"72118d02-4cb7-4dd4-bec1-1de6ba1614b1\") " pod="openshift-cluster-node-tuning-operator/tuned-zp9rb"
Sep 18 19:14:02.633785 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.633766    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rddgw\" (UniqueName: \"kubernetes.io/projected/438a5295-6f51-49ed-99d1-96a8fe037805-kube-api-access-rddgw\") pod \"node-ca-hmfqw\" (UID: \"438a5295-6f51-49ed-99d1-96a8fe037805\") " pod="openshift-image-registry/node-ca-hmfqw"
Sep 18 19:14:02.636878 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.636860    2522 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-6wf2f"
Sep 18 19:14:02.715754 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.715722    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eda05f97-2fcc-4f43-af17-9f5d514486bf-etc-openvswitch\") pod \"ovnkube-node-hbhqv\" (UID: \"eda05f97-2fcc-4f43-af17-9f5d514486bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:02.715902 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.715766    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eda05f97-2fcc-4f43-af17-9f5d514486bf-ovnkube-script-lib\") pod \"ovnkube-node-hbhqv\" (UID: \"eda05f97-2fcc-4f43-af17-9f5d514486bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:02.715902 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.715791    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e8cd8a19-7cbb-4585-b5b3-6d940ebe41c6-registration-dir\") pod \"aws-ebs-csi-driver-node-2cql5\" (UID: \"e8cd8a19-7cbb-4585-b5b3-6d940ebe41c6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cql5"
Sep 18 19:14:02.715902 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.715819    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eda05f97-2fcc-4f43-af17-9f5d514486bf-host-slash\") pod \"ovnkube-node-hbhqv\" (UID: \"eda05f97-2fcc-4f43-af17-9f5d514486bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:02.715902 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.715843    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rkvw\" (UniqueName: \"kubernetes.io/projected/eda05f97-2fcc-4f43-af17-9f5d514486bf-kube-api-access-9rkvw\") pod \"ovnkube-node-hbhqv\" (UID: \"eda05f97-2fcc-4f43-af17-9f5d514486bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:02.715902 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.715866    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bw4n\" (UniqueName: \"kubernetes.io/projected/e8cd8a19-7cbb-4585-b5b3-6d940ebe41c6-kube-api-access-9bw4n\") pod \"aws-ebs-csi-driver-node-2cql5\" (UID: \"e8cd8a19-7cbb-4585-b5b3-6d940ebe41c6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cql5"
Sep 18 19:14:02.715902 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.715892    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eda05f97-2fcc-4f43-af17-9f5d514486bf-ovn-node-metrics-cert\") pod \"ovnkube-node-hbhqv\" (UID: \"eda05f97-2fcc-4f43-af17-9f5d514486bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:02.716175 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.715915    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e8cd8a19-7cbb-4585-b5b3-6d940ebe41c6-device-dir\") pod \"aws-ebs-csi-driver-node-2cql5\" (UID: \"e8cd8a19-7cbb-4585-b5b3-6d940ebe41c6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cql5"
Sep 18 19:14:02.716175 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.715939    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eda05f97-2fcc-4f43-af17-9f5d514486bf-systemd-units\") pod \"ovnkube-node-hbhqv\" (UID: \"eda05f97-2fcc-4f43-af17-9f5d514486bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:02.716175 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.715982    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eda05f97-2fcc-4f43-af17-9f5d514486bf-host-run-ovn-kubernetes\") pod \"ovnkube-node-hbhqv\" (UID: \"eda05f97-2fcc-4f43-af17-9f5d514486bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:02.716175 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.716009    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eda05f97-2fcc-4f43-af17-9f5d514486bf-log-socket\") pod \"ovnkube-node-hbhqv\" (UID: \"eda05f97-2fcc-4f43-af17-9f5d514486bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:02.716175 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.716057    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eda05f97-2fcc-4f43-af17-9f5d514486bf-env-overrides\") pod \"ovnkube-node-hbhqv\" (UID: \"eda05f97-2fcc-4f43-af17-9f5d514486bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:02.716175 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.716079    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6c2a6c5f-990b-4ceb-9308-f7a6e29e22a2-system-cni-dir\") pod \"multus-additional-cni-plugins-jrg4l\" (UID: \"6c2a6c5f-990b-4ceb-9308-f7a6e29e22a2\") " pod="openshift-multus/multus-additional-cni-plugins-jrg4l"
Sep 18 19:14:02.716175 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.716104    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc5xq\" (UniqueName: \"kubernetes.io/projected/6c2a6c5f-990b-4ceb-9308-f7a6e29e22a2-kube-api-access-lc5xq\") pod \"multus-additional-cni-plugins-jrg4l\" (UID: \"6c2a6c5f-990b-4ceb-9308-f7a6e29e22a2\") " pod="openshift-multus/multus-additional-cni-plugins-jrg4l"
Sep 18 19:14:02.716175 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.716126    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eda05f97-2fcc-4f43-af17-9f5d514486bf-host-run-netns\") pod \"ovnkube-node-hbhqv\" (UID: \"eda05f97-2fcc-4f43-af17-9f5d514486bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:02.716175 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.716147    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eda05f97-2fcc-4f43-af17-9f5d514486bf-run-openvswitch\") pod \"ovnkube-node-hbhqv\" (UID: \"eda05f97-2fcc-4f43-af17-9f5d514486bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:02.716175 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.716170    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eda05f97-2fcc-4f43-af17-9f5d514486bf-run-ovn\") pod \"ovnkube-node-hbhqv\" (UID: \"eda05f97-2fcc-4f43-af17-9f5d514486bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:02.716483 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.716192    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eda05f97-2fcc-4f43-af17-9f5d514486bf-node-log\") pod \"ovnkube-node-hbhqv\" (UID: \"eda05f97-2fcc-4f43-af17-9f5d514486bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:02.716483 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.716216    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6c2a6c5f-990b-4ceb-9308-f7a6e29e22a2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jrg4l\" (UID: \"6c2a6c5f-990b-4ceb-9308-f7a6e29e22a2\") " pod="openshift-multus/multus-additional-cni-plugins-jrg4l"
Sep 18 19:14:02.716483 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.716242    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e8cd8a19-7cbb-4585-b5b3-6d940ebe41c6-socket-dir\") pod \"aws-ebs-csi-driver-node-2cql5\" (UID: \"e8cd8a19-7cbb-4585-b5b3-6d940ebe41c6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cql5"
Sep 18 19:14:02.716483 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.716265    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/24ec319c-fec4-4b8b-bcb6-6331ec8574fa-dbus\") pod \"global-pull-secret-syncer-g679p\" (UID: \"24ec319c-fec4-4b8b-bcb6-6331ec8574fa\") " pod="kube-system/global-pull-secret-syncer-g679p"
Sep 18 19:14:02.716483 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.716287    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eda05f97-2fcc-4f43-af17-9f5d514486bf-var-lib-openvswitch\") pod \"ovnkube-node-hbhqv\" (UID: \"eda05f97-2fcc-4f43-af17-9f5d514486bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:02.716483 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.716317    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eda05f97-2fcc-4f43-af17-9f5d514486bf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hbhqv\" (UID: \"eda05f97-2fcc-4f43-af17-9f5d514486bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:02.716483 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.716356    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b796f3a-2d43-47a8-8d73-a34a2af2d62b-tmp-dir\") pod \"node-resolver-w4tjm\" (UID: \"5b796f3a-2d43-47a8-8d73-a34a2af2d62b\") " pod="openshift-dns/node-resolver-w4tjm"
Sep 18 19:14:02.716483 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.716378    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eda05f97-2fcc-4f43-af17-9f5d514486bf-run-systemd\") pod \"ovnkube-node-hbhqv\" (UID: \"eda05f97-2fcc-4f43-af17-9f5d514486bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:02.716483 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.716401    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eda05f97-2fcc-4f43-af17-9f5d514486bf-host-cni-bin\") pod \"ovnkube-node-hbhqv\" (UID: \"eda05f97-2fcc-4f43-af17-9f5d514486bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:02.716483 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.716432    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6c2a6c5f-990b-4ceb-9308-f7a6e29e22a2-cnibin\") pod \"multus-additional-cni-plugins-jrg4l\" (UID: \"6c2a6c5f-990b-4ceb-9308-f7a6e29e22a2\") " pod="openshift-multus/multus-additional-cni-plugins-jrg4l"
Sep 18 19:14:02.716483 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.716474    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/24ec319c-fec4-4b8b-bcb6-6331ec8574fa-kubelet-config\") pod \"global-pull-secret-syncer-g679p\" (UID: \"24ec319c-fec4-4b8b-bcb6-6331ec8574fa\") " pod="kube-system/global-pull-secret-syncer-g679p"
Sep 18 19:14:02.716962 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.716496    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6c2a6c5f-990b-4ceb-9308-f7a6e29e22a2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jrg4l\" (UID: \"6c2a6c5f-990b-4ceb-9308-f7a6e29e22a2\") " pod="openshift-multus/multus-additional-cni-plugins-jrg4l"
Sep 18 19:14:02.716962 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.716522    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6c2a6c5f-990b-4ceb-9308-f7a6e29e22a2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jrg4l\" (UID: \"6c2a6c5f-990b-4ceb-9308-f7a6e29e22a2\") " pod="openshift-multus/multus-additional-cni-plugins-jrg4l"
Sep 18 19:14:02.716962 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.716551    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e8cd8a19-7cbb-4585-b5b3-6d940ebe41c6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2cql5\" (UID: \"e8cd8a19-7cbb-4585-b5b3-6d940ebe41c6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cql5"
Sep 18 19:14:02.716962 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.716595    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eda05f97-2fcc-4f43-af17-9f5d514486bf-host-kubelet\") pod \"ovnkube-node-hbhqv\" (UID: \"eda05f97-2fcc-4f43-af17-9f5d514486bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:02.716962 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.716619    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eda05f97-2fcc-4f43-af17-9f5d514486bf-host-cni-netd\") pod \"ovnkube-node-hbhqv\" (UID: \"eda05f97-2fcc-4f43-af17-9f5d514486bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:02.716962 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.716636    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eda05f97-2fcc-4f43-af17-9f5d514486bf-host-run-netns\") pod \"ovnkube-node-hbhqv\" (UID: \"eda05f97-2fcc-4f43-af17-9f5d514486bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:02.716962 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.716659    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eda05f97-2fcc-4f43-af17-9f5d514486bf-run-openvswitch\") pod \"ovnkube-node-hbhqv\" (UID: \"eda05f97-2fcc-4f43-af17-9f5d514486bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:02.716962 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.716643    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eda05f97-2fcc-4f43-af17-9f5d514486bf-ovnkube-config\") pod \"ovnkube-node-hbhqv\" (UID: \"eda05f97-2fcc-4f43-af17-9f5d514486bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:02.716962 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.716696    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eda05f97-2fcc-4f43-af17-9f5d514486bf-etc-openvswitch\") pod \"ovnkube-node-hbhqv\" (UID: \"eda05f97-2fcc-4f43-af17-9f5d514486bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:02.716962 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.716717    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24ec319c-fec4-4b8b-bcb6-6331ec8574fa-original-pull-secret\") pod \"global-pull-secret-syncer-g679p\" (UID: \"24ec319c-fec4-4b8b-bcb6-6331ec8574fa\") " pod="kube-system/global-pull-secret-syncer-g679p"
Sep 18 19:14:02.716962 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.716747    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5b796f3a-2d43-47a8-8d73-a34a2af2d62b-hosts-file\") pod \"node-resolver-w4tjm\" (UID: \"5b796f3a-2d43-47a8-8d73-a34a2af2d62b\") " pod="openshift-dns/node-resolver-w4tjm"
Sep 18 19:14:02.716962 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.716775    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6c2a6c5f-990b-4ceb-9308-f7a6e29e22a2-cni-binary-copy\") pod \"multus-additional-cni-plugins-jrg4l\" (UID: \"6c2a6c5f-990b-4ceb-9308-f7a6e29e22a2\") " pod="openshift-multus/multus-additional-cni-plugins-jrg4l"
Sep 18 19:14:02.716962 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.716801    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e8cd8a19-7cbb-4585-b5b3-6d940ebe41c6-etc-selinux\") pod \"aws-ebs-csi-driver-node-2cql5\" (UID: \"e8cd8a19-7cbb-4585-b5b3-6d940ebe41c6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cql5"
Sep 18 19:14:02.716962 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.716828    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e8cd8a19-7cbb-4585-b5b3-6d940ebe41c6-sys-fs\") pod \"aws-ebs-csi-driver-node-2cql5\" (UID: \"e8cd8a19-7cbb-4585-b5b3-6d940ebe41c6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cql5"
Sep 18 19:14:02.716962 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.716856    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5dhq\" (UniqueName: \"kubernetes.io/projected/5b796f3a-2d43-47a8-8d73-a34a2af2d62b-kube-api-access-j5dhq\") pod \"node-resolver-w4tjm\" (UID: \"5b796f3a-2d43-47a8-8d73-a34a2af2d62b\") " pod="openshift-dns/node-resolver-w4tjm"
Sep 18 19:14:02.716962 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.716941    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b796f3a-2d43-47a8-8d73-a34a2af2d62b-tmp-dir\") pod \"node-resolver-w4tjm\" (UID: \"5b796f3a-2d43-47a8-8d73-a34a2af2d62b\") " pod="openshift-dns/node-resolver-w4tjm"
Sep 18 19:14:02.717722 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.716995    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eda05f97-2fcc-4f43-af17-9f5d514486bf-run-systemd\") pod \"ovnkube-node-hbhqv\" (UID: \"eda05f97-2fcc-4f43-af17-9f5d514486bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:02.717722 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.717042    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eda05f97-2fcc-4f43-af17-9f5d514486bf-host-cni-bin\") pod \"ovnkube-node-hbhqv\" (UID: \"eda05f97-2fcc-4f43-af17-9f5d514486bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:02.717722 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.717082    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6c2a6c5f-990b-4ceb-9308-f7a6e29e22a2-cnibin\") pod \"multus-additional-cni-plugins-jrg4l\" (UID: \"6c2a6c5f-990b-4ceb-9308-f7a6e29e22a2\") " pod="openshift-multus/multus-additional-cni-plugins-jrg4l"
Sep 18 19:14:02.717722 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.717126    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/24ec319c-fec4-4b8b-bcb6-6331ec8574fa-kubelet-config\") pod \"global-pull-secret-syncer-g679p\" (UID: \"24ec319c-fec4-4b8b-bcb6-6331ec8574fa\") " pod="kube-system/global-pull-secret-syncer-g679p"
Sep 18 19:14:02.717722 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.717148    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eda05f97-2fcc-4f43-af17-9f5d514486bf-run-ovn\") pod \"ovnkube-node-hbhqv\" (UID: \"eda05f97-2fcc-4f43-af17-9f5d514486bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:02.717722 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.717193    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eda05f97-2fcc-4f43-af17-9f5d514486bf-node-log\") pod \"ovnkube-node-hbhqv\" (UID: \"eda05f97-2fcc-4f43-af17-9f5d514486bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:02.717722 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.717256    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6c2a6c5f-990b-4ceb-9308-f7a6e29e22a2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jrg4l\" (UID: \"6c2a6c5f-990b-4ceb-9308-f7a6e29e22a2\") " pod="openshift-multus/multus-additional-cni-plugins-jrg4l"
Sep 18 19:14:02.717722 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.717261    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eda05f97-2fcc-4f43-af17-9f5d514486bf-ovnkube-config\") pod \"ovnkube-node-hbhqv\" (UID: \"eda05f97-2fcc-4f43-af17-9f5d514486bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:02.717722 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.717329    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e8cd8a19-7cbb-4585-b5b3-6d940ebe41c6-registration-dir\") pod \"aws-ebs-csi-driver-node-2cql5\" (UID: \"e8cd8a19-7cbb-4585-b5b3-6d940ebe41c6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cql5"
Sep 18 19:14:02.717722 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.717369    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eda05f97-2fcc-4f43-af17-9f5d514486bf-host-slash\") pod \"ovnkube-node-hbhqv\" (UID: \"eda05f97-2fcc-4f43-af17-9f5d514486bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:02.717722 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.717707    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eda05f97-2fcc-4f43-af17-9f5d514486bf-log-socket\") pod \"ovnkube-node-hbhqv\" (UID: \"eda05f97-2fcc-4f43-af17-9f5d514486bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:02.718265 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.717748    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6c2a6c5f-990b-4ceb-9308-f7a6e29e22a2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jrg4l\" (UID: \"6c2a6c5f-990b-4ceb-9308-f7a6e29e22a2\") " pod="openshift-multus/multus-additional-cni-plugins-jrg4l"
Sep 18 19:14:02.718265 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.717771    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6c2a6c5f-990b-4ceb-9308-f7a6e29e22a2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jrg4l\" (UID: \"6c2a6c5f-990b-4ceb-9308-f7a6e29e22a2\") " pod="openshift-multus/multus-additional-cni-plugins-jrg4l"
Sep 18 19:14:02.718265 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.717828    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/24ec319c-fec4-4b8b-bcb6-6331ec8574fa-dbus\") pod \"global-pull-secret-syncer-g679p\" (UID: \"24ec319c-fec4-4b8b-bcb6-6331ec8574fa\") " pod="kube-system/global-pull-secret-syncer-g679p"
Sep 18 19:14:02.718265 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.717871    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eda05f97-2fcc-4f43-af17-9f5d514486bf-var-lib-openvswitch\") pod \"ovnkube-node-hbhqv\" (UID: \"eda05f97-2fcc-4f43-af17-9f5d514486bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:02.718265 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:02.717944    2522 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:02.718265 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:02.717993    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24ec319c-fec4-4b8b-bcb6-6331ec8574fa-original-pull-secret podName:24ec319c-fec4-4b8b-bcb6-6331ec8574fa nodeName:}" failed. No retries permitted until 2026-09-18 19:14:03.217976589 +0000 UTC m=+3.101503047 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/24ec319c-fec4-4b8b-bcb6-6331ec8574fa-original-pull-secret") pod "global-pull-secret-syncer-g679p" (UID: "24ec319c-fec4-4b8b-bcb6-6331ec8574fa") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:02.718265 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.718141    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eda05f97-2fcc-4f43-af17-9f5d514486bf-env-overrides\") pod \"ovnkube-node-hbhqv\" (UID: \"eda05f97-2fcc-4f43-af17-9f5d514486bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:02.718265 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.718181    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5b796f3a-2d43-47a8-8d73-a34a2af2d62b-hosts-file\") pod \"node-resolver-w4tjm\" (UID: \"5b796f3a-2d43-47a8-8d73-a34a2af2d62b\") " pod="openshift-dns/node-resolver-w4tjm"
Sep 18 19:14:02.718265 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.718252    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e8cd8a19-7cbb-4585-b5b3-6d940ebe41c6-socket-dir\") pod \"aws-ebs-csi-driver-node-2cql5\" (UID: \"e8cd8a19-7cbb-4585-b5b3-6d940ebe41c6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cql5"
Sep 18 19:14:02.718742 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.718299    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6c2a6c5f-990b-4ceb-9308-f7a6e29e22a2-system-cni-dir\") pod \"multus-additional-cni-plugins-jrg4l\" (UID: \"6c2a6c5f-990b-4ceb-9308-f7a6e29e22a2\") " pod="openshift-multus/multus-additional-cni-plugins-jrg4l"
Sep 18 19:14:02.718742 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.718308    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eda05f97-2fcc-4f43-af17-9f5d514486bf-ovnkube-script-lib\") pod \"ovnkube-node-hbhqv\" (UID: \"eda05f97-2fcc-4f43-af17-9f5d514486bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:02.718742 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.718381    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e8cd8a19-7cbb-4585-b5b3-6d940ebe41c6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2cql5\" (UID: \"e8cd8a19-7cbb-4585-b5b3-6d940ebe41c6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cql5"
Sep 18 19:14:02.718742 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.718430    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eda05f97-2fcc-4f43-af17-9f5d514486bf-host-kubelet\") pod \"ovnkube-node-hbhqv\" (UID: \"eda05f97-2fcc-4f43-af17-9f5d514486bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:02.718742 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.718467    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eda05f97-2fcc-4f43-af17-9f5d514486bf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hbhqv\" (UID: \"eda05f97-2fcc-4f43-af17-9f5d514486bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:02.718742 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.718529    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eda05f97-2fcc-4f43-af17-9f5d514486bf-host-cni-netd\") pod \"ovnkube-node-hbhqv\" (UID: \"eda05f97-2fcc-4f43-af17-9f5d514486bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:02.718742 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.718603    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eda05f97-2fcc-4f43-af17-9f5d514486bf-systemd-units\") pod \"ovnkube-node-hbhqv\" (UID: \"eda05f97-2fcc-4f43-af17-9f5d514486bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:02.718742 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.718649    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e8cd8a19-7cbb-4585-b5b3-6d940ebe41c6-device-dir\") pod \"aws-ebs-csi-driver-node-2cql5\" (UID: \"e8cd8a19-7cbb-4585-b5b3-6d940ebe41c6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cql5"
Sep 18 19:14:02.718742 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.718652    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6c2a6c5f-990b-4ceb-9308-f7a6e29e22a2-cni-binary-copy\") pod \"multus-additional-cni-plugins-jrg4l\" (UID: \"6c2a6c5f-990b-4ceb-9308-f7a6e29e22a2\") " pod="openshift-multus/multus-additional-cni-plugins-jrg4l"
Sep 18 19:14:02.718742 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.718710    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e8cd8a19-7cbb-4585-b5b3-6d940ebe41c6-etc-selinux\") pod \"aws-ebs-csi-driver-node-2cql5\" (UID: \"e8cd8a19-7cbb-4585-b5b3-6d940ebe41c6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cql5"
Sep 18 19:14:02.718742 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.718717    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e8cd8a19-7cbb-4585-b5b3-6d940ebe41c6-sys-fs\") pod \"aws-ebs-csi-driver-node-2cql5\" (UID: \"e8cd8a19-7cbb-4585-b5b3-6d940ebe41c6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cql5"
Sep 18 19:14:02.719241 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.718766    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eda05f97-2fcc-4f43-af17-9f5d514486bf-host-run-ovn-kubernetes\") pod \"ovnkube-node-hbhqv\" (UID: \"eda05f97-2fcc-4f43-af17-9f5d514486bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:02.721112 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.720379    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eda05f97-2fcc-4f43-af17-9f5d514486bf-ovn-node-metrics-cert\") pod \"ovnkube-node-hbhqv\" (UID: \"eda05f97-2fcc-4f43-af17-9f5d514486bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:02.721648 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.721245    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-221.ec2.internal" event={"ID":"523614729ae9b688582b641ea2fafde6","Type":"ContainerStarted","Data":"2444e6710fada79f35297c75a6776359456cffc6f5a44b91abdcdced828250ca"}
Sep 18 19:14:02.722519 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.722490    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-221.ec2.internal" event={"ID":"31c0bde74e3407e1ddada35fdb2a70e7","Type":"ContainerStarted","Data":"2c3e2a798798b126ab56c253da80704f92df72862f7f525fb61339ca39e96790"}
Sep 18 19:14:02.741150 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.741089    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5dhq\" (UniqueName: \"kubernetes.io/projected/5b796f3a-2d43-47a8-8d73-a34a2af2d62b-kube-api-access-j5dhq\") pod \"node-resolver-w4tjm\" (UID: \"5b796f3a-2d43-47a8-8d73-a34a2af2d62b\") " pod="openshift-dns/node-resolver-w4tjm"
Sep 18 19:14:02.741247 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.741205    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bw4n\" (UniqueName: \"kubernetes.io/projected/e8cd8a19-7cbb-4585-b5b3-6d940ebe41c6-kube-api-access-9bw4n\") pod \"aws-ebs-csi-driver-node-2cql5\" (UID: \"e8cd8a19-7cbb-4585-b5b3-6d940ebe41c6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cql5"
Sep 18 19:14:02.741942 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.741919    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rkvw\" (UniqueName: \"kubernetes.io/projected/eda05f97-2fcc-4f43-af17-9f5d514486bf-kube-api-access-9rkvw\") pod \"ovnkube-node-hbhqv\" (UID: \"eda05f97-2fcc-4f43-af17-9f5d514486bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:02.743208 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.743186    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc5xq\" (UniqueName: \"kubernetes.io/projected/6c2a6c5f-990b-4ceb-9308-f7a6e29e22a2-kube-api-access-lc5xq\") pod \"multus-additional-cni-plugins-jrg4l\" (UID: \"6c2a6c5f-990b-4ceb-9308-f7a6e29e22a2\") " pod="openshift-multus/multus-additional-cni-plugins-jrg4l"
Sep 18 19:14:02.799999 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.799969    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zp9rb"
Sep 18 19:14:02.807067 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.807044    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-d8dg5"
Sep 18 19:14:02.810989 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:14:02.810822    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72118d02_4cb7_4dd4_bec1_1de6ba1614b1.slice/crio-b892c9d526f968b8dac94ba78e2dddcf441d3bd30d61a027f4fff91baff568f6 WatchSource:0}: Error finding container b892c9d526f968b8dac94ba78e2dddcf441d3bd30d61a027f4fff91baff568f6: Status 404 returned error can't find the container with id b892c9d526f968b8dac94ba78e2dddcf441d3bd30d61a027f4fff91baff568f6
Sep 18 19:14:02.816726 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.816703    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-s7ckz"
Sep 18 19:14:02.817038 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:14:02.817010    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5f32ffd_68fe_47ad_9b94_2d59b1fbe3e7.slice/crio-63f1a968c63c009d6e59ba0e801e9666b886b0e03d01b7ecdbf04fff24bb32b0 WatchSource:0}: Error finding container 63f1a968c63c009d6e59ba0e801e9666b886b0e03d01b7ecdbf04fff24bb32b0: Status 404 returned error can't find the container with id 63f1a968c63c009d6e59ba0e801e9666b886b0e03d01b7ecdbf04fff24bb32b0
Sep 18 19:14:02.824156 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.824140    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hmfqw"
Sep 18 19:14:02.826960 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:14:02.826936    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode31f28a0_45fc_45ae_a307_578e281565a9.slice/crio-e10f81087da6e99838eac586335afcf5244c99ae727ee32db5143dcceaa336d8 WatchSource:0}: Error finding container e10f81087da6e99838eac586335afcf5244c99ae727ee32db5143dcceaa336d8: Status 404 returned error can't find the container with id e10f81087da6e99838eac586335afcf5244c99ae727ee32db5143dcceaa336d8
Sep 18 19:14:02.833194 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.833152    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-844d4"
Sep 18 19:14:02.833930 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:14:02.833901    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod438a5295_6f51_49ed_99d1_96a8fe037805.slice/crio-1459718edcde896c7c72684237d8f0561fff347851d807276b632165af738f50 WatchSource:0}: Error finding container 1459718edcde896c7c72684237d8f0561fff347851d807276b632165af738f50: Status 404 returned error can't find the container with id 1459718edcde896c7c72684237d8f0561fff347851d807276b632165af738f50
Sep 18 19:14:02.841805 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.841785    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jrg4l"
Sep 18 19:14:02.843622 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:14:02.843590    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfd5ec3a_8ede_49e3_b8e2_00cc159b65aa.slice/crio-c7cddfaad6ba9de8d967efed7143b53b348c28c4ee29ee865a73a8f7312d8cb9 WatchSource:0}: Error finding container c7cddfaad6ba9de8d967efed7143b53b348c28c4ee29ee865a73a8f7312d8cb9: Status 404 returned error can't find the container with id c7cddfaad6ba9de8d967efed7143b53b348c28c4ee29ee865a73a8f7312d8cb9
Sep 18 19:14:02.848553 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.848511    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cql5"
Sep 18 19:14:02.851192 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:14:02.851170    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c2a6c5f_990b_4ceb_9308_f7a6e29e22a2.slice/crio-5879c0de1e96570741f71a399ea0c02eca31c7088997023850a5ce370798e933 WatchSource:0}: Error finding container 5879c0de1e96570741f71a399ea0c02eca31c7088997023850a5ce370798e933: Status 404 returned error can't find the container with id 5879c0de1e96570741f71a399ea0c02eca31c7088997023850a5ce370798e933
Sep 18 19:14:02.851499 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.851481    2522 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:14:02.855726 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.855702    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-w4tjm"
Sep 18 19:14:02.858137 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:14:02.858112    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8cd8a19_7cbb_4585_b5b3_6d940ebe41c6.slice/crio-dd3ddcd2f15552901ee3c1d97440a9352105c18296d14fe9dbbb24b23c7ba013 WatchSource:0}: Error finding container dd3ddcd2f15552901ee3c1d97440a9352105c18296d14fe9dbbb24b23c7ba013: Status 404 returned error can't find the container with id dd3ddcd2f15552901ee3c1d97440a9352105c18296d14fe9dbbb24b23c7ba013
Sep 18 19:14:02.863200 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:02.863184    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:02.867863 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:14:02.867841    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b796f3a_2d43_47a8_8d73_a34a2af2d62b.slice/crio-ed63d5447558cbc6e6b3c501fad6f8a775a98d5c26e00834016e97fba8aeb662 WatchSource:0}: Error finding container ed63d5447558cbc6e6b3c501fad6f8a775a98d5c26e00834016e97fba8aeb662: Status 404 returned error can't find the container with id ed63d5447558cbc6e6b3c501fad6f8a775a98d5c26e00834016e97fba8aeb662
Sep 18 19:14:02.874762 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:14:02.874735    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeda05f97_2fcc_4f43_af17_9f5d514486bf.slice/crio-0e852a5b1f7c1c6dbb3e07338a20585c987df5a09e4764a08a8b7aea66fc167f WatchSource:0}: Error finding container 0e852a5b1f7c1c6dbb3e07338a20585c987df5a09e4764a08a8b7aea66fc167f: Status 404 returned error can't find the container with id 0e852a5b1f7c1c6dbb3e07338a20585c987df5a09e4764a08a8b7aea66fc167f
Sep 18 19:14:03.119538 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:03.119446    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f516e39e-b125-4617-94ce-7f0dccfae7bb-metrics-certs\") pod \"network-metrics-daemon-5j97v\" (UID: \"f516e39e-b125-4617-94ce-7f0dccfae7bb\") " pod="openshift-multus/network-metrics-daemon-5j97v"
Sep 18 19:14:03.119709 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:03.119606    2522 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:03.119709 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:03.119668    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f516e39e-b125-4617-94ce-7f0dccfae7bb-metrics-certs podName:f516e39e-b125-4617-94ce-7f0dccfae7bb nodeName:}" failed. No retries permitted until 2026-09-18 19:14:04.119650669 +0000 UTC m=+4.003177113 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f516e39e-b125-4617-94ce-7f0dccfae7bb-metrics-certs") pod "network-metrics-daemon-5j97v" (UID: "f516e39e-b125-4617-94ce-7f0dccfae7bb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:03.219963 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:03.219931    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7f72\" (UniqueName: \"kubernetes.io/projected/3f0563df-380a-41eb-bb44-902bf8ceb4f6-kube-api-access-t7f72\") pod \"network-check-target-dk2hl\" (UID: \"3f0563df-380a-41eb-bb44-902bf8ceb4f6\") " pod="openshift-network-diagnostics/network-check-target-dk2hl"
Sep 18 19:14:03.220128 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:03.219996    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24ec319c-fec4-4b8b-bcb6-6331ec8574fa-original-pull-secret\") pod \"global-pull-secret-syncer-g679p\" (UID: \"24ec319c-fec4-4b8b-bcb6-6331ec8574fa\") " pod="kube-system/global-pull-secret-syncer-g679p"
Sep 18 19:14:03.220128 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:03.220092    2522 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:03.220229 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:03.220145    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24ec319c-fec4-4b8b-bcb6-6331ec8574fa-original-pull-secret podName:24ec319c-fec4-4b8b-bcb6-6331ec8574fa nodeName:}" failed. No retries permitted until 2026-09-18 19:14:04.220132547 +0000 UTC m=+4.103659000 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/24ec319c-fec4-4b8b-bcb6-6331ec8574fa-original-pull-secret") pod "global-pull-secret-syncer-g679p" (UID: "24ec319c-fec4-4b8b-bcb6-6331ec8574fa") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:03.220229 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:03.220095    2522 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:14:03.220229 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:03.220217    2522 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:14:03.220385 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:03.220235    2522 projected.go:196] Error preparing data for projected volume kube-api-access-t7f72 for pod openshift-network-diagnostics/network-check-target-dk2hl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:03.220385 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:03.220310    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f0563df-380a-41eb-bb44-902bf8ceb4f6-kube-api-access-t7f72 podName:3f0563df-380a-41eb-bb44-902bf8ceb4f6 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:04.220276609 +0000 UTC m=+4.103803073 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-t7f72" (UniqueName: "kubernetes.io/projected/3f0563df-380a-41eb-bb44-902bf8ceb4f6-kube-api-access-t7f72") pod "network-check-target-dk2hl" (UID: "3f0563df-380a-41eb-bb44-902bf8ceb4f6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:03.640843 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:03.640799    2522 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 19:09:02 +0000 UTC" deadline="2028-05-14 03:37:54.468205838 +0000 UTC"
Sep 18 19:14:03.640843 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:03.640842    2522 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14480h23m50.827368238s"
Sep 18 19:14:03.716224 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:03.716197    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g679p"
Sep 18 19:14:03.716389 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:03.716299    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g679p" podUID="24ec319c-fec4-4b8b-bcb6-6331ec8574fa"
Sep 18 19:14:03.725689 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:03.725658    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jrg4l" event={"ID":"6c2a6c5f-990b-4ceb-9308-f7a6e29e22a2","Type":"ContainerStarted","Data":"5879c0de1e96570741f71a399ea0c02eca31c7088997023850a5ce370798e933"}
Sep 18 19:14:03.726831 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:03.726805    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hmfqw" event={"ID":"438a5295-6f51-49ed-99d1-96a8fe037805","Type":"ContainerStarted","Data":"1459718edcde896c7c72684237d8f0561fff347851d807276b632165af738f50"}
Sep 18 19:14:03.728342 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:03.728312    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-s7ckz" event={"ID":"e31f28a0-45fc-45ae-a307-578e281565a9","Type":"ContainerStarted","Data":"e10f81087da6e99838eac586335afcf5244c99ae727ee32db5143dcceaa336d8"}
Sep 18 19:14:03.729643 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:03.729621    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-844d4" event={"ID":"bfd5ec3a-8ede-49e3-b8e2-00cc159b65aa","Type":"ContainerStarted","Data":"c7cddfaad6ba9de8d967efed7143b53b348c28c4ee29ee865a73a8f7312d8cb9"}
Sep 18 19:14:03.730802 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:03.730779    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-d8dg5" event={"ID":"f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7","Type":"ContainerStarted","Data":"63f1a968c63c009d6e59ba0e801e9666b886b0e03d01b7ecdbf04fff24bb32b0"}
Sep 18 19:14:03.732180 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:03.732157    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zp9rb" event={"ID":"72118d02-4cb7-4dd4-bec1-1de6ba1614b1","Type":"ContainerStarted","Data":"b892c9d526f968b8dac94ba78e2dddcf441d3bd30d61a027f4fff91baff568f6"}
Sep 18 19:14:03.733641 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:03.733619    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv" event={"ID":"eda05f97-2fcc-4f43-af17-9f5d514486bf","Type":"ContainerStarted","Data":"0e852a5b1f7c1c6dbb3e07338a20585c987df5a09e4764a08a8b7aea66fc167f"}
Sep 18 19:14:03.734911 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:03.734889    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-w4tjm" event={"ID":"5b796f3a-2d43-47a8-8d73-a34a2af2d62b","Type":"ContainerStarted","Data":"ed63d5447558cbc6e6b3c501fad6f8a775a98d5c26e00834016e97fba8aeb662"}
Sep 18 19:14:03.736438 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:03.735956    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cql5" event={"ID":"e8cd8a19-7cbb-4585-b5b3-6d940ebe41c6","Type":"ContainerStarted","Data":"dd3ddcd2f15552901ee3c1d97440a9352105c18296d14fe9dbbb24b23c7ba013"}
Sep 18 19:14:04.127534 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:04.127457    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f516e39e-b125-4617-94ce-7f0dccfae7bb-metrics-certs\") pod \"network-metrics-daemon-5j97v\" (UID: \"f516e39e-b125-4617-94ce-7f0dccfae7bb\") " pod="openshift-multus/network-metrics-daemon-5j97v"
Sep 18 19:14:04.127696 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:04.127602    2522 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:04.127696 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:04.127661    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f516e39e-b125-4617-94ce-7f0dccfae7bb-metrics-certs podName:f516e39e-b125-4617-94ce-7f0dccfae7bb nodeName:}" failed. No retries permitted until 2026-09-18 19:14:06.12764261 +0000 UTC m=+6.011169063 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f516e39e-b125-4617-94ce-7f0dccfae7bb-metrics-certs") pod "network-metrics-daemon-5j97v" (UID: "f516e39e-b125-4617-94ce-7f0dccfae7bb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:04.228845 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:04.228812    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7f72\" (UniqueName: \"kubernetes.io/projected/3f0563df-380a-41eb-bb44-902bf8ceb4f6-kube-api-access-t7f72\") pod \"network-check-target-dk2hl\" (UID: \"3f0563df-380a-41eb-bb44-902bf8ceb4f6\") " pod="openshift-network-diagnostics/network-check-target-dk2hl"
Sep 18 19:14:04.229018 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:04.228882    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24ec319c-fec4-4b8b-bcb6-6331ec8574fa-original-pull-secret\") pod \"global-pull-secret-syncer-g679p\" (UID: \"24ec319c-fec4-4b8b-bcb6-6331ec8574fa\") " pod="kube-system/global-pull-secret-syncer-g679p"
Sep 18 19:14:04.229018 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:04.229000    2522 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:04.229132 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:04.229057    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24ec319c-fec4-4b8b-bcb6-6331ec8574fa-original-pull-secret podName:24ec319c-fec4-4b8b-bcb6-6331ec8574fa nodeName:}" failed. No retries permitted until 2026-09-18 19:14:06.229038571 +0000 UTC m=+6.112565016 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/24ec319c-fec4-4b8b-bcb6-6331ec8574fa-original-pull-secret") pod "global-pull-secret-syncer-g679p" (UID: "24ec319c-fec4-4b8b-bcb6-6331ec8574fa") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:04.229434 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:04.229417    2522 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:14:04.229507 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:04.229440    2522 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:14:04.229507 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:04.229452    2522 projected.go:196] Error preparing data for projected volume kube-api-access-t7f72 for pod openshift-network-diagnostics/network-check-target-dk2hl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:04.229507 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:04.229497    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f0563df-380a-41eb-bb44-902bf8ceb4f6-kube-api-access-t7f72 podName:3f0563df-380a-41eb-bb44-902bf8ceb4f6 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:06.229482501 +0000 UTC m=+6.113008957 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-t7f72" (UniqueName: "kubernetes.io/projected/3f0563df-380a-41eb-bb44-902bf8ceb4f6-kube-api-access-t7f72") pod "network-check-target-dk2hl" (UID: "3f0563df-380a-41eb-bb44-902bf8ceb4f6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:04.641670 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:04.641635    2522 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 19:09:02 +0000 UTC" deadline="2028-03-24 23:40:33.772908674 +0000 UTC"
Sep 18 19:14:04.641670 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:04.641670    2522 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13276h26m29.131243064s"
Sep 18 19:14:04.720266 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:04.720234    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5j97v"
Sep 18 19:14:04.720428 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:04.720366    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5j97v" podUID="f516e39e-b125-4617-94ce-7f0dccfae7bb"
Sep 18 19:14:04.720492 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:04.720445    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dk2hl"
Sep 18 19:14:04.720549 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:04.720521    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dk2hl" podUID="3f0563df-380a-41eb-bb44-902bf8ceb4f6"
Sep 18 19:14:05.716601 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:05.716496    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g679p"
Sep 18 19:14:05.717154 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:05.716664    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g679p" podUID="24ec319c-fec4-4b8b-bcb6-6331ec8574fa"
Sep 18 19:14:06.149865 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:06.149781    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f516e39e-b125-4617-94ce-7f0dccfae7bb-metrics-certs\") pod \"network-metrics-daemon-5j97v\" (UID: \"f516e39e-b125-4617-94ce-7f0dccfae7bb\") " pod="openshift-multus/network-metrics-daemon-5j97v"
Sep 18 19:14:06.150022 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:06.149955    2522 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:06.150022 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:06.150016    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f516e39e-b125-4617-94ce-7f0dccfae7bb-metrics-certs podName:f516e39e-b125-4617-94ce-7f0dccfae7bb nodeName:}" failed. No retries permitted until 2026-09-18 19:14:10.149998939 +0000 UTC m=+10.033525381 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f516e39e-b125-4617-94ce-7f0dccfae7bb-metrics-certs") pod "network-metrics-daemon-5j97v" (UID: "f516e39e-b125-4617-94ce-7f0dccfae7bb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:06.250825 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:06.250787    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24ec319c-fec4-4b8b-bcb6-6331ec8574fa-original-pull-secret\") pod \"global-pull-secret-syncer-g679p\" (UID: \"24ec319c-fec4-4b8b-bcb6-6331ec8574fa\") " pod="kube-system/global-pull-secret-syncer-g679p"
Sep 18 19:14:06.251002 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:06.250868    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7f72\" (UniqueName: \"kubernetes.io/projected/3f0563df-380a-41eb-bb44-902bf8ceb4f6-kube-api-access-t7f72\") pod \"network-check-target-dk2hl\" (UID: \"3f0563df-380a-41eb-bb44-902bf8ceb4f6\") " pod="openshift-network-diagnostics/network-check-target-dk2hl"
Sep 18 19:14:06.251073 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:06.251005    2522 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:14:06.251073 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:06.251022    2522 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:14:06.251073 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:06.251034    2522 projected.go:196] Error preparing data for projected volume kube-api-access-t7f72 for pod openshift-network-diagnostics/network-check-target-dk2hl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:06.251225 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:06.251093    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f0563df-380a-41eb-bb44-902bf8ceb4f6-kube-api-access-t7f72 podName:3f0563df-380a-41eb-bb44-902bf8ceb4f6 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:10.2510758 +0000 UTC m=+10.134602243 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-t7f72" (UniqueName: "kubernetes.io/projected/3f0563df-380a-41eb-bb44-902bf8ceb4f6-kube-api-access-t7f72") pod "network-check-target-dk2hl" (UID: "3f0563df-380a-41eb-bb44-902bf8ceb4f6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:06.251550 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:06.251466    2522 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:06.251550 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:06.251517    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24ec319c-fec4-4b8b-bcb6-6331ec8574fa-original-pull-secret podName:24ec319c-fec4-4b8b-bcb6-6331ec8574fa nodeName:}" failed. No retries permitted until 2026-09-18 19:14:10.251502283 +0000 UTC m=+10.135028725 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/24ec319c-fec4-4b8b-bcb6-6331ec8574fa-original-pull-secret") pod "global-pull-secret-syncer-g679p" (UID: "24ec319c-fec4-4b8b-bcb6-6331ec8574fa") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:06.719418 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:06.716481    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dk2hl"
Sep 18 19:14:06.719418 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:06.716610    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dk2hl" podUID="3f0563df-380a-41eb-bb44-902bf8ceb4f6"
Sep 18 19:14:06.719418 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:06.716966    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5j97v"
Sep 18 19:14:06.719418 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:06.717058    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5j97v" podUID="f516e39e-b125-4617-94ce-7f0dccfae7bb"
Sep 18 19:14:07.716603 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:07.716572    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g679p"
Sep 18 19:14:07.716787 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:07.716704    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g679p" podUID="24ec319c-fec4-4b8b-bcb6-6331ec8574fa"
Sep 18 19:14:08.718699 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:08.716634    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dk2hl"
Sep 18 19:14:08.718699 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:08.716763    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dk2hl" podUID="3f0563df-380a-41eb-bb44-902bf8ceb4f6"
Sep 18 19:14:08.718699 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:08.717159    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5j97v"
Sep 18 19:14:08.718699 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:08.717254    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5j97v" podUID="f516e39e-b125-4617-94ce-7f0dccfae7bb"
Sep 18 19:14:09.717046 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:09.716500    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g679p"
Sep 18 19:14:09.717046 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:09.716655    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g679p" podUID="24ec319c-fec4-4b8b-bcb6-6331ec8574fa"
Sep 18 19:14:10.185937 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:10.185857    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f516e39e-b125-4617-94ce-7f0dccfae7bb-metrics-certs\") pod \"network-metrics-daemon-5j97v\" (UID: \"f516e39e-b125-4617-94ce-7f0dccfae7bb\") " pod="openshift-multus/network-metrics-daemon-5j97v"
Sep 18 19:14:10.186489 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:10.185996    2522 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:10.186489 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:10.186064    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f516e39e-b125-4617-94ce-7f0dccfae7bb-metrics-certs podName:f516e39e-b125-4617-94ce-7f0dccfae7bb nodeName:}" failed. No retries permitted until 2026-09-18 19:14:18.186044369 +0000 UTC m=+18.069570827 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f516e39e-b125-4617-94ce-7f0dccfae7bb-metrics-certs") pod "network-metrics-daemon-5j97v" (UID: "f516e39e-b125-4617-94ce-7f0dccfae7bb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:10.287222 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:10.286665    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7f72\" (UniqueName: \"kubernetes.io/projected/3f0563df-380a-41eb-bb44-902bf8ceb4f6-kube-api-access-t7f72\") pod \"network-check-target-dk2hl\" (UID: \"3f0563df-380a-41eb-bb44-902bf8ceb4f6\") " pod="openshift-network-diagnostics/network-check-target-dk2hl"
Sep 18 19:14:10.287222 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:10.286745    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24ec319c-fec4-4b8b-bcb6-6331ec8574fa-original-pull-secret\") pod \"global-pull-secret-syncer-g679p\" (UID: \"24ec319c-fec4-4b8b-bcb6-6331ec8574fa\") " pod="kube-system/global-pull-secret-syncer-g679p"
Sep 18 19:14:10.287222 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:10.286827    2522 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:10.287222 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:10.286830    2522 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:14:10.287222 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:10.286852    2522 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:14:10.287222 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:10.286883    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24ec319c-fec4-4b8b-bcb6-6331ec8574fa-original-pull-secret podName:24ec319c-fec4-4b8b-bcb6-6331ec8574fa nodeName:}" failed. No retries permitted until 2026-09-18 19:14:18.286865205 +0000 UTC m=+18.170391658 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/24ec319c-fec4-4b8b-bcb6-6331ec8574fa-original-pull-secret") pod "global-pull-secret-syncer-g679p" (UID: "24ec319c-fec4-4b8b-bcb6-6331ec8574fa") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:10.287222 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:10.286894    2522 projected.go:196] Error preparing data for projected volume kube-api-access-t7f72 for pod openshift-network-diagnostics/network-check-target-dk2hl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:10.287222 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:10.286928    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f0563df-380a-41eb-bb44-902bf8ceb4f6-kube-api-access-t7f72 podName:3f0563df-380a-41eb-bb44-902bf8ceb4f6 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:18.286918708 +0000 UTC m=+18.170445153 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-t7f72" (UniqueName: "kubernetes.io/projected/3f0563df-380a-41eb-bb44-902bf8ceb4f6-kube-api-access-t7f72") pod "network-check-target-dk2hl" (UID: "3f0563df-380a-41eb-bb44-902bf8ceb4f6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:10.717256 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:10.717159    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5j97v"
Sep 18 19:14:10.717446 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:10.717289    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5j97v" podUID="f516e39e-b125-4617-94ce-7f0dccfae7bb"
Sep 18 19:14:10.717446 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:10.717332    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dk2hl"
Sep 18 19:14:10.717446 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:10.717396    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dk2hl" podUID="3f0563df-380a-41eb-bb44-902bf8ceb4f6"
Sep 18 19:14:11.715923 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:11.715893    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g679p"
Sep 18 19:14:11.716375 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:11.716006    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g679p" podUID="24ec319c-fec4-4b8b-bcb6-6331ec8574fa"
Sep 18 19:14:12.715753 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:12.715715    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dk2hl"
Sep 18 19:14:12.715935 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:12.715848    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dk2hl" podUID="3f0563df-380a-41eb-bb44-902bf8ceb4f6"
Sep 18 19:14:12.715935 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:12.715915    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5j97v"
Sep 18 19:14:12.716327 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:12.716039    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5j97v" podUID="f516e39e-b125-4617-94ce-7f0dccfae7bb"
Sep 18 19:14:13.716638 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:13.716539    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g679p"
Sep 18 19:14:13.716982 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:13.716666    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g679p" podUID="24ec319c-fec4-4b8b-bcb6-6331ec8574fa"
Sep 18 19:14:14.716208 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:14.716174    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dk2hl"
Sep 18 19:14:14.716364 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:14.716186    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5j97v"
Sep 18 19:14:14.716364 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:14.716309    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dk2hl" podUID="3f0563df-380a-41eb-bb44-902bf8ceb4f6"
Sep 18 19:14:14.716364 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:14.716351    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5j97v" podUID="f516e39e-b125-4617-94ce-7f0dccfae7bb"
Sep 18 19:14:15.715745 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:15.715710    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g679p"
Sep 18 19:14:15.716200 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:15.715837    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g679p" podUID="24ec319c-fec4-4b8b-bcb6-6331ec8574fa"
Sep 18 19:14:16.716498 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:16.716454    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dk2hl"
Sep 18 19:14:16.716979 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:16.716463    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5j97v"
Sep 18 19:14:16.716979 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:16.716615    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dk2hl" podUID="3f0563df-380a-41eb-bb44-902bf8ceb4f6"
Sep 18 19:14:16.716979 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:16.716676    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5j97v" podUID="f516e39e-b125-4617-94ce-7f0dccfae7bb"
Sep 18 19:14:17.715830 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:17.715798    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g679p"
Sep 18 19:14:17.716032 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:17.715915    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g679p" podUID="24ec319c-fec4-4b8b-bcb6-6331ec8574fa"
Sep 18 19:14:18.241931 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:18.241893    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f516e39e-b125-4617-94ce-7f0dccfae7bb-metrics-certs\") pod \"network-metrics-daemon-5j97v\" (UID: \"f516e39e-b125-4617-94ce-7f0dccfae7bb\") " pod="openshift-multus/network-metrics-daemon-5j97v"
Sep 18 19:14:18.242387 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:18.242029    2522 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:18.242387 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:18.242106    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f516e39e-b125-4617-94ce-7f0dccfae7bb-metrics-certs podName:f516e39e-b125-4617-94ce-7f0dccfae7bb nodeName:}" failed. No retries permitted until 2026-09-18 19:14:34.242084212 +0000 UTC m=+34.125610658 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f516e39e-b125-4617-94ce-7f0dccfae7bb-metrics-certs") pod "network-metrics-daemon-5j97v" (UID: "f516e39e-b125-4617-94ce-7f0dccfae7bb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:18.342814 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:18.342770    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7f72\" (UniqueName: \"kubernetes.io/projected/3f0563df-380a-41eb-bb44-902bf8ceb4f6-kube-api-access-t7f72\") pod \"network-check-target-dk2hl\" (UID: \"3f0563df-380a-41eb-bb44-902bf8ceb4f6\") " pod="openshift-network-diagnostics/network-check-target-dk2hl"
Sep 18 19:14:18.343003 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:18.342834    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24ec319c-fec4-4b8b-bcb6-6331ec8574fa-original-pull-secret\") pod \"global-pull-secret-syncer-g679p\" (UID: \"24ec319c-fec4-4b8b-bcb6-6331ec8574fa\") " pod="kube-system/global-pull-secret-syncer-g679p"
Sep 18 19:14:18.343003 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:18.342955    2522 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:14:18.343003 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:18.342966    2522 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:18.343139 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:18.343040    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24ec319c-fec4-4b8b-bcb6-6331ec8574fa-original-pull-secret podName:24ec319c-fec4-4b8b-bcb6-6331ec8574fa nodeName:}" failed. No retries permitted until 2026-09-18 19:14:34.343023867 +0000 UTC m=+34.226550320 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/24ec319c-fec4-4b8b-bcb6-6331ec8574fa-original-pull-secret") pod "global-pull-secret-syncer-g679p" (UID: "24ec319c-fec4-4b8b-bcb6-6331ec8574fa") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:18.343139 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:18.342975    2522 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:14:18.343139 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:18.343085    2522 projected.go:196] Error preparing data for projected volume kube-api-access-t7f72 for pod openshift-network-diagnostics/network-check-target-dk2hl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:18.343139 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:18.343138    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f0563df-380a-41eb-bb44-902bf8ceb4f6-kube-api-access-t7f72 podName:3f0563df-380a-41eb-bb44-902bf8ceb4f6 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:34.343122996 +0000 UTC m=+34.226649442 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-t7f72" (UniqueName: "kubernetes.io/projected/3f0563df-380a-41eb-bb44-902bf8ceb4f6-kube-api-access-t7f72") pod "network-check-target-dk2hl" (UID: "3f0563df-380a-41eb-bb44-902bf8ceb4f6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:18.716114 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:18.715767    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5j97v"
Sep 18 19:14:18.716114 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:18.715906    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dk2hl"
Sep 18 19:14:18.716114 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:18.715907    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5j97v" podUID="f516e39e-b125-4617-94ce-7f0dccfae7bb"
Sep 18 19:14:18.716114 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:18.715996    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dk2hl" podUID="3f0563df-380a-41eb-bb44-902bf8ceb4f6"
Sep 18 19:14:19.715677 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:19.715643    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g679p"
Sep 18 19:14:19.716083 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:19.715772    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g679p" podUID="24ec319c-fec4-4b8b-bcb6-6331ec8574fa"
Sep 18 19:14:20.716861 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:20.716828    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dk2hl"
Sep 18 19:14:20.717334 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:20.716922    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dk2hl" podUID="3f0563df-380a-41eb-bb44-902bf8ceb4f6"
Sep 18 19:14:20.717334 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:20.716971    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5j97v"
Sep 18 19:14:20.717334 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:20.717051    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5j97v" podUID="f516e39e-b125-4617-94ce-7f0dccfae7bb"
Sep 18 19:14:21.716429 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:21.716152    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g679p"
Sep 18 19:14:21.716522 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:21.716504    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g679p" podUID="24ec319c-fec4-4b8b-bcb6-6331ec8574fa"
Sep 18 19:14:21.771764 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:21.771741    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-d8dg5" event={"ID":"f5f32ffd-68fe-47ad-9b94-2d59b1fbe3e7","Type":"ContainerStarted","Data":"e6fe4b962b252feda6729b084d3940473c9873454cd913f51caf59b7b06c8aae"}
Sep 18 19:14:21.773336 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:21.773312    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zp9rb" event={"ID":"72118d02-4cb7-4dd4-bec1-1de6ba1614b1","Type":"ContainerStarted","Data":"eb30da15c23fcabfd40ef618b63720e55792e2356d3ffe7b3693ffa97f3c7b36"}
Sep 18 19:14:21.774891 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:21.774866    2522 generic.go:356] "Generic (PLEG): container finished" podID="31c0bde74e3407e1ddada35fdb2a70e7" containerID="172fc6661299b1b1d7fe43c1e4f3636ed0acc685d2f38fdb0553fe6bbd91315e" exitCode=0
Sep 18 19:14:21.774965 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:21.774940    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-221.ec2.internal" event={"ID":"31c0bde74e3407e1ddada35fdb2a70e7","Type":"ContainerDied","Data":"172fc6661299b1b1d7fe43c1e4f3636ed0acc685d2f38fdb0553fe6bbd91315e"}
Sep 18 19:14:21.776461 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:21.776443    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-221.ec2.internal" event={"ID":"523614729ae9b688582b641ea2fafde6","Type":"ContainerStarted","Data":"d2f59d1eeb818233001db60d82e0be5659339e8a2f200570ccb2b119455a3d58"}
Sep 18 19:14:21.779144 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:21.779090    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv" event={"ID":"eda05f97-2fcc-4f43-af17-9f5d514486bf","Type":"ContainerStarted","Data":"41ecdfb7da794f876179934660b55f8aa1991ccb1b3033381e5975deea46f38d"}
Sep 18 19:14:21.779193 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:21.779148    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv" event={"ID":"eda05f97-2fcc-4f43-af17-9f5d514486bf","Type":"ContainerStarted","Data":"02d3f0bd5bb763f440e4a4576f90f86a9acc4114dca6cb565edd0f1bec02a1d6"}
Sep 18 19:14:21.779193 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:21.779182    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv" event={"ID":"eda05f97-2fcc-4f43-af17-9f5d514486bf","Type":"ContainerStarted","Data":"d9de4656f59bda2a28f352f4f007deef425d44f0cb66a92a691b24ecfbcf07c8"}
Sep 18 19:14:21.779284 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:21.779218    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv" event={"ID":"eda05f97-2fcc-4f43-af17-9f5d514486bf","Type":"ContainerStarted","Data":"fdb914ed6e67a3ebfdbaf6dd8e4836f9529ae5effcd67b10fda005f9f8e1ca4e"}
Sep 18 19:14:21.780749 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:21.780723    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-w4tjm" event={"ID":"5b796f3a-2d43-47a8-8d73-a34a2af2d62b","Type":"ContainerStarted","Data":"92081138726b543f7d806d31f4d07791e5baf885240cf8f0919473de965ed09b"}
Sep 18 19:14:21.782229 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:21.782206    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cql5" event={"ID":"e8cd8a19-7cbb-4585-b5b3-6d940ebe41c6","Type":"ContainerStarted","Data":"27539d503f1c6579cd365e321c2a04d15e2f56fea8d575bd5502f27fb0d0d8c9"}
Sep 18 19:14:21.783902 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:21.783874    2522 generic.go:356] "Generic (PLEG): container finished" podID="6c2a6c5f-990b-4ceb-9308-f7a6e29e22a2" containerID="f2810cc2eecf2fd4acf4fe6c9c2d0798b29a0547ae71a9af30bb38fb371214a5" exitCode=0
Sep 18 19:14:21.783979 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:21.783950    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jrg4l" event={"ID":"6c2a6c5f-990b-4ceb-9308-f7a6e29e22a2","Type":"ContainerDied","Data":"f2810cc2eecf2fd4acf4fe6c9c2d0798b29a0547ae71a9af30bb38fb371214a5"}
Sep 18 19:14:21.785414 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:21.785388    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hmfqw" event={"ID":"438a5295-6f51-49ed-99d1-96a8fe037805","Type":"ContainerStarted","Data":"72c90f7a6d81b313c0f3df39d5eb6631a78cd6bc8b6bce27eddf7a62bc9b7e31"}
Sep 18 19:14:21.786879 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:21.786858    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-s7ckz" event={"ID":"e31f28a0-45fc-45ae-a307-578e281565a9","Type":"ContainerStarted","Data":"f3bfbe292ccb69c7986d2323ffdffcd25e2e19d46a70fabc7c2c989c99272136"}
Sep 18 19:14:21.802518 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:21.802468    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-d8dg5" podStartSLOduration=3.895637798 podStartE2EDuration="21.802453094s" totalImagesPullingTime="17.906815296s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:21.797229965 +0000 UTC m=+21.680756429" watchObservedRunningTime="2026-09-18 19:14:21.802453094 +0000 UTC m=+21.685979556"
Sep 18 19:14:21.817160 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:21.817124    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-hmfqw" podStartSLOduration=7.465772252 podStartE2EDuration="21.817113246s" totalImagesPullingTime="14.351340994s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:21.816852383 +0000 UTC m=+21.700378846" watchObservedRunningTime="2026-09-18 19:14:21.817113246 +0000 UTC m=+21.700639738"
Sep 18 19:14:21.834120 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:21.834082    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-221.ec2.internal" podStartSLOduration=20.834070407 podStartE2EDuration="20.834070407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:21.833940583 +0000 UTC m=+21.717467057" watchObservedRunningTime="2026-09-18 19:14:21.834070407 +0000 UTC m=+21.717596869"
Sep 18 19:14:21.848213 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:21.848179    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-s7ckz" podStartSLOduration=3.968012646 podStartE2EDuration="21.848168905s" totalImagesPullingTime="17.880156259s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:14:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:21.848062613 +0000 UTC m=+21.731589087" watchObservedRunningTime="2026-09-18 19:14:21.848168905 +0000 UTC m=+21.731695366"
Sep 18 19:14:21.909948 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:21.909908    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-w4tjm" podStartSLOduration=6.59296518 podStartE2EDuration="20.909895253s" totalImagesPullingTime="14.316930073s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:21.884352517 +0000 UTC m=+21.767878979" watchObservedRunningTime="2026-09-18 19:14:21.909895253 +0000 UTC m=+21.793421712"
Sep 18 19:14:21.910280 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:21.910261    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zp9rb" podStartSLOduration=4.011413437 podStartE2EDuration="21.910256671s" totalImagesPullingTime="17.898843234s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:21.909378564 +0000 UTC m=+21.792905025" watchObservedRunningTime="2026-09-18 19:14:21.910256671 +0000 UTC m=+21.793783132"
Sep 18 19:14:22.716220 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:22.716194    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dk2hl"
Sep 18 19:14:22.716374 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:22.716196    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5j97v"
Sep 18 19:14:22.716439 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:22.716401    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5j97v" podUID="f516e39e-b125-4617-94ce-7f0dccfae7bb"
Sep 18 19:14:22.716439 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:22.716307    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dk2hl" podUID="3f0563df-380a-41eb-bb44-902bf8ceb4f6"
Sep 18 19:14:22.776463 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:22.776283    2522 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 18 19:14:22.793290 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:22.793261    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv" event={"ID":"eda05f97-2fcc-4f43-af17-9f5d514486bf","Type":"ContainerStarted","Data":"c0aafcbe2ac77d3dd810062a21370d61b0ff38cd40115e3fc364ec1de0afc299"}
Sep 18 19:14:22.793394 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:22.793349    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv" event={"ID":"eda05f97-2fcc-4f43-af17-9f5d514486bf","Type":"ContainerStarted","Data":"1e1ccdb9664ccaec879c559648f7d7e172af2f861506a5a53af9d12314e7a5a0"}
Sep 18 19:14:22.795062 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:22.795042    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cql5" event={"ID":"e8cd8a19-7cbb-4585-b5b3-6d940ebe41c6","Type":"ContainerStarted","Data":"1de7786bb5fa665b33f349cf7814f78e051c4e3d8f41423e0d223f491ab91f9b"}
Sep 18 19:14:22.796620 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:22.796597    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-844d4" event={"ID":"bfd5ec3a-8ede-49e3-b8e2-00cc159b65aa","Type":"ContainerStarted","Data":"b798df6dbbe2e54ec113b915b1dd32952c53749537ceb47afc2987d4b3c45b73"}
Sep 18 19:14:22.837739 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:22.837694    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-844d4" podStartSLOduration=4.995771475 podStartE2EDuration="22.837674165s" totalImagesPullingTime="17.84190269s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:22.819516247 +0000 UTC m=+22.703042710" watchObservedRunningTime="2026-09-18 19:14:22.837674165 +0000 UTC m=+22.721200627"
Sep 18 19:14:23.580257 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:23.580169    2522 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-18T19:14:22.776463144Z","UUID":"917308bc-3006-40b2-91b3-26f95291d52d","Handler":null,"Name":"","Endpoint":""}
Sep 18 19:14:23.581999 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:23.581981    2522 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 18 19:14:23.582132 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:23.582008    2522 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 18 19:14:23.716371 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:23.716343    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g679p"
Sep 18 19:14:23.716530 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:23.716451    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g679p" podUID="24ec319c-fec4-4b8b-bcb6-6331ec8574fa"
Sep 18 19:14:23.800729 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:23.800646    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-221.ec2.internal" event={"ID":"31c0bde74e3407e1ddada35fdb2a70e7","Type":"ContainerStarted","Data":"401f00ec9f77e748e3f5f3f7414099e6384f614e47a8650416f7f87577e4273a"}
Sep 18 19:14:23.803087 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:23.803064    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cql5" event={"ID":"e8cd8a19-7cbb-4585-b5b3-6d940ebe41c6","Type":"ContainerStarted","Data":"9b09ac0ea0a2bb09eadacb8cfbb3432e899695571d95c06f13ca192cb42a8bc2"}
Sep 18 19:14:23.864195 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:23.864153    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-221.ec2.internal" podStartSLOduration=22.864138749 podStartE2EDuration="22.864138749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:23.838775797 +0000 UTC m=+23.722302259" watchObservedRunningTime="2026-09-18 19:14:23.864138749 +0000 UTC m=+23.747665210"
Sep 18 19:14:24.716103 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:24.716061    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dk2hl"
Sep 18 19:14:24.716359 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:24.716190    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dk2hl" podUID="3f0563df-380a-41eb-bb44-902bf8ceb4f6"
Sep 18 19:14:24.716359 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:24.716252    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5j97v"
Sep 18 19:14:24.716546 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:24.716365    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5j97v" podUID="f516e39e-b125-4617-94ce-7f0dccfae7bb"
Sep 18 19:14:24.808877 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:24.808842    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv" event={"ID":"eda05f97-2fcc-4f43-af17-9f5d514486bf","Type":"ContainerStarted","Data":"027d07316c9953e1ffc446bc5583c2a405abbb1da50f08925186cfa2edbe055d"}
Sep 18 19:14:25.716078 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:25.716049    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g679p"
Sep 18 19:14:25.716231 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:25.716142    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g679p" podUID="24ec319c-fec4-4b8b-bcb6-6331ec8574fa"
Sep 18 19:14:25.998836 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:25.998814    2522 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-s7ckz"
Sep 18 19:14:25.999507 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:25.999488    2522 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-s7ckz"
Sep 18 19:14:26.039320 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:26.039276    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cql5" podStartSLOduration=5.423593419 podStartE2EDuration="25.039260458s" totalImagesPullingTime="19.615667039s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:23.864770052 +0000 UTC m=+23.748296517" watchObservedRunningTime="2026-09-18 19:14:26.039260458 +0000 UTC m=+25.922786910"
Sep 18 19:14:26.716551 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:26.716382    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5j97v"
Sep 18 19:14:26.716551 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:26.716397    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dk2hl"
Sep 18 19:14:26.716731 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:26.716653    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5j97v" podUID="f516e39e-b125-4617-94ce-7f0dccfae7bb"
Sep 18 19:14:26.716775 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:26.716762    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dk2hl" podUID="3f0563df-380a-41eb-bb44-902bf8ceb4f6"
Sep 18 19:14:26.815102 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:26.815077    2522 generic.go:356] "Generic (PLEG): container finished" podID="6c2a6c5f-990b-4ceb-9308-f7a6e29e22a2" containerID="e007c633f641572d5b3ce58ae4831ca85b59215962674ca06091498c6dafa937" exitCode=0
Sep 18 19:14:26.815301 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:26.815167    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jrg4l" event={"ID":"6c2a6c5f-990b-4ceb-9308-f7a6e29e22a2","Type":"ContainerDied","Data":"e007c633f641572d5b3ce58ae4831ca85b59215962674ca06091498c6dafa937"}
Sep 18 19:14:26.819040 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:26.819011    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv" event={"ID":"eda05f97-2fcc-4f43-af17-9f5d514486bf","Type":"ContainerStarted","Data":"10ea6292daf0ea0a0b2495f06ccc95d1c45b3db8480ea99d3a5dd3df40f472d6"}
Sep 18 19:14:26.819345 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:26.819315    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-s7ckz"
Sep 18 19:14:26.819601 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:26.819588    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-s7ckz"
Sep 18 19:14:26.902395 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:26.902353    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv" podStartSLOduration=7.614688942 podStartE2EDuration="25.902340443s" totalImagesPullingTime="18.287651501s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:26.902071464 +0000 UTC m=+26.785597925" watchObservedRunningTime="2026-09-18 19:14:26.902340443 +0000 UTC m=+26.785866925"
Sep 18 19:14:27.533746 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:27.533722    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:27.715760 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:27.715729    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g679p"
Sep 18 19:14:27.715893 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:27.715868    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g679p" podUID="24ec319c-fec4-4b8b-bcb6-6331ec8574fa"
Sep 18 19:14:27.823505 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:27.823429    2522 generic.go:356] "Generic (PLEG): container finished" podID="6c2a6c5f-990b-4ceb-9308-f7a6e29e22a2" containerID="c430fbd6bc9f0d1651edff3737e74392c221e5214effdd80f99318aef786d328" exitCode=0
Sep 18 19:14:27.823659 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:27.823507    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jrg4l" event={"ID":"6c2a6c5f-990b-4ceb-9308-f7a6e29e22a2","Type":"ContainerDied","Data":"c430fbd6bc9f0d1651edff3737e74392c221e5214effdd80f99318aef786d328"}
Sep 18 19:14:27.824713 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:27.824248    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:27.824713 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:27.824323    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:27.841148 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:27.841119    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:27.841634 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:27.841613    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:28.209938 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:28.209428    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dk2hl"]
Sep 18 19:14:28.209938 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:28.209553    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dk2hl"
Sep 18 19:14:28.209938 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:28.209669    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dk2hl" podUID="3f0563df-380a-41eb-bb44-902bf8ceb4f6"
Sep 18 19:14:28.212767 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:28.212717    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5j97v"]
Sep 18 19:14:28.212972 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:28.212880    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5j97v"
Sep 18 19:14:28.212972 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:28.212984    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5j97v" podUID="f516e39e-b125-4617-94ce-7f0dccfae7bb"
Sep 18 19:14:28.213148 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:28.213050    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-g679p"]
Sep 18 19:14:28.213200 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:28.213147    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g679p"
Sep 18 19:14:28.213316 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:28.213274    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g679p" podUID="24ec319c-fec4-4b8b-bcb6-6331ec8574fa"
Sep 18 19:14:28.827847 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:28.827813    2522 generic.go:356] "Generic (PLEG): container finished" podID="6c2a6c5f-990b-4ceb-9308-f7a6e29e22a2" containerID="73822e70beda66929d33199845a40d243017c59562bdb6faa536e3b1582e52a4" exitCode=0
Sep 18 19:14:28.828614 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:28.827891    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jrg4l" event={"ID":"6c2a6c5f-990b-4ceb-9308-f7a6e29e22a2","Type":"ContainerDied","Data":"73822e70beda66929d33199845a40d243017c59562bdb6faa536e3b1582e52a4"}
Sep 18 19:14:29.716014 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:29.715978    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5j97v"
Sep 18 19:14:29.716014 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:29.716006    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dk2hl"
Sep 18 19:14:29.716014 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:29.716012    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g679p"
Sep 18 19:14:29.716260 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:29.716187    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5j97v" podUID="f516e39e-b125-4617-94ce-7f0dccfae7bb"
Sep 18 19:14:29.716313 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:29.716263    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dk2hl" podUID="3f0563df-380a-41eb-bb44-902bf8ceb4f6"
Sep 18 19:14:29.716370 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:29.716352    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g679p" podUID="24ec319c-fec4-4b8b-bcb6-6331ec8574fa"
Sep 18 19:14:31.716086 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:31.715901    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dk2hl"
Sep 18 19:14:31.716472 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:31.715898    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g679p"
Sep 18 19:14:31.716472 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:31.715956    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5j97v"
Sep 18 19:14:31.716472 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:31.716184    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dk2hl" podUID="3f0563df-380a-41eb-bb44-902bf8ceb4f6"
Sep 18 19:14:31.716472 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:31.716293    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5j97v" podUID="f516e39e-b125-4617-94ce-7f0dccfae7bb"
Sep 18 19:14:31.716472 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:31.716353    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g679p" podUID="24ec319c-fec4-4b8b-bcb6-6331ec8574fa"
Sep 18 19:14:33.715807 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:33.715783    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5j97v"
Sep 18 19:14:33.716196 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:33.715781    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dk2hl"
Sep 18 19:14:33.716196 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:33.715914    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5j97v" podUID="f516e39e-b125-4617-94ce-7f0dccfae7bb"
Sep 18 19:14:33.716196 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:33.715785    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g679p"
Sep 18 19:14:33.716196 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:33.715937    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dk2hl" podUID="3f0563df-380a-41eb-bb44-902bf8ceb4f6"
Sep 18 19:14:33.716196 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:33.716001    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g679p" podUID="24ec319c-fec4-4b8b-bcb6-6331ec8574fa"
Sep 18 19:14:33.840750 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:33.840690    2522 generic.go:356] "Generic (PLEG): container finished" podID="6c2a6c5f-990b-4ceb-9308-f7a6e29e22a2" containerID="baeb3cceff87e8a731a1bf19eb7533970d61a0316ddf72efb69329fa58b7c1f1" exitCode=0
Sep 18 19:14:33.840750 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:33.840727    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jrg4l" event={"ID":"6c2a6c5f-990b-4ceb-9308-f7a6e29e22a2","Type":"ContainerDied","Data":"baeb3cceff87e8a731a1bf19eb7533970d61a0316ddf72efb69329fa58b7c1f1"}
Sep 18 19:14:33.862425 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:33.862410    2522 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-221.ec2.internal" event="NodeReady"
Sep 18 19:14:33.862510 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:33.862491    2522 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 18 19:14:34.004392 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.004368    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-sszkh"]
Sep 18 19:14:34.015231 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.015210    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-sszkh"
Sep 18 19:14:34.022932 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.022912    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 18 19:14:34.023160 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.023140    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-z9rbg\""
Sep 18 19:14:34.030755 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.030738    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7b77b6b64d-dgr86"]
Sep 18 19:14:34.033619 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.033602    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 18 19:14:34.042932 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.042916    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b77b6b64d-dgr86"
Sep 18 19:14:34.046247 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.046230    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 18 19:14:34.046332 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.046288    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 18 19:14:34.046394 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.046328    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 18 19:14:34.046394 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.046357    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-jqgxp\""
Sep 18 19:14:34.061112 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.061087    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-sszkh"]
Sep 18 19:14:34.063766 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.063749    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 18 19:14:34.088822 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.088801    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7b77b6b64d-dgr86"]
Sep 18 19:14:34.089591 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.089574    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-pmrmr"]
Sep 18 19:14:34.098286 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.098243    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pmrmr"
Sep 18 19:14:34.102631 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.102614    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 18 19:14:34.102709 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.102614    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 18 19:14:34.103632 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.103617    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-pmm6s\""
Sep 18 19:14:34.107133 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.107118    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 18 19:14:34.125657 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.125638    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pmrmr"]
Sep 18 19:14:34.162521 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.162499    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxcmr\" (UniqueName: \"kubernetes.io/projected/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-kube-api-access-rxcmr\") pod \"image-registry-7b77b6b64d-dgr86\" (UID: \"63dfbfc5-e1e8-4e79-8df9-d47e16ba5289\") " pod="openshift-image-registry/image-registry-7b77b6b64d-dgr86"
Sep 18 19:14:34.162642 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.162524    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-registry-certificates\") pod \"image-registry-7b77b6b64d-dgr86\" (UID: \"63dfbfc5-e1e8-4e79-8df9-d47e16ba5289\") " pod="openshift-image-registry/image-registry-7b77b6b64d-dgr86"
Sep 18 19:14:34.162642 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.162545    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-bound-sa-token\") pod \"image-registry-7b77b6b64d-dgr86\" (UID: \"63dfbfc5-e1e8-4e79-8df9-d47e16ba5289\") " pod="openshift-image-registry/image-registry-7b77b6b64d-dgr86"
Sep 18 19:14:34.162726 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.162655    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-image-registry-private-configuration\") pod \"image-registry-7b77b6b64d-dgr86\" (UID: \"63dfbfc5-e1e8-4e79-8df9-d47e16ba5289\") " pod="openshift-image-registry/image-registry-7b77b6b64d-dgr86"
Sep 18 19:14:34.162726 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.162681    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-installation-pull-secrets\") pod \"image-registry-7b77b6b64d-dgr86\" (UID: \"63dfbfc5-e1e8-4e79-8df9-d47e16ba5289\") " pod="openshift-image-registry/image-registry-7b77b6b64d-dgr86"
Sep 18 19:14:34.162726 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.162703    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/00d2da2b-3370-4f0a-8104-9bc429576c69-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-sszkh\" (UID: \"00d2da2b-3370-4f0a-8104-9bc429576c69\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-sszkh"
Sep 18 19:14:34.162825 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.162726    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-registry-tls\") pod \"image-registry-7b77b6b64d-dgr86\" (UID: \"63dfbfc5-e1e8-4e79-8df9-d47e16ba5289\") " pod="openshift-image-registry/image-registry-7b77b6b64d-dgr86"
Sep 18 19:14:34.162825 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.162761    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/00d2da2b-3370-4f0a-8104-9bc429576c69-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-sszkh\" (UID: \"00d2da2b-3370-4f0a-8104-9bc429576c69\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-sszkh"
Sep 18 19:14:34.162825 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.162787    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-trusted-ca\") pod \"image-registry-7b77b6b64d-dgr86\" (UID: \"63dfbfc5-e1e8-4e79-8df9-d47e16ba5289\") " pod="openshift-image-registry/image-registry-7b77b6b64d-dgr86"
Sep 18 19:14:34.162825 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.162811    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-ca-trust-extracted\") pod \"image-registry-7b77b6b64d-dgr86\" (UID: \"63dfbfc5-e1e8-4e79-8df9-d47e16ba5289\") " pod="openshift-image-registry/image-registry-7b77b6b64d-dgr86"
Sep 18 19:14:34.174239 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.174217    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-mrd75"]
Sep 18 19:14:34.187848 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.187830    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mrd75"
Sep 18 19:14:34.192266 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.192248    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 18 19:14:34.192357 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.192280    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 18 19:14:34.192417 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.192404    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6m5tz\""
Sep 18 19:14:34.263386 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.263363    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f516e39e-b125-4617-94ce-7f0dccfae7bb-metrics-certs\") pod \"network-metrics-daemon-5j97v\" (UID: \"f516e39e-b125-4617-94ce-7f0dccfae7bb\") " pod="openshift-multus/network-metrics-daemon-5j97v"
Sep 18 19:14:34.263502 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.263390    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-image-registry-private-configuration\") pod \"image-registry-7b77b6b64d-dgr86\" (UID: \"63dfbfc5-e1e8-4e79-8df9-d47e16ba5289\") " pod="openshift-image-registry/image-registry-7b77b6b64d-dgr86"
Sep 18 19:14:34.263502 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.263408    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-installation-pull-secrets\") pod \"image-registry-7b77b6b64d-dgr86\" (UID: \"63dfbfc5-e1e8-4e79-8df9-d47e16ba5289\") " pod="openshift-image-registry/image-registry-7b77b6b64d-dgr86"
Sep 18 19:14:34.263502 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.263424    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/72d866b0-9634-442e-bd20-d90321a00452-metrics-tls\") pod \"dns-default-mrd75\" (UID: \"72d866b0-9634-442e-bd20-d90321a00452\") " pod="openshift-dns/dns-default-mrd75"
Sep 18 19:14:34.263502 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.263441    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/00d2da2b-3370-4f0a-8104-9bc429576c69-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-sszkh\" (UID: \"00d2da2b-3370-4f0a-8104-9bc429576c69\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-sszkh"
Sep 18 19:14:34.263679 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:34.263535    2522 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:34.263679 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.263594    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-registry-tls\") pod \"image-registry-7b77b6b64d-dgr86\" (UID: \"63dfbfc5-e1e8-4e79-8df9-d47e16ba5289\") " pod="openshift-image-registry/image-registry-7b77b6b64d-dgr86"
Sep 18 19:14:34.263679 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:34.263609    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f516e39e-b125-4617-94ce-7f0dccfae7bb-metrics-certs podName:f516e39e-b125-4617-94ce-7f0dccfae7bb nodeName:}" failed. No retries permitted until 2026-09-18 19:15:06.263592258 +0000 UTC m=+66.147118714 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f516e39e-b125-4617-94ce-7f0dccfae7bb-metrics-certs") pod "network-metrics-daemon-5j97v" (UID: "f516e39e-b125-4617-94ce-7f0dccfae7bb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:34.263679 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.263639    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92c79\" (UniqueName: \"kubernetes.io/projected/72d866b0-9634-442e-bd20-d90321a00452-kube-api-access-92c79\") pod \"dns-default-mrd75\" (UID: \"72d866b0-9634-442e-bd20-d90321a00452\") " pod="openshift-dns/dns-default-mrd75"
Sep 18 19:14:34.263679 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:34.263663    2522 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:14:34.263679 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:34.263679    2522 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b77b6b64d-dgr86: secret "image-registry-tls" not found
Sep 18 19:14:34.263970 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.263667    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/47bbbb88-fe11-4227-bddc-4e124994d36a-cert\") pod \"ingress-canary-pmrmr\" (UID: \"47bbbb88-fe11-4227-bddc-4e124994d36a\") " pod="openshift-ingress-canary/ingress-canary-pmrmr"
Sep 18 19:14:34.263970 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.263738    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/00d2da2b-3370-4f0a-8104-9bc429576c69-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-sszkh\" (UID: \"00d2da2b-3370-4f0a-8104-9bc429576c69\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-sszkh"
Sep 18 19:14:34.263970 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.263760    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-trusted-ca\") pod \"image-registry-7b77b6b64d-dgr86\" (UID: \"63dfbfc5-e1e8-4e79-8df9-d47e16ba5289\") " pod="openshift-image-registry/image-registry-7b77b6b64d-dgr86"
Sep 18 19:14:34.263970 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.263789    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-ca-trust-extracted\") pod \"image-registry-7b77b6b64d-dgr86\" (UID: \"63dfbfc5-e1e8-4e79-8df9-d47e16ba5289\") " pod="openshift-image-registry/image-registry-7b77b6b64d-dgr86"
Sep 18 19:14:34.263970 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.263812    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxcmr\" (UniqueName: \"kubernetes.io/projected/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-kube-api-access-rxcmr\") pod \"image-registry-7b77b6b64d-dgr86\" (UID: \"63dfbfc5-e1e8-4e79-8df9-d47e16ba5289\") " pod="openshift-image-registry/image-registry-7b77b6b64d-dgr86"
Sep 18 19:14:34.263970 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.263836    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/72d866b0-9634-442e-bd20-d90321a00452-config-volume\") pod \"dns-default-mrd75\" (UID: \"72d866b0-9634-442e-bd20-d90321a00452\") " pod="openshift-dns/dns-default-mrd75"
Sep 18 19:14:34.263970 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.263862    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-registry-certificates\") pod \"image-registry-7b77b6b64d-dgr86\" (UID: \"63dfbfc5-e1e8-4e79-8df9-d47e16ba5289\") " pod="openshift-image-registry/image-registry-7b77b6b64d-dgr86"
Sep 18 19:14:34.263970 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.263884    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72d866b0-9634-442e-bd20-d90321a00452-tmp-dir\") pod \"dns-default-mrd75\" (UID: \"72d866b0-9634-442e-bd20-d90321a00452\") " pod="openshift-dns/dns-default-mrd75"
Sep 18 19:14:34.263970 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:34.263898    2522 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 18 19:14:34.263970 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.263910    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-bound-sa-token\") pod \"image-registry-7b77b6b64d-dgr86\" (UID: \"63dfbfc5-e1e8-4e79-8df9-d47e16ba5289\") " pod="openshift-image-registry/image-registry-7b77b6b64d-dgr86"
Sep 18 19:14:34.263970 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.263933    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4bc4\" (UniqueName: \"kubernetes.io/projected/47bbbb88-fe11-4227-bddc-4e124994d36a-kube-api-access-z4bc4\") pod \"ingress-canary-pmrmr\" (UID: \"47bbbb88-fe11-4227-bddc-4e124994d36a\") " pod="openshift-ingress-canary/ingress-canary-pmrmr"
Sep 18 19:14:34.263970 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:34.263962    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00d2da2b-3370-4f0a-8104-9bc429576c69-networking-console-plugin-cert podName:00d2da2b-3370-4f0a-8104-9bc429576c69 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:34.763947484 +0000 UTC m=+34.647473933 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/00d2da2b-3370-4f0a-8104-9bc429576c69-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-sszkh" (UID: "00d2da2b-3370-4f0a-8104-9bc429576c69") : secret "networking-console-plugin-cert" not found
Sep 18 19:14:34.264554 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:34.264050    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-registry-tls podName:63dfbfc5-e1e8-4e79-8df9-d47e16ba5289 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:34.764036224 +0000 UTC m=+34.647562673 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-registry-tls") pod "image-registry-7b77b6b64d-dgr86" (UID: "63dfbfc5-e1e8-4e79-8df9-d47e16ba5289") : secret "image-registry-tls" not found
Sep 18 19:14:34.264554 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.264185    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/00d2da2b-3370-4f0a-8104-9bc429576c69-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-sszkh\" (UID: \"00d2da2b-3370-4f0a-8104-9bc429576c69\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-sszkh"
Sep 18 19:14:34.264554 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.264356    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-ca-trust-extracted\") pod \"image-registry-7b77b6b64d-dgr86\" (UID: \"63dfbfc5-e1e8-4e79-8df9-d47e16ba5289\") " pod="openshift-image-registry/image-registry-7b77b6b64d-dgr86"
Sep 18 19:14:34.264554 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.264440    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-registry-certificates\") pod \"image-registry-7b77b6b64d-dgr86\" (UID: \"63dfbfc5-e1e8-4e79-8df9-d47e16ba5289\") " pod="openshift-image-registry/image-registry-7b77b6b64d-dgr86"
Sep 18 19:14:34.264893 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.264872    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-trusted-ca\") pod \"image-registry-7b77b6b64d-dgr86\" (UID: \"63dfbfc5-e1e8-4e79-8df9-d47e16ba5289\") " pod="openshift-image-registry/image-registry-7b77b6b64d-dgr86"
Sep 18 19:14:34.267322 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.267305    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-image-registry-private-configuration\") pod \"image-registry-7b77b6b64d-dgr86\" (UID: \"63dfbfc5-e1e8-4e79-8df9-d47e16ba5289\") " pod="openshift-image-registry/image-registry-7b77b6b64d-dgr86"
Sep 18 19:14:34.267404 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.267319    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-installation-pull-secrets\") pod \"image-registry-7b77b6b64d-dgr86\" (UID: \"63dfbfc5-e1e8-4e79-8df9-d47e16ba5289\") " pod="openshift-image-registry/image-registry-7b77b6b64d-dgr86"
Sep 18 19:14:34.313906 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.313879    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-bound-sa-token\") pod \"image-registry-7b77b6b64d-dgr86\" (UID: \"63dfbfc5-e1e8-4e79-8df9-d47e16ba5289\") " pod="openshift-image-registry/image-registry-7b77b6b64d-dgr86"
Sep 18 19:14:34.318320 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.318302    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxcmr\" (UniqueName: \"kubernetes.io/projected/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-kube-api-access-rxcmr\") pod \"image-registry-7b77b6b64d-dgr86\" (UID: \"63dfbfc5-e1e8-4e79-8df9-d47e16ba5289\") " pod="openshift-image-registry/image-registry-7b77b6b64d-dgr86"
Sep 18 19:14:34.358227 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.358180    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mrd75"]
Sep 18 19:14:34.364813 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.364795    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/72d866b0-9634-442e-bd20-d90321a00452-metrics-tls\") pod \"dns-default-mrd75\" (UID: \"72d866b0-9634-442e-bd20-d90321a00452\") " pod="openshift-dns/dns-default-mrd75"
Sep 18 19:14:34.364865 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.364823    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24ec319c-fec4-4b8b-bcb6-6331ec8574fa-original-pull-secret\") pod \"global-pull-secret-syncer-g679p\" (UID: \"24ec319c-fec4-4b8b-bcb6-6331ec8574fa\") " pod="kube-system/global-pull-secret-syncer-g679p"
Sep 18 19:14:34.364865 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.364845    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92c79\" (UniqueName: \"kubernetes.io/projected/72d866b0-9634-442e-bd20-d90321a00452-kube-api-access-92c79\") pod \"dns-default-mrd75\" (UID: \"72d866b0-9634-442e-bd20-d90321a00452\") " pod="openshift-dns/dns-default-mrd75"
Sep 18 19:14:34.364865 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.364861    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/47bbbb88-fe11-4227-bddc-4e124994d36a-cert\") pod \"ingress-canary-pmrmr\" (UID: \"47bbbb88-fe11-4227-bddc-4e124994d36a\") " pod="openshift-ingress-canary/ingress-canary-pmrmr"
Sep 18 19:14:34.364961 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.364900    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/72d866b0-9634-442e-bd20-d90321a00452-config-volume\") pod \"dns-default-mrd75\" (UID: \"72d866b0-9634-442e-bd20-d90321a00452\") " pod="openshift-dns/dns-default-mrd75"
Sep 18 19:14:34.364961 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:34.364923    2522 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:14:34.364961 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:34.364940    2522 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:34.365063 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:34.364974    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72d866b0-9634-442e-bd20-d90321a00452-metrics-tls podName:72d866b0-9634-442e-bd20-d90321a00452 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:34.864958704 +0000 UTC m=+34.748485148 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/72d866b0-9634-442e-bd20-d90321a00452-metrics-tls") pod "dns-default-mrd75" (UID: "72d866b0-9634-442e-bd20-d90321a00452") : secret "dns-default-metrics-tls" not found
Sep 18 19:14:34.365063 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:34.364995    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24ec319c-fec4-4b8b-bcb6-6331ec8574fa-original-pull-secret podName:24ec319c-fec4-4b8b-bcb6-6331ec8574fa nodeName:}" failed. No retries permitted until 2026-09-18 19:15:06.364980025 +0000 UTC m=+66.248506465 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/24ec319c-fec4-4b8b-bcb6-6331ec8574fa-original-pull-secret") pod "global-pull-secret-syncer-g679p" (UID: "24ec319c-fec4-4b8b-bcb6-6331ec8574fa") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:34.365063 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.364926    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72d866b0-9634-442e-bd20-d90321a00452-tmp-dir\") pod \"dns-default-mrd75\" (UID: \"72d866b0-9634-442e-bd20-d90321a00452\") " pod="openshift-dns/dns-default-mrd75"
Sep 18 19:14:34.365063 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.365041    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4bc4\" (UniqueName: \"kubernetes.io/projected/47bbbb88-fe11-4227-bddc-4e124994d36a-kube-api-access-z4bc4\") pod \"ingress-canary-pmrmr\" (UID: \"47bbbb88-fe11-4227-bddc-4e124994d36a\") " pod="openshift-ingress-canary/ingress-canary-pmrmr"
Sep 18 19:14:34.365253 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.365079    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7f72\" (UniqueName: \"kubernetes.io/projected/3f0563df-380a-41eb-bb44-902bf8ceb4f6-kube-api-access-t7f72\") pod \"network-check-target-dk2hl\" (UID: \"3f0563df-380a-41eb-bb44-902bf8ceb4f6\") " pod="openshift-network-diagnostics/network-check-target-dk2hl"
Sep 18 19:14:34.365253 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.365189    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72d866b0-9634-442e-bd20-d90321a00452-tmp-dir\") pod \"dns-default-mrd75\" (UID: \"72d866b0-9634-442e-bd20-d90321a00452\") " pod="openshift-dns/dns-default-mrd75"
Sep 18 19:14:34.365253 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:34.365194    2522 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:14:34.365253 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:34.365211    2522 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:14:34.365253 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:34.365224    2522 projected.go:196] Error preparing data for projected volume kube-api-access-t7f72 for pod openshift-network-diagnostics/network-check-target-dk2hl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:34.365437 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:34.365269    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f0563df-380a-41eb-bb44-902bf8ceb4f6-kube-api-access-t7f72 podName:3f0563df-380a-41eb-bb44-902bf8ceb4f6 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:06.365256544 +0000 UTC m=+66.248782988 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-t7f72" (UniqueName: "kubernetes.io/projected/3f0563df-380a-41eb-bb44-902bf8ceb4f6-kube-api-access-t7f72") pod "network-check-target-dk2hl" (UID: "3f0563df-380a-41eb-bb44-902bf8ceb4f6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:34.365475 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.365439    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/72d866b0-9634-442e-bd20-d90321a00452-config-volume\") pod \"dns-default-mrd75\" (UID: \"72d866b0-9634-442e-bd20-d90321a00452\") " pod="openshift-dns/dns-default-mrd75"
Sep 18 19:14:34.366917 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.366900    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/47bbbb88-fe11-4227-bddc-4e124994d36a-cert\") pod \"ingress-canary-pmrmr\" (UID: \"47bbbb88-fe11-4227-bddc-4e124994d36a\") " pod="openshift-ingress-canary/ingress-canary-pmrmr"
Sep 18 19:14:34.395715 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.395692    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92c79\" (UniqueName: \"kubernetes.io/projected/72d866b0-9634-442e-bd20-d90321a00452-kube-api-access-92c79\") pod \"dns-default-mrd75\" (UID: \"72d866b0-9634-442e-bd20-d90321a00452\") " pod="openshift-dns/dns-default-mrd75"
Sep 18 19:14:34.399772 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.399756    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4bc4\" (UniqueName: \"kubernetes.io/projected/47bbbb88-fe11-4227-bddc-4e124994d36a-kube-api-access-z4bc4\") pod \"ingress-canary-pmrmr\" (UID: \"47bbbb88-fe11-4227-bddc-4e124994d36a\") " pod="openshift-ingress-canary/ingress-canary-pmrmr"
Sep 18 19:14:34.406277 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.406262    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pmrmr"
Sep 18 19:14:34.725832 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.725760    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pmrmr"]
Sep 18 19:14:34.729508 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:14:34.729489    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47bbbb88_fe11_4227_bddc_4e124994d36a.slice/crio-93723387befbf730033f8f2e8eb1fe7b0d534686c49ec95cddeb480b11589459 WatchSource:0}: Error finding container 93723387befbf730033f8f2e8eb1fe7b0d534686c49ec95cddeb480b11589459: Status 404 returned error can't find the container with id 93723387befbf730033f8f2e8eb1fe7b0d534686c49ec95cddeb480b11589459
Sep 18 19:14:34.768961 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.768938    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-registry-tls\") pod \"image-registry-7b77b6b64d-dgr86\" (UID: \"63dfbfc5-e1e8-4e79-8df9-d47e16ba5289\") " pod="openshift-image-registry/image-registry-7b77b6b64d-dgr86"
Sep 18 19:14:34.769044 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.768968    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/00d2da2b-3370-4f0a-8104-9bc429576c69-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-sszkh\" (UID: \"00d2da2b-3370-4f0a-8104-9bc429576c69\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-sszkh"
Sep 18 19:14:34.769095 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:34.769048    2522 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:14:34.769095 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:34.769065    2522 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b77b6b64d-dgr86: secret "image-registry-tls" not found
Sep 18 19:14:34.769173 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:34.769107    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-registry-tls podName:63dfbfc5-e1e8-4e79-8df9-d47e16ba5289 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:35.769093097 +0000 UTC m=+35.652619536 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-registry-tls") pod "image-registry-7b77b6b64d-dgr86" (UID: "63dfbfc5-e1e8-4e79-8df9-d47e16ba5289") : secret "image-registry-tls" not found
Sep 18 19:14:34.769173 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:34.769055    2522 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 18 19:14:34.769173 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:34.769163    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00d2da2b-3370-4f0a-8104-9bc429576c69-networking-console-plugin-cert podName:00d2da2b-3370-4f0a-8104-9bc429576c69 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:35.769152293 +0000 UTC m=+35.652678749 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/00d2da2b-3370-4f0a-8104-9bc429576c69-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-sszkh" (UID: "00d2da2b-3370-4f0a-8104-9bc429576c69") : secret "networking-console-plugin-cert" not found
Sep 18 19:14:34.846391 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.846355    2522 generic.go:356] "Generic (PLEG): container finished" podID="6c2a6c5f-990b-4ceb-9308-f7a6e29e22a2" containerID="08b06e02e485fbabe24f0b75c8db8545f80439df24fe9bfef07d45f6bd5edc7b" exitCode=0
Sep 18 19:14:34.846545 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.846428    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jrg4l" event={"ID":"6c2a6c5f-990b-4ceb-9308-f7a6e29e22a2","Type":"ContainerDied","Data":"08b06e02e485fbabe24f0b75c8db8545f80439df24fe9bfef07d45f6bd5edc7b"}
Sep 18 19:14:34.847579 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.847544    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pmrmr" event={"ID":"47bbbb88-fe11-4227-bddc-4e124994d36a","Type":"ContainerStarted","Data":"93723387befbf730033f8f2e8eb1fe7b0d534686c49ec95cddeb480b11589459"}
Sep 18 19:14:34.869331 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:34.869309    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/72d866b0-9634-442e-bd20-d90321a00452-metrics-tls\") pod \"dns-default-mrd75\" (UID: \"72d866b0-9634-442e-bd20-d90321a00452\") " pod="openshift-dns/dns-default-mrd75"
Sep 18 19:14:34.869428 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:34.869416    2522 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:14:34.869471 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:34.869458    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72d866b0-9634-442e-bd20-d90321a00452-metrics-tls podName:72d866b0-9634-442e-bd20-d90321a00452 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:35.869446346 +0000 UTC m=+35.752972799 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/72d866b0-9634-442e-bd20-d90321a00452-metrics-tls") pod "dns-default-mrd75" (UID: "72d866b0-9634-442e-bd20-d90321a00452") : secret "dns-default-metrics-tls" not found
Sep 18 19:14:35.716141 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:35.716106    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g679p"
Sep 18 19:14:35.716141 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:35.716143    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dk2hl"
Sep 18 19:14:35.716381 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:35.716143    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5j97v"
Sep 18 19:14:35.723491 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:35.723470    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xjcwp\""
Sep 18 19:14:35.724611 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:35.724585    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 19:14:35.724918 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:35.724903    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-frfgm\""
Sep 18 19:14:35.726065 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:35.726047    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 19:14:35.726604 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:35.726582    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 19:14:35.727074 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:35.727056    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 19:14:35.776391 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:35.776366    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-registry-tls\") pod \"image-registry-7b77b6b64d-dgr86\" (UID: \"63dfbfc5-e1e8-4e79-8df9-d47e16ba5289\") " pod="openshift-image-registry/image-registry-7b77b6b64d-dgr86"
Sep 18 19:14:35.776494 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:35.776407    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/00d2da2b-3370-4f0a-8104-9bc429576c69-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-sszkh\" (UID: \"00d2da2b-3370-4f0a-8104-9bc429576c69\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-sszkh"
Sep 18 19:14:35.776494 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:35.776471    2522 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:14:35.776494 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:35.776489    2522 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b77b6b64d-dgr86: secret "image-registry-tls" not found
Sep 18 19:14:35.776672 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:35.776515    2522 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 18 19:14:35.776672 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:35.776548    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-registry-tls podName:63dfbfc5-e1e8-4e79-8df9-d47e16ba5289 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:37.776529975 +0000 UTC m=+37.660056416 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-registry-tls") pod "image-registry-7b77b6b64d-dgr86" (UID: "63dfbfc5-e1e8-4e79-8df9-d47e16ba5289") : secret "image-registry-tls" not found
Sep 18 19:14:35.776672 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:35.776583    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00d2da2b-3370-4f0a-8104-9bc429576c69-networking-console-plugin-cert podName:00d2da2b-3370-4f0a-8104-9bc429576c69 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:37.776556263 +0000 UTC m=+37.660082708 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/00d2da2b-3370-4f0a-8104-9bc429576c69-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-sszkh" (UID: "00d2da2b-3370-4f0a-8104-9bc429576c69") : secret "networking-console-plugin-cert" not found
Sep 18 19:14:35.853449 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:35.853423    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jrg4l" event={"ID":"6c2a6c5f-990b-4ceb-9308-f7a6e29e22a2","Type":"ContainerStarted","Data":"32f91c786c6cf9535bbcf50075f114aa0405cf4e3d8502554166604d422a7fe9"}
Sep 18 19:14:35.877693 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:35.877660    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/72d866b0-9634-442e-bd20-d90321a00452-metrics-tls\") pod \"dns-default-mrd75\" (UID: \"72d866b0-9634-442e-bd20-d90321a00452\") " pod="openshift-dns/dns-default-mrd75"
Sep 18 19:14:35.877854 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:35.877821    2522 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:14:35.877918 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:35.877883    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72d866b0-9634-442e-bd20-d90321a00452-metrics-tls podName:72d866b0-9634-442e-bd20-d90321a00452 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:37.87786637 +0000 UTC m=+37.761392810 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/72d866b0-9634-442e-bd20-d90321a00452-metrics-tls") pod "dns-default-mrd75" (UID: "72d866b0-9634-442e-bd20-d90321a00452") : secret "dns-default-metrics-tls" not found
Sep 18 19:14:35.899518 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:35.899476    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jrg4l" podStartSLOduration=7.399105891 podStartE2EDuration="34.899461605s" totalImagesPullingTime="27.500355714s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:01 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:35.899141612 +0000 UTC m=+35.782668087" watchObservedRunningTime="2026-09-18 19:14:35.899461605 +0000 UTC m=+35.782988066"
Sep 18 19:14:37.792738 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:37.792699    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/00d2da2b-3370-4f0a-8104-9bc429576c69-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-sszkh\" (UID: \"00d2da2b-3370-4f0a-8104-9bc429576c69\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-sszkh"
Sep 18 19:14:37.793253 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:37.792801    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-registry-tls\") pod \"image-registry-7b77b6b64d-dgr86\" (UID: \"63dfbfc5-e1e8-4e79-8df9-d47e16ba5289\") " pod="openshift-image-registry/image-registry-7b77b6b64d-dgr86"
Sep 18 19:14:37.793253 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:37.792845    2522 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 18 19:14:37.793253 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:37.792884    2522 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:14:37.793253 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:37.792895    2522 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b77b6b64d-dgr86: secret "image-registry-tls" not found
Sep 18 19:14:37.793253 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:37.792932    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00d2da2b-3370-4f0a-8104-9bc429576c69-networking-console-plugin-cert podName:00d2da2b-3370-4f0a-8104-9bc429576c69 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:41.792901543 +0000 UTC m=+41.676428003 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/00d2da2b-3370-4f0a-8104-9bc429576c69-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-sszkh" (UID: "00d2da2b-3370-4f0a-8104-9bc429576c69") : secret "networking-console-plugin-cert" not found
Sep 18 19:14:37.793253 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:37.792955    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-registry-tls podName:63dfbfc5-e1e8-4e79-8df9-d47e16ba5289 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:41.792945577 +0000 UTC m=+41.676472021 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-registry-tls") pod "image-registry-7b77b6b64d-dgr86" (UID: "63dfbfc5-e1e8-4e79-8df9-d47e16ba5289") : secret "image-registry-tls" not found
Sep 18 19:14:37.859816 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:37.859786    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pmrmr" event={"ID":"47bbbb88-fe11-4227-bddc-4e124994d36a","Type":"ContainerStarted","Data":"b841dcbe8855c361e4ef2c8f8c85e77d18e8677a6e50dbfa73b66567567658c5"}
Sep 18 19:14:37.893698 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:37.893672    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/72d866b0-9634-442e-bd20-d90321a00452-metrics-tls\") pod \"dns-default-mrd75\" (UID: \"72d866b0-9634-442e-bd20-d90321a00452\") " pod="openshift-dns/dns-default-mrd75"
Sep 18 19:14:37.893782 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:37.893768    2522 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:14:37.893831 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:37.893820    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72d866b0-9634-442e-bd20-d90321a00452-metrics-tls podName:72d866b0-9634-442e-bd20-d90321a00452 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:41.893803632 +0000 UTC m=+41.777330089 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/72d866b0-9634-442e-bd20-d90321a00452-metrics-tls") pod "dns-default-mrd75" (UID: "72d866b0-9634-442e-bd20-d90321a00452") : secret "dns-default-metrics-tls" not found
Sep 18 19:14:41.826705 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:41.826665    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-registry-tls\") pod \"image-registry-7b77b6b64d-dgr86\" (UID: \"63dfbfc5-e1e8-4e79-8df9-d47e16ba5289\") " pod="openshift-image-registry/image-registry-7b77b6b64d-dgr86"
Sep 18 19:14:41.826705 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:41.826710    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/00d2da2b-3370-4f0a-8104-9bc429576c69-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-sszkh\" (UID: \"00d2da2b-3370-4f0a-8104-9bc429576c69\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-sszkh"
Sep 18 19:14:41.827127 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:41.826804    2522 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 18 19:14:41.827127 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:41.826810    2522 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:14:41.827127 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:41.826830    2522 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b77b6b64d-dgr86: secret "image-registry-tls" not found
Sep 18 19:14:41.827127 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:41.826855    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00d2da2b-3370-4f0a-8104-9bc429576c69-networking-console-plugin-cert podName:00d2da2b-3370-4f0a-8104-9bc429576c69 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:49.826841426 +0000 UTC m=+49.710367879 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/00d2da2b-3370-4f0a-8104-9bc429576c69-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-sszkh" (UID: "00d2da2b-3370-4f0a-8104-9bc429576c69") : secret "networking-console-plugin-cert" not found
Sep 18 19:14:41.827127 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:41.826873    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-registry-tls podName:63dfbfc5-e1e8-4e79-8df9-d47e16ba5289 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:49.826859918 +0000 UTC m=+49.710386359 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-registry-tls") pod "image-registry-7b77b6b64d-dgr86" (UID: "63dfbfc5-e1e8-4e79-8df9-d47e16ba5289") : secret "image-registry-tls" not found
Sep 18 19:14:41.927870 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:41.927839    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/72d866b0-9634-442e-bd20-d90321a00452-metrics-tls\") pod \"dns-default-mrd75\" (UID: \"72d866b0-9634-442e-bd20-d90321a00452\") " pod="openshift-dns/dns-default-mrd75"
Sep 18 19:14:41.928001 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:41.927959    2522 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:14:41.928034 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:41.928011    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72d866b0-9634-442e-bd20-d90321a00452-metrics-tls podName:72d866b0-9634-442e-bd20-d90321a00452 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:49.927997123 +0000 UTC m=+49.811523564 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/72d866b0-9634-442e-bd20-d90321a00452-metrics-tls") pod "dns-default-mrd75" (UID: "72d866b0-9634-442e-bd20-d90321a00452") : secret "dns-default-metrics-tls" not found
Sep 18 19:14:49.883855 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:49.883826    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-registry-tls\") pod \"image-registry-7b77b6b64d-dgr86\" (UID: \"63dfbfc5-e1e8-4e79-8df9-d47e16ba5289\") " pod="openshift-image-registry/image-registry-7b77b6b64d-dgr86"
Sep 18 19:14:49.883855 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:49.883861    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/00d2da2b-3370-4f0a-8104-9bc429576c69-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-sszkh\" (UID: \"00d2da2b-3370-4f0a-8104-9bc429576c69\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-sszkh"
Sep 18 19:14:49.884284 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:49.883970    2522 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 18 19:14:49.884284 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:49.883976    2522 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:14:49.884284 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:49.883996    2522 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b77b6b64d-dgr86: secret "image-registry-tls" not found
Sep 18 19:14:49.884284 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:49.884023    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00d2da2b-3370-4f0a-8104-9bc429576c69-networking-console-plugin-cert podName:00d2da2b-3370-4f0a-8104-9bc429576c69 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:05.884007653 +0000 UTC m=+65.767534092 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/00d2da2b-3370-4f0a-8104-9bc429576c69-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-sszkh" (UID: "00d2da2b-3370-4f0a-8104-9bc429576c69") : secret "networking-console-plugin-cert" not found
Sep 18 19:14:49.884284 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:49.884057    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-registry-tls podName:63dfbfc5-e1e8-4e79-8df9-d47e16ba5289 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:05.884029744 +0000 UTC m=+65.767556201 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-registry-tls") pod "image-registry-7b77b6b64d-dgr86" (UID: "63dfbfc5-e1e8-4e79-8df9-d47e16ba5289") : secret "image-registry-tls" not found
Sep 18 19:14:49.984873 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:49.984841    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/72d866b0-9634-442e-bd20-d90321a00452-metrics-tls\") pod \"dns-default-mrd75\" (UID: \"72d866b0-9634-442e-bd20-d90321a00452\") " pod="openshift-dns/dns-default-mrd75"
Sep 18 19:14:49.985021 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:49.985004    2522 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:14:49.985118 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:14:49.985067    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72d866b0-9634-442e-bd20-d90321a00452-metrics-tls podName:72d866b0-9634-442e-bd20-d90321a00452 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:05.985052027 +0000 UTC m=+65.868578483 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/72d866b0-9634-442e-bd20-d90321a00452-metrics-tls") pod "dns-default-mrd75" (UID: "72d866b0-9634-442e-bd20-d90321a00452") : secret "dns-default-metrics-tls" not found
Sep 18 19:14:59.840166 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:59.840141    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hbhqv"
Sep 18 19:14:59.919602 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:14:59.919542    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-pmrmr" podStartSLOduration=23.554004172 podStartE2EDuration="25.919527089s" totalImagesPullingTime="2.365522917s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:14:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:37.926471283 +0000 UTC m=+37.809997757" watchObservedRunningTime="2026-09-18 19:14:59.919527089 +0000 UTC m=+59.803053552"
Sep 18 19:15:01.767911 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:15:01.767873    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d544df478-5nch5"]
Sep 18 19:15:01.770427 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:15:01.770406    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d544df478-5nch5"
Sep 18 19:15:01.774164 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:15:01.774135    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 18 19:15:01.774377 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:15:01.774362    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 18 19:15:01.774580 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:15:01.774546    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-xs92n\""
Sep 18 19:15:01.775258 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:15:01.775238    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 18 19:15:01.775952 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:15:01.775933    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 18 19:15:01.780595 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:15:01.780552    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d544df478-5nch5"]
Sep 18 19:15:01.868742 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:15:01.868703    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-752dx\" (UniqueName: \"kubernetes.io/projected/5a5340ed-d845-4599-a58c-efbd5f71a12b-kube-api-access-752dx\") pod \"managed-serviceaccount-addon-agent-6d544df478-5nch5\" (UID: \"5a5340ed-d845-4599-a58c-efbd5f71a12b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d544df478-5nch5"
Sep 18 19:15:01.868742 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:15:01.868746    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5a5340ed-d845-4599-a58c-efbd5f71a12b-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6d544df478-5nch5\" (UID: \"5a5340ed-d845-4599-a58c-efbd5f71a12b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d544df478-5nch5"
Sep 18 19:15:01.969755 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:15:01.969716    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5a5340ed-d845-4599-a58c-efbd5f71a12b-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6d544df478-5nch5\" (UID: \"5a5340ed-d845-4599-a58c-efbd5f71a12b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d544df478-5nch5"
Sep 18 19:15:01.969882 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:15:01.969844    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-752dx\" (UniqueName: \"kubernetes.io/projected/5a5340ed-d845-4599-a58c-efbd5f71a12b-kube-api-access-752dx\") pod \"managed-serviceaccount-addon-agent-6d544df478-5nch5\" (UID: \"5a5340ed-d845-4599-a58c-efbd5f71a12b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d544df478-5nch5"
Sep 18 19:15:01.972870 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:15:01.972851    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5a5340ed-d845-4599-a58c-efbd5f71a12b-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6d544df478-5nch5\" (UID: \"5a5340ed-d845-4599-a58c-efbd5f71a12b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d544df478-5nch5"
Sep 18 19:15:01.978761 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:15:01.978740    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-752dx\" (UniqueName: \"kubernetes.io/projected/5a5340ed-d845-4599-a58c-efbd5f71a12b-kube-api-access-752dx\") pod \"managed-serviceaccount-addon-agent-6d544df478-5nch5\" (UID: \"5a5340ed-d845-4599-a58c-efbd5f71a12b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d544df478-5nch5"
Sep 18 19:15:02.089330 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:15:02.089310    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d544df478-5nch5"
Sep 18 19:15:02.355999 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:15:02.355857    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d544df478-5nch5"]
Sep 18 19:15:02.361430 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:15:02.361402    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a5340ed_d845_4599_a58c_efbd5f71a12b.slice/crio-de23f1b29e318ab1a894b4efad5c904c9fdc8ebe5543f4fbdb62264bda42c5b0 WatchSource:0}: Error finding container de23f1b29e318ab1a894b4efad5c904c9fdc8ebe5543f4fbdb62264bda42c5b0: Status 404 returned error can't find the container with id de23f1b29e318ab1a894b4efad5c904c9fdc8ebe5543f4fbdb62264bda42c5b0
Sep 18 19:15:02.916432 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:15:02.916398    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d544df478-5nch5" event={"ID":"5a5340ed-d845-4599-a58c-efbd5f71a12b","Type":"ContainerStarted","Data":"de23f1b29e318ab1a894b4efad5c904c9fdc8ebe5543f4fbdb62264bda42c5b0"}
Sep 18 19:15:05.899712 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:15:05.899670    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-registry-tls\") pod \"image-registry-7b77b6b64d-dgr86\" (UID: \"63dfbfc5-e1e8-4e79-8df9-d47e16ba5289\") " pod="openshift-image-registry/image-registry-7b77b6b64d-dgr86"
Sep 18 19:15:05.899712 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:15:05.899722    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/00d2da2b-3370-4f0a-8104-9bc429576c69-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-sszkh\" (UID: \"00d2da2b-3370-4f0a-8104-9bc429576c69\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-sszkh"
Sep 18 19:15:05.900124 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:15:05.899836    2522 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 18 19:15:05.900124 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:15:05.899842    2522 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:15:05.900124 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:15:05.899860    2522 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b77b6b64d-dgr86: secret "image-registry-tls" not found
Sep 18 19:15:05.900124 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:15:05.899888    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00d2da2b-3370-4f0a-8104-9bc429576c69-networking-console-plugin-cert podName:00d2da2b-3370-4f0a-8104-9bc429576c69 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:37.89987395 +0000 UTC m=+97.783400389 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/00d2da2b-3370-4f0a-8104-9bc429576c69-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-sszkh" (UID: "00d2da2b-3370-4f0a-8104-9bc429576c69") : secret "networking-console-plugin-cert" not found
Sep 18 19:15:05.900124 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:15:05.899902    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-registry-tls podName:63dfbfc5-e1e8-4e79-8df9-d47e16ba5289 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:37.899896194 +0000 UTC m=+97.783422634 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-registry-tls") pod "image-registry-7b77b6b64d-dgr86" (UID: "63dfbfc5-e1e8-4e79-8df9-d47e16ba5289") : secret "image-registry-tls" not found
Sep 18 19:15:05.924622 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:15:05.924591    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d544df478-5nch5" event={"ID":"5a5340ed-d845-4599-a58c-efbd5f71a12b","Type":"ContainerStarted","Data":"a4219d58de8a75f885f8738dd7f79564916a23a8c4f14936a5e123b679f05f73"}
Sep 18 19:15:06.000741 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:15:06.000718    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/72d866b0-9634-442e-bd20-d90321a00452-metrics-tls\") pod \"dns-default-mrd75\" (UID: \"72d866b0-9634-442e-bd20-d90321a00452\") " pod="openshift-dns/dns-default-mrd75"
Sep 18 19:15:06.000873 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:15:06.000856    2522 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:15:06.000935 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:15:06.000908    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72d866b0-9634-442e-bd20-d90321a00452-metrics-tls podName:72d866b0-9634-442e-bd20-d90321a00452 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:38.000894669 +0000 UTC m=+97.884421143 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/72d866b0-9634-442e-bd20-d90321a00452-metrics-tls") pod "dns-default-mrd75" (UID: "72d866b0-9634-442e-bd20-d90321a00452") : secret "dns-default-metrics-tls" not found
Sep 18 19:15:06.303141 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:15:06.303042    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f516e39e-b125-4617-94ce-7f0dccfae7bb-metrics-certs\") pod \"network-metrics-daemon-5j97v\" (UID: \"f516e39e-b125-4617-94ce-7f0dccfae7bb\") " pod="openshift-multus/network-metrics-daemon-5j97v"
Sep 18 19:15:06.307760 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:15:06.307738    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 19:15:06.313803 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:15:06.313788    2522 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 18 19:15:06.313863 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:15:06.313838    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f516e39e-b125-4617-94ce-7f0dccfae7bb-metrics-certs podName:f516e39e-b125-4617-94ce-7f0dccfae7bb nodeName:}" failed. No retries permitted until 2026-09-18 19:16:10.313824148 +0000 UTC m=+130.197350588 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f516e39e-b125-4617-94ce-7f0dccfae7bb-metrics-certs") pod "network-metrics-daemon-5j97v" (UID: "f516e39e-b125-4617-94ce-7f0dccfae7bb") : secret "metrics-daemon-secret" not found
Sep 18 19:15:06.403534 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:15:06.403510    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7f72\" (UniqueName: \"kubernetes.io/projected/3f0563df-380a-41eb-bb44-902bf8ceb4f6-kube-api-access-t7f72\") pod \"network-check-target-dk2hl\" (UID: \"3f0563df-380a-41eb-bb44-902bf8ceb4f6\") " pod="openshift-network-diagnostics/network-check-target-dk2hl"
Sep 18 19:15:06.403643 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:15:06.403557    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24ec319c-fec4-4b8b-bcb6-6331ec8574fa-original-pull-secret\") pod \"global-pull-secret-syncer-g679p\" (UID: \"24ec319c-fec4-4b8b-bcb6-6331ec8574fa\") " pod="kube-system/global-pull-secret-syncer-g679p"
Sep 18 19:15:06.406353 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:15:06.406335    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 19:15:06.406428 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:15:06.406418    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 19:15:06.416266 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:15:06.416236    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24ec319c-fec4-4b8b-bcb6-6331ec8574fa-original-pull-secret\") pod \"global-pull-secret-syncer-g679p\" (UID: \"24ec319c-fec4-4b8b-bcb6-6331ec8574fa\") " pod="kube-system/global-pull-secret-syncer-g679p"
Sep 18 19:15:06.417042 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:15:06.417027    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 19:15:06.426260 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:15:06.426239    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7f72\" (UniqueName: \"kubernetes.io/projected/3f0563df-380a-41eb-bb44-902bf8ceb4f6-kube-api-access-t7f72\") pod \"network-check-target-dk2hl\" (UID: \"3f0563df-380a-41eb-bb44-902bf8ceb4f6\") " pod="openshift-network-diagnostics/network-check-target-dk2hl"
Sep 18 19:15:06.627692 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:15:06.627665    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g679p"
Sep 18 19:15:06.637984 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:15:06.637965    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-frfgm\""
Sep 18 19:15:06.645594 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:15:06.645555    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dk2hl"
Sep 18 19:15:06.912520 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:15:06.912435    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d544df478-5nch5" podStartSLOduration=2.889059251 podStartE2EDuration="5.912415988s" totalImagesPullingTime="3.023356737s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:15:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:15:05.956062533 +0000 UTC m=+65.839588996" watchObservedRunningTime="2026-09-18 19:15:06.912415988 +0000 UTC m=+66.795942450"
Sep 18 19:15:06.913029 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:15:06.913010    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-g679p"]
Sep 18 19:15:06.915973 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:15:06.915952    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24ec319c_fec4_4b8b_bcb6_6331ec8574fa.slice/crio-8040334557ecaeccc677dcd5b3e1c1285bbd33e72cd1478df066d462959ac818 WatchSource:0}: Error finding container 8040334557ecaeccc677dcd5b3e1c1285bbd33e72cd1478df066d462959ac818: Status 404 returned error can't find the container with id 8040334557ecaeccc677dcd5b3e1c1285bbd33e72cd1478df066d462959ac818
Sep 18 19:15:06.927433 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:15:06.927409    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-g679p" event={"ID":"24ec319c-fec4-4b8b-bcb6-6331ec8574fa","Type":"ContainerStarted","Data":"8040334557ecaeccc677dcd5b3e1c1285bbd33e72cd1478df066d462959ac818"}
Sep 18 19:15:06.932023 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:15:06.932001    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dk2hl"]
Sep 18 19:15:06.934055 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:15:06.934031    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f0563df_380a_41eb_bb44_902bf8ceb4f6.slice/crio-7bdf61b8fc86c8592c3347918c6f3b96b12315a3ffee1ab0c3e98ddcfcf9e350 WatchSource:0}: Error finding container 7bdf61b8fc86c8592c3347918c6f3b96b12315a3ffee1ab0c3e98ddcfcf9e350: Status 404 returned error can't find the container with id 7bdf61b8fc86c8592c3347918c6f3b96b12315a3ffee1ab0c3e98ddcfcf9e350
Sep 18 19:15:07.931955 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:15:07.931916    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dk2hl" event={"ID":"3f0563df-380a-41eb-bb44-902bf8ceb4f6","Type":"ContainerStarted","Data":"7bdf61b8fc86c8592c3347918c6f3b96b12315a3ffee1ab0c3e98ddcfcf9e350"}
Sep 18 19:15:11.943703 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:15:11.943665    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dk2hl" event={"ID":"3f0563df-380a-41eb-bb44-902bf8ceb4f6","Type":"ContainerStarted","Data":"4203a8534867f9b8108fe88672ffbf34e66a521c21691f1ba0685f5ece42f67d"}
Sep 18 19:15:11.944141 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:15:11.943814    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dk2hl"
Sep 18 19:15:11.945138 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:15:11.945116    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-g679p" event={"ID":"24ec319c-fec4-4b8b-bcb6-6331ec8574fa","Type":"ContainerStarted","Data":"0181782bc01507cfc2374ce52653db08dc056eebeae47caf6b428c20e587a451"}
Sep 18 19:15:11.961359 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:15:11.961313    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-dk2hl" podStartSLOduration=67.675613459 podStartE2EDuration="1m11.96130155s" totalImagesPullingTime="4.285688091s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:14:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:15:11.959874146 +0000 UTC m=+71.843400608" watchObservedRunningTime="2026-09-18 19:15:11.96130155 +0000 UTC m=+71.844828009"
Sep 18 19:15:37.921350 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:15:37.921307    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-registry-tls\") pod \"image-registry-7b77b6b64d-dgr86\" (UID: \"63dfbfc5-e1e8-4e79-8df9-d47e16ba5289\") " pod="openshift-image-registry/image-registry-7b77b6b64d-dgr86"
Sep 18 19:15:37.921350 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:15:37.921354    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/00d2da2b-3370-4f0a-8104-9bc429576c69-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-sszkh\" (UID: \"00d2da2b-3370-4f0a-8104-9bc429576c69\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-sszkh"
Sep 18 19:15:37.921854 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:15:37.921475    2522 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 18 19:15:37.921854 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:15:37.921527    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00d2da2b-3370-4f0a-8104-9bc429576c69-networking-console-plugin-cert podName:00d2da2b-3370-4f0a-8104-9bc429576c69 nodeName:}" failed. No retries permitted until 2026-09-18 19:16:41.921514856 +0000 UTC m=+161.805041295 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/00d2da2b-3370-4f0a-8104-9bc429576c69-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-sszkh" (UID: "00d2da2b-3370-4f0a-8104-9bc429576c69") : secret "networking-console-plugin-cert" not found
Sep 18 19:15:37.921854 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:15:37.921473    2522 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:15:37.921854 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:15:37.921594    2522 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b77b6b64d-dgr86: secret "image-registry-tls" not found
Sep 18 19:15:37.921854 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:15:37.921642    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-registry-tls podName:63dfbfc5-e1e8-4e79-8df9-d47e16ba5289 nodeName:}" failed. No retries permitted until 2026-09-18 19:16:41.921628436 +0000 UTC m=+161.805154890 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-registry-tls") pod "image-registry-7b77b6b64d-dgr86" (UID: "63dfbfc5-e1e8-4e79-8df9-d47e16ba5289") : secret "image-registry-tls" not found
Sep 18 19:15:38.022453 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:15:38.022374    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/72d866b0-9634-442e-bd20-d90321a00452-metrics-tls\") pod \"dns-default-mrd75\" (UID: \"72d866b0-9634-442e-bd20-d90321a00452\") " pod="openshift-dns/dns-default-mrd75"
Sep 18 19:15:38.022599 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:15:38.022476    2522 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:15:38.022599 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:15:38.022548    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72d866b0-9634-442e-bd20-d90321a00452-metrics-tls podName:72d866b0-9634-442e-bd20-d90321a00452 nodeName:}" failed. No retries permitted until 2026-09-18 19:16:42.022530586 +0000 UTC m=+161.906057041 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/72d866b0-9634-442e-bd20-d90321a00452-metrics-tls") pod "dns-default-mrd75" (UID: "72d866b0-9634-442e-bd20-d90321a00452") : secret "dns-default-metrics-tls" not found
Sep 18 19:15:42.949058 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:15:42.949027    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-dk2hl"
Sep 18 19:15:42.972252 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:15:42.972208    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-g679p" podStartSLOduration=97.655908819 podStartE2EDuration="1m41.972193294s" totalImagesPullingTime="4.316284475s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:15:11.975038935 +0000 UTC m=+71.858565408" watchObservedRunningTime="2026-09-18 19:15:42.972193294 +0000 UTC m=+102.855719810"
Sep 18 19:16:10.339850 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:10.339811    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f516e39e-b125-4617-94ce-7f0dccfae7bb-metrics-certs\") pod \"network-metrics-daemon-5j97v\" (UID: \"f516e39e-b125-4617-94ce-7f0dccfae7bb\") " pod="openshift-multus/network-metrics-daemon-5j97v"
Sep 18 19:16:10.340318 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:16:10.339920    2522 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 18 19:16:10.340318 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:16:10.339980    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f516e39e-b125-4617-94ce-7f0dccfae7bb-metrics-certs podName:f516e39e-b125-4617-94ce-7f0dccfae7bb nodeName:}" failed. No retries permitted until 2026-09-18 19:18:12.339965824 +0000 UTC m=+252.223492264 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f516e39e-b125-4617-94ce-7f0dccfae7bb-metrics-certs") pod "network-metrics-daemon-5j97v" (UID: "f516e39e-b125-4617-94ce-7f0dccfae7bb") : secret "metrics-daemon-secret" not found
Sep 18 19:16:29.977748 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:29.977706    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-d4tvc"]
Sep 18 19:16:29.980591 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:29.980557    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-d4tvc"
Sep 18 19:16:29.988774 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:29.988755    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-2tkck\""
Sep 18 19:16:30.006001 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:30.005982    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-d4tvc"]
Sep 18 19:16:30.078838 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:30.078806    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6fld\" (UniqueName: \"kubernetes.io/projected/60e6e041-fd32-4494-b891-11eb4082d868-kube-api-access-m6fld\") pod \"network-check-source-7cbc45fddc-d4tvc\" (UID: \"60e6e041-fd32-4494-b891-11eb4082d868\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-d4tvc"
Sep 18 19:16:30.101303 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:30.101273    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-s69xh"]
Sep 18 19:16:30.104082 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:30.104068    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-qkrw2"]
Sep 18 19:16:30.104206 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:30.104189    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-s69xh"
Sep 18 19:16:30.106795 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:30.106774    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-qkrw2"
Sep 18 19:16:30.110555 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:30.110441    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 18 19:16:30.111177 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:30.111158    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:16:30.111255 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:30.111220    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 18 19:16:30.111315 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:30.111253    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-xr9gt\""
Sep 18 19:16:30.111315 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:30.111255    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:16:30.111457 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:30.111443    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 18 19:16:30.111642 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:30.111627    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 18 19:16:30.111691 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:30.111667    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-ff2jk\""
Sep 18 19:16:30.126693 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:30.126670    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-s69xh"]
Sep 18 19:16:30.127592 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:30.127557    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-qkrw2"]
Sep 18 19:16:30.180248 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:30.180217    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6fld\" (UniqueName: \"kubernetes.io/projected/60e6e041-fd32-4494-b891-11eb4082d868-kube-api-access-m6fld\") pod \"network-check-source-7cbc45fddc-d4tvc\" (UID: \"60e6e041-fd32-4494-b891-11eb4082d868\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-d4tvc"
Sep 18 19:16:30.247581 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:30.247485    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6fld\" (UniqueName: \"kubernetes.io/projected/60e6e041-fd32-4494-b891-11eb4082d868-kube-api-access-m6fld\") pod \"network-check-source-7cbc45fddc-d4tvc\" (UID: \"60e6e041-fd32-4494-b891-11eb4082d868\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-d4tvc"
Sep 18 19:16:30.281444 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:30.281407    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdj5k\" (UniqueName: \"kubernetes.io/projected/879188dd-2181-4f6c-9337-1356aa59fdc0-kube-api-access-kdj5k\") pod \"volume-data-source-validator-5c4db8bdf9-s69xh\" (UID: \"879188dd-2181-4f6c-9337-1356aa59fdc0\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-s69xh"
Sep 18 19:16:30.281630 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:30.281465    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4b4t\" (UniqueName: \"kubernetes.io/projected/09a74ca7-303a-49c9-9967-98074bbfbb17-kube-api-access-w4b4t\") pod \"kube-storage-version-migrator-operator-55889f7b47-qkrw2\" (UID: \"09a74ca7-303a-49c9-9967-98074bbfbb17\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-qkrw2"
Sep 18 19:16:30.281630 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:30.281520    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/09a74ca7-303a-49c9-9967-98074bbfbb17-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-qkrw2\" (UID: \"09a74ca7-303a-49c9-9967-98074bbfbb17\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-qkrw2"
Sep 18 19:16:30.281630 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:30.281550    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/09a74ca7-303a-49c9-9967-98074bbfbb17-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-qkrw2\" (UID: \"09a74ca7-303a-49c9-9967-98074bbfbb17\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-qkrw2"
Sep 18 19:16:30.288416 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:30.288394    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-d4tvc"
Sep 18 19:16:30.382178 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:30.382146    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/09a74ca7-303a-49c9-9967-98074bbfbb17-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-qkrw2\" (UID: \"09a74ca7-303a-49c9-9967-98074bbfbb17\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-qkrw2"
Sep 18 19:16:30.382353 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:30.382189    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/09a74ca7-303a-49c9-9967-98074bbfbb17-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-qkrw2\" (UID: \"09a74ca7-303a-49c9-9967-98074bbfbb17\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-qkrw2"
Sep 18 19:16:30.382353 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:30.382260    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdj5k\" (UniqueName: \"kubernetes.io/projected/879188dd-2181-4f6c-9337-1356aa59fdc0-kube-api-access-kdj5k\") pod \"volume-data-source-validator-5c4db8bdf9-s69xh\" (UID: \"879188dd-2181-4f6c-9337-1356aa59fdc0\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-s69xh"
Sep 18 19:16:30.382353 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:30.382294    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4b4t\" (UniqueName: \"kubernetes.io/projected/09a74ca7-303a-49c9-9967-98074bbfbb17-kube-api-access-w4b4t\") pod \"kube-storage-version-migrator-operator-55889f7b47-qkrw2\" (UID: \"09a74ca7-303a-49c9-9967-98074bbfbb17\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-qkrw2"
Sep 18 19:16:30.382873 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:30.382855    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/09a74ca7-303a-49c9-9967-98074bbfbb17-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-qkrw2\" (UID: \"09a74ca7-303a-49c9-9967-98074bbfbb17\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-qkrw2"
Sep 18 19:16:30.384348 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:30.384327    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/09a74ca7-303a-49c9-9967-98074bbfbb17-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-qkrw2\" (UID: \"09a74ca7-303a-49c9-9967-98074bbfbb17\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-qkrw2"
Sep 18 19:16:30.416782 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:30.416750    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdj5k\" (UniqueName: \"kubernetes.io/projected/879188dd-2181-4f6c-9337-1356aa59fdc0-kube-api-access-kdj5k\") pod \"volume-data-source-validator-5c4db8bdf9-s69xh\" (UID: \"879188dd-2181-4f6c-9337-1356aa59fdc0\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-s69xh"
Sep 18 19:16:30.419183 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:30.419161    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4b4t\" (UniqueName: \"kubernetes.io/projected/09a74ca7-303a-49c9-9967-98074bbfbb17-kube-api-access-w4b4t\") pod \"kube-storage-version-migrator-operator-55889f7b47-qkrw2\" (UID: \"09a74ca7-303a-49c9-9967-98074bbfbb17\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-qkrw2"
Sep 18 19:16:30.557535 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:30.557456    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-d4tvc"]
Sep 18 19:16:30.560424 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:16:30.560400    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60e6e041_fd32_4494_b891_11eb4082d868.slice/crio-1d99a0ee98f327de7d70025723ce21bb2a64de961692f159a4b9cbd6d7ad8c3b WatchSource:0}: Error finding container 1d99a0ee98f327de7d70025723ce21bb2a64de961692f159a4b9cbd6d7ad8c3b: Status 404 returned error can't find the container with id 1d99a0ee98f327de7d70025723ce21bb2a64de961692f159a4b9cbd6d7ad8c3b
Sep 18 19:16:30.713650 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:30.713617    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-s69xh"
Sep 18 19:16:30.718164 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:30.718138    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-qkrw2"
Sep 18 19:16:30.986669 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:30.986640    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-s69xh"]
Sep 18 19:16:30.989143 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:16:30.989118    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod879188dd_2181_4f6c_9337_1356aa59fdc0.slice/crio-d6fdbe57ef44730b63574c25f574286b49b8f41d47a9efb39609aa783f9addb9 WatchSource:0}: Error finding container d6fdbe57ef44730b63574c25f574286b49b8f41d47a9efb39609aa783f9addb9: Status 404 returned error can't find the container with id d6fdbe57ef44730b63574c25f574286b49b8f41d47a9efb39609aa783f9addb9
Sep 18 19:16:31.009243 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:31.009224    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-qkrw2"]
Sep 18 19:16:31.011477 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:16:31.011456    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09a74ca7_303a_49c9_9967_98074bbfbb17.slice/crio-28aadb8b6301c63791c4c3f1aab404c3811ce279fefc27f0e4c7a4b0d256075f WatchSource:0}: Error finding container 28aadb8b6301c63791c4c3f1aab404c3811ce279fefc27f0e4c7a4b0d256075f: Status 404 returned error can't find the container with id 28aadb8b6301c63791c4c3f1aab404c3811ce279fefc27f0e4c7a4b0d256075f
Sep 18 19:16:31.131353 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:31.131312    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-qkrw2" event={"ID":"09a74ca7-303a-49c9-9967-98074bbfbb17","Type":"ContainerStarted","Data":"28aadb8b6301c63791c4c3f1aab404c3811ce279fefc27f0e4c7a4b0d256075f"}
Sep 18 19:16:31.132542 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:31.132512    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-s69xh" event={"ID":"879188dd-2181-4f6c-9337-1356aa59fdc0","Type":"ContainerStarted","Data":"d6fdbe57ef44730b63574c25f574286b49b8f41d47a9efb39609aa783f9addb9"}
Sep 18 19:16:31.134002 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:31.133972    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-d4tvc" event={"ID":"60e6e041-fd32-4494-b891-11eb4082d868","Type":"ContainerStarted","Data":"e698f48cfee750c35eb4d2ababf043bea672fe4b1a64cc63b5b6663d397b53a3"}
Sep 18 19:16:31.134098 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:31.134013    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-d4tvc" event={"ID":"60e6e041-fd32-4494-b891-11eb4082d868","Type":"ContainerStarted","Data":"1d99a0ee98f327de7d70025723ce21bb2a64de961692f159a4b9cbd6d7ad8c3b"}
Sep 18 19:16:31.150462 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:31.150423    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-d4tvc" podStartSLOduration=2.150410143 podStartE2EDuration="2.150410143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:16:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:16:31.149905592 +0000 UTC m=+151.033432055" watchObservedRunningTime="2026-09-18 19:16:31.150410143 +0000 UTC m=+151.033936984"
Sep 18 19:16:33.141578 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:33.141478    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-qkrw2" event={"ID":"09a74ca7-303a-49c9-9967-98074bbfbb17","Type":"ContainerStarted","Data":"331a67b5626d3d6aac2a2276d9ecbdd480078cb7a43906cef7b3dadf8cb2b9b9"}
Sep 18 19:16:33.143030 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:33.143007    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-s69xh" event={"ID":"879188dd-2181-4f6c-9337-1356aa59fdc0","Type":"ContainerStarted","Data":"bb53db96024cc2dbdfca2f882e87497f78565484f5202765f72ec7eb9c2fddd9"}
Sep 18 19:16:33.163031 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:33.162988    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-qkrw2" podStartSLOduration=1.2883059000000001 podStartE2EDuration="3.162973931s" totalImagesPullingTime="1.874668031s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:16:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:16:33.162194726 +0000 UTC m=+153.045721187" watchObservedRunningTime="2026-09-18 19:16:33.162973931 +0000 UTC m=+153.046500392"
Sep 18 19:16:33.183967 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:33.183921    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-s69xh" podStartSLOduration=2.032883317 podStartE2EDuration="3.183905018s" totalImagesPullingTime="1.151021701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:16:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:16:33.182357534 +0000 UTC m=+153.065884007" watchObservedRunningTime="2026-09-18 19:16:33.183905018 +0000 UTC m=+153.067431477"
Sep 18 19:16:34.913355 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:34.913319    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8bdr7"]
Sep 18 19:16:34.916347 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:34.916330    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8bdr7"
Sep 18 19:16:34.922956 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:34.922936    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 18 19:16:34.923274 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:34.923259    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 18 19:16:34.924273 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:34.924259    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xd5sl\""
Sep 18 19:16:34.925773 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:34.925759    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 18 19:16:34.930277 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:34.930258    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 18 19:16:34.931990 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:34.931971    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8bdr7"]
Sep 18 19:16:35.016782 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:35.016753    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/399aa0ce-d6ee-4fd0-a027-edf66721c7f2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8bdr7\" (UID: \"399aa0ce-d6ee-4fd0-a027-edf66721c7f2\") " pod="openshift-insights/insights-runtime-extractor-8bdr7"
Sep 18 19:16:35.016782 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:35.016784    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/399aa0ce-d6ee-4fd0-a027-edf66721c7f2-crio-socket\") pod \"insights-runtime-extractor-8bdr7\" (UID: \"399aa0ce-d6ee-4fd0-a027-edf66721c7f2\") " pod="openshift-insights/insights-runtime-extractor-8bdr7"
Sep 18 19:16:35.017007 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:35.016804    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/399aa0ce-d6ee-4fd0-a027-edf66721c7f2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8bdr7\" (UID: \"399aa0ce-d6ee-4fd0-a027-edf66721c7f2\") " pod="openshift-insights/insights-runtime-extractor-8bdr7"
Sep 18 19:16:35.017007 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:35.016825    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc889\" (UniqueName: \"kubernetes.io/projected/399aa0ce-d6ee-4fd0-a027-edf66721c7f2-kube-api-access-xc889\") pod \"insights-runtime-extractor-8bdr7\" (UID: \"399aa0ce-d6ee-4fd0-a027-edf66721c7f2\") " pod="openshift-insights/insights-runtime-extractor-8bdr7"
Sep 18 19:16:35.017007 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:35.016849    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/399aa0ce-d6ee-4fd0-a027-edf66721c7f2-data-volume\") pod \"insights-runtime-extractor-8bdr7\" (UID: \"399aa0ce-d6ee-4fd0-a027-edf66721c7f2\") " pod="openshift-insights/insights-runtime-extractor-8bdr7"
Sep 18 19:16:35.117455 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:35.117417    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/399aa0ce-d6ee-4fd0-a027-edf66721c7f2-crio-socket\") pod \"insights-runtime-extractor-8bdr7\" (UID: \"399aa0ce-d6ee-4fd0-a027-edf66721c7f2\") " pod="openshift-insights/insights-runtime-extractor-8bdr7"
Sep 18 19:16:35.117455 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:35.117455    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/399aa0ce-d6ee-4fd0-a027-edf66721c7f2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8bdr7\" (UID: \"399aa0ce-d6ee-4fd0-a027-edf66721c7f2\") " pod="openshift-insights/insights-runtime-extractor-8bdr7"
Sep 18 19:16:35.117719 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:35.117471    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc889\" (UniqueName: \"kubernetes.io/projected/399aa0ce-d6ee-4fd0-a027-edf66721c7f2-kube-api-access-xc889\") pod \"insights-runtime-extractor-8bdr7\" (UID: \"399aa0ce-d6ee-4fd0-a027-edf66721c7f2\") " pod="openshift-insights/insights-runtime-extractor-8bdr7"
Sep 18 19:16:35.117719 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:35.117496    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/399aa0ce-d6ee-4fd0-a027-edf66721c7f2-data-volume\") pod \"insights-runtime-extractor-8bdr7\" (UID: \"399aa0ce-d6ee-4fd0-a027-edf66721c7f2\") " pod="openshift-insights/insights-runtime-extractor-8bdr7"
Sep 18 19:16:35.117719 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:35.117554    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/399aa0ce-d6ee-4fd0-a027-edf66721c7f2-crio-socket\") pod \"insights-runtime-extractor-8bdr7\" (UID: \"399aa0ce-d6ee-4fd0-a027-edf66721c7f2\") " pod="openshift-insights/insights-runtime-extractor-8bdr7"
Sep 18 19:16:35.117719 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:35.117584    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/399aa0ce-d6ee-4fd0-a027-edf66721c7f2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8bdr7\" (UID: \"399aa0ce-d6ee-4fd0-a027-edf66721c7f2\") " pod="openshift-insights/insights-runtime-extractor-8bdr7"
Sep 18 19:16:35.117719 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:16:35.117619    2522 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:16:35.117719 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:16:35.117653    2522 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:16:35.117719 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:16:35.117697    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/399aa0ce-d6ee-4fd0-a027-edf66721c7f2-insights-runtime-extractor-tls podName:399aa0ce-d6ee-4fd0-a027-edf66721c7f2 nodeName:}" failed. No retries permitted until 2026-09-18 19:16:35.617676111 +0000 UTC m=+155.501202584 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/399aa0ce-d6ee-4fd0-a027-edf66721c7f2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8bdr7" (UID: "399aa0ce-d6ee-4fd0-a027-edf66721c7f2") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:16:35.118020 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:16:35.117746    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/399aa0ce-d6ee-4fd0-a027-edf66721c7f2-kube-rbac-proxy-cm podName:399aa0ce-d6ee-4fd0-a027-edf66721c7f2 nodeName:}" failed. No retries permitted until 2026-09-18 19:16:35.617728297 +0000 UTC m=+155.501254767 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/399aa0ce-d6ee-4fd0-a027-edf66721c7f2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8bdr7" (UID: "399aa0ce-d6ee-4fd0-a027-edf66721c7f2") : configmap "kube-rbac-proxy" not found
Sep 18 19:16:35.118020 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:35.117931    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/399aa0ce-d6ee-4fd0-a027-edf66721c7f2-data-volume\") pod \"insights-runtime-extractor-8bdr7\" (UID: \"399aa0ce-d6ee-4fd0-a027-edf66721c7f2\") " pod="openshift-insights/insights-runtime-extractor-8bdr7"
Sep 18 19:16:35.128333 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:35.128310    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc889\" (UniqueName: \"kubernetes.io/projected/399aa0ce-d6ee-4fd0-a027-edf66721c7f2-kube-api-access-xc889\") pod \"insights-runtime-extractor-8bdr7\" (UID: \"399aa0ce-d6ee-4fd0-a027-edf66721c7f2\") " pod="openshift-insights/insights-runtime-extractor-8bdr7"
Sep 18 19:16:35.621418 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:35.621386    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/399aa0ce-d6ee-4fd0-a027-edf66721c7f2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8bdr7\" (UID: \"399aa0ce-d6ee-4fd0-a027-edf66721c7f2\") " pod="openshift-insights/insights-runtime-extractor-8bdr7"
Sep 18 19:16:35.621418 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:35.621417    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/399aa0ce-d6ee-4fd0-a027-edf66721c7f2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8bdr7\" (UID: \"399aa0ce-d6ee-4fd0-a027-edf66721c7f2\") " pod="openshift-insights/insights-runtime-extractor-8bdr7"
Sep 18 19:16:35.621716 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:16:35.621507    2522 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:16:35.621716 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:16:35.621523    2522 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:16:35.621716 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:16:35.621579    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/399aa0ce-d6ee-4fd0-a027-edf66721c7f2-kube-rbac-proxy-cm podName:399aa0ce-d6ee-4fd0-a027-edf66721c7f2 nodeName:}" failed. No retries permitted until 2026-09-18 19:16:36.621549472 +0000 UTC m=+156.505075938 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/399aa0ce-d6ee-4fd0-a027-edf66721c7f2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8bdr7" (UID: "399aa0ce-d6ee-4fd0-a027-edf66721c7f2") : configmap "kube-rbac-proxy" not found
Sep 18 19:16:35.621716 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:16:35.621593    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/399aa0ce-d6ee-4fd0-a027-edf66721c7f2-insights-runtime-extractor-tls podName:399aa0ce-d6ee-4fd0-a027-edf66721c7f2 nodeName:}" failed. No retries permitted until 2026-09-18 19:16:36.621587369 +0000 UTC m=+156.505113809 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/399aa0ce-d6ee-4fd0-a027-edf66721c7f2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8bdr7" (UID: "399aa0ce-d6ee-4fd0-a027-edf66721c7f2") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:16:36.626233 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:36.626199    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/399aa0ce-d6ee-4fd0-a027-edf66721c7f2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8bdr7\" (UID: \"399aa0ce-d6ee-4fd0-a027-edf66721c7f2\") " pod="openshift-insights/insights-runtime-extractor-8bdr7"
Sep 18 19:16:36.626698 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:36.626241    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/399aa0ce-d6ee-4fd0-a027-edf66721c7f2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8bdr7\" (UID: \"399aa0ce-d6ee-4fd0-a027-edf66721c7f2\") " pod="openshift-insights/insights-runtime-extractor-8bdr7"
Sep 18 19:16:36.626698 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:16:36.626316    2522 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:16:36.626698 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:16:36.626370    2522 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:16:36.626698 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:16:36.626385    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/399aa0ce-d6ee-4fd0-a027-edf66721c7f2-kube-rbac-proxy-cm podName:399aa0ce-d6ee-4fd0-a027-edf66721c7f2 nodeName:}" failed. No retries permitted until 2026-09-18 19:16:38.626369783 +0000 UTC m=+158.509896223 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/399aa0ce-d6ee-4fd0-a027-edf66721c7f2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8bdr7" (UID: "399aa0ce-d6ee-4fd0-a027-edf66721c7f2") : configmap "kube-rbac-proxy" not found
Sep 18 19:16:36.626698 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:16:36.626426    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/399aa0ce-d6ee-4fd0-a027-edf66721c7f2-insights-runtime-extractor-tls podName:399aa0ce-d6ee-4fd0-a027-edf66721c7f2 nodeName:}" failed. No retries permitted until 2026-09-18 19:16:38.626410546 +0000 UTC m=+158.509937003 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/399aa0ce-d6ee-4fd0-a027-edf66721c7f2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8bdr7" (UID: "399aa0ce-d6ee-4fd0-a027-edf66721c7f2") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:16:37.061996 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:16:37.061914    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-6446c8d75d-sszkh" podUID="00d2da2b-3370-4f0a-8104-9bc429576c69"
Sep 18 19:16:37.067035 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:16:37.067012    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-7b77b6b64d-dgr86" podUID="63dfbfc5-e1e8-4e79-8df9-d47e16ba5289"
Sep 18 19:16:37.154694 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:37.154658    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-sszkh"
Sep 18 19:16:37.154869 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:37.154661    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b77b6b64d-dgr86"
Sep 18 19:16:37.198285 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:16:37.198248    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-mrd75" podUID="72d866b0-9634-442e-bd20-d90321a00452"
Sep 18 19:16:37.204207 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:37.204179    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-752fm"]
Sep 18 19:16:37.207418 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:37.207400    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-752fm"
Sep 18 19:16:37.210970 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:37.210941    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 18 19:16:37.211073 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:37.210988    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 18 19:16:37.211136 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:37.211085    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-fsnrn\""
Sep 18 19:16:37.212023 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:37.212003    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 18 19:16:37.212115 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:37.212038    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 18 19:16:37.224060 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:37.224038    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-752fm"]
Sep 18 19:16:37.231496 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:37.231472    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d965e394-2035-42b7-a36b-51022cba76b9-signing-key\") pod \"service-ca-55f6fc7b86-752fm\" (UID: \"d965e394-2035-42b7-a36b-51022cba76b9\") " pod="openshift-service-ca/service-ca-55f6fc7b86-752fm"
Sep 18 19:16:37.231606 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:37.231532    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d965e394-2035-42b7-a36b-51022cba76b9-signing-cabundle\") pod \"service-ca-55f6fc7b86-752fm\" (UID: \"d965e394-2035-42b7-a36b-51022cba76b9\") " pod="openshift-service-ca/service-ca-55f6fc7b86-752fm"
Sep 18 19:16:37.231723 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:37.231704    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck8qc\" (UniqueName: \"kubernetes.io/projected/d965e394-2035-42b7-a36b-51022cba76b9-kube-api-access-ck8qc\") pod \"service-ca-55f6fc7b86-752fm\" (UID: \"d965e394-2035-42b7-a36b-51022cba76b9\") " pod="openshift-service-ca/service-ca-55f6fc7b86-752fm"
Sep 18 19:16:37.332769 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:37.332744    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d965e394-2035-42b7-a36b-51022cba76b9-signing-cabundle\") pod \"service-ca-55f6fc7b86-752fm\" (UID: \"d965e394-2035-42b7-a36b-51022cba76b9\") " pod="openshift-service-ca/service-ca-55f6fc7b86-752fm"
Sep 18 19:16:37.332904 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:37.332786    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck8qc\" (UniqueName: \"kubernetes.io/projected/d965e394-2035-42b7-a36b-51022cba76b9-kube-api-access-ck8qc\") pod \"service-ca-55f6fc7b86-752fm\" (UID: \"d965e394-2035-42b7-a36b-51022cba76b9\") " pod="openshift-service-ca/service-ca-55f6fc7b86-752fm"
Sep 18 19:16:37.332985 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:37.332962    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d965e394-2035-42b7-a36b-51022cba76b9-signing-key\") pod \"service-ca-55f6fc7b86-752fm\" (UID: \"d965e394-2035-42b7-a36b-51022cba76b9\") " pod="openshift-service-ca/service-ca-55f6fc7b86-752fm"
Sep 18 19:16:37.333386 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:37.333361    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d965e394-2035-42b7-a36b-51022cba76b9-signing-cabundle\") pod \"service-ca-55f6fc7b86-752fm\" (UID: \"d965e394-2035-42b7-a36b-51022cba76b9\") " pod="openshift-service-ca/service-ca-55f6fc7b86-752fm"
Sep 18 19:16:37.335226 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:37.335205    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d965e394-2035-42b7-a36b-51022cba76b9-signing-key\") pod \"service-ca-55f6fc7b86-752fm\" (UID: \"d965e394-2035-42b7-a36b-51022cba76b9\") " pod="openshift-service-ca/service-ca-55f6fc7b86-752fm"
Sep 18 19:16:37.343627 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:37.343608    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck8qc\" (UniqueName: \"kubernetes.io/projected/d965e394-2035-42b7-a36b-51022cba76b9-kube-api-access-ck8qc\") pod \"service-ca-55f6fc7b86-752fm\" (UID: \"d965e394-2035-42b7-a36b-51022cba76b9\") " pod="openshift-service-ca/service-ca-55f6fc7b86-752fm"
Sep 18 19:16:37.516715 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:37.516677    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-752fm"
Sep 18 19:16:37.798546 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:37.798519    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-752fm"]
Sep 18 19:16:37.801664 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:16:37.801635    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd965e394_2035_42b7_a36b_51022cba76b9.slice/crio-e36c0d1fc33a17770cd8b3be3d5631a0c27941b02651b502fac5ea2f0a2c68e4 WatchSource:0}: Error finding container e36c0d1fc33a17770cd8b3be3d5631a0c27941b02651b502fac5ea2f0a2c68e4: Status 404 returned error can't find the container with id e36c0d1fc33a17770cd8b3be3d5631a0c27941b02651b502fac5ea2f0a2c68e4
Sep 18 19:16:37.803534 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:37.803496    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-w4tjm_5b796f3a-2d43-47a8-8d73-a34a2af2d62b/dns-node-resolver/0.log"
Sep 18 19:16:38.159211 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:38.159169    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-752fm" event={"ID":"d965e394-2035-42b7-a36b-51022cba76b9","Type":"ContainerStarted","Data":"e36c0d1fc33a17770cd8b3be3d5631a0c27941b02651b502fac5ea2f0a2c68e4"}
Sep 18 19:16:38.159211 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:38.159188    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mrd75"
Sep 18 19:16:38.406639 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:38.406609    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hmfqw_438a5295-6f51-49ed-99d1-96a8fe037805/node-ca/0.log"
Sep 18 19:16:38.643794 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:38.643757    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/399aa0ce-d6ee-4fd0-a027-edf66721c7f2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8bdr7\" (UID: \"399aa0ce-d6ee-4fd0-a027-edf66721c7f2\") " pod="openshift-insights/insights-runtime-extractor-8bdr7"
Sep 18 19:16:38.643988 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:38.643808    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/399aa0ce-d6ee-4fd0-a027-edf66721c7f2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8bdr7\" (UID: \"399aa0ce-d6ee-4fd0-a027-edf66721c7f2\") " pod="openshift-insights/insights-runtime-extractor-8bdr7"
Sep 18 19:16:38.643988 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:16:38.643912    2522 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:16:38.643988 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:16:38.643979    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/399aa0ce-d6ee-4fd0-a027-edf66721c7f2-kube-rbac-proxy-cm podName:399aa0ce-d6ee-4fd0-a027-edf66721c7f2 nodeName:}" failed. No retries permitted until 2026-09-18 19:16:42.64396156 +0000 UTC m=+162.527488020 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/399aa0ce-d6ee-4fd0-a027-edf66721c7f2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8bdr7" (UID: "399aa0ce-d6ee-4fd0-a027-edf66721c7f2") : configmap "kube-rbac-proxy" not found
Sep 18 19:16:38.644155 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:16:38.643913    2522 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:16:38.644155 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:16:38.644076    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/399aa0ce-d6ee-4fd0-a027-edf66721c7f2-insights-runtime-extractor-tls podName:399aa0ce-d6ee-4fd0-a027-edf66721c7f2 nodeName:}" failed. No retries permitted until 2026-09-18 19:16:42.644059725 +0000 UTC m=+162.527586170 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/399aa0ce-d6ee-4fd0-a027-edf66721c7f2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8bdr7" (UID: "399aa0ce-d6ee-4fd0-a027-edf66721c7f2") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:16:38.741213 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:16:38.741174    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-5j97v" podUID="f516e39e-b125-4617-94ce-7f0dccfae7bb"
Sep 18 19:16:39.409394 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:39.409361    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pmrmr_47bbbb88-fe11-4227-bddc-4e124994d36a/serve-healthcheck-canary/0.log"
Sep 18 19:16:40.167287 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:40.167255    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-752fm" event={"ID":"d965e394-2035-42b7-a36b-51022cba76b9","Type":"ContainerStarted","Data":"e520201dd5392d41b955b56d4dee6ccd72399115dfab292a61cd5fbffd60f8b0"}
Sep 18 19:16:40.198927 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:40.198882    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-55f6fc7b86-752fm" podStartSLOduration=1.388807904 podStartE2EDuration="3.19886855s" totalImagesPullingTime="1.810060646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:16:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:16:40.198833934 +0000 UTC m=+160.082360396" watchObservedRunningTime="2026-09-18 19:16:40.19886855 +0000 UTC m=+160.082395014"
Sep 18 19:16:40.209799 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:40.209778    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-qkrw2_09a74ca7-303a-49c9-9967-98074bbfbb17/kube-storage-version-migrator-operator/0.log"
Sep 18 19:16:41.971749 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:41.971716    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/00d2da2b-3370-4f0a-8104-9bc429576c69-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-sszkh\" (UID: \"00d2da2b-3370-4f0a-8104-9bc429576c69\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-sszkh"
Sep 18 19:16:41.972151 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:41.971772    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-registry-tls\") pod \"image-registry-7b77b6b64d-dgr86\" (UID: \"63dfbfc5-e1e8-4e79-8df9-d47e16ba5289\") " pod="openshift-image-registry/image-registry-7b77b6b64d-dgr86"
Sep 18 19:16:41.972151 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:16:41.971857    2522 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 18 19:16:41.972151 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:16:41.971916    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00d2da2b-3370-4f0a-8104-9bc429576c69-networking-console-plugin-cert podName:00d2da2b-3370-4f0a-8104-9bc429576c69 nodeName:}" failed. No retries permitted until 2026-09-18 19:18:43.971901866 +0000 UTC m=+283.855428306 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/00d2da2b-3370-4f0a-8104-9bc429576c69-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-sszkh" (UID: "00d2da2b-3370-4f0a-8104-9bc429576c69") : secret "networking-console-plugin-cert" not found
Sep 18 19:16:41.972151 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:16:41.971863    2522 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:16:41.972151 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:16:41.971957    2522 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b77b6b64d-dgr86: secret "image-registry-tls" not found
Sep 18 19:16:41.972151 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:16:41.972006    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-registry-tls podName:63dfbfc5-e1e8-4e79-8df9-d47e16ba5289 nodeName:}" failed. No retries permitted until 2026-09-18 19:18:43.971995806 +0000 UTC m=+283.855522263 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-registry-tls") pod "image-registry-7b77b6b64d-dgr86" (UID: "63dfbfc5-e1e8-4e79-8df9-d47e16ba5289") : secret "image-registry-tls" not found
Sep 18 19:16:42.072999 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:42.072953    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/72d866b0-9634-442e-bd20-d90321a00452-metrics-tls\") pod \"dns-default-mrd75\" (UID: \"72d866b0-9634-442e-bd20-d90321a00452\") " pod="openshift-dns/dns-default-mrd75"
Sep 18 19:16:42.073139 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:16:42.073107    2522 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:16:42.073176 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:16:42.073170    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72d866b0-9634-442e-bd20-d90321a00452-metrics-tls podName:72d866b0-9634-442e-bd20-d90321a00452 nodeName:}" failed. No retries permitted until 2026-09-18 19:18:44.073155408 +0000 UTC m=+283.956681852 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/72d866b0-9634-442e-bd20-d90321a00452-metrics-tls") pod "dns-default-mrd75" (UID: "72d866b0-9634-442e-bd20-d90321a00452") : secret "dns-default-metrics-tls" not found
Sep 18 19:16:42.677779 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:42.677741    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/399aa0ce-d6ee-4fd0-a027-edf66721c7f2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8bdr7\" (UID: \"399aa0ce-d6ee-4fd0-a027-edf66721c7f2\") " pod="openshift-insights/insights-runtime-extractor-8bdr7"
Sep 18 19:16:42.677779 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:42.677785    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/399aa0ce-d6ee-4fd0-a027-edf66721c7f2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8bdr7\" (UID: \"399aa0ce-d6ee-4fd0-a027-edf66721c7f2\") " pod="openshift-insights/insights-runtime-extractor-8bdr7"
Sep 18 19:16:42.677995 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:16:42.677910    2522 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:16:42.677995 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:16:42.677946    2522 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:16:42.677995 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:16:42.677970    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/399aa0ce-d6ee-4fd0-a027-edf66721c7f2-insights-runtime-extractor-tls podName:399aa0ce-d6ee-4fd0-a027-edf66721c7f2 nodeName:}" failed. No retries permitted until 2026-09-18 19:16:50.677955149 +0000 UTC m=+170.561481594 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/399aa0ce-d6ee-4fd0-a027-edf66721c7f2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8bdr7" (UID: "399aa0ce-d6ee-4fd0-a027-edf66721c7f2") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:16:42.678092 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:16:42.677998    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/399aa0ce-d6ee-4fd0-a027-edf66721c7f2-kube-rbac-proxy-cm podName:399aa0ce-d6ee-4fd0-a027-edf66721c7f2 nodeName:}" failed. No retries permitted until 2026-09-18 19:16:50.677985252 +0000 UTC m=+170.561511708 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/399aa0ce-d6ee-4fd0-a027-edf66721c7f2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8bdr7" (UID: "399aa0ce-d6ee-4fd0-a027-edf66721c7f2") : configmap "kube-rbac-proxy" not found
Sep 18 19:16:50.741809 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:50.741777    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/399aa0ce-d6ee-4fd0-a027-edf66721c7f2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8bdr7\" (UID: \"399aa0ce-d6ee-4fd0-a027-edf66721c7f2\") " pod="openshift-insights/insights-runtime-extractor-8bdr7"
Sep 18 19:16:50.741809 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:50.741814    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/399aa0ce-d6ee-4fd0-a027-edf66721c7f2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8bdr7\" (UID: \"399aa0ce-d6ee-4fd0-a027-edf66721c7f2\") " pod="openshift-insights/insights-runtime-extractor-8bdr7"
Sep 18 19:16:50.742224 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:16:50.741940    2522 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:16:50.742224 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:16:50.742004    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/399aa0ce-d6ee-4fd0-a027-edf66721c7f2-insights-runtime-extractor-tls podName:399aa0ce-d6ee-4fd0-a027-edf66721c7f2 nodeName:}" failed. No retries permitted until 2026-09-18 19:17:06.741989411 +0000 UTC m=+186.625515851 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/399aa0ce-d6ee-4fd0-a027-edf66721c7f2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8bdr7" (UID: "399aa0ce-d6ee-4fd0-a027-edf66721c7f2") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:16:50.742438 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:50.742419    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/399aa0ce-d6ee-4fd0-a027-edf66721c7f2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8bdr7\" (UID: \"399aa0ce-d6ee-4fd0-a027-edf66721c7f2\") " pod="openshift-insights/insights-runtime-extractor-8bdr7"
Sep 18 19:16:53.716748 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:16:53.716660    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5j97v"
Sep 18 19:17:04.892108 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:04.892071    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-zm6d2"]
Sep 18 19:17:04.895636 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:04.895605    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-zm6d2"
Sep 18 19:17:04.898918 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:04.898895    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 18 19:17:04.899029 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:04.898944    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-9cbww\""
Sep 18 19:17:04.925618 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:04.925594    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-zm6d2"]
Sep 18 19:17:04.951824 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:04.951794    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/533ce2b0-2a99-4011-a5d9-c85e3cd87170-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-zm6d2\" (UID: \"533ce2b0-2a99-4011-a5d9-c85e3cd87170\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-zm6d2"
Sep 18 19:17:05.052435 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:05.052393    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/533ce2b0-2a99-4011-a5d9-c85e3cd87170-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-zm6d2\" (UID: \"533ce2b0-2a99-4011-a5d9-c85e3cd87170\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-zm6d2"
Sep 18 19:17:05.052632 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:17:05.052537    2522 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 18 19:17:05.052632 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:17:05.052614    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/533ce2b0-2a99-4011-a5d9-c85e3cd87170-tls-certificates podName:533ce2b0-2a99-4011-a5d9-c85e3cd87170 nodeName:}" failed. No retries permitted until 2026-09-18 19:17:05.552599034 +0000 UTC m=+185.436125487 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/533ce2b0-2a99-4011-a5d9-c85e3cd87170-tls-certificates") pod "prometheus-operator-admission-webhook-5c6747dc7b-zm6d2" (UID: "533ce2b0-2a99-4011-a5d9-c85e3cd87170") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 18 19:17:05.557687 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:05.557660    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/533ce2b0-2a99-4011-a5d9-c85e3cd87170-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-zm6d2\" (UID: \"533ce2b0-2a99-4011-a5d9-c85e3cd87170\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-zm6d2"
Sep 18 19:17:05.559817 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:05.559799    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/533ce2b0-2a99-4011-a5d9-c85e3cd87170-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-zm6d2\" (UID: \"533ce2b0-2a99-4011-a5d9-c85e3cd87170\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-zm6d2"
Sep 18 19:17:05.804484 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:05.804390    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-zm6d2"
Sep 18 19:17:06.089175 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:06.089124    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-zm6d2"]
Sep 18 19:17:06.092005 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:17:06.091982    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod533ce2b0_2a99_4011_a5d9_c85e3cd87170.slice/crio-817f3a1c179e35581d0003d57cccfeed3c7991476345d5caf572db17d4371fb2 WatchSource:0}: Error finding container 817f3a1c179e35581d0003d57cccfeed3c7991476345d5caf572db17d4371fb2: Status 404 returned error can't find the container with id 817f3a1c179e35581d0003d57cccfeed3c7991476345d5caf572db17d4371fb2
Sep 18 19:17:06.246959 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:06.246924    2522 generic.go:356] "Generic (PLEG): container finished" podID="5a5340ed-d845-4599-a58c-efbd5f71a12b" containerID="a4219d58de8a75f885f8738dd7f79564916a23a8c4f14936a5e123b679f05f73" exitCode=255
Sep 18 19:17:06.247127 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:06.246997    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d544df478-5nch5" event={"ID":"5a5340ed-d845-4599-a58c-efbd5f71a12b","Type":"ContainerDied","Data":"a4219d58de8a75f885f8738dd7f79564916a23a8c4f14936a5e123b679f05f73"}
Sep 18 19:17:06.248210 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:06.248183    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-zm6d2" event={"ID":"533ce2b0-2a99-4011-a5d9-c85e3cd87170","Type":"ContainerStarted","Data":"817f3a1c179e35581d0003d57cccfeed3c7991476345d5caf572db17d4371fb2"}
Sep 18 19:17:06.253044 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:06.253027    2522 scope.go:117] "RemoveContainer" containerID="a4219d58de8a75f885f8738dd7f79564916a23a8c4f14936a5e123b679f05f73"
Sep 18 19:17:06.767231 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:06.767201    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/399aa0ce-d6ee-4fd0-a027-edf66721c7f2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8bdr7\" (UID: \"399aa0ce-d6ee-4fd0-a027-edf66721c7f2\") " pod="openshift-insights/insights-runtime-extractor-8bdr7"
Sep 18 19:17:06.769694 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:06.769671    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/399aa0ce-d6ee-4fd0-a027-edf66721c7f2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8bdr7\" (UID: \"399aa0ce-d6ee-4fd0-a027-edf66721c7f2\") " pod="openshift-insights/insights-runtime-extractor-8bdr7"
Sep 18 19:17:07.030898 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:07.030815    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xd5sl\""
Sep 18 19:17:07.034807 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:07.034774    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8bdr7"
Sep 18 19:17:07.254834 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:07.254801    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d544df478-5nch5" event={"ID":"5a5340ed-d845-4599-a58c-efbd5f71a12b","Type":"ContainerStarted","Data":"bfba6624241375d6ef28b0ba09c8633a9bfedc2a798088dde5cf105dc07e58d4"}
Sep 18 19:17:07.484931 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:07.484855    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8bdr7"]
Sep 18 19:17:07.487317 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:17:07.487293    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod399aa0ce_d6ee_4fd0_a027_edf66721c7f2.slice/crio-343197894af563a3bee2726af96a7f4f045f0e5c117673f6c05d261a185df9e5 WatchSource:0}: Error finding container 343197894af563a3bee2726af96a7f4f045f0e5c117673f6c05d261a185df9e5: Status 404 returned error can't find the container with id 343197894af563a3bee2726af96a7f4f045f0e5c117673f6c05d261a185df9e5
Sep 18 19:17:08.259160 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:08.259123    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8bdr7" event={"ID":"399aa0ce-d6ee-4fd0-a027-edf66721c7f2","Type":"ContainerStarted","Data":"85684aabd1fd75129cca5341cbb9951e2fdd06d22cab9e2bd0ef8d49fcf0622c"}
Sep 18 19:17:08.259552 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:08.259190    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8bdr7" event={"ID":"399aa0ce-d6ee-4fd0-a027-edf66721c7f2","Type":"ContainerStarted","Data":"542e8789769421a13876d1107b8ea2cb0fdd85bec80464d9b4248004a29a354f"}
Sep 18 19:17:08.259552 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:08.259212    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8bdr7" event={"ID":"399aa0ce-d6ee-4fd0-a027-edf66721c7f2","Type":"ContainerStarted","Data":"343197894af563a3bee2726af96a7f4f045f0e5c117673f6c05d261a185df9e5"}
Sep 18 19:17:08.260501 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:08.260475    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-zm6d2" event={"ID":"533ce2b0-2a99-4011-a5d9-c85e3cd87170","Type":"ContainerStarted","Data":"70f017a7ef5505aa9a64fdeffc67f3ddc5bb103e3720e5a374fe8b65fdc1e16f"}
Sep 18 19:17:08.260648 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:08.260632    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-zm6d2"
Sep 18 19:17:08.265396 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:08.265367    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-zm6d2"
Sep 18 19:17:08.282199 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:08.282150    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-zm6d2" podStartSLOduration=3.143274756 podStartE2EDuration="4.282135009s" totalImagesPullingTime="1.138860253s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:17:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:17:08.28037931 +0000 UTC m=+188.163905772" watchObservedRunningTime="2026-09-18 19:17:08.282135009 +0000 UTC m=+188.165661500"
Sep 18 19:17:10.269043 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:10.269005    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8bdr7" event={"ID":"399aa0ce-d6ee-4fd0-a027-edf66721c7f2","Type":"ContainerStarted","Data":"5dbff0c474f55e8fb62082c0db5dbf7baed4cbd965cee378fb98c1cff76684c7"}
Sep 18 19:17:10.292894 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:10.292841    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-8bdr7" podStartSLOduration=34.531420252 podStartE2EDuration="36.292823734s" totalImagesPullingTime="1.761403482s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:16:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:17:10.292610414 +0000 UTC m=+190.176136875" watchObservedRunningTime="2026-09-18 19:17:10.292823734 +0000 UTC m=+190.176350195"
Sep 18 19:17:12.687498 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:12.687462    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6594f4c75-f57kg"]
Sep 18 19:17:12.690704 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:12.690686    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-f57kg"
Sep 18 19:17:12.693379 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:12.693359    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-b6ls4\""
Sep 18 19:17:12.693667 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:12.693652    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 18 19:17:12.694102 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:12.694089    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 18 19:17:12.711238 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:12.711199    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-f57kg"]
Sep 18 19:17:12.816582 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:12.816542    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kskmc\" (UniqueName: \"kubernetes.io/projected/25102f99-24b3-4c22-81ac-ee32e1fceabf-kube-api-access-kskmc\") pod \"downloads-6594f4c75-f57kg\" (UID: \"25102f99-24b3-4c22-81ac-ee32e1fceabf\") " pod="openshift-console/downloads-6594f4c75-f57kg"
Sep 18 19:17:12.916872 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:12.916838    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kskmc\" (UniqueName: \"kubernetes.io/projected/25102f99-24b3-4c22-81ac-ee32e1fceabf-kube-api-access-kskmc\") pod \"downloads-6594f4c75-f57kg\" (UID: \"25102f99-24b3-4c22-81ac-ee32e1fceabf\") " pod="openshift-console/downloads-6594f4c75-f57kg"
Sep 18 19:17:12.930976 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:12.930946    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kskmc\" (UniqueName: \"kubernetes.io/projected/25102f99-24b3-4c22-81ac-ee32e1fceabf-kube-api-access-kskmc\") pod \"downloads-6594f4c75-f57kg\" (UID: \"25102f99-24b3-4c22-81ac-ee32e1fceabf\") " pod="openshift-console/downloads-6594f4c75-f57kg"
Sep 18 19:17:12.999270 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:12.999175    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-f57kg"
Sep 18 19:17:13.348161 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:13.348129    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-f57kg"]
Sep 18 19:17:13.351230 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:17:13.351199    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25102f99_24b3_4c22_81ac_ee32e1fceabf.slice/crio-f3ba483b42d03d9969a575df8a9f64c2993541f281bf473160da10696491ae53 WatchSource:0}: Error finding container f3ba483b42d03d9969a575df8a9f64c2993541f281bf473160da10696491ae53: Status 404 returned error can't find the container with id f3ba483b42d03d9969a575df8a9f64c2993541f281bf473160da10696491ae53
Sep 18 19:17:13.747087 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:13.747004    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-q9q7d"]
Sep 18 19:17:13.751764 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:13.751743    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-q9q7d"
Sep 18 19:17:13.752674 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:13.752642    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-58tlh"]
Sep 18 19:17:13.755148 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:13.755130    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 18 19:17:13.755332 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:13.755316    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 18 19:17:13.755905 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:13.755890    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-58tlh"
Sep 18 19:17:13.756474 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:13.756354    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 18 19:17:13.756474 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:13.756371    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 18 19:17:13.756474 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:13.756403    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 18 19:17:13.756474 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:13.756464    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-t5klp\""
Sep 18 19:17:13.758441 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:13.758423    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-mg7jn\""
Sep 18 19:17:13.758777 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:13.758756    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 18 19:17:13.758885 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:13.758863    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 18 19:17:13.760015 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:13.759725    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 18 19:17:13.773680 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:13.773641    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-q9q7d"]
Sep 18 19:17:13.774699 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:13.774676    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-58tlh"]
Sep 18 19:17:13.790070 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:13.790039    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-cfpmj"]
Sep 18 19:17:13.793294 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:13.793277    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cfpmj"
Sep 18 19:17:13.796654 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:13.796633    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 18 19:17:13.796855 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:13.796665    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 18 19:17:13.797520 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:13.797499    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2n9jz\""
Sep 18 19:17:13.797825 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:13.797805    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 18 19:17:13.824204 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:13.824165    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2317359e-59e6-4734-8ac4-f01b0bb27d57-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-58tlh\" (UID: \"2317359e-59e6-4734-8ac4-f01b0bb27d57\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-58tlh"
Sep 18 19:17:13.824374 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:13.824220    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2317359e-59e6-4734-8ac4-f01b0bb27d57-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-58tlh\" (UID: \"2317359e-59e6-4734-8ac4-f01b0bb27d57\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-58tlh"
Sep 18 19:17:13.824374 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:13.824248    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1d034d6d-be14-44c8-a72b-b7351a3f4a40-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-q9q7d\" (UID: \"1d034d6d-be14-44c8-a72b-b7351a3f4a40\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-q9q7d"
Sep 18 19:17:13.824374 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:13.824273    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d034d6d-be14-44c8-a72b-b7351a3f4a40-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-q9q7d\" (UID: \"1d034d6d-be14-44c8-a72b-b7351a3f4a40\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-q9q7d"
Sep 18 19:17:13.824374 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:13.824314    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2317359e-59e6-4734-8ac4-f01b0bb27d57-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-58tlh\" (UID: \"2317359e-59e6-4734-8ac4-f01b0bb27d57\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-58tlh"
Sep 18 19:17:13.824374 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:13.824340    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2fk8\" (UniqueName: \"kubernetes.io/projected/2317359e-59e6-4734-8ac4-f01b0bb27d57-kube-api-access-l2fk8\") pod \"kube-state-metrics-84c885658c-58tlh\" (UID: \"2317359e-59e6-4734-8ac4-f01b0bb27d57\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-58tlh"
Sep 18 19:17:13.824374 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:13.824371    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2317359e-59e6-4734-8ac4-f01b0bb27d57-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-58tlh\" (UID: \"2317359e-59e6-4734-8ac4-f01b0bb27d57\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-58tlh"
Sep 18 19:17:13.824715 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:13.824469    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d034d6d-be14-44c8-a72b-b7351a3f4a40-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-q9q7d\" (UID: \"1d034d6d-be14-44c8-a72b-b7351a3f4a40\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-q9q7d"
Sep 18 19:17:13.824715 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:13.824502    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2317359e-59e6-4734-8ac4-f01b0bb27d57-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-58tlh\" (UID: \"2317359e-59e6-4734-8ac4-f01b0bb27d57\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-58tlh"
Sep 18 19:17:13.824715 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:13.824527    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbhvv\" (UniqueName: \"kubernetes.io/projected/1d034d6d-be14-44c8-a72b-b7351a3f4a40-kube-api-access-wbhvv\") pod \"openshift-state-metrics-86c5bc79d6-q9q7d\" (UID: \"1d034d6d-be14-44c8-a72b-b7351a3f4a40\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-q9q7d"
Sep 18 19:17:13.924965 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:13.924927    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/896188ce-60fe-4f42-bced-50927342f530-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cfpmj\" (UID: \"896188ce-60fe-4f42-bced-50927342f530\") " pod="openshift-monitoring/node-exporter-cfpmj"
Sep 18 19:17:13.924965 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:13.924968    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/896188ce-60fe-4f42-bced-50927342f530-node-exporter-tls\") pod \"node-exporter-cfpmj\" (UID: \"896188ce-60fe-4f42-bced-50927342f530\") " pod="openshift-monitoring/node-exporter-cfpmj"
Sep 18 19:17:13.925215 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:13.925002    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2317359e-59e6-4734-8ac4-f01b0bb27d57-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-58tlh\" (UID: \"2317359e-59e6-4734-8ac4-f01b0bb27d57\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-58tlh"
Sep 18 19:17:13.925215 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:13.925031    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1d034d6d-be14-44c8-a72b-b7351a3f4a40-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-q9q7d\" (UID: \"1d034d6d-be14-44c8-a72b-b7351a3f4a40\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-q9q7d"
Sep 18 19:17:13.925215 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:13.925052    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d034d6d-be14-44c8-a72b-b7351a3f4a40-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-q9q7d\" (UID: \"1d034d6d-be14-44c8-a72b-b7351a3f4a40\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-q9q7d"
Sep 18 19:17:13.925215 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:13.925079    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/896188ce-60fe-4f42-bced-50927342f530-node-exporter-textfile\") pod \"node-exporter-cfpmj\" (UID: \"896188ce-60fe-4f42-bced-50927342f530\") " pod="openshift-monitoring/node-exporter-cfpmj"
Sep 18 19:17:13.925215 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:13.925119    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2317359e-59e6-4734-8ac4-f01b0bb27d57-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-58tlh\" (UID: \"2317359e-59e6-4734-8ac4-f01b0bb27d57\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-58tlh"
Sep 18 19:17:13.925215 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:13.925145    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2fk8\" (UniqueName: \"kubernetes.io/projected/2317359e-59e6-4734-8ac4-f01b0bb27d57-kube-api-access-l2fk8\") pod \"kube-state-metrics-84c885658c-58tlh\" (UID: \"2317359e-59e6-4734-8ac4-f01b0bb27d57\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-58tlh"
Sep 18 19:17:13.925215 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:13.925198    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2317359e-59e6-4734-8ac4-f01b0bb27d57-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-58tlh\" (UID: \"2317359e-59e6-4734-8ac4-f01b0bb27d57\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-58tlh"
Sep 18 19:17:13.925521 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:17:13.925390    2522 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 18 19:17:13.925521 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:17:13.925458    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2317359e-59e6-4734-8ac4-f01b0bb27d57-kube-state-metrics-tls podName:2317359e-59e6-4734-8ac4-f01b0bb27d57 nodeName:}" failed. No retries permitted until 2026-09-18 19:17:14.425437451 +0000 UTC m=+194.308963900 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/2317359e-59e6-4734-8ac4-f01b0bb27d57-kube-state-metrics-tls") pod "kube-state-metrics-84c885658c-58tlh" (UID: "2317359e-59e6-4734-8ac4-f01b0bb27d57") : secret "kube-state-metrics-tls" not found
Sep 18 19:17:13.925923 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:13.925901    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d034d6d-be14-44c8-a72b-b7351a3f4a40-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-q9q7d\" (UID: \"1d034d6d-be14-44c8-a72b-b7351a3f4a40\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-q9q7d"
Sep 18 19:17:13.926025 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:13.925923    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2317359e-59e6-4734-8ac4-f01b0bb27d57-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-58tlh\" (UID: \"2317359e-59e6-4734-8ac4-f01b0bb27d57\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-58tlh"
Sep 18 19:17:13.926025 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:13.926000    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/896188ce-60fe-4f42-bced-50927342f530-sys\") pod \"node-exporter-cfpmj\" (UID: \"896188ce-60fe-4f42-bced-50927342f530\") " pod="openshift-monitoring/node-exporter-cfpmj"
Sep 18 19:17:13.926136 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:13.926031    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gxpl\" (UniqueName: \"kubernetes.io/projected/896188ce-60fe-4f42-bced-50927342f530-kube-api-access-2gxpl\") pod \"node-exporter-cfpmj\" (UID: \"896188ce-60fe-4f42-bced-50927342f530\") " pod="openshift-monitoring/node-exporter-cfpmj"
Sep 18 19:17:13.926136 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:13.926067    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/896188ce-60fe-4f42-bced-50927342f530-node-exporter-accelerators-collector-config\") pod \"node-exporter-cfpmj\" (UID: \"896188ce-60fe-4f42-bced-50927342f530\") " pod="openshift-monitoring/node-exporter-cfpmj"
Sep 18 19:17:13.926248 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:13.926138    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/896188ce-60fe-4f42-bced-50927342f530-root\") pod \"node-exporter-cfpmj\" (UID: \"896188ce-60fe-4f42-bced-50927342f530\") " pod="openshift-monitoring/node-exporter-cfpmj"
Sep 18 19:17:13.926248 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:13.926169    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2317359e-59e6-4734-8ac4-f01b0bb27d57-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-58tlh\" (UID: \"2317359e-59e6-4734-8ac4-f01b0bb27d57\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-58tlh"
Sep 18 19:17:13.926248 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:13.926174    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/896188ce-60fe-4f42-bced-50927342f530-metrics-client-ca\") pod \"node-exporter-cfpmj\" (UID: \"896188ce-60fe-4f42-bced-50927342f530\") " pod="openshift-monitoring/node-exporter-cfpmj"
Sep 18 19:17:13.926248 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:13.926232    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d034d6d-be14-44c8-a72b-b7351a3f4a40-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-q9q7d\" (UID: \"1d034d6d-be14-44c8-a72b-b7351a3f4a40\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-q9q7d"
Sep 18 19:17:13.926441 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:13.926263    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/896188ce-60fe-4f42-bced-50927342f530-node-exporter-wtmp\") pod \"node-exporter-cfpmj\" (UID: \"896188ce-60fe-4f42-bced-50927342f530\") " pod="openshift-monitoring/node-exporter-cfpmj"
Sep 18 19:17:13.926441 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:13.926293    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2317359e-59e6-4734-8ac4-f01b0bb27d57-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-58tlh\" (UID: \"2317359e-59e6-4734-8ac4-f01b0bb27d57\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-58tlh"
Sep 18 19:17:13.926441 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:17:13.926308    2522 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 18 19:17:13.926441 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:13.926316    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbhvv\" (UniqueName: \"kubernetes.io/projected/1d034d6d-be14-44c8-a72b-b7351a3f4a40-kube-api-access-wbhvv\") pod \"openshift-state-metrics-86c5bc79d6-q9q7d\" (UID: \"1d034d6d-be14-44c8-a72b-b7351a3f4a40\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-q9q7d"
Sep 18 19:17:13.926441 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:17:13.926357    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d034d6d-be14-44c8-a72b-b7351a3f4a40-openshift-state-metrics-tls podName:1d034d6d-be14-44c8-a72b-b7351a3f4a40 nodeName:}" failed. No retries permitted until 2026-09-18 19:17:14.426339413 +0000 UTC m=+194.309865852 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/1d034d6d-be14-44c8-a72b-b7351a3f4a40-openshift-state-metrics-tls") pod "openshift-state-metrics-86c5bc79d6-q9q7d" (UID: "1d034d6d-be14-44c8-a72b-b7351a3f4a40") : secret "openshift-state-metrics-tls" not found
Sep 18 19:17:13.926441 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:13.926385    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2317359e-59e6-4734-8ac4-f01b0bb27d57-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-58tlh\" (UID: \"2317359e-59e6-4734-8ac4-f01b0bb27d57\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-58tlh"
Sep 18 19:17:13.926768 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:13.926648    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2317359e-59e6-4734-8ac4-f01b0bb27d57-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-58tlh\" (UID: \"2317359e-59e6-4734-8ac4-f01b0bb27d57\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-58tlh"
Sep 18 19:17:13.927866 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:13.927836    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1d034d6d-be14-44c8-a72b-b7351a3f4a40-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-q9q7d\" (UID: \"1d034d6d-be14-44c8-a72b-b7351a3f4a40\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-q9q7d"
Sep 18 19:17:13.928490 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:13.928472    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2317359e-59e6-4734-8ac4-f01b0bb27d57-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-58tlh\" (UID: \"2317359e-59e6-4734-8ac4-f01b0bb27d57\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-58tlh"
Sep 18 19:17:13.946952 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:13.946900    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbhvv\" (UniqueName: \"kubernetes.io/projected/1d034d6d-be14-44c8-a72b-b7351a3f4a40-kube-api-access-wbhvv\") pod \"openshift-state-metrics-86c5bc79d6-q9q7d\" (UID: \"1d034d6d-be14-44c8-a72b-b7351a3f4a40\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-q9q7d"
Sep 18 19:17:13.947758 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:13.947735    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2fk8\" (UniqueName: \"kubernetes.io/projected/2317359e-59e6-4734-8ac4-f01b0bb27d57-kube-api-access-l2fk8\") pod \"kube-state-metrics-84c885658c-58tlh\" (UID: \"2317359e-59e6-4734-8ac4-f01b0bb27d57\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-58tlh"
Sep 18 19:17:14.027741 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.027654    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/896188ce-60fe-4f42-bced-50927342f530-node-exporter-wtmp\") pod \"node-exporter-cfpmj\" (UID: \"896188ce-60fe-4f42-bced-50927342f530\") " pod="openshift-monitoring/node-exporter-cfpmj"
Sep 18 19:17:14.027741 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.027733    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/896188ce-60fe-4f42-bced-50927342f530-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cfpmj\" (UID: \"896188ce-60fe-4f42-bced-50927342f530\") " pod="openshift-monitoring/node-exporter-cfpmj"
Sep 18 19:17:14.028011 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.027760    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/896188ce-60fe-4f42-bced-50927342f530-node-exporter-tls\") pod \"node-exporter-cfpmj\" (UID: \"896188ce-60fe-4f42-bced-50927342f530\") " pod="openshift-monitoring/node-exporter-cfpmj"
Sep 18 19:17:14.028011 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.027794    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/896188ce-60fe-4f42-bced-50927342f530-node-exporter-textfile\") pod \"node-exporter-cfpmj\" (UID: \"896188ce-60fe-4f42-bced-50927342f530\") " pod="openshift-monitoring/node-exporter-cfpmj"
Sep 18 19:17:14.028011 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.027810    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/896188ce-60fe-4f42-bced-50927342f530-node-exporter-wtmp\") pod \"node-exporter-cfpmj\" (UID: \"896188ce-60fe-4f42-bced-50927342f530\") " pod="openshift-monitoring/node-exporter-cfpmj"
Sep 18 19:17:14.028011 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.027895    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/896188ce-60fe-4f42-bced-50927342f530-sys\") pod \"node-exporter-cfpmj\" (UID: \"896188ce-60fe-4f42-bced-50927342f530\") " pod="openshift-monitoring/node-exporter-cfpmj"
Sep 18 19:17:14.028011 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.027925    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gxpl\" (UniqueName: \"kubernetes.io/projected/896188ce-60fe-4f42-bced-50927342f530-kube-api-access-2gxpl\") pod \"node-exporter-cfpmj\" (UID: \"896188ce-60fe-4f42-bced-50927342f530\") " pod="openshift-monitoring/node-exporter-cfpmj"
Sep 18 19:17:14.028011 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.027958    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/896188ce-60fe-4f42-bced-50927342f530-node-exporter-accelerators-collector-config\") pod \"node-exporter-cfpmj\" (UID: \"896188ce-60fe-4f42-bced-50927342f530\") " pod="openshift-monitoring/node-exporter-cfpmj"
Sep 18 19:17:14.028011 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.028003    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/896188ce-60fe-4f42-bced-50927342f530-root\") pod \"node-exporter-cfpmj\" (UID: \"896188ce-60fe-4f42-bced-50927342f530\") " pod="openshift-monitoring/node-exporter-cfpmj"
Sep 18 19:17:14.028359 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.028038    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/896188ce-60fe-4f42-bced-50927342f530-metrics-client-ca\") pod \"node-exporter-cfpmj\" (UID: \"896188ce-60fe-4f42-bced-50927342f530\") " pod="openshift-monitoring/node-exporter-cfpmj"
Sep 18 19:17:14.028359 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.028202    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/896188ce-60fe-4f42-bced-50927342f530-node-exporter-textfile\") pod \"node-exporter-cfpmj\" (UID: \"896188ce-60fe-4f42-bced-50927342f530\") " pod="openshift-monitoring/node-exporter-cfpmj"
Sep 18 19:17:14.028359 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.028214    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/896188ce-60fe-4f42-bced-50927342f530-sys\") pod \"node-exporter-cfpmj\" (UID: \"896188ce-60fe-4f42-bced-50927342f530\") " pod="openshift-monitoring/node-exporter-cfpmj"
Sep 18 19:17:14.028359 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.028267    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/896188ce-60fe-4f42-bced-50927342f530-root\") pod \"node-exporter-cfpmj\" (UID: \"896188ce-60fe-4f42-bced-50927342f530\") " pod="openshift-monitoring/node-exporter-cfpmj"
Sep 18 19:17:14.028683 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.028629    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/896188ce-60fe-4f42-bced-50927342f530-metrics-client-ca\") pod \"node-exporter-cfpmj\" (UID: \"896188ce-60fe-4f42-bced-50927342f530\") " pod="openshift-monitoring/node-exporter-cfpmj"
Sep 18 19:17:14.028980 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.028689    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/896188ce-60fe-4f42-bced-50927342f530-node-exporter-accelerators-collector-config\") pod \"node-exporter-cfpmj\" (UID: \"896188ce-60fe-4f42-bced-50927342f530\") " pod="openshift-monitoring/node-exporter-cfpmj"
Sep 18 19:17:14.030865 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.030641    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/896188ce-60fe-4f42-bced-50927342f530-node-exporter-tls\") pod \"node-exporter-cfpmj\" (UID: \"896188ce-60fe-4f42-bced-50927342f530\") " pod="openshift-monitoring/node-exporter-cfpmj"
Sep 18 19:17:14.030865 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.030745    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/896188ce-60fe-4f42-bced-50927342f530-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cfpmj\" (UID: \"896188ce-60fe-4f42-bced-50927342f530\") " pod="openshift-monitoring/node-exporter-cfpmj"
Sep 18 19:17:14.041794 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.041766    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gxpl\" (UniqueName: \"kubernetes.io/projected/896188ce-60fe-4f42-bced-50927342f530-kube-api-access-2gxpl\") pod \"node-exporter-cfpmj\" (UID: \"896188ce-60fe-4f42-bced-50927342f530\") " pod="openshift-monitoring/node-exporter-cfpmj"
Sep 18 19:17:14.103367 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.103336    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cfpmj"
Sep 18 19:17:14.116638 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:17:14.116603    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod896188ce_60fe_4f42_bced_50927342f530.slice/crio-4ff8dd861c541483abdb86ec6110b6c1e7d5cc83a8345c7cb266ee85003f3b2e WatchSource:0}: Error finding container 4ff8dd861c541483abdb86ec6110b6c1e7d5cc83a8345c7cb266ee85003f3b2e: Status 404 returned error can't find the container with id 4ff8dd861c541483abdb86ec6110b6c1e7d5cc83a8345c7cb266ee85003f3b2e
Sep 18 19:17:14.283451 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.283359    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cfpmj" event={"ID":"896188ce-60fe-4f42-bced-50927342f530","Type":"ContainerStarted","Data":"4ff8dd861c541483abdb86ec6110b6c1e7d5cc83a8345c7cb266ee85003f3b2e"}
Sep 18 19:17:14.284547 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.284516    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-f57kg" event={"ID":"25102f99-24b3-4c22-81ac-ee32e1fceabf","Type":"ContainerStarted","Data":"f3ba483b42d03d9969a575df8a9f64c2993541f281bf473160da10696491ae53"}
Sep 18 19:17:14.432375 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.432344    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2317359e-59e6-4734-8ac4-f01b0bb27d57-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-58tlh\" (UID: \"2317359e-59e6-4734-8ac4-f01b0bb27d57\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-58tlh"
Sep 18 19:17:14.432703 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.432417    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d034d6d-be14-44c8-a72b-b7351a3f4a40-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-q9q7d\" (UID: \"1d034d6d-be14-44c8-a72b-b7351a3f4a40\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-q9q7d"
Sep 18 19:17:14.435492 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.435464    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2317359e-59e6-4734-8ac4-f01b0bb27d57-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-58tlh\" (UID: \"2317359e-59e6-4734-8ac4-f01b0bb27d57\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-58tlh"
Sep 18 19:17:14.435630 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.435464    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d034d6d-be14-44c8-a72b-b7351a3f4a40-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-q9q7d\" (UID: \"1d034d6d-be14-44c8-a72b-b7351a3f4a40\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-q9q7d"
Sep 18 19:17:14.632977 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.632935    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 19:17:14.638844 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.638818    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:14.645583 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.645518    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 18 19:17:14.645719 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.645682    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 18 19:17:14.645787 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.645760    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 18 19:17:14.646176 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.646018    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 18 19:17:14.646176 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.646067    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 18 19:17:14.646351 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.646307    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 18 19:17:14.646680 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.646502    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 18 19:17:14.646680 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.646646    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 18 19:17:14.650119 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.650094    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-dfddl\""
Sep 18 19:17:14.654542 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.654342    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 18 19:17:14.664061 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.663460    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-q9q7d"
Sep 18 19:17:14.667449 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.667426    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-58tlh"
Sep 18 19:17:14.675996 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.675971    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 19:17:14.737335 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.736737    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hvxw\" (UniqueName: \"kubernetes.io/projected/357238c4-1440-486a-b32e-716b8f889807-kube-api-access-5hvxw\") pod \"alertmanager-main-0\" (UID: \"357238c4-1440-486a-b32e-716b8f889807\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:14.737335 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.736789    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/357238c4-1440-486a-b32e-716b8f889807-config-out\") pod \"alertmanager-main-0\" (UID: \"357238c4-1440-486a-b32e-716b8f889807\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:14.737335 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.736819    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/357238c4-1440-486a-b32e-716b8f889807-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"357238c4-1440-486a-b32e-716b8f889807\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:14.737335 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.736876    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/357238c4-1440-486a-b32e-716b8f889807-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"357238c4-1440-486a-b32e-716b8f889807\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:14.737335 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.736901    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/357238c4-1440-486a-b32e-716b8f889807-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"357238c4-1440-486a-b32e-716b8f889807\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:14.737335 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.736930    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/357238c4-1440-486a-b32e-716b8f889807-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"357238c4-1440-486a-b32e-716b8f889807\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:14.737335 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.736987    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/357238c4-1440-486a-b32e-716b8f889807-config-volume\") pod \"alertmanager-main-0\" (UID: \"357238c4-1440-486a-b32e-716b8f889807\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:14.737335 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.737014    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/357238c4-1440-486a-b32e-716b8f889807-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"357238c4-1440-486a-b32e-716b8f889807\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:14.737335 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.737043    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/357238c4-1440-486a-b32e-716b8f889807-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"357238c4-1440-486a-b32e-716b8f889807\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:14.737335 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.737071    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/357238c4-1440-486a-b32e-716b8f889807-web-config\") pod \"alertmanager-main-0\" (UID: \"357238c4-1440-486a-b32e-716b8f889807\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:14.737335 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.737100    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/357238c4-1440-486a-b32e-716b8f889807-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"357238c4-1440-486a-b32e-716b8f889807\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:14.737335 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.737134    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/357238c4-1440-486a-b32e-716b8f889807-tls-assets\") pod \"alertmanager-main-0\" (UID: \"357238c4-1440-486a-b32e-716b8f889807\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:14.737335 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.737166    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/357238c4-1440-486a-b32e-716b8f889807-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"357238c4-1440-486a-b32e-716b8f889807\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:14.839589 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.838231    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hvxw\" (UniqueName: \"kubernetes.io/projected/357238c4-1440-486a-b32e-716b8f889807-kube-api-access-5hvxw\") pod \"alertmanager-main-0\" (UID: \"357238c4-1440-486a-b32e-716b8f889807\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:14.839589 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.838287    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/357238c4-1440-486a-b32e-716b8f889807-config-out\") pod \"alertmanager-main-0\" (UID: \"357238c4-1440-486a-b32e-716b8f889807\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:14.839589 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.838317    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/357238c4-1440-486a-b32e-716b8f889807-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"357238c4-1440-486a-b32e-716b8f889807\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:14.839589 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.838371    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/357238c4-1440-486a-b32e-716b8f889807-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"357238c4-1440-486a-b32e-716b8f889807\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:14.839589 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.838398    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/357238c4-1440-486a-b32e-716b8f889807-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"357238c4-1440-486a-b32e-716b8f889807\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:14.839589 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.838425    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/357238c4-1440-486a-b32e-716b8f889807-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"357238c4-1440-486a-b32e-716b8f889807\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:14.839589 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.838489    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/357238c4-1440-486a-b32e-716b8f889807-config-volume\") pod \"alertmanager-main-0\" (UID: \"357238c4-1440-486a-b32e-716b8f889807\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:14.839589 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.838518    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/357238c4-1440-486a-b32e-716b8f889807-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"357238c4-1440-486a-b32e-716b8f889807\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:14.839589 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.838547    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/357238c4-1440-486a-b32e-716b8f889807-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"357238c4-1440-486a-b32e-716b8f889807\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:14.839589 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.838588    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/357238c4-1440-486a-b32e-716b8f889807-web-config\") pod \"alertmanager-main-0\" (UID: \"357238c4-1440-486a-b32e-716b8f889807\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:14.839589 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.838619    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/357238c4-1440-486a-b32e-716b8f889807-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"357238c4-1440-486a-b32e-716b8f889807\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:14.839589 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.838655    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/357238c4-1440-486a-b32e-716b8f889807-tls-assets\") pod \"alertmanager-main-0\" (UID: \"357238c4-1440-486a-b32e-716b8f889807\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:14.839589 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.838686    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/357238c4-1440-486a-b32e-716b8f889807-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"357238c4-1440-486a-b32e-716b8f889807\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:14.839589 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.839501    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/357238c4-1440-486a-b32e-716b8f889807-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"357238c4-1440-486a-b32e-716b8f889807\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:14.843531 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.841349    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/357238c4-1440-486a-b32e-716b8f889807-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"357238c4-1440-486a-b32e-716b8f889807\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:14.843531 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:17:14.841772    2522 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 18 19:17:14.843531 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:17:14.841850    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/357238c4-1440-486a-b32e-716b8f889807-secret-alertmanager-main-tls podName:357238c4-1440-486a-b32e-716b8f889807 nodeName:}" failed. No retries permitted until 2026-09-18 19:17:15.341829985 +0000 UTC m=+195.225356431 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/357238c4-1440-486a-b32e-716b8f889807-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "357238c4-1440-486a-b32e-716b8f889807") : secret "alertmanager-main-tls" not found
Sep 18 19:17:14.844047 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.844018    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/357238c4-1440-486a-b32e-716b8f889807-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"357238c4-1440-486a-b32e-716b8f889807\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:14.847846 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.847799    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/357238c4-1440-486a-b32e-716b8f889807-web-config\") pod \"alertmanager-main-0\" (UID: \"357238c4-1440-486a-b32e-716b8f889807\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:14.849074 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.848978    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/357238c4-1440-486a-b32e-716b8f889807-config-out\") pod \"alertmanager-main-0\" (UID: \"357238c4-1440-486a-b32e-716b8f889807\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:14.850617 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.849433    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/357238c4-1440-486a-b32e-716b8f889807-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"357238c4-1440-486a-b32e-716b8f889807\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:14.850617 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.850340    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/357238c4-1440-486a-b32e-716b8f889807-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"357238c4-1440-486a-b32e-716b8f889807\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:14.851628 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.850960    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/357238c4-1440-486a-b32e-716b8f889807-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"357238c4-1440-486a-b32e-716b8f889807\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:14.851628 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.851335    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/357238c4-1440-486a-b32e-716b8f889807-config-volume\") pod \"alertmanager-main-0\" (UID: \"357238c4-1440-486a-b32e-716b8f889807\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:14.851963 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.851917    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/357238c4-1440-486a-b32e-716b8f889807-tls-assets\") pod \"alertmanager-main-0\" (UID: \"357238c4-1440-486a-b32e-716b8f889807\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:14.852735 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.852694    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/357238c4-1440-486a-b32e-716b8f889807-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"357238c4-1440-486a-b32e-716b8f889807\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:14.858651 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.858629    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hvxw\" (UniqueName: \"kubernetes.io/projected/357238c4-1440-486a-b32e-716b8f889807-kube-api-access-5hvxw\") pod \"alertmanager-main-0\" (UID: \"357238c4-1440-486a-b32e-716b8f889807\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:14.984489 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.984445    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-q9q7d"]
Sep 18 19:17:14.987123 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:17:14.987094    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d034d6d_be14_44c8_a72b_b7351a3f4a40.slice/crio-b4c60bf590a5c66a28ac934ed52f8dc79b530b46fd9ee89be8f3b20757a8cc5e WatchSource:0}: Error finding container b4c60bf590a5c66a28ac934ed52f8dc79b530b46fd9ee89be8f3b20757a8cc5e: Status 404 returned error can't find the container with id b4c60bf590a5c66a28ac934ed52f8dc79b530b46fd9ee89be8f3b20757a8cc5e
Sep 18 19:17:14.997698 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:14.997589    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-58tlh"]
Sep 18 19:17:15.000979 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:17:15.000949    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2317359e_59e6_4734_8ac4_f01b0bb27d57.slice/crio-b54822ac2fb95fb7d1c14d7747def4dd27c09fa519a6ace516e9465d650d9162 WatchSource:0}: Error finding container b54822ac2fb95fb7d1c14d7747def4dd27c09fa519a6ace516e9465d650d9162: Status 404 returned error can't find the container with id b54822ac2fb95fb7d1c14d7747def4dd27c09fa519a6ace516e9465d650d9162
Sep 18 19:17:15.290808 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:15.290777    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-58tlh" event={"ID":"2317359e-59e6-4734-8ac4-f01b0bb27d57","Type":"ContainerStarted","Data":"b54822ac2fb95fb7d1c14d7747def4dd27c09fa519a6ace516e9465d650d9162"}
Sep 18 19:17:15.292540 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:15.292509    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-q9q7d" event={"ID":"1d034d6d-be14-44c8-a72b-b7351a3f4a40","Type":"ContainerStarted","Data":"5b3ec3fdd9f8b80d3cd22ecc2ed1ad224a528952b8602161ade6230120f8c164"}
Sep 18 19:17:15.292657 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:15.292576    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-q9q7d" event={"ID":"1d034d6d-be14-44c8-a72b-b7351a3f4a40","Type":"ContainerStarted","Data":"b4c60bf590a5c66a28ac934ed52f8dc79b530b46fd9ee89be8f3b20757a8cc5e"}
Sep 18 19:17:15.344531 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:15.344502    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/357238c4-1440-486a-b32e-716b8f889807-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"357238c4-1440-486a-b32e-716b8f889807\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:15.347247 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:15.347223    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/357238c4-1440-486a-b32e-716b8f889807-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"357238c4-1440-486a-b32e-716b8f889807\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:15.555063 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:15.554983    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:17:15.913263 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:15.913155    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 19:17:16.241611 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:17:16.241533    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod357238c4_1440_486a_b32e_716b8f889807.slice/crio-b2f275745acfb4c2c59a13f081238471e7ea11ba6cc452d9b83fa3f4ada8d035 WatchSource:0}: Error finding container b2f275745acfb4c2c59a13f081238471e7ea11ba6cc452d9b83fa3f4ada8d035: Status 404 returned error can't find the container with id b2f275745acfb4c2c59a13f081238471e7ea11ba6cc452d9b83fa3f4ada8d035
Sep 18 19:17:16.300074 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:16.300032    2522 generic.go:356] "Generic (PLEG): container finished" podID="896188ce-60fe-4f42-bced-50927342f530" containerID="849f193a346c1fbcf43de474456d0cb063bb74b384b4787028de59eb70a9dd5f" exitCode=0
Sep 18 19:17:16.300264 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:16.300091    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cfpmj" event={"ID":"896188ce-60fe-4f42-bced-50927342f530","Type":"ContainerDied","Data":"849f193a346c1fbcf43de474456d0cb063bb74b384b4787028de59eb70a9dd5f"}
Sep 18 19:17:16.302626 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:16.302592    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-q9q7d" event={"ID":"1d034d6d-be14-44c8-a72b-b7351a3f4a40","Type":"ContainerStarted","Data":"479b3a3eece59e48e62d63e86cc8cde268f1d3180c5d07b7b19aef91a83bf031"}
Sep 18 19:17:16.303966 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:16.303944    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"357238c4-1440-486a-b32e-716b8f889807","Type":"ContainerStarted","Data":"b2f275745acfb4c2c59a13f081238471e7ea11ba6cc452d9b83fa3f4ada8d035"}
Sep 18 19:17:17.312896 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:17.312846    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cfpmj" event={"ID":"896188ce-60fe-4f42-bced-50927342f530","Type":"ContainerStarted","Data":"106beeabc510a17f184de5d62d7ca53c5b2e1b23211c7fe7cb2ffb4514696c1c"}
Sep 18 19:17:17.315676 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:17.314687    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cfpmj" event={"ID":"896188ce-60fe-4f42-bced-50927342f530","Type":"ContainerStarted","Data":"368315cc41c380f0e21d94941c756c63f0d887fefd7de7768469342022184a56"}
Sep 18 19:17:17.323174 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:17.323142    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-58tlh" event={"ID":"2317359e-59e6-4734-8ac4-f01b0bb27d57","Type":"ContainerStarted","Data":"a4172bc8cab3586b7972d3457f1787a21f2a0a11843372549343d9847613e6e7"}
Sep 18 19:17:17.323342 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:17.323213    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-58tlh" event={"ID":"2317359e-59e6-4734-8ac4-f01b0bb27d57","Type":"ContainerStarted","Data":"79151b42f85929c2c92ff0c23f3680e69395b49b3a058703fec16cc203970294"}
Sep 18 19:17:17.323342 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:17.323275    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-58tlh" event={"ID":"2317359e-59e6-4734-8ac4-f01b0bb27d57","Type":"ContainerStarted","Data":"5e670a759d0b4eef1f8871bb9603b3c25c052c12d7537984b075bd9da667ae29"}
Sep 18 19:17:17.326714 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:17.326688    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-q9q7d" event={"ID":"1d034d6d-be14-44c8-a72b-b7351a3f4a40","Type":"ContainerStarted","Data":"21fd8becb0257064bdb0c48978c5fab2c3fbf320df8dc92574b1804994a74dc7"}
Sep 18 19:17:17.393496 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:17.393431    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-cfpmj" podStartSLOduration=3.099449098 podStartE2EDuration="4.393408438s" totalImagesPullingTime="1.216899395s" totalInitContainerRuntime="77.059945ms" isStatefulPod=true podCreationTimestamp="2026-09-18 19:17:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:17:17.355395704 +0000 UTC m=+197.238922179" watchObservedRunningTime="2026-09-18 19:17:17.393408438 +0000 UTC m=+197.276934903"
Sep 18 19:17:17.427166 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:17.426057    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-84c885658c-58tlh" podStartSLOduration=2.88196883 podStartE2EDuration="4.426035537s" totalImagesPullingTime="1.544066707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:17:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:17:17.425015967 +0000 UTC m=+197.308542429" watchObservedRunningTime="2026-09-18 19:17:17.426035537 +0000 UTC m=+197.309562000"
Sep 18 19:17:17.427166 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:17.426717    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-q9q7d" podStartSLOduration=3.219993022 podStartE2EDuration="4.426710059s" totalImagesPullingTime="1.206717037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:17:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:17:17.392184828 +0000 UTC m=+197.275711300" watchObservedRunningTime="2026-09-18 19:17:17.426710059 +0000 UTC m=+197.310236516"
Sep 18 19:17:18.292191 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:18.292149    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-fdd59846c-7qgqt"]
Sep 18 19:17:18.295587 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:18.295551    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-fdd59846c-7qgqt"
Sep 18 19:17:18.299648 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:18.299585    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-fw7fd\""
Sep 18 19:17:18.299648 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:18.299618    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 18 19:17:18.299836 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:18.299695    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-c73dl6mp1t7lj\""
Sep 18 19:17:18.299913 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:18.299898    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 18 19:17:18.299995 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:18.299981    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 18 19:17:18.300083 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:18.300060    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 18 19:17:18.332662 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:18.332625    2522 generic.go:356] "Generic (PLEG): container finished" podID="357238c4-1440-486a-b32e-716b8f889807" containerID="ea754f8ffe36ff5f22037c673a37a712ec75b357c9fe4e0116a132d7534b5c4e" exitCode=0
Sep 18 19:17:18.333185 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:18.333051    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"357238c4-1440-486a-b32e-716b8f889807","Type":"ContainerDied","Data":"ea754f8ffe36ff5f22037c673a37a712ec75b357c9fe4e0116a132d7534b5c4e"}
Sep 18 19:17:18.336088 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:18.336065    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-fdd59846c-7qgqt"]
Sep 18 19:17:18.474764 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:18.474723    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ae2fbef4-e449-47e1-9017-199dec3639b2-metrics-server-audit-profiles\") pod \"metrics-server-fdd59846c-7qgqt\" (UID: \"ae2fbef4-e449-47e1-9017-199dec3639b2\") " pod="openshift-monitoring/metrics-server-fdd59846c-7qgqt"
Sep 18 19:17:18.474962 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:18.474823    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae2fbef4-e449-47e1-9017-199dec3639b2-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-fdd59846c-7qgqt\" (UID: \"ae2fbef4-e449-47e1-9017-199dec3639b2\") " pod="openshift-monitoring/metrics-server-fdd59846c-7qgqt"
Sep 18 19:17:18.474962 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:18.474945    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ae2fbef4-e449-47e1-9017-199dec3639b2-client-ca-bundle\") pod \"metrics-server-fdd59846c-7qgqt\" (UID: \"ae2fbef4-e449-47e1-9017-199dec3639b2\") " pod="openshift-monitoring/metrics-server-fdd59846c-7qgqt"
Sep 18 19:17:18.475088 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:18.475072    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ae2fbef4-e449-47e1-9017-199dec3639b2-audit-log\") pod \"metrics-server-fdd59846c-7qgqt\" (UID: \"ae2fbef4-e449-47e1-9017-199dec3639b2\") " pod="openshift-monitoring/metrics-server-fdd59846c-7qgqt"
Sep 18 19:17:18.475761 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:18.475496    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ae2fbef4-e449-47e1-9017-199dec3639b2-secret-metrics-server-client-certs\") pod \"metrics-server-fdd59846c-7qgqt\" (UID: \"ae2fbef4-e449-47e1-9017-199dec3639b2\") " pod="openshift-monitoring/metrics-server-fdd59846c-7qgqt"
Sep 18 19:17:18.475761 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:18.475600    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvkph\" (UniqueName: \"kubernetes.io/projected/ae2fbef4-e449-47e1-9017-199dec3639b2-kube-api-access-kvkph\") pod \"metrics-server-fdd59846c-7qgqt\" (UID: \"ae2fbef4-e449-47e1-9017-199dec3639b2\") " pod="openshift-monitoring/metrics-server-fdd59846c-7qgqt"
Sep 18 19:17:18.475761 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:18.475683    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ae2fbef4-e449-47e1-9017-199dec3639b2-secret-metrics-server-tls\") pod \"metrics-server-fdd59846c-7qgqt\" (UID: \"ae2fbef4-e449-47e1-9017-199dec3639b2\") " pod="openshift-monitoring/metrics-server-fdd59846c-7qgqt"
Sep 18 19:17:18.576619 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:18.576583    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ae2fbef4-e449-47e1-9017-199dec3639b2-secret-metrics-server-client-certs\") pod \"metrics-server-fdd59846c-7qgqt\" (UID: \"ae2fbef4-e449-47e1-9017-199dec3639b2\") " pod="openshift-monitoring/metrics-server-fdd59846c-7qgqt"
Sep 18 19:17:18.576814 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:18.576630    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvkph\" (UniqueName: \"kubernetes.io/projected/ae2fbef4-e449-47e1-9017-199dec3639b2-kube-api-access-kvkph\") pod \"metrics-server-fdd59846c-7qgqt\" (UID: \"ae2fbef4-e449-47e1-9017-199dec3639b2\") " pod="openshift-monitoring/metrics-server-fdd59846c-7qgqt"
Sep 18 19:17:18.576814 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:18.576649    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ae2fbef4-e449-47e1-9017-199dec3639b2-secret-metrics-server-tls\") pod \"metrics-server-fdd59846c-7qgqt\" (UID: \"ae2fbef4-e449-47e1-9017-199dec3639b2\") " pod="openshift-monitoring/metrics-server-fdd59846c-7qgqt"
Sep 18 19:17:18.576814 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:18.576683    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ae2fbef4-e449-47e1-9017-199dec3639b2-metrics-server-audit-profiles\") pod \"metrics-server-fdd59846c-7qgqt\" (UID: \"ae2fbef4-e449-47e1-9017-199dec3639b2\") " pod="openshift-monitoring/metrics-server-fdd59846c-7qgqt"
Sep 18 19:17:18.576814 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:18.576727    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae2fbef4-e449-47e1-9017-199dec3639b2-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-fdd59846c-7qgqt\" (UID: \"ae2fbef4-e449-47e1-9017-199dec3639b2\") " pod="openshift-monitoring/metrics-server-fdd59846c-7qgqt"
Sep 18 19:17:18.576814 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:18.576759    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ae2fbef4-e449-47e1-9017-199dec3639b2-client-ca-bundle\") pod \"metrics-server-fdd59846c-7qgqt\" (UID: \"ae2fbef4-e449-47e1-9017-199dec3639b2\") " pod="openshift-monitoring/metrics-server-fdd59846c-7qgqt"
Sep 18 19:17:18.576814 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:18.576792    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ae2fbef4-e449-47e1-9017-199dec3639b2-audit-log\") pod \"metrics-server-fdd59846c-7qgqt\" (UID: \"ae2fbef4-e449-47e1-9017-199dec3639b2\") " pod="openshift-monitoring/metrics-server-fdd59846c-7qgqt"
Sep 18 19:17:18.577280 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:18.577212    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ae2fbef4-e449-47e1-9017-199dec3639b2-audit-log\") pod \"metrics-server-fdd59846c-7qgqt\" (UID: \"ae2fbef4-e449-47e1-9017-199dec3639b2\") " pod="openshift-monitoring/metrics-server-fdd59846c-7qgqt"
Sep 18 19:17:18.577670 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:18.577630    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae2fbef4-e449-47e1-9017-199dec3639b2-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-fdd59846c-7qgqt\" (UID: \"ae2fbef4-e449-47e1-9017-199dec3639b2\") " pod="openshift-monitoring/metrics-server-fdd59846c-7qgqt"
Sep 18 19:17:18.577906 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:18.577886    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ae2fbef4-e449-47e1-9017-199dec3639b2-metrics-server-audit-profiles\") pod \"metrics-server-fdd59846c-7qgqt\" (UID: \"ae2fbef4-e449-47e1-9017-199dec3639b2\") " pod="openshift-monitoring/metrics-server-fdd59846c-7qgqt"
Sep 18 19:17:18.579405 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:18.579363    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ae2fbef4-e449-47e1-9017-199dec3639b2-secret-metrics-server-tls\") pod \"metrics-server-fdd59846c-7qgqt\" (UID: \"ae2fbef4-e449-47e1-9017-199dec3639b2\") " pod="openshift-monitoring/metrics-server-fdd59846c-7qgqt"
Sep 18 19:17:18.579553 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:18.579531    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ae2fbef4-e449-47e1-9017-199dec3639b2-secret-metrics-server-client-certs\") pod \"metrics-server-fdd59846c-7qgqt\" (UID: \"ae2fbef4-e449-47e1-9017-199dec3639b2\") " pod="openshift-monitoring/metrics-server-fdd59846c-7qgqt"
Sep 18 19:17:18.579627 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:18.579574    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ae2fbef4-e449-47e1-9017-199dec3639b2-client-ca-bundle\") pod \"metrics-server-fdd59846c-7qgqt\" (UID: \"ae2fbef4-e449-47e1-9017-199dec3639b2\") " pod="openshift-monitoring/metrics-server-fdd59846c-7qgqt"
Sep 18 19:17:18.586355 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:18.586333    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvkph\" (UniqueName: \"kubernetes.io/projected/ae2fbef4-e449-47e1-9017-199dec3639b2-kube-api-access-kvkph\") pod \"metrics-server-fdd59846c-7qgqt\" (UID: \"ae2fbef4-e449-47e1-9017-199dec3639b2\") " pod="openshift-monitoring/metrics-server-fdd59846c-7qgqt"
Sep 18 19:17:18.607263 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:18.607233    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-fdd59846c-7qgqt"
Sep 18 19:17:18.927978 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:18.927947    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-fdd59846c-7qgqt"]
Sep 18 19:17:18.929619 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:17:18.929590    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae2fbef4_e449_47e1_9017_199dec3639b2.slice/crio-c682c719265e47502cf044fd0e0f9a1dc6b9e1bb534b31de99fbad708ce910ed WatchSource:0}: Error finding container c682c719265e47502cf044fd0e0f9a1dc6b9e1bb534b31de99fbad708ce910ed: Status 404 returned error can't find the container with id c682c719265e47502cf044fd0e0f9a1dc6b9e1bb534b31de99fbad708ce910ed
Sep 18 19:17:19.338644 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:19.338611    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-fdd59846c-7qgqt" event={"ID":"ae2fbef4-e449-47e1-9017-199dec3639b2","Type":"ContainerStarted","Data":"c682c719265e47502cf044fd0e0f9a1dc6b9e1bb534b31de99fbad708ce910ed"}
Sep 18 19:17:21.352270 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:21.352232    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"357238c4-1440-486a-b32e-716b8f889807","Type":"ContainerStarted","Data":"264b193e3ed0eb780f6b0b84a39530f36dbef838d54ffe6a6089b0c1bfd9bf98"}
Sep 18 19:17:21.352735 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:21.352313    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"357238c4-1440-486a-b32e-716b8f889807","Type":"ContainerStarted","Data":"300f03e2b19351c8234b01aa7b6ae709b7f77caafb5b663bcff67482a1e4db21"}
Sep 18 19:17:21.352735 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:21.352352    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"357238c4-1440-486a-b32e-716b8f889807","Type":"ContainerStarted","Data":"4aba7dfdbd4be2e427f8ccd520a39954d987462471299e2616948f125ee716a1"}
Sep 18 19:17:21.352735 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:21.352392    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"357238c4-1440-486a-b32e-716b8f889807","Type":"ContainerStarted","Data":"c81564c8e60403017c272ccec2afd5ff55b85521f6e58749b2449e176f2d44ae"}
Sep 18 19:17:21.352735 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:21.352435    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"357238c4-1440-486a-b32e-716b8f889807","Type":"ContainerStarted","Data":"d20643fcbf1c9b749355e6bc988a7e2adf3e89adc115618c3b701bbd0b6fb71a"}
Sep 18 19:17:21.353993 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:21.353965    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-fdd59846c-7qgqt" event={"ID":"ae2fbef4-e449-47e1-9017-199dec3639b2","Type":"ContainerStarted","Data":"5c39dc67ba3fa544b045b4a2f4d32d6001d77a756ec9327676e8a1225efc5fc9"}
Sep 18 19:17:21.380475 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:21.380368    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-fdd59846c-7qgqt" podStartSLOduration=1.737349861 podStartE2EDuration="3.380347934s" totalImagesPullingTime="1.642998073s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:17:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:17:21.379611547 +0000 UTC m=+201.263138012" watchObservedRunningTime="2026-09-18 19:17:21.380347934 +0000 UTC m=+201.263874400"
Sep 18 19:17:22.362937 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:22.362895    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"357238c4-1440-486a-b32e-716b8f889807","Type":"ContainerStarted","Data":"313bf6de775f0b0de4a200149b214f9414ae5cc42b32b2809df9dca3a6fc1cfb"}
Sep 18 19:17:24.305054 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:24.304994    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=5.977710268 podStartE2EDuration="10.304970294s" totalImagesPullingTime="4.305122082s" totalInitContainerRuntime="22.137944ms" isStatefulPod=false podCreationTimestamp="2026-09-18 19:17:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:17:22.416865954 +0000 UTC m=+202.300392417" watchObservedRunningTime="2026-09-18 19:17:24.304970294 +0000 UTC m=+204.188496773"
Sep 18 19:17:24.305801 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:24.305781    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7b77b6b64d-dgr86"]
Sep 18 19:17:24.306071 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:17:24.306051    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-7b77b6b64d-dgr86" podUID="63dfbfc5-e1e8-4e79-8df9-d47e16ba5289"
Sep 18 19:17:24.372409 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:24.372377    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b77b6b64d-dgr86"
Sep 18 19:17:24.380214 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:24.380190    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b77b6b64d-dgr86"
Sep 18 19:17:24.438273 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:24.438236    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-ca-trust-extracted\") pod \"63dfbfc5-e1e8-4e79-8df9-d47e16ba5289\" (UID: \"63dfbfc5-e1e8-4e79-8df9-d47e16ba5289\") "
Sep 18 19:17:24.438467 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:24.438296    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-image-registry-private-configuration\") pod \"63dfbfc5-e1e8-4e79-8df9-d47e16ba5289\" (UID: \"63dfbfc5-e1e8-4e79-8df9-d47e16ba5289\") "
Sep 18 19:17:24.438467 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:24.438349    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-installation-pull-secrets\") pod \"63dfbfc5-e1e8-4e79-8df9-d47e16ba5289\" (UID: \"63dfbfc5-e1e8-4e79-8df9-d47e16ba5289\") "
Sep 18 19:17:24.438467 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:24.438386    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rxcmr\" (UniqueName: \"kubernetes.io/projected/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-kube-api-access-rxcmr\") pod \"63dfbfc5-e1e8-4e79-8df9-d47e16ba5289\" (UID: \"63dfbfc5-e1e8-4e79-8df9-d47e16ba5289\") "
Sep 18 19:17:24.438467 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:24.438411    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-registry-certificates\") pod \"63dfbfc5-e1e8-4e79-8df9-d47e16ba5289\" (UID: \"63dfbfc5-e1e8-4e79-8df9-d47e16ba5289\") "
Sep 18 19:17:24.438467 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:24.438438    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-bound-sa-token\") pod \"63dfbfc5-e1e8-4e79-8df9-d47e16ba5289\" (UID: \"63dfbfc5-e1e8-4e79-8df9-d47e16ba5289\") "
Sep 18 19:17:24.438467 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:24.438467    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-trusted-ca\") pod \"63dfbfc5-e1e8-4e79-8df9-d47e16ba5289\" (UID: \"63dfbfc5-e1e8-4e79-8df9-d47e16ba5289\") "
Sep 18 19:17:24.438839 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:24.438537    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "63dfbfc5-e1e8-4e79-8df9-d47e16ba5289" (UID: "63dfbfc5-e1e8-4e79-8df9-d47e16ba5289"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:17:24.438839 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:24.438716    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "63dfbfc5-e1e8-4e79-8df9-d47e16ba5289" (UID: "63dfbfc5-e1e8-4e79-8df9-d47e16ba5289"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:17:24.438943 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:24.438878    2522 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-ca-trust-extracted\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:17:24.438943 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:24.438897    2522 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-registry-certificates\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:17:24.439023 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:24.438952    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "63dfbfc5-e1e8-4e79-8df9-d47e16ba5289" (UID: "63dfbfc5-e1e8-4e79-8df9-d47e16ba5289"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:17:24.441481 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:24.441436    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "63dfbfc5-e1e8-4e79-8df9-d47e16ba5289" (UID: "63dfbfc5-e1e8-4e79-8df9-d47e16ba5289"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:17:24.441609 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:24.441493    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-kube-api-access-rxcmr" (OuterVolumeSpecName: "kube-api-access-rxcmr") pod "63dfbfc5-e1e8-4e79-8df9-d47e16ba5289" (UID: "63dfbfc5-e1e8-4e79-8df9-d47e16ba5289"). InnerVolumeSpecName "kube-api-access-rxcmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:17:24.441609 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:24.441577    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "63dfbfc5-e1e8-4e79-8df9-d47e16ba5289" (UID: "63dfbfc5-e1e8-4e79-8df9-d47e16ba5289"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:17:24.441722 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:24.441631    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "63dfbfc5-e1e8-4e79-8df9-d47e16ba5289" (UID: "63dfbfc5-e1e8-4e79-8df9-d47e16ba5289"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:17:24.539984 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:24.539939    2522 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-image-registry-private-configuration\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:17:24.539984 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:24.539979    2522 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-installation-pull-secrets\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:17:24.539984 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:24.539990    2522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxcmr\" (UniqueName: \"kubernetes.io/projected/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-kube-api-access-rxcmr\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:17:24.540225 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:24.539999    2522 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-bound-sa-token\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:17:24.540225 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:24.540012    2522 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-trusted-ca\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:17:25.377063 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:25.377029    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b77b6b64d-dgr86"
Sep 18 19:17:25.414430 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:25.414387    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7b77b6b64d-dgr86"]
Sep 18 19:17:25.423079 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:25.423047    2522 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7b77b6b64d-dgr86"]
Sep 18 19:17:25.448386 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:25.448350    2522 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289-registry-tls\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:17:26.721110 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:26.721073    2522 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63dfbfc5-e1e8-4e79-8df9-d47e16ba5289" path="/var/lib/kubelet/pods/63dfbfc5-e1e8-4e79-8df9-d47e16ba5289/volumes"
Sep 18 19:17:31.404575 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:31.404513    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-f57kg" event={"ID":"25102f99-24b3-4c22-81ac-ee32e1fceabf","Type":"ContainerStarted","Data":"becdfbbd9c71c8ce7dff23a7193104f13bc3fa7af6bd8eb15060b08273d5b6f1"}
Sep 18 19:17:31.405029 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:31.404753    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6594f4c75-f57kg"
Sep 18 19:17:31.427270 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:31.427238    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6594f4c75-f57kg"
Sep 18 19:17:31.467438 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:31.467375    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-6594f4c75-f57kg" podStartSLOduration=2.454950644 podStartE2EDuration="19.467358826s" totalImagesPullingTime="17.012408182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:17:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:17:31.43673137 +0000 UTC m=+211.320257833" watchObservedRunningTime="2026-09-18 19:17:31.467358826 +0000 UTC m=+211.350885282"
Sep 18 19:17:34.187388 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:34.187352    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-d67bc5c57-fgwxg"]
Sep 18 19:17:34.217534 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:34.217448    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d67bc5c57-fgwxg"]
Sep 18 19:17:34.217703 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:34.217580    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d67bc5c57-fgwxg"
Sep 18 19:17:34.222715 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:34.222690    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 18 19:17:34.222948 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:34.222930    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 18 19:17:34.224720 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:34.224696    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-nvq4k\""
Sep 18 19:17:34.225088 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:34.224932    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 18 19:17:34.225088 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:34.224959    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 18 19:17:34.225088 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:34.224961    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 18 19:17:34.235072 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:34.235051    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 18 19:17:34.332491 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:34.332454    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cb74c5b7-6640-4747-92f0-03bdde39c5a9-oauth-serving-cert\") pod \"console-d67bc5c57-fgwxg\" (UID: \"cb74c5b7-6640-4747-92f0-03bdde39c5a9\") " pod="openshift-console/console-d67bc5c57-fgwxg"
Sep 18 19:17:34.332685 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:34.332550    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cb74c5b7-6640-4747-92f0-03bdde39c5a9-service-ca\") pod \"console-d67bc5c57-fgwxg\" (UID: \"cb74c5b7-6640-4747-92f0-03bdde39c5a9\") " pod="openshift-console/console-d67bc5c57-fgwxg"
Sep 18 19:17:34.332685 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:34.332622    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb74c5b7-6640-4747-92f0-03bdde39c5a9-trusted-ca-bundle\") pod \"console-d67bc5c57-fgwxg\" (UID: \"cb74c5b7-6640-4747-92f0-03bdde39c5a9\") " pod="openshift-console/console-d67bc5c57-fgwxg"
Sep 18 19:17:34.332802 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:34.332699    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cb74c5b7-6640-4747-92f0-03bdde39c5a9-console-oauth-config\") pod \"console-d67bc5c57-fgwxg\" (UID: \"cb74c5b7-6640-4747-92f0-03bdde39c5a9\") " pod="openshift-console/console-d67bc5c57-fgwxg"
Sep 18 19:17:34.332802 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:34.332765    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmsv9\" (UniqueName: \"kubernetes.io/projected/cb74c5b7-6640-4747-92f0-03bdde39c5a9-kube-api-access-jmsv9\") pod \"console-d67bc5c57-fgwxg\" (UID: \"cb74c5b7-6640-4747-92f0-03bdde39c5a9\") " pod="openshift-console/console-d67bc5c57-fgwxg"
Sep 18 19:17:34.332802 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:34.332796    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cb74c5b7-6640-4747-92f0-03bdde39c5a9-console-serving-cert\") pod \"console-d67bc5c57-fgwxg\" (UID: \"cb74c5b7-6640-4747-92f0-03bdde39c5a9\") " pod="openshift-console/console-d67bc5c57-fgwxg"
Sep 18 19:17:34.332922 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:34.332813    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cb74c5b7-6640-4747-92f0-03bdde39c5a9-console-config\") pod \"console-d67bc5c57-fgwxg\" (UID: \"cb74c5b7-6640-4747-92f0-03bdde39c5a9\") " pod="openshift-console/console-d67bc5c57-fgwxg"
Sep 18 19:17:34.433426 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:34.433395    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmsv9\" (UniqueName: \"kubernetes.io/projected/cb74c5b7-6640-4747-92f0-03bdde39c5a9-kube-api-access-jmsv9\") pod \"console-d67bc5c57-fgwxg\" (UID: \"cb74c5b7-6640-4747-92f0-03bdde39c5a9\") " pod="openshift-console/console-d67bc5c57-fgwxg"
Sep 18 19:17:34.433619 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:34.433434    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cb74c5b7-6640-4747-92f0-03bdde39c5a9-console-serving-cert\") pod \"console-d67bc5c57-fgwxg\" (UID: \"cb74c5b7-6640-4747-92f0-03bdde39c5a9\") " pod="openshift-console/console-d67bc5c57-fgwxg"
Sep 18 19:17:34.433619 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:34.433458    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cb74c5b7-6640-4747-92f0-03bdde39c5a9-console-config\") pod \"console-d67bc5c57-fgwxg\" (UID: \"cb74c5b7-6640-4747-92f0-03bdde39c5a9\") " pod="openshift-console/console-d67bc5c57-fgwxg"
Sep 18 19:17:34.433726 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:34.433618    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cb74c5b7-6640-4747-92f0-03bdde39c5a9-oauth-serving-cert\") pod \"console-d67bc5c57-fgwxg\" (UID: \"cb74c5b7-6640-4747-92f0-03bdde39c5a9\") " pod="openshift-console/console-d67bc5c57-fgwxg"
Sep 18 19:17:34.433726 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:34.433697    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cb74c5b7-6640-4747-92f0-03bdde39c5a9-service-ca\") pod \"console-d67bc5c57-fgwxg\" (UID: \"cb74c5b7-6640-4747-92f0-03bdde39c5a9\") " pod="openshift-console/console-d67bc5c57-fgwxg"
Sep 18 19:17:34.433835 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:34.433738    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb74c5b7-6640-4747-92f0-03bdde39c5a9-trusted-ca-bundle\") pod \"console-d67bc5c57-fgwxg\" (UID: \"cb74c5b7-6640-4747-92f0-03bdde39c5a9\") " pod="openshift-console/console-d67bc5c57-fgwxg"
Sep 18 19:17:34.433835 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:34.433800    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cb74c5b7-6640-4747-92f0-03bdde39c5a9-console-oauth-config\") pod \"console-d67bc5c57-fgwxg\" (UID: \"cb74c5b7-6640-4747-92f0-03bdde39c5a9\") " pod="openshift-console/console-d67bc5c57-fgwxg"
Sep 18 19:17:34.434459 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:34.434323    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cb74c5b7-6640-4747-92f0-03bdde39c5a9-console-config\") pod \"console-d67bc5c57-fgwxg\" (UID: \"cb74c5b7-6640-4747-92f0-03bdde39c5a9\") " pod="openshift-console/console-d67bc5c57-fgwxg"
Sep 18 19:17:34.434459 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:34.434346    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cb74c5b7-6640-4747-92f0-03bdde39c5a9-oauth-serving-cert\") pod \"console-d67bc5c57-fgwxg\" (UID: \"cb74c5b7-6640-4747-92f0-03bdde39c5a9\") " pod="openshift-console/console-d67bc5c57-fgwxg"
Sep 18 19:17:34.434823 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:34.434779    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb74c5b7-6640-4747-92f0-03bdde39c5a9-trusted-ca-bundle\") pod \"console-d67bc5c57-fgwxg\" (UID: \"cb74c5b7-6640-4747-92f0-03bdde39c5a9\") " pod="openshift-console/console-d67bc5c57-fgwxg"
Sep 18 19:17:34.434823 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:34.434785    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cb74c5b7-6640-4747-92f0-03bdde39c5a9-service-ca\") pod \"console-d67bc5c57-fgwxg\" (UID: \"cb74c5b7-6640-4747-92f0-03bdde39c5a9\") " pod="openshift-console/console-d67bc5c57-fgwxg"
Sep 18 19:17:34.437131 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:34.437104    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cb74c5b7-6640-4747-92f0-03bdde39c5a9-console-serving-cert\") pod \"console-d67bc5c57-fgwxg\" (UID: \"cb74c5b7-6640-4747-92f0-03bdde39c5a9\") " pod="openshift-console/console-d67bc5c57-fgwxg"
Sep 18 19:17:34.437227 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:34.437179    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cb74c5b7-6640-4747-92f0-03bdde39c5a9-console-oauth-config\") pod \"console-d67bc5c57-fgwxg\" (UID: \"cb74c5b7-6640-4747-92f0-03bdde39c5a9\") " pod="openshift-console/console-d67bc5c57-fgwxg"
Sep 18 19:17:34.447133 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:34.447077    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmsv9\" (UniqueName: \"kubernetes.io/projected/cb74c5b7-6640-4747-92f0-03bdde39c5a9-kube-api-access-jmsv9\") pod \"console-d67bc5c57-fgwxg\" (UID: \"cb74c5b7-6640-4747-92f0-03bdde39c5a9\") " pod="openshift-console/console-d67bc5c57-fgwxg"
Sep 18 19:17:34.529344 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:34.529310    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d67bc5c57-fgwxg"
Sep 18 19:17:34.839756 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:34.839731    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d67bc5c57-fgwxg"]
Sep 18 19:17:34.841793 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:17:34.841766    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb74c5b7_6640_4747_92f0_03bdde39c5a9.slice/crio-5a6ea2c1bad2afbe8e93fd5159264c7c9c6524fd72114f8209921ddbf728cb29 WatchSource:0}: Error finding container 5a6ea2c1bad2afbe8e93fd5159264c7c9c6524fd72114f8209921ddbf728cb29: Status 404 returned error can't find the container with id 5a6ea2c1bad2afbe8e93fd5159264c7c9c6524fd72114f8209921ddbf728cb29
Sep 18 19:17:35.429647 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:35.429481    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d67bc5c57-fgwxg" event={"ID":"cb74c5b7-6640-4747-92f0-03bdde39c5a9","Type":"ContainerStarted","Data":"5a6ea2c1bad2afbe8e93fd5159264c7c9c6524fd72114f8209921ddbf728cb29"}
Sep 18 19:17:38.607686 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:38.607645    2522 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-fdd59846c-7qgqt"
Sep 18 19:17:38.608136 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:38.607721    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-fdd59846c-7qgqt"
Sep 18 19:17:40.457113 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:40.457078    2522 generic.go:356] "Generic (PLEG): container finished" podID="09a74ca7-303a-49c9-9967-98074bbfbb17" containerID="331a67b5626d3d6aac2a2276d9ecbdd480078cb7a43906cef7b3dadf8cb2b9b9" exitCode=0
Sep 18 19:17:40.457618 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:40.457155    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-qkrw2" event={"ID":"09a74ca7-303a-49c9-9967-98074bbfbb17","Type":"ContainerDied","Data":"331a67b5626d3d6aac2a2276d9ecbdd480078cb7a43906cef7b3dadf8cb2b9b9"}
Sep 18 19:17:40.457618 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:40.457503    2522 scope.go:117] "RemoveContainer" containerID="331a67b5626d3d6aac2a2276d9ecbdd480078cb7a43906cef7b3dadf8cb2b9b9"
Sep 18 19:17:40.459186 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:40.459163    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d67bc5c57-fgwxg" event={"ID":"cb74c5b7-6640-4747-92f0-03bdde39c5a9","Type":"ContainerStarted","Data":"24d6220e1ea8e7487df5df2145d2370d59aa6e25d96f9af49d031eda2336ff28"}
Sep 18 19:17:40.516476 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:40.516425    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-d67bc5c57-fgwxg" podStartSLOduration=1.041452439 podStartE2EDuration="6.516407393s" totalImagesPullingTime="5.474954954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:17:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:17:40.516159518 +0000 UTC m=+220.399685992" watchObservedRunningTime="2026-09-18 19:17:40.516407393 +0000 UTC m=+220.399933855"
Sep 18 19:17:41.464376 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:41.464337    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-qkrw2" event={"ID":"09a74ca7-303a-49c9-9967-98074bbfbb17","Type":"ContainerStarted","Data":"20b5b96e775557ddc0e6241e12b0840535051a97ffcfee0b4f4f6c777aa829be"}
Sep 18 19:17:44.530286 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:44.530255    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-d67bc5c57-fgwxg"
Sep 18 19:17:44.530784 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:44.530312    2522 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-d67bc5c57-fgwxg"
Sep 18 19:17:44.534989 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:44.534960    2522 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-d67bc5c57-fgwxg"
Sep 18 19:17:45.482115 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:45.482089    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-d67bc5c57-fgwxg"
Sep 18 19:17:58.613072 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:58.613043    2522 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-fdd59846c-7qgqt"
Sep 18 19:17:58.616829 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:17:58.616808    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-fdd59846c-7qgqt"
Sep 18 19:18:12.373506 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:12.373466    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f516e39e-b125-4617-94ce-7f0dccfae7bb-metrics-certs\") pod \"network-metrics-daemon-5j97v\" (UID: \"f516e39e-b125-4617-94ce-7f0dccfae7bb\") " pod="openshift-multus/network-metrics-daemon-5j97v"
Sep 18 19:18:12.375732 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:12.375713    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f516e39e-b125-4617-94ce-7f0dccfae7bb-metrics-certs\") pod \"network-metrics-daemon-5j97v\" (UID: \"f516e39e-b125-4617-94ce-7f0dccfae7bb\") " pod="openshift-multus/network-metrics-daemon-5j97v"
Sep 18 19:18:12.628191 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:12.628114    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xjcwp\""
Sep 18 19:18:12.637749 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:12.637728    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5j97v"
Sep 18 19:18:13.090285 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:13.090259    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5j97v"]
Sep 18 19:18:13.090395 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:18:13.090337    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf516e39e_b125_4617_94ce_7f0dccfae7bb.slice/crio-1e19712b56b6cd68eec6f033c7d826324ba80502576485c7a3ce7744c189f819 WatchSource:0}: Error finding container 1e19712b56b6cd68eec6f033c7d826324ba80502576485c7a3ce7744c189f819: Status 404 returned error can't find the container with id 1e19712b56b6cd68eec6f033c7d826324ba80502576485c7a3ce7744c189f819
Sep 18 19:18:13.595201 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:13.595167    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5j97v" event={"ID":"f516e39e-b125-4617-94ce-7f0dccfae7bb","Type":"ContainerStarted","Data":"1e19712b56b6cd68eec6f033c7d826324ba80502576485c7a3ce7744c189f819"}
Sep 18 19:18:15.604814 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:15.604776    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5j97v" event={"ID":"f516e39e-b125-4617-94ce-7f0dccfae7bb","Type":"ContainerStarted","Data":"9d32b8f94c71970c02a4f47c3ddcffb3af68d132218f105c6297c86faf04ee51"}
Sep 18 19:18:15.605262 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:15.604835    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5j97v" event={"ID":"f516e39e-b125-4617-94ce-7f0dccfae7bb","Type":"ContainerStarted","Data":"b2de73659082f406e022f257ef236421b68be28edf5bef80958f0c87fd60a149"}
Sep 18 19:18:15.635249 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:15.635201    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-5j97v" podStartSLOduration=254.160581546 podStartE2EDuration="4m15.635186994s" totalImagesPullingTime="1.474605448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:14:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:18:15.633740395 +0000 UTC m=+255.517266860" watchObservedRunningTime="2026-09-18 19:18:15.635186994 +0000 UTC m=+255.518713455"
Sep 18 19:18:35.124717 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:35.124633    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 19:18:35.125205 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:35.124931    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="357238c4-1440-486a-b32e-716b8f889807" containerName="alertmanager" containerID="cri-o://d20643fcbf1c9b749355e6bc988a7e2adf3e89adc115618c3b701bbd0b6fb71a" gracePeriod=120
Sep 18 19:18:35.125205 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:35.125012    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="357238c4-1440-486a-b32e-716b8f889807" containerName="kube-rbac-proxy-metric" containerID="cri-o://264b193e3ed0eb780f6b0b84a39530f36dbef838d54ffe6a6089b0c1bfd9bf98" gracePeriod=120
Sep 18 19:18:35.125205 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:35.125038    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="357238c4-1440-486a-b32e-716b8f889807" containerName="kube-rbac-proxy-web" containerID="cri-o://4aba7dfdbd4be2e427f8ccd520a39954d987462471299e2616948f125ee716a1" gracePeriod=120
Sep 18 19:18:35.125205 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:35.125056    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="357238c4-1440-486a-b32e-716b8f889807" containerName="config-reloader" containerID="cri-o://c81564c8e60403017c272ccec2afd5ff55b85521f6e58749b2449e176f2d44ae" gracePeriod=120
Sep 18 19:18:35.125205 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:35.125094    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="357238c4-1440-486a-b32e-716b8f889807" containerName="kube-rbac-proxy" containerID="cri-o://300f03e2b19351c8234b01aa7b6ae709b7f77caafb5b663bcff67482a1e4db21" gracePeriod=120
Sep 18 19:18:35.125205 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:35.125087    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="357238c4-1440-486a-b32e-716b8f889807" containerName="prom-label-proxy" containerID="cri-o://313bf6de775f0b0de4a200149b214f9414ae5cc42b32b2809df9dca3a6fc1cfb" gracePeriod=120
Sep 18 19:18:35.678941 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:35.678909    2522 generic.go:356] "Generic (PLEG): container finished" podID="357238c4-1440-486a-b32e-716b8f889807" containerID="313bf6de775f0b0de4a200149b214f9414ae5cc42b32b2809df9dca3a6fc1cfb" exitCode=0
Sep 18 19:18:35.678941 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:35.678934    2522 generic.go:356] "Generic (PLEG): container finished" podID="357238c4-1440-486a-b32e-716b8f889807" containerID="300f03e2b19351c8234b01aa7b6ae709b7f77caafb5b663bcff67482a1e4db21" exitCode=0
Sep 18 19:18:35.678941 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:35.678942    2522 generic.go:356] "Generic (PLEG): container finished" podID="357238c4-1440-486a-b32e-716b8f889807" containerID="c81564c8e60403017c272ccec2afd5ff55b85521f6e58749b2449e176f2d44ae" exitCode=0
Sep 18 19:18:35.678941 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:35.678949    2522 generic.go:356] "Generic (PLEG): container finished" podID="357238c4-1440-486a-b32e-716b8f889807" containerID="d20643fcbf1c9b749355e6bc988a7e2adf3e89adc115618c3b701bbd0b6fb71a" exitCode=0
Sep 18 19:18:35.679215 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:35.678981    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"357238c4-1440-486a-b32e-716b8f889807","Type":"ContainerDied","Data":"313bf6de775f0b0de4a200149b214f9414ae5cc42b32b2809df9dca3a6fc1cfb"}
Sep 18 19:18:35.679215 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:35.679045    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"357238c4-1440-486a-b32e-716b8f889807","Type":"ContainerDied","Data":"300f03e2b19351c8234b01aa7b6ae709b7f77caafb5b663bcff67482a1e4db21"}
Sep 18 19:18:35.679215 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:35.679071    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"357238c4-1440-486a-b32e-716b8f889807","Type":"ContainerDied","Data":"c81564c8e60403017c272ccec2afd5ff55b85521f6e58749b2449e176f2d44ae"}
Sep 18 19:18:35.679215 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:35.679094    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"357238c4-1440-486a-b32e-716b8f889807","Type":"ContainerDied","Data":"d20643fcbf1c9b749355e6bc988a7e2adf3e89adc115618c3b701bbd0b6fb71a"}
Sep 18 19:18:36.365923 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.365896    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:18:36.463172 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.463078    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/357238c4-1440-486a-b32e-716b8f889807-alertmanager-trusted-ca-bundle\") pod \"357238c4-1440-486a-b32e-716b8f889807\" (UID: \"357238c4-1440-486a-b32e-716b8f889807\") "
Sep 18 19:18:36.463172 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.463127    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/357238c4-1440-486a-b32e-716b8f889807-secret-alertmanager-kube-rbac-proxy-web\") pod \"357238c4-1440-486a-b32e-716b8f889807\" (UID: \"357238c4-1440-486a-b32e-716b8f889807\") "
Sep 18 19:18:36.463172 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.463155    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/357238c4-1440-486a-b32e-716b8f889807-config-volume\") pod \"357238c4-1440-486a-b32e-716b8f889807\" (UID: \"357238c4-1440-486a-b32e-716b8f889807\") "
Sep 18 19:18:36.463451 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.463184    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/357238c4-1440-486a-b32e-716b8f889807-alertmanager-main-db\") pod \"357238c4-1440-486a-b32e-716b8f889807\" (UID: \"357238c4-1440-486a-b32e-716b8f889807\") "
Sep 18 19:18:36.463451 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.463201    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/357238c4-1440-486a-b32e-716b8f889807-secret-alertmanager-main-tls\") pod \"357238c4-1440-486a-b32e-716b8f889807\" (UID: \"357238c4-1440-486a-b32e-716b8f889807\") "
Sep 18 19:18:36.463451 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.463219    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/357238c4-1440-486a-b32e-716b8f889807-web-config\") pod \"357238c4-1440-486a-b32e-716b8f889807\" (UID: \"357238c4-1440-486a-b32e-716b8f889807\") "
Sep 18 19:18:36.463451 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.463246    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/357238c4-1440-486a-b32e-716b8f889807-secret-alertmanager-kube-rbac-proxy-metric\") pod \"357238c4-1440-486a-b32e-716b8f889807\" (UID: \"357238c4-1440-486a-b32e-716b8f889807\") "
Sep 18 19:18:36.463451 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.463272    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/357238c4-1440-486a-b32e-716b8f889807-tls-assets\") pod \"357238c4-1440-486a-b32e-716b8f889807\" (UID: \"357238c4-1440-486a-b32e-716b8f889807\") "
Sep 18 19:18:36.463451 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.463301    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/357238c4-1440-486a-b32e-716b8f889807-secret-alertmanager-kube-rbac-proxy\") pod \"357238c4-1440-486a-b32e-716b8f889807\" (UID: \"357238c4-1440-486a-b32e-716b8f889807\") "
Sep 18 19:18:36.463451 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.463327    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/357238c4-1440-486a-b32e-716b8f889807-metrics-client-ca\") pod \"357238c4-1440-486a-b32e-716b8f889807\" (UID: \"357238c4-1440-486a-b32e-716b8f889807\") "
Sep 18 19:18:36.463451 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.463370    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5hvxw\" (UniqueName: \"kubernetes.io/projected/357238c4-1440-486a-b32e-716b8f889807-kube-api-access-5hvxw\") pod \"357238c4-1440-486a-b32e-716b8f889807\" (UID: \"357238c4-1440-486a-b32e-716b8f889807\") "
Sep 18 19:18:36.463451 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.463400    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/357238c4-1440-486a-b32e-716b8f889807-cluster-tls-config\") pod \"357238c4-1440-486a-b32e-716b8f889807\" (UID: \"357238c4-1440-486a-b32e-716b8f889807\") "
Sep 18 19:18:36.463905 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.463484    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/357238c4-1440-486a-b32e-716b8f889807-config-out\") pod \"357238c4-1440-486a-b32e-716b8f889807\" (UID: \"357238c4-1440-486a-b32e-716b8f889807\") "
Sep 18 19:18:36.463905 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.463557    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/357238c4-1440-486a-b32e-716b8f889807-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "357238c4-1440-486a-b32e-716b8f889807" (UID: "357238c4-1440-486a-b32e-716b8f889807"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:18:36.463905 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.463859    2522 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/357238c4-1440-486a-b32e-716b8f889807-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:18:36.463905 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.463592    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/357238c4-1440-486a-b32e-716b8f889807-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "357238c4-1440-486a-b32e-716b8f889807" (UID: "357238c4-1440-486a-b32e-716b8f889807"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:18:36.464102 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.463970    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/357238c4-1440-486a-b32e-716b8f889807-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "357238c4-1440-486a-b32e-716b8f889807" (UID: "357238c4-1440-486a-b32e-716b8f889807"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:18:36.466606 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.466452    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/357238c4-1440-486a-b32e-716b8f889807-kube-api-access-5hvxw" (OuterVolumeSpecName: "kube-api-access-5hvxw") pod "357238c4-1440-486a-b32e-716b8f889807" (UID: "357238c4-1440-486a-b32e-716b8f889807"). InnerVolumeSpecName "kube-api-access-5hvxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:18:36.466606 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.466542    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/357238c4-1440-486a-b32e-716b8f889807-config-out" (OuterVolumeSpecName: "config-out") pod "357238c4-1440-486a-b32e-716b8f889807" (UID: "357238c4-1440-486a-b32e-716b8f889807"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:18:36.466606 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.466545    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/357238c4-1440-486a-b32e-716b8f889807-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "357238c4-1440-486a-b32e-716b8f889807" (UID: "357238c4-1440-486a-b32e-716b8f889807"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:18:36.466841 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.466619    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/357238c4-1440-486a-b32e-716b8f889807-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "357238c4-1440-486a-b32e-716b8f889807" (UID: "357238c4-1440-486a-b32e-716b8f889807"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:18:36.466841 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.466647    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/357238c4-1440-486a-b32e-716b8f889807-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "357238c4-1440-486a-b32e-716b8f889807" (UID: "357238c4-1440-486a-b32e-716b8f889807"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:18:36.466963 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.466935    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/357238c4-1440-486a-b32e-716b8f889807-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "357238c4-1440-486a-b32e-716b8f889807" (UID: "357238c4-1440-486a-b32e-716b8f889807"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:18:36.467335 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.467308    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/357238c4-1440-486a-b32e-716b8f889807-config-volume" (OuterVolumeSpecName: "config-volume") pod "357238c4-1440-486a-b32e-716b8f889807" (UID: "357238c4-1440-486a-b32e-716b8f889807"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:18:36.467941 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.467914    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/357238c4-1440-486a-b32e-716b8f889807-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "357238c4-1440-486a-b32e-716b8f889807" (UID: "357238c4-1440-486a-b32e-716b8f889807"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:18:36.470985 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.470963    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/357238c4-1440-486a-b32e-716b8f889807-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "357238c4-1440-486a-b32e-716b8f889807" (UID: "357238c4-1440-486a-b32e-716b8f889807"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:18:36.476885 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.476863    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/357238c4-1440-486a-b32e-716b8f889807-web-config" (OuterVolumeSpecName: "web-config") pod "357238c4-1440-486a-b32e-716b8f889807" (UID: "357238c4-1440-486a-b32e-716b8f889807"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:18:36.500299 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.500270    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6bd78c47cf-5hjvg"]
Sep 18 19:18:36.501279 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.501192    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="357238c4-1440-486a-b32e-716b8f889807" containerName="kube-rbac-proxy"
Sep 18 19:18:36.501279 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.501221    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="357238c4-1440-486a-b32e-716b8f889807" containerName="kube-rbac-proxy"
Sep 18 19:18:36.503255 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.501233    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="357238c4-1440-486a-b32e-716b8f889807" containerName="prom-label-proxy"
Sep 18 19:18:36.503255 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.501496    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="357238c4-1440-486a-b32e-716b8f889807" containerName="prom-label-proxy"
Sep 18 19:18:36.503255 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.501526    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="357238c4-1440-486a-b32e-716b8f889807" containerName="config-reloader"
Sep 18 19:18:36.503255 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.501536    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="357238c4-1440-486a-b32e-716b8f889807" containerName="config-reloader"
Sep 18 19:18:36.503255 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.501549    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="357238c4-1440-486a-b32e-716b8f889807" containerName="kube-rbac-proxy-web"
Sep 18 19:18:36.503255 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.501577    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="357238c4-1440-486a-b32e-716b8f889807" containerName="kube-rbac-proxy-web"
Sep 18 19:18:36.503255 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.501600    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="357238c4-1440-486a-b32e-716b8f889807" containerName="kube-rbac-proxy-metric"
Sep 18 19:18:36.503255 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.501610    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="357238c4-1440-486a-b32e-716b8f889807" containerName="kube-rbac-proxy-metric"
Sep 18 19:18:36.503255 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.501622    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="357238c4-1440-486a-b32e-716b8f889807" containerName="alertmanager"
Sep 18 19:18:36.503255 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.501631    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="357238c4-1440-486a-b32e-716b8f889807" containerName="alertmanager"
Sep 18 19:18:36.503255 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.501677    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="357238c4-1440-486a-b32e-716b8f889807" containerName="init-config-reloader"
Sep 18 19:18:36.503255 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.501687    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="357238c4-1440-486a-b32e-716b8f889807" containerName="init-config-reloader"
Sep 18 19:18:36.503255 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.501839    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="357238c4-1440-486a-b32e-716b8f889807" containerName="kube-rbac-proxy-metric"
Sep 18 19:18:36.503255 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.501850    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="357238c4-1440-486a-b32e-716b8f889807" containerName="prom-label-proxy"
Sep 18 19:18:36.503255 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.501866    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="357238c4-1440-486a-b32e-716b8f889807" containerName="kube-rbac-proxy"
Sep 18 19:18:36.503255 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.501875    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="357238c4-1440-486a-b32e-716b8f889807" containerName="alertmanager"
Sep 18 19:18:36.503255 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.501884    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="357238c4-1440-486a-b32e-716b8f889807" containerName="config-reloader"
Sep 18 19:18:36.503255 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.501897    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="357238c4-1440-486a-b32e-716b8f889807" containerName="kube-rbac-proxy-web"
Sep 18 19:18:36.506641 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.506618    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bd78c47cf-5hjvg"
Sep 18 19:18:36.564683 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.564645    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e87fd35d-f7a7-429f-a997-0267c01d7405-console-oauth-config\") pod \"console-6bd78c47cf-5hjvg\" (UID: \"e87fd35d-f7a7-429f-a997-0267c01d7405\") " pod="openshift-console/console-6bd78c47cf-5hjvg"
Sep 18 19:18:36.564683 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.564681    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e87fd35d-f7a7-429f-a997-0267c01d7405-oauth-serving-cert\") pod \"console-6bd78c47cf-5hjvg\" (UID: \"e87fd35d-f7a7-429f-a997-0267c01d7405\") " pod="openshift-console/console-6bd78c47cf-5hjvg"
Sep 18 19:18:36.564938 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.564711    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e87fd35d-f7a7-429f-a997-0267c01d7405-console-serving-cert\") pod \"console-6bd78c47cf-5hjvg\" (UID: \"e87fd35d-f7a7-429f-a997-0267c01d7405\") " pod="openshift-console/console-6bd78c47cf-5hjvg"
Sep 18 19:18:36.564938 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.564854    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e87fd35d-f7a7-429f-a997-0267c01d7405-console-config\") pod \"console-6bd78c47cf-5hjvg\" (UID: \"e87fd35d-f7a7-429f-a997-0267c01d7405\") " pod="openshift-console/console-6bd78c47cf-5hjvg"
Sep 18 19:18:36.565013 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.564933    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lf4g\" (UniqueName: \"kubernetes.io/projected/e87fd35d-f7a7-429f-a997-0267c01d7405-kube-api-access-5lf4g\") pod \"console-6bd78c47cf-5hjvg\" (UID: \"e87fd35d-f7a7-429f-a997-0267c01d7405\") " pod="openshift-console/console-6bd78c47cf-5hjvg"
Sep 18 19:18:36.565013 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.564975    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e87fd35d-f7a7-429f-a997-0267c01d7405-service-ca\") pod \"console-6bd78c47cf-5hjvg\" (UID: \"e87fd35d-f7a7-429f-a997-0267c01d7405\") " pod="openshift-console/console-6bd78c47cf-5hjvg"
Sep 18 19:18:36.565013 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.564993    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e87fd35d-f7a7-429f-a997-0267c01d7405-trusted-ca-bundle\") pod \"console-6bd78c47cf-5hjvg\" (UID: \"e87fd35d-f7a7-429f-a997-0267c01d7405\") " pod="openshift-console/console-6bd78c47cf-5hjvg"
Sep 18 19:18:36.565149 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.565030    2522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hvxw\" (UniqueName: \"kubernetes.io/projected/357238c4-1440-486a-b32e-716b8f889807-kube-api-access-5hvxw\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:18:36.565149 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.565041    2522 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/357238c4-1440-486a-b32e-716b8f889807-cluster-tls-config\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:18:36.565149 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.565085    2522 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/357238c4-1440-486a-b32e-716b8f889807-config-out\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:18:36.565149 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.565105    2522 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/357238c4-1440-486a-b32e-716b8f889807-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:18:36.565149 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.565122    2522 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/357238c4-1440-486a-b32e-716b8f889807-config-volume\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:18:36.565149 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.565136    2522 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/357238c4-1440-486a-b32e-716b8f889807-alertmanager-main-db\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:18:36.565149 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.565150    2522 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/357238c4-1440-486a-b32e-716b8f889807-secret-alertmanager-main-tls\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:18:36.565382 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.565163    2522 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/357238c4-1440-486a-b32e-716b8f889807-web-config\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:18:36.565382 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.565178    2522 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/357238c4-1440-486a-b32e-716b8f889807-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:18:36.565382 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.565191    2522 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/357238c4-1440-486a-b32e-716b8f889807-tls-assets\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:18:36.565382 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.565204    2522 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/357238c4-1440-486a-b32e-716b8f889807-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:18:36.565382 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.565218    2522 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/357238c4-1440-486a-b32e-716b8f889807-metrics-client-ca\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:18:36.571618 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.571594    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bd78c47cf-5hjvg"]
Sep 18 19:18:36.666037 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.666003    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lf4g\" (UniqueName: \"kubernetes.io/projected/e87fd35d-f7a7-429f-a997-0267c01d7405-kube-api-access-5lf4g\") pod \"console-6bd78c47cf-5hjvg\" (UID: \"e87fd35d-f7a7-429f-a997-0267c01d7405\") " pod="openshift-console/console-6bd78c47cf-5hjvg"
Sep 18 19:18:36.666207 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.666045    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e87fd35d-f7a7-429f-a997-0267c01d7405-service-ca\") pod \"console-6bd78c47cf-5hjvg\" (UID: \"e87fd35d-f7a7-429f-a997-0267c01d7405\") " pod="openshift-console/console-6bd78c47cf-5hjvg"
Sep 18 19:18:36.666207 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.666066    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e87fd35d-f7a7-429f-a997-0267c01d7405-trusted-ca-bundle\") pod \"console-6bd78c47cf-5hjvg\" (UID: \"e87fd35d-f7a7-429f-a997-0267c01d7405\") " pod="openshift-console/console-6bd78c47cf-5hjvg"
Sep 18 19:18:36.666318 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.666199    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e87fd35d-f7a7-429f-a997-0267c01d7405-console-oauth-config\") pod \"console-6bd78c47cf-5hjvg\" (UID: \"e87fd35d-f7a7-429f-a997-0267c01d7405\") " pod="openshift-console/console-6bd78c47cf-5hjvg"
Sep 18 19:18:36.666318 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.666240    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e87fd35d-f7a7-429f-a997-0267c01d7405-oauth-serving-cert\") pod \"console-6bd78c47cf-5hjvg\" (UID: \"e87fd35d-f7a7-429f-a997-0267c01d7405\") " pod="openshift-console/console-6bd78c47cf-5hjvg"
Sep 18 19:18:36.666318 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.666303    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e87fd35d-f7a7-429f-a997-0267c01d7405-console-serving-cert\") pod \"console-6bd78c47cf-5hjvg\" (UID: \"e87fd35d-f7a7-429f-a997-0267c01d7405\") " pod="openshift-console/console-6bd78c47cf-5hjvg"
Sep 18 19:18:36.666494 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.666385    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e87fd35d-f7a7-429f-a997-0267c01d7405-console-config\") pod \"console-6bd78c47cf-5hjvg\" (UID: \"e87fd35d-f7a7-429f-a997-0267c01d7405\") " pod="openshift-console/console-6bd78c47cf-5hjvg"
Sep 18 19:18:36.666885 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.666857    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e87fd35d-f7a7-429f-a997-0267c01d7405-service-ca\") pod \"console-6bd78c47cf-5hjvg\" (UID: \"e87fd35d-f7a7-429f-a997-0267c01d7405\") " pod="openshift-console/console-6bd78c47cf-5hjvg"
Sep 18 19:18:36.667088 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.667040    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e87fd35d-f7a7-429f-a997-0267c01d7405-trusted-ca-bundle\") pod \"console-6bd78c47cf-5hjvg\" (UID: \"e87fd35d-f7a7-429f-a997-0267c01d7405\") " pod="openshift-console/console-6bd78c47cf-5hjvg"
Sep 18 19:18:36.667088 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.667066    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e87fd35d-f7a7-429f-a997-0267c01d7405-console-config\") pod \"console-6bd78c47cf-5hjvg\" (UID: \"e87fd35d-f7a7-429f-a997-0267c01d7405\") " pod="openshift-console/console-6bd78c47cf-5hjvg"
Sep 18 19:18:36.667251 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.667038    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e87fd35d-f7a7-429f-a997-0267c01d7405-oauth-serving-cert\") pod \"console-6bd78c47cf-5hjvg\" (UID: \"e87fd35d-f7a7-429f-a997-0267c01d7405\") " pod="openshift-console/console-6bd78c47cf-5hjvg"
Sep 18 19:18:36.668877 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.668854    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e87fd35d-f7a7-429f-a997-0267c01d7405-console-oauth-config\") pod \"console-6bd78c47cf-5hjvg\" (UID: \"e87fd35d-f7a7-429f-a997-0267c01d7405\") " pod="openshift-console/console-6bd78c47cf-5hjvg"
Sep 18 19:18:36.668955 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.668921    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e87fd35d-f7a7-429f-a997-0267c01d7405-console-serving-cert\") pod \"console-6bd78c47cf-5hjvg\" (UID: \"e87fd35d-f7a7-429f-a997-0267c01d7405\") " pod="openshift-console/console-6bd78c47cf-5hjvg"
Sep 18 19:18:36.687233 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.687202    2522 generic.go:356] "Generic (PLEG): container finished" podID="357238c4-1440-486a-b32e-716b8f889807" containerID="264b193e3ed0eb780f6b0b84a39530f36dbef838d54ffe6a6089b0c1bfd9bf98" exitCode=0
Sep 18 19:18:36.687233 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.687230    2522 generic.go:356] "Generic (PLEG): container finished" podID="357238c4-1440-486a-b32e-716b8f889807" containerID="4aba7dfdbd4be2e427f8ccd520a39954d987462471299e2616948f125ee716a1" exitCode=0
Sep 18 19:18:36.687455 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.687286    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:18:36.687455 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.687287    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"357238c4-1440-486a-b32e-716b8f889807","Type":"ContainerDied","Data":"264b193e3ed0eb780f6b0b84a39530f36dbef838d54ffe6a6089b0c1bfd9bf98"}
Sep 18 19:18:36.687455 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.687355    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"357238c4-1440-486a-b32e-716b8f889807","Type":"ContainerDied","Data":"4aba7dfdbd4be2e427f8ccd520a39954d987462471299e2616948f125ee716a1"}
Sep 18 19:18:36.687455 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.687382    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"357238c4-1440-486a-b32e-716b8f889807","Type":"ContainerDied","Data":"b2f275745acfb4c2c59a13f081238471e7ea11ba6cc452d9b83fa3f4ada8d035"}
Sep 18 19:18:36.687455 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.687412    2522 scope.go:117] "RemoveContainer" containerID="313bf6de775f0b0de4a200149b214f9414ae5cc42b32b2809df9dca3a6fc1cfb"
Sep 18 19:18:36.696015 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.695996    2522 scope.go:117] "RemoveContainer" containerID="264b193e3ed0eb780f6b0b84a39530f36dbef838d54ffe6a6089b0c1bfd9bf98"
Sep 18 19:18:36.703871 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.703852    2522 scope.go:117] "RemoveContainer" containerID="300f03e2b19351c8234b01aa7b6ae709b7f77caafb5b663bcff67482a1e4db21"
Sep 18 19:18:36.704305 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.704284    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lf4g\" (UniqueName: \"kubernetes.io/projected/e87fd35d-f7a7-429f-a997-0267c01d7405-kube-api-access-5lf4g\") pod \"console-6bd78c47cf-5hjvg\" (UID: \"e87fd35d-f7a7-429f-a997-0267c01d7405\") " pod="openshift-console/console-6bd78c47cf-5hjvg"
Sep 18 19:18:36.710726 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.710708    2522 scope.go:117] "RemoveContainer" containerID="4aba7dfdbd4be2e427f8ccd520a39954d987462471299e2616948f125ee716a1"
Sep 18 19:18:36.717239 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.717219    2522 scope.go:117] "RemoveContainer" containerID="c81564c8e60403017c272ccec2afd5ff55b85521f6e58749b2449e176f2d44ae"
Sep 18 19:18:36.724200 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.724184    2522 scope.go:117] "RemoveContainer" containerID="d20643fcbf1c9b749355e6bc988a7e2adf3e89adc115618c3b701bbd0b6fb71a"
Sep 18 19:18:36.731027 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.731010    2522 scope.go:117] "RemoveContainer" containerID="ea754f8ffe36ff5f22037c673a37a712ec75b357c9fe4e0116a132d7534b5c4e"
Sep 18 19:18:36.735925 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.735902    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 19:18:36.738167 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.738152    2522 scope.go:117] "RemoveContainer" containerID="313bf6de775f0b0de4a200149b214f9414ae5cc42b32b2809df9dca3a6fc1cfb"
Sep 18 19:18:36.738440 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:18:36.738420    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"313bf6de775f0b0de4a200149b214f9414ae5cc42b32b2809df9dca3a6fc1cfb\": container with ID starting with 313bf6de775f0b0de4a200149b214f9414ae5cc42b32b2809df9dca3a6fc1cfb not found: ID does not exist" containerID="313bf6de775f0b0de4a200149b214f9414ae5cc42b32b2809df9dca3a6fc1cfb"
Sep 18 19:18:36.738499 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.738449    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"313bf6de775f0b0de4a200149b214f9414ae5cc42b32b2809df9dca3a6fc1cfb"} err="failed to get container status \"313bf6de775f0b0de4a200149b214f9414ae5cc42b32b2809df9dca3a6fc1cfb\": rpc error: code = NotFound desc = could not find container \"313bf6de775f0b0de4a200149b214f9414ae5cc42b32b2809df9dca3a6fc1cfb\": container with ID starting with 313bf6de775f0b0de4a200149b214f9414ae5cc42b32b2809df9dca3a6fc1cfb not found: ID does not exist"
Sep 18 19:18:36.738499 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.738490    2522 scope.go:117] "RemoveContainer" containerID="264b193e3ed0eb780f6b0b84a39530f36dbef838d54ffe6a6089b0c1bfd9bf98"
Sep 18 19:18:36.738784 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:18:36.738768    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"264b193e3ed0eb780f6b0b84a39530f36dbef838d54ffe6a6089b0c1bfd9bf98\": container with ID starting with 264b193e3ed0eb780f6b0b84a39530f36dbef838d54ffe6a6089b0c1bfd9bf98 not found: ID does not exist" containerID="264b193e3ed0eb780f6b0b84a39530f36dbef838d54ffe6a6089b0c1bfd9bf98"
Sep 18 19:18:36.738823 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.738791    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"264b193e3ed0eb780f6b0b84a39530f36dbef838d54ffe6a6089b0c1bfd9bf98"} err="failed to get container status \"264b193e3ed0eb780f6b0b84a39530f36dbef838d54ffe6a6089b0c1bfd9bf98\": rpc error: code = NotFound desc = could not find container \"264b193e3ed0eb780f6b0b84a39530f36dbef838d54ffe6a6089b0c1bfd9bf98\": container with ID starting with 264b193e3ed0eb780f6b0b84a39530f36dbef838d54ffe6a6089b0c1bfd9bf98 not found: ID does not exist"
Sep 18 19:18:36.738823 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.738806    2522 scope.go:117] "RemoveContainer" containerID="300f03e2b19351c8234b01aa7b6ae709b7f77caafb5b663bcff67482a1e4db21"
Sep 18 19:18:36.739042 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:18:36.739027    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"300f03e2b19351c8234b01aa7b6ae709b7f77caafb5b663bcff67482a1e4db21\": container with ID starting with 300f03e2b19351c8234b01aa7b6ae709b7f77caafb5b663bcff67482a1e4db21 not found: ID does not exist" containerID="300f03e2b19351c8234b01aa7b6ae709b7f77caafb5b663bcff67482a1e4db21"
Sep 18 19:18:36.739081 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.739053    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"300f03e2b19351c8234b01aa7b6ae709b7f77caafb5b663bcff67482a1e4db21"} err="failed to get container status \"300f03e2b19351c8234b01aa7b6ae709b7f77caafb5b663bcff67482a1e4db21\": rpc error: code = NotFound desc = could not find container \"300f03e2b19351c8234b01aa7b6ae709b7f77caafb5b663bcff67482a1e4db21\": container with ID starting with 300f03e2b19351c8234b01aa7b6ae709b7f77caafb5b663bcff67482a1e4db21 not found: ID does not exist"
Sep 18 19:18:36.739081 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.739066    2522 scope.go:117] "RemoveContainer" containerID="4aba7dfdbd4be2e427f8ccd520a39954d987462471299e2616948f125ee716a1"
Sep 18 19:18:36.739281 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:18:36.739264    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4aba7dfdbd4be2e427f8ccd520a39954d987462471299e2616948f125ee716a1\": container with ID starting with 4aba7dfdbd4be2e427f8ccd520a39954d987462471299e2616948f125ee716a1 not found: ID does not exist" containerID="4aba7dfdbd4be2e427f8ccd520a39954d987462471299e2616948f125ee716a1"
Sep 18 19:18:36.739317 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.739286    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4aba7dfdbd4be2e427f8ccd520a39954d987462471299e2616948f125ee716a1"} err="failed to get container status \"4aba7dfdbd4be2e427f8ccd520a39954d987462471299e2616948f125ee716a1\": rpc error: code = NotFound desc = could not find container \"4aba7dfdbd4be2e427f8ccd520a39954d987462471299e2616948f125ee716a1\": container with ID starting with 4aba7dfdbd4be2e427f8ccd520a39954d987462471299e2616948f125ee716a1 not found: ID does not exist"
Sep 18 19:18:36.739317 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.739300    2522 scope.go:117] "RemoveContainer" containerID="c81564c8e60403017c272ccec2afd5ff55b85521f6e58749b2449e176f2d44ae"
Sep 18 19:18:36.739551 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:18:36.739533    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c81564c8e60403017c272ccec2afd5ff55b85521f6e58749b2449e176f2d44ae\": container with ID starting with c81564c8e60403017c272ccec2afd5ff55b85521f6e58749b2449e176f2d44ae not found: ID does not exist" containerID="c81564c8e60403017c272ccec2afd5ff55b85521f6e58749b2449e176f2d44ae"
Sep 18 19:18:36.739642 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.739558    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c81564c8e60403017c272ccec2afd5ff55b85521f6e58749b2449e176f2d44ae"} err="failed to get container status \"c81564c8e60403017c272ccec2afd5ff55b85521f6e58749b2449e176f2d44ae\": rpc error: code = NotFound desc = could not find container \"c81564c8e60403017c272ccec2afd5ff55b85521f6e58749b2449e176f2d44ae\": container with ID starting with c81564c8e60403017c272ccec2afd5ff55b85521f6e58749b2449e176f2d44ae not found: ID does not exist"
Sep 18 19:18:36.739642 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.739592    2522 scope.go:117] "RemoveContainer" containerID="d20643fcbf1c9b749355e6bc988a7e2adf3e89adc115618c3b701bbd0b6fb71a"
Sep 18 19:18:36.739859 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:18:36.739844    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d20643fcbf1c9b749355e6bc988a7e2adf3e89adc115618c3b701bbd0b6fb71a\": container with ID starting with d20643fcbf1c9b749355e6bc988a7e2adf3e89adc115618c3b701bbd0b6fb71a not found: ID does not exist" containerID="d20643fcbf1c9b749355e6bc988a7e2adf3e89adc115618c3b701bbd0b6fb71a"
Sep 18 19:18:36.739898 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.739862    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d20643fcbf1c9b749355e6bc988a7e2adf3e89adc115618c3b701bbd0b6fb71a"} err="failed to get container status \"d20643fcbf1c9b749355e6bc988a7e2adf3e89adc115618c3b701bbd0b6fb71a\": rpc error: code = NotFound desc = could not find container \"d20643fcbf1c9b749355e6bc988a7e2adf3e89adc115618c3b701bbd0b6fb71a\": container with ID starting with d20643fcbf1c9b749355e6bc988a7e2adf3e89adc115618c3b701bbd0b6fb71a not found: ID does not exist"
Sep 18 19:18:36.739898 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.739885    2522 scope.go:117] "RemoveContainer" containerID="ea754f8ffe36ff5f22037c673a37a712ec75b357c9fe4e0116a132d7534b5c4e"
Sep 18 19:18:36.740105 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:18:36.740086    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea754f8ffe36ff5f22037c673a37a712ec75b357c9fe4e0116a132d7534b5c4e\": container with ID starting with ea754f8ffe36ff5f22037c673a37a712ec75b357c9fe4e0116a132d7534b5c4e not found: ID does not exist" containerID="ea754f8ffe36ff5f22037c673a37a712ec75b357c9fe4e0116a132d7534b5c4e"
Sep 18 19:18:36.740163 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.740112    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea754f8ffe36ff5f22037c673a37a712ec75b357c9fe4e0116a132d7534b5c4e"} err="failed to get container status \"ea754f8ffe36ff5f22037c673a37a712ec75b357c9fe4e0116a132d7534b5c4e\": rpc error: code = NotFound desc = could not find container \"ea754f8ffe36ff5f22037c673a37a712ec75b357c9fe4e0116a132d7534b5c4e\": container with ID starting with ea754f8ffe36ff5f22037c673a37a712ec75b357c9fe4e0116a132d7534b5c4e not found: ID does not exist"
Sep 18 19:18:36.740163 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.740131    2522 scope.go:117] "RemoveContainer" containerID="313bf6de775f0b0de4a200149b214f9414ae5cc42b32b2809df9dca3a6fc1cfb"
Sep 18 19:18:36.740355 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.740338    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"313bf6de775f0b0de4a200149b214f9414ae5cc42b32b2809df9dca3a6fc1cfb"} err="failed to get container status \"313bf6de775f0b0de4a200149b214f9414ae5cc42b32b2809df9dca3a6fc1cfb\": rpc error: code = NotFound desc = could not find container \"313bf6de775f0b0de4a200149b214f9414ae5cc42b32b2809df9dca3a6fc1cfb\": container with ID starting with 313bf6de775f0b0de4a200149b214f9414ae5cc42b32b2809df9dca3a6fc1cfb not found: ID does not exist"
Sep 18 19:18:36.740422 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.740356    2522 scope.go:117] "RemoveContainer" containerID="264b193e3ed0eb780f6b0b84a39530f36dbef838d54ffe6a6089b0c1bfd9bf98"
Sep 18 19:18:36.740625 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.740603    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"264b193e3ed0eb780f6b0b84a39530f36dbef838d54ffe6a6089b0c1bfd9bf98"} err="failed to get container status \"264b193e3ed0eb780f6b0b84a39530f36dbef838d54ffe6a6089b0c1bfd9bf98\": rpc error: code = NotFound desc = could not find container \"264b193e3ed0eb780f6b0b84a39530f36dbef838d54ffe6a6089b0c1bfd9bf98\": container with ID starting with 264b193e3ed0eb780f6b0b84a39530f36dbef838d54ffe6a6089b0c1bfd9bf98 not found: ID does not exist"
Sep 18 19:18:36.740710 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.740626    2522 scope.go:117] "RemoveContainer" containerID="300f03e2b19351c8234b01aa7b6ae709b7f77caafb5b663bcff67482a1e4db21"
Sep 18 19:18:36.740844 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.740826    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"300f03e2b19351c8234b01aa7b6ae709b7f77caafb5b663bcff67482a1e4db21"} err="failed to get container status \"300f03e2b19351c8234b01aa7b6ae709b7f77caafb5b663bcff67482a1e4db21\": rpc error: code = NotFound desc = could not find container \"300f03e2b19351c8234b01aa7b6ae709b7f77caafb5b663bcff67482a1e4db21\": container with ID starting with 300f03e2b19351c8234b01aa7b6ae709b7f77caafb5b663bcff67482a1e4db21 not found: ID does not exist"
Sep 18 19:18:36.740889 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.740846    2522 scope.go:117] "RemoveContainer" containerID="4aba7dfdbd4be2e427f8ccd520a39954d987462471299e2616948f125ee716a1"
Sep 18 19:18:36.741042 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.741027    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4aba7dfdbd4be2e427f8ccd520a39954d987462471299e2616948f125ee716a1"} err="failed to get container status \"4aba7dfdbd4be2e427f8ccd520a39954d987462471299e2616948f125ee716a1\": rpc error: code = NotFound desc = could not find container \"4aba7dfdbd4be2e427f8ccd520a39954d987462471299e2616948f125ee716a1\": container with ID starting with 4aba7dfdbd4be2e427f8ccd520a39954d987462471299e2616948f125ee716a1 not found: ID does not exist"
Sep 18 19:18:36.741085 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.741042    2522 scope.go:117] "RemoveContainer" containerID="c81564c8e60403017c272ccec2afd5ff55b85521f6e58749b2449e176f2d44ae"
Sep 18 19:18:36.741268 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.741248    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c81564c8e60403017c272ccec2afd5ff55b85521f6e58749b2449e176f2d44ae"} err="failed to get container status \"c81564c8e60403017c272ccec2afd5ff55b85521f6e58749b2449e176f2d44ae\": rpc error: code = NotFound desc = could not find container \"c81564c8e60403017c272ccec2afd5ff55b85521f6e58749b2449e176f2d44ae\": container with ID starting with c81564c8e60403017c272ccec2afd5ff55b85521f6e58749b2449e176f2d44ae not found: ID does not exist"
Sep 18 19:18:36.741317 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.741270    2522 scope.go:117] "RemoveContainer" containerID="d20643fcbf1c9b749355e6bc988a7e2adf3e89adc115618c3b701bbd0b6fb71a"
Sep 18 19:18:36.741495 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.741476    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d20643fcbf1c9b749355e6bc988a7e2adf3e89adc115618c3b701bbd0b6fb71a"} err="failed to get container status \"d20643fcbf1c9b749355e6bc988a7e2adf3e89adc115618c3b701bbd0b6fb71a\": rpc error: code = NotFound desc = could not find container \"d20643fcbf1c9b749355e6bc988a7e2adf3e89adc115618c3b701bbd0b6fb71a\": container with ID starting with d20643fcbf1c9b749355e6bc988a7e2adf3e89adc115618c3b701bbd0b6fb71a not found: ID does not exist"
Sep 18 19:18:36.741577 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.741496    2522 scope.go:117] "RemoveContainer" containerID="ea754f8ffe36ff5f22037c673a37a712ec75b357c9fe4e0116a132d7534b5c4e"
Sep 18 19:18:36.741744 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.741723    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea754f8ffe36ff5f22037c673a37a712ec75b357c9fe4e0116a132d7534b5c4e"} err="failed to get container status \"ea754f8ffe36ff5f22037c673a37a712ec75b357c9fe4e0116a132d7534b5c4e\": rpc error: code = NotFound desc = could not find container \"ea754f8ffe36ff5f22037c673a37a712ec75b357c9fe4e0116a132d7534b5c4e\": container with ID starting with ea754f8ffe36ff5f22037c673a37a712ec75b357c9fe4e0116a132d7534b5c4e not found: ID does not exist"
Sep 18 19:18:36.749864 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.749843    2522 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 19:18:36.802074 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.802043    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 19:18:36.807299 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.807281    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:18:36.814126 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.813984    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 18 19:18:36.814126 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.814032    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 18 19:18:36.814126 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.814040    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 18 19:18:36.814126 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.814036    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 18 19:18:36.814126 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.814092    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 18 19:18:36.814126 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.813983    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 18 19:18:36.814126 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.814115    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 18 19:18:36.814449 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.814099    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 18 19:18:36.814449 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.814167    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-dfddl\""
Sep 18 19:18:36.815211 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.815196    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bd78c47cf-5hjvg"
Sep 18 19:18:36.837786 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.837756    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 18 19:18:36.838162 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.838137    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 19:18:36.869187 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.869156    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0c8d507-b773-4b4e-b84c-3d387bbd2223-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d0c8d507-b773-4b4e-b84c-3d387bbd2223\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:18:36.869291 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.869248    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0c8d507-b773-4b4e-b84c-3d387bbd2223-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d0c8d507-b773-4b4e-b84c-3d387bbd2223\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:18:36.869291 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.869280    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d0c8d507-b773-4b4e-b84c-3d387bbd2223-config-out\") pod \"alertmanager-main-0\" (UID: \"d0c8d507-b773-4b4e-b84c-3d387bbd2223\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:18:36.869366 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.869312    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d0c8d507-b773-4b4e-b84c-3d387bbd2223-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d0c8d507-b773-4b4e-b84c-3d387bbd2223\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:18:36.869366 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.869338    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d0c8d507-b773-4b4e-b84c-3d387bbd2223-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d0c8d507-b773-4b4e-b84c-3d387bbd2223\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:18:36.869423 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.869382    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d0c8d507-b773-4b4e-b84c-3d387bbd2223-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d0c8d507-b773-4b4e-b84c-3d387bbd2223\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:18:36.869423 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.869401    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d0c8d507-b773-4b4e-b84c-3d387bbd2223-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d0c8d507-b773-4b4e-b84c-3d387bbd2223\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:18:36.869423 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.869417    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d0c8d507-b773-4b4e-b84c-3d387bbd2223-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d0c8d507-b773-4b4e-b84c-3d387bbd2223\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:18:36.869513 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.869456    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d0c8d507-b773-4b4e-b84c-3d387bbd2223-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d0c8d507-b773-4b4e-b84c-3d387bbd2223\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:18:36.869513 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.869480    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d0c8d507-b773-4b4e-b84c-3d387bbd2223-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d0c8d507-b773-4b4e-b84c-3d387bbd2223\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:18:36.869513 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.869498    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d0c8d507-b773-4b4e-b84c-3d387bbd2223-web-config\") pod \"alertmanager-main-0\" (UID: \"d0c8d507-b773-4b4e-b84c-3d387bbd2223\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:18:36.869643 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.869512    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzf98\" (UniqueName: \"kubernetes.io/projected/d0c8d507-b773-4b4e-b84c-3d387bbd2223-kube-api-access-pzf98\") pod \"alertmanager-main-0\" (UID: \"d0c8d507-b773-4b4e-b84c-3d387bbd2223\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:18:36.869643 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.869530    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d0c8d507-b773-4b4e-b84c-3d387bbd2223-config-volume\") pod \"alertmanager-main-0\" (UID: \"d0c8d507-b773-4b4e-b84c-3d387bbd2223\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:18:36.970352 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.970259    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d0c8d507-b773-4b4e-b84c-3d387bbd2223-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d0c8d507-b773-4b4e-b84c-3d387bbd2223\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:18:36.970352 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.970290    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d0c8d507-b773-4b4e-b84c-3d387bbd2223-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d0c8d507-b773-4b4e-b84c-3d387bbd2223\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:18:36.970622 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.970309    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d0c8d507-b773-4b4e-b84c-3d387bbd2223-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d0c8d507-b773-4b4e-b84c-3d387bbd2223\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:18:36.970622 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.970439    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d0c8d507-b773-4b4e-b84c-3d387bbd2223-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d0c8d507-b773-4b4e-b84c-3d387bbd2223\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:18:36.970622 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.970481    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d0c8d507-b773-4b4e-b84c-3d387bbd2223-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d0c8d507-b773-4b4e-b84c-3d387bbd2223\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:18:36.970622 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.970511    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d0c8d507-b773-4b4e-b84c-3d387bbd2223-web-config\") pod \"alertmanager-main-0\" (UID: \"d0c8d507-b773-4b4e-b84c-3d387bbd2223\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:18:36.970622 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.970533    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzf98\" (UniqueName: \"kubernetes.io/projected/d0c8d507-b773-4b4e-b84c-3d387bbd2223-kube-api-access-pzf98\") pod \"alertmanager-main-0\" (UID: \"d0c8d507-b773-4b4e-b84c-3d387bbd2223\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:18:36.970622 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.970583    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d0c8d507-b773-4b4e-b84c-3d387bbd2223-config-volume\") pod \"alertmanager-main-0\" (UID: \"d0c8d507-b773-4b4e-b84c-3d387bbd2223\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:18:36.970622 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.970609    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0c8d507-b773-4b4e-b84c-3d387bbd2223-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d0c8d507-b773-4b4e-b84c-3d387bbd2223\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:18:36.970976 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.970647    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0c8d507-b773-4b4e-b84c-3d387bbd2223-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d0c8d507-b773-4b4e-b84c-3d387bbd2223\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:18:36.970976 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.970673    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d0c8d507-b773-4b4e-b84c-3d387bbd2223-config-out\") pod \"alertmanager-main-0\" (UID: \"d0c8d507-b773-4b4e-b84c-3d387bbd2223\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:18:36.970976 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.970683    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d0c8d507-b773-4b4e-b84c-3d387bbd2223-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d0c8d507-b773-4b4e-b84c-3d387bbd2223\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:18:36.970976 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.970718    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d0c8d507-b773-4b4e-b84c-3d387bbd2223-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d0c8d507-b773-4b4e-b84c-3d387bbd2223\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:18:36.970976 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.970752    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d0c8d507-b773-4b4e-b84c-3d387bbd2223-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d0c8d507-b773-4b4e-b84c-3d387bbd2223\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:18:36.972040 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.972012    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0c8d507-b773-4b4e-b84c-3d387bbd2223-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d0c8d507-b773-4b4e-b84c-3d387bbd2223\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:18:36.972667 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.972644    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0c8d507-b773-4b4e-b84c-3d387bbd2223-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d0c8d507-b773-4b4e-b84c-3d387bbd2223\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:18:36.973543 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.973480    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d0c8d507-b773-4b4e-b84c-3d387bbd2223-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d0c8d507-b773-4b4e-b84c-3d387bbd2223\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:18:36.973543 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.973491    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d0c8d507-b773-4b4e-b84c-3d387bbd2223-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d0c8d507-b773-4b4e-b84c-3d387bbd2223\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:18:36.973543 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.973489    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d0c8d507-b773-4b4e-b84c-3d387bbd2223-web-config\") pod \"alertmanager-main-0\" (UID: \"d0c8d507-b773-4b4e-b84c-3d387bbd2223\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:18:36.973778 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.973624    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d0c8d507-b773-4b4e-b84c-3d387bbd2223-config-volume\") pod \"alertmanager-main-0\" (UID: \"d0c8d507-b773-4b4e-b84c-3d387bbd2223\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:18:36.973778 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.973650    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d0c8d507-b773-4b4e-b84c-3d387bbd2223-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d0c8d507-b773-4b4e-b84c-3d387bbd2223\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:18:36.973778 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.973730    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d0c8d507-b773-4b4e-b84c-3d387bbd2223-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d0c8d507-b773-4b4e-b84c-3d387bbd2223\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:18:36.974555 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.974536    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d0c8d507-b773-4b4e-b84c-3d387bbd2223-config-out\") pod \"alertmanager-main-0\" (UID: \"d0c8d507-b773-4b4e-b84c-3d387bbd2223\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:18:36.975032 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.975012    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d0c8d507-b773-4b4e-b84c-3d387bbd2223-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d0c8d507-b773-4b4e-b84c-3d387bbd2223\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:18:36.975500 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.975485    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d0c8d507-b773-4b4e-b84c-3d387bbd2223-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d0c8d507-b773-4b4e-b84c-3d387bbd2223\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:18:36.982515 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:36.982498    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzf98\" (UniqueName: \"kubernetes.io/projected/d0c8d507-b773-4b4e-b84c-3d387bbd2223-kube-api-access-pzf98\") pod \"alertmanager-main-0\" (UID: \"d0c8d507-b773-4b4e-b84c-3d387bbd2223\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:18:37.107335 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:37.107292    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bd78c47cf-5hjvg"]
Sep 18 19:18:37.110093 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:18:37.110067    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode87fd35d_f7a7_429f_a997_0267c01d7405.slice/crio-9daa93334f22b8e5a6c15bad9deb9aa5f2add8d11f2c6072290dde9169e5469a WatchSource:0}: Error finding container 9daa93334f22b8e5a6c15bad9deb9aa5f2add8d11f2c6072290dde9169e5469a: Status 404 returned error can't find the container with id 9daa93334f22b8e5a6c15bad9deb9aa5f2add8d11f2c6072290dde9169e5469a
Sep 18 19:18:37.117036 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:37.117011    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:18:37.418387 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:37.418083    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 19:18:37.420919 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:18:37.420895    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0c8d507_b773_4b4e_b84c_3d387bbd2223.slice/crio-0056b9c66b93a07e5ac54bbd1b9cdb05bedcc28190ecb8cb946051764f070158 WatchSource:0}: Error finding container 0056b9c66b93a07e5ac54bbd1b9cdb05bedcc28190ecb8cb946051764f070158: Status 404 returned error can't find the container with id 0056b9c66b93a07e5ac54bbd1b9cdb05bedcc28190ecb8cb946051764f070158
Sep 18 19:18:37.693300 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:37.693264    2522 generic.go:356] "Generic (PLEG): container finished" podID="d0c8d507-b773-4b4e-b84c-3d387bbd2223" containerID="2c464134fc0818d07a9d64c55c3b83edcf42378e67f069426ed6b6de22678284" exitCode=0
Sep 18 19:18:37.693480 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:37.693348    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d0c8d507-b773-4b4e-b84c-3d387bbd2223","Type":"ContainerDied","Data":"2c464134fc0818d07a9d64c55c3b83edcf42378e67f069426ed6b6de22678284"}
Sep 18 19:18:37.693480 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:37.693430    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d0c8d507-b773-4b4e-b84c-3d387bbd2223","Type":"ContainerStarted","Data":"0056b9c66b93a07e5ac54bbd1b9cdb05bedcc28190ecb8cb946051764f070158"}
Sep 18 19:18:37.695074 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:37.695054    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bd78c47cf-5hjvg" event={"ID":"e87fd35d-f7a7-429f-a997-0267c01d7405","Type":"ContainerStarted","Data":"33aeba817ba7617eb9c185fa049336e631586ee21590bf04b59b210d06306a53"}
Sep 18 19:18:37.695177 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:37.695102    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bd78c47cf-5hjvg" event={"ID":"e87fd35d-f7a7-429f-a997-0267c01d7405","Type":"ContainerStarted","Data":"9daa93334f22b8e5a6c15bad9deb9aa5f2add8d11f2c6072290dde9169e5469a"}
Sep 18 19:18:37.802106 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:37.802050    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6bd78c47cf-5hjvg" podStartSLOduration=1.8020360800000002 podStartE2EDuration="1.80203608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:18:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:18:37.801757193 +0000 UTC m=+277.685283668" watchObservedRunningTime="2026-09-18 19:18:37.80203608 +0000 UTC m=+277.685562543"
Sep 18 19:18:38.703289 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:38.703256    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d0c8d507-b773-4b4e-b84c-3d387bbd2223","Type":"ContainerStarted","Data":"3d274d5ec038e0c72558a7b6d1f33aede9a1b3e129504e6f04b0b022edf50770"}
Sep 18 19:18:38.703804 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:38.703367    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d0c8d507-b773-4b4e-b84c-3d387bbd2223","Type":"ContainerStarted","Data":"1ebdbc5f4200cf8843a86764b7ccc1a8d3cbd96408bb4daeea6006c7e09b186a"}
Sep 18 19:18:38.703804 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:38.703415    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d0c8d507-b773-4b4e-b84c-3d387bbd2223","Type":"ContainerStarted","Data":"07b9b552ce3e3100cb5a32ebeaee242bb7e099944a351461e6b621198ed64074"}
Sep 18 19:18:38.703804 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:38.703458    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d0c8d507-b773-4b4e-b84c-3d387bbd2223","Type":"ContainerStarted","Data":"87b4eed8aa03af2b3dcaedcd07820052b07080504d4f7b324cd502146e7aaa12"}
Sep 18 19:18:38.703804 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:38.703499    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d0c8d507-b773-4b4e-b84c-3d387bbd2223","Type":"ContainerStarted","Data":"4540caa7fe7850a194a24c4055419d9c510ab1276c5e5a3ea42e391f79fd0c01"}
Sep 18 19:18:38.703804 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:38.703544    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d0c8d507-b773-4b4e-b84c-3d387bbd2223","Type":"ContainerStarted","Data":"384d92fc3146481b8832dfe8cf4399f4ee9855bc3886aa387b92774190a3de15"}
Sep 18 19:18:38.721009 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:38.720980    2522 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="357238c4-1440-486a-b32e-716b8f889807" path="/var/lib/kubelet/pods/357238c4-1440-486a-b32e-716b8f889807/volumes"
Sep 18 19:18:38.751210 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:38.751159    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.735767214 podStartE2EDuration="2.751142087s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.374873ms" isStatefulPod=false podCreationTimestamp="2026-09-18 19:18:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:18:38.748264013 +0000 UTC m=+278.631790476" watchObservedRunningTime="2026-09-18 19:18:38.751142087 +0000 UTC m=+278.634668549"
Sep 18 19:18:40.155436 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:18:40.155393    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-6446c8d75d-sszkh" podUID="00d2da2b-3370-4f0a-8104-9bc429576c69"
Sep 18 19:18:40.715831 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:40.713442    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-sszkh"
Sep 18 19:18:41.159970 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:18:41.159924    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-mrd75" podUID="72d866b0-9634-442e-bd20-d90321a00452"
Sep 18 19:18:41.717017 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:41.716987    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mrd75"
Sep 18 19:18:44.032846 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:44.032800    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/00d2da2b-3370-4f0a-8104-9bc429576c69-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-sszkh\" (UID: \"00d2da2b-3370-4f0a-8104-9bc429576c69\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-sszkh"
Sep 18 19:18:44.035212 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:44.035189    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/00d2da2b-3370-4f0a-8104-9bc429576c69-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-sszkh\" (UID: \"00d2da2b-3370-4f0a-8104-9bc429576c69\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-sszkh"
Sep 18 19:18:44.134042 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:44.134007    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/72d866b0-9634-442e-bd20-d90321a00452-metrics-tls\") pod \"dns-default-mrd75\" (UID: \"72d866b0-9634-442e-bd20-d90321a00452\") " pod="openshift-dns/dns-default-mrd75"
Sep 18 19:18:44.136308 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:44.136286    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/72d866b0-9634-442e-bd20-d90321a00452-metrics-tls\") pod \"dns-default-mrd75\" (UID: \"72d866b0-9634-442e-bd20-d90321a00452\") " pod="openshift-dns/dns-default-mrd75"
Sep 18 19:18:44.318473 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:44.318435    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-z9rbg\""
Sep 18 19:18:44.325158 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:44.325127    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-sszkh"
Sep 18 19:18:44.420888 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:44.420857    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6m5tz\""
Sep 18 19:18:44.428811 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:44.428782    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mrd75"
Sep 18 19:18:44.604177 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:44.604152    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-sszkh"]
Sep 18 19:18:44.606577 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:18:44.606537    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00d2da2b_3370_4f0a_8104_9bc429576c69.slice/crio-035d21e586aaf1b0c324ce3c42c0e43c075b809c19502a49a7665a4117bdf0ce WatchSource:0}: Error finding container 035d21e586aaf1b0c324ce3c42c0e43c075b809c19502a49a7665a4117bdf0ce: Status 404 returned error can't find the container with id 035d21e586aaf1b0c324ce3c42c0e43c075b809c19502a49a7665a4117bdf0ce
Sep 18 19:18:44.704399 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:44.704268    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mrd75"]
Sep 18 19:18:44.706818 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:18:44.706791    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72d866b0_9634_442e_bd20_d90321a00452.slice/crio-d07de6de859546f49b893d64b104725211a500020d5ac416278b680a85848396 WatchSource:0}: Error finding container d07de6de859546f49b893d64b104725211a500020d5ac416278b680a85848396: Status 404 returned error can't find the container with id d07de6de859546f49b893d64b104725211a500020d5ac416278b680a85848396
Sep 18 19:18:44.728944 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:44.728912    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mrd75" event={"ID":"72d866b0-9634-442e-bd20-d90321a00452","Type":"ContainerStarted","Data":"d07de6de859546f49b893d64b104725211a500020d5ac416278b680a85848396"}
Sep 18 19:18:44.730002 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:44.729974    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-sszkh" event={"ID":"00d2da2b-3370-4f0a-8104-9bc429576c69","Type":"ContainerStarted","Data":"035d21e586aaf1b0c324ce3c42c0e43c075b809c19502a49a7665a4117bdf0ce"}
Sep 18 19:18:46.740539 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:46.740447    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mrd75" event={"ID":"72d866b0-9634-442e-bd20-d90321a00452","Type":"ContainerStarted","Data":"2a8d355c08df381aaf0f68e8c2af62b6397173c114e371ae55ea7a20fde7dd38"}
Sep 18 19:18:46.740539 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:46.740499    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mrd75" event={"ID":"72d866b0-9634-442e-bd20-d90321a00452","Type":"ContainerStarted","Data":"6b5b7d7dcdc452f0c31e4bb74658fdac72ec6ca9513781d9e6d2397f44a74e39"}
Sep 18 19:18:46.741054 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:46.740595    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-mrd75"
Sep 18 19:18:46.741976 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:46.741952    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-sszkh" event={"ID":"00d2da2b-3370-4f0a-8104-9bc429576c69","Type":"ContainerStarted","Data":"dbed3435b930eeaac7e0d854e72ff5af424fdcfe9acd2a2f65944d7514b0cf15"}
Sep 18 19:18:46.771024 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:46.770980    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-mrd75" podStartSLOduration=251.032138352 podStartE2EDuration="4m12.770963934s" totalImagesPullingTime="1.738825582s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:14:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:18:46.769699926 +0000 UTC m=+286.653226389" watchObservedRunningTime="2026-09-18 19:18:46.770963934 +0000 UTC m=+286.654490395"
Sep 18 19:18:46.812221 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:46.812174    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6446c8d75d-sszkh" podStartSLOduration=279.694366013 podStartE2EDuration="4m40.812156338s" totalImagesPullingTime="1.117790325s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:14:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:18:46.811766103 +0000 UTC m=+286.695292568" watchObservedRunningTime="2026-09-18 19:18:46.812156338 +0000 UTC m=+286.695682800"
Sep 18 19:18:46.815862 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:46.815831    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6bd78c47cf-5hjvg"
Sep 18 19:18:46.815968 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:46.815881    2522 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6bd78c47cf-5hjvg"
Sep 18 19:18:46.820229 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:46.820209    2522 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6bd78c47cf-5hjvg"
Sep 18 19:18:47.750078 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:47.750049    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6bd78c47cf-5hjvg"
Sep 18 19:18:47.840691 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:47.840658    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d67bc5c57-fgwxg"]
Sep 18 19:18:56.749510 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:18:56.749481    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mrd75"
Sep 18 19:19:00.527994 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:19:00.527970    2522 kubelet.go:1648] "Image garbage collection succeeded"
Sep 18 19:19:12.859142 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:19:12.859084    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-d67bc5c57-fgwxg" podUID="cb74c5b7-6640-4747-92f0-03bdde39c5a9" containerName="console" containerID="cri-o://24d6220e1ea8e7487df5df2145d2370d59aa6e25d96f9af49d031eda2336ff28" gracePeriod=15
Sep 18 19:19:13.108583 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:19:13.108539    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d67bc5c57-fgwxg"
Sep 18 19:19:13.171345 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:19:13.171280    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb74c5b7-6640-4747-92f0-03bdde39c5a9-trusted-ca-bundle\") pod \"cb74c5b7-6640-4747-92f0-03bdde39c5a9\" (UID: \"cb74c5b7-6640-4747-92f0-03bdde39c5a9\") "
Sep 18 19:19:13.171345 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:19:13.171327    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jmsv9\" (UniqueName: \"kubernetes.io/projected/cb74c5b7-6640-4747-92f0-03bdde39c5a9-kube-api-access-jmsv9\") pod \"cb74c5b7-6640-4747-92f0-03bdde39c5a9\" (UID: \"cb74c5b7-6640-4747-92f0-03bdde39c5a9\") "
Sep 18 19:19:13.171506 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:19:13.171359    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cb74c5b7-6640-4747-92f0-03bdde39c5a9-console-oauth-config\") pod \"cb74c5b7-6640-4747-92f0-03bdde39c5a9\" (UID: \"cb74c5b7-6640-4747-92f0-03bdde39c5a9\") "
Sep 18 19:19:13.171506 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:19:13.171458    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cb74c5b7-6640-4747-92f0-03bdde39c5a9-service-ca\") pod \"cb74c5b7-6640-4747-92f0-03bdde39c5a9\" (UID: \"cb74c5b7-6640-4747-92f0-03bdde39c5a9\") "
Sep 18 19:19:13.171636 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:19:13.171513    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cb74c5b7-6640-4747-92f0-03bdde39c5a9-console-serving-cert\") pod \"cb74c5b7-6640-4747-92f0-03bdde39c5a9\" (UID: \"cb74c5b7-6640-4747-92f0-03bdde39c5a9\") "
Sep 18 19:19:13.171636 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:19:13.171539    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cb74c5b7-6640-4747-92f0-03bdde39c5a9-console-config\") pod \"cb74c5b7-6640-4747-92f0-03bdde39c5a9\" (UID: \"cb74c5b7-6640-4747-92f0-03bdde39c5a9\") "
Sep 18 19:19:13.171636 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:19:13.171626    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cb74c5b7-6640-4747-92f0-03bdde39c5a9-oauth-serving-cert\") pod \"cb74c5b7-6640-4747-92f0-03bdde39c5a9\" (UID: \"cb74c5b7-6640-4747-92f0-03bdde39c5a9\") "
Sep 18 19:19:13.171880 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:19:13.171840    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb74c5b7-6640-4747-92f0-03bdde39c5a9-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "cb74c5b7-6640-4747-92f0-03bdde39c5a9" (UID: "cb74c5b7-6640-4747-92f0-03bdde39c5a9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:19:13.171998 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:19:13.171959    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb74c5b7-6640-4747-92f0-03bdde39c5a9-service-ca" (OuterVolumeSpecName: "service-ca") pod "cb74c5b7-6640-4747-92f0-03bdde39c5a9" (UID: "cb74c5b7-6640-4747-92f0-03bdde39c5a9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:19:13.171998 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:19:13.171968    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb74c5b7-6640-4747-92f0-03bdde39c5a9-console-config" (OuterVolumeSpecName: "console-config") pod "cb74c5b7-6640-4747-92f0-03bdde39c5a9" (UID: "cb74c5b7-6640-4747-92f0-03bdde39c5a9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:19:13.172193 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:19:13.172171    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb74c5b7-6640-4747-92f0-03bdde39c5a9-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "cb74c5b7-6640-4747-92f0-03bdde39c5a9" (UID: "cb74c5b7-6640-4747-92f0-03bdde39c5a9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:19:13.173599 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:19:13.173557    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb74c5b7-6640-4747-92f0-03bdde39c5a9-kube-api-access-jmsv9" (OuterVolumeSpecName: "kube-api-access-jmsv9") pod "cb74c5b7-6640-4747-92f0-03bdde39c5a9" (UID: "cb74c5b7-6640-4747-92f0-03bdde39c5a9"). InnerVolumeSpecName "kube-api-access-jmsv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:19:13.173599 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:19:13.173590    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb74c5b7-6640-4747-92f0-03bdde39c5a9-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "cb74c5b7-6640-4747-92f0-03bdde39c5a9" (UID: "cb74c5b7-6640-4747-92f0-03bdde39c5a9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:19:13.173764 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:19:13.173700    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb74c5b7-6640-4747-92f0-03bdde39c5a9-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "cb74c5b7-6640-4747-92f0-03bdde39c5a9" (UID: "cb74c5b7-6640-4747-92f0-03bdde39c5a9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:19:13.272243 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:19:13.272198    2522 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cb74c5b7-6640-4747-92f0-03bdde39c5a9-oauth-serving-cert\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:19:13.272243 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:19:13.272237    2522 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb74c5b7-6640-4747-92f0-03bdde39c5a9-trusted-ca-bundle\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:19:13.272243 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:19:13.272250    2522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmsv9\" (UniqueName: \"kubernetes.io/projected/cb74c5b7-6640-4747-92f0-03bdde39c5a9-kube-api-access-jmsv9\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:19:13.272452 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:19:13.272260    2522 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cb74c5b7-6640-4747-92f0-03bdde39c5a9-console-oauth-config\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:19:13.272452 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:19:13.272269    2522 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cb74c5b7-6640-4747-92f0-03bdde39c5a9-service-ca\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:19:13.272452 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:19:13.272277    2522 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cb74c5b7-6640-4747-92f0-03bdde39c5a9-console-serving-cert\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:19:13.272452 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:19:13.272285    2522 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cb74c5b7-6640-4747-92f0-03bdde39c5a9-console-config\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:19:13.853577 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:19:13.853542    2522 generic.go:356] "Generic (PLEG): container finished" podID="cb74c5b7-6640-4747-92f0-03bdde39c5a9" containerID="24d6220e1ea8e7487df5df2145d2370d59aa6e25d96f9af49d031eda2336ff28" exitCode=0
Sep 18 19:19:13.853745 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:19:13.853605    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d67bc5c57-fgwxg"
Sep 18 19:19:13.853745 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:19:13.853628    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d67bc5c57-fgwxg" event={"ID":"cb74c5b7-6640-4747-92f0-03bdde39c5a9","Type":"ContainerDied","Data":"24d6220e1ea8e7487df5df2145d2370d59aa6e25d96f9af49d031eda2336ff28"}
Sep 18 19:19:13.853745 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:19:13.853680    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d67bc5c57-fgwxg" event={"ID":"cb74c5b7-6640-4747-92f0-03bdde39c5a9","Type":"ContainerDied","Data":"5a6ea2c1bad2afbe8e93fd5159264c7c9c6524fd72114f8209921ddbf728cb29"}
Sep 18 19:19:13.853745 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:19:13.853699    2522 scope.go:117] "RemoveContainer" containerID="24d6220e1ea8e7487df5df2145d2370d59aa6e25d96f9af49d031eda2336ff28"
Sep 18 19:19:13.862140 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:19:13.862122    2522 scope.go:117] "RemoveContainer" containerID="24d6220e1ea8e7487df5df2145d2370d59aa6e25d96f9af49d031eda2336ff28"
Sep 18 19:19:13.862404 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:19:13.862379    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24d6220e1ea8e7487df5df2145d2370d59aa6e25d96f9af49d031eda2336ff28\": container with ID starting with 24d6220e1ea8e7487df5df2145d2370d59aa6e25d96f9af49d031eda2336ff28 not found: ID does not exist" containerID="24d6220e1ea8e7487df5df2145d2370d59aa6e25d96f9af49d031eda2336ff28"
Sep 18 19:19:13.862445 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:19:13.862404    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24d6220e1ea8e7487df5df2145d2370d59aa6e25d96f9af49d031eda2336ff28"} err="failed to get container status \"24d6220e1ea8e7487df5df2145d2370d59aa6e25d96f9af49d031eda2336ff28\": rpc error: code = NotFound desc = could not find container \"24d6220e1ea8e7487df5df2145d2370d59aa6e25d96f9af49d031eda2336ff28\": container with ID starting with 24d6220e1ea8e7487df5df2145d2370d59aa6e25d96f9af49d031eda2336ff28 not found: ID does not exist"
Sep 18 19:19:13.876249 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:19:13.876226    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d67bc5c57-fgwxg"]
Sep 18 19:19:13.881759 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:19:13.881738    2522 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-d67bc5c57-fgwxg"]
Sep 18 19:19:14.721406 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:19:14.721378    2522 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb74c5b7-6640-4747-92f0-03bdde39c5a9" path="/var/lib/kubelet/pods/cb74c5b7-6640-4747-92f0-03bdde39c5a9/volumes"
Sep 18 19:19:47.078276 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:19:47.078244    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ab4c7n"]
Sep 18 19:19:47.078795 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:19:47.078612    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb74c5b7-6640-4747-92f0-03bdde39c5a9" containerName="console"
Sep 18 19:19:47.078795 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:19:47.078627    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb74c5b7-6640-4747-92f0-03bdde39c5a9" containerName="console"
Sep 18 19:19:47.078795 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:19:47.078686    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb74c5b7-6640-4747-92f0-03bdde39c5a9" containerName="console"
Sep 18 19:19:47.082092 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:19:47.082076    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ab4c7n"
Sep 18 19:19:47.085635 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:19:47.085613    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 18 19:19:47.085773 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:19:47.085705    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-2fkjt\""
Sep 18 19:19:47.086741 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:19:47.086723    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 18 19:19:47.100512 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:19:47.100490    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ab4c7n"]
Sep 18 19:19:47.241950 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:19:47.241896    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhzx8\" (UniqueName: \"kubernetes.io/projected/360c9c37-3a55-4736-a6bf-f8af19bc5879-kube-api-access-xhzx8\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ab4c7n\" (UID: \"360c9c37-3a55-4736-a6bf-f8af19bc5879\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ab4c7n"
Sep 18 19:19:47.241950 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:19:47.241957    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/360c9c37-3a55-4736-a6bf-f8af19bc5879-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ab4c7n\" (UID: \"360c9c37-3a55-4736-a6bf-f8af19bc5879\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ab4c7n"
Sep 18 19:19:47.242171 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:19:47.241981    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/360c9c37-3a55-4736-a6bf-f8af19bc5879-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ab4c7n\" (UID: \"360c9c37-3a55-4736-a6bf-f8af19bc5879\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ab4c7n"
Sep 18 19:19:47.342901 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:19:47.342831    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhzx8\" (UniqueName: \"kubernetes.io/projected/360c9c37-3a55-4736-a6bf-f8af19bc5879-kube-api-access-xhzx8\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ab4c7n\" (UID: \"360c9c37-3a55-4736-a6bf-f8af19bc5879\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ab4c7n"
Sep 18 19:19:47.342901 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:19:47.342869    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/360c9c37-3a55-4736-a6bf-f8af19bc5879-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ab4c7n\" (UID: \"360c9c37-3a55-4736-a6bf-f8af19bc5879\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ab4c7n"
Sep 18 19:19:47.342901 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:19:47.342891    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/360c9c37-3a55-4736-a6bf-f8af19bc5879-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ab4c7n\" (UID: \"360c9c37-3a55-4736-a6bf-f8af19bc5879\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ab4c7n"
Sep 18 19:19:47.343216 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:19:47.343201    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/360c9c37-3a55-4736-a6bf-f8af19bc5879-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ab4c7n\" (UID: \"360c9c37-3a55-4736-a6bf-f8af19bc5879\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ab4c7n"
Sep 18 19:19:47.343253 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:19:47.343235    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/360c9c37-3a55-4736-a6bf-f8af19bc5879-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ab4c7n\" (UID: \"360c9c37-3a55-4736-a6bf-f8af19bc5879\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ab4c7n"
Sep 18 19:19:47.354130 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:19:47.354102    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhzx8\" (UniqueName: \"kubernetes.io/projected/360c9c37-3a55-4736-a6bf-f8af19bc5879-kube-api-access-xhzx8\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ab4c7n\" (UID: \"360c9c37-3a55-4736-a6bf-f8af19bc5879\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ab4c7n"
Sep 18 19:19:47.391370 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:19:47.391343    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ab4c7n"
Sep 18 19:19:47.681386 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:19:47.681358    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ab4c7n"]
Sep 18 19:19:47.684211 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:19:47.684185    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod360c9c37_3a55_4736_a6bf_f8af19bc5879.slice/crio-13406a359c22eaa5433e4578e5a1da6b115cf4f1995a0ab818f9ac51bc164561 WatchSource:0}: Error finding container 13406a359c22eaa5433e4578e5a1da6b115cf4f1995a0ab818f9ac51bc164561: Status 404 returned error can't find the container with id 13406a359c22eaa5433e4578e5a1da6b115cf4f1995a0ab818f9ac51bc164561
Sep 18 19:19:47.686199 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:19:47.686175    2522 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:19:47.995972 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:19:47.995893    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ab4c7n" event={"ID":"360c9c37-3a55-4736-a6bf-f8af19bc5879","Type":"ContainerStarted","Data":"13406a359c22eaa5433e4578e5a1da6b115cf4f1995a0ab818f9ac51bc164561"}
Sep 18 19:19:54.024052 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:19:54.024020    2522 generic.go:356] "Generic (PLEG): container finished" podID="360c9c37-3a55-4736-a6bf-f8af19bc5879" containerID="b9baaaa0e4f2d5aacaf025ff7c56f7dec676e434336f6b2a92c95be6477a298b" exitCode=0
Sep 18 19:19:54.024448 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:19:54.024084    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ab4c7n" event={"ID":"360c9c37-3a55-4736-a6bf-f8af19bc5879","Type":"ContainerDied","Data":"b9baaaa0e4f2d5aacaf025ff7c56f7dec676e434336f6b2a92c95be6477a298b"}
Sep 18 19:20:00.050757 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:00.050718    2522 generic.go:356] "Generic (PLEG): container finished" podID="360c9c37-3a55-4736-a6bf-f8af19bc5879" containerID="28f950baed1f3f30c3e132c118e71138112fa067141909850a823daab07a191f" exitCode=0
Sep 18 19:20:00.051181 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:00.050802    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ab4c7n" event={"ID":"360c9c37-3a55-4736-a6bf-f8af19bc5879","Type":"ContainerDied","Data":"28f950baed1f3f30c3e132c118e71138112fa067141909850a823daab07a191f"}
Sep 18 19:20:08.085436 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:08.085399    2522 generic.go:356] "Generic (PLEG): container finished" podID="360c9c37-3a55-4736-a6bf-f8af19bc5879" containerID="64bf9edf7332fbc6b7f91d50ac3f72c38173260679852cde4806739819bd6e18" exitCode=0
Sep 18 19:20:08.085436 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:08.085440    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ab4c7n" event={"ID":"360c9c37-3a55-4736-a6bf-f8af19bc5879","Type":"ContainerDied","Data":"64bf9edf7332fbc6b7f91d50ac3f72c38173260679852cde4806739819bd6e18"}
Sep 18 19:20:09.208802 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:09.208776    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ab4c7n"
Sep 18 19:20:09.333437 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:09.333407    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/360c9c37-3a55-4736-a6bf-f8af19bc5879-util\") pod \"360c9c37-3a55-4736-a6bf-f8af19bc5879\" (UID: \"360c9c37-3a55-4736-a6bf-f8af19bc5879\") "
Sep 18 19:20:09.333592 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:09.333469    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xhzx8\" (UniqueName: \"kubernetes.io/projected/360c9c37-3a55-4736-a6bf-f8af19bc5879-kube-api-access-xhzx8\") pod \"360c9c37-3a55-4736-a6bf-f8af19bc5879\" (UID: \"360c9c37-3a55-4736-a6bf-f8af19bc5879\") "
Sep 18 19:20:09.333592 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:09.333533    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/360c9c37-3a55-4736-a6bf-f8af19bc5879-bundle\") pod \"360c9c37-3a55-4736-a6bf-f8af19bc5879\" (UID: \"360c9c37-3a55-4736-a6bf-f8af19bc5879\") "
Sep 18 19:20:09.334172 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:09.334145    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/360c9c37-3a55-4736-a6bf-f8af19bc5879-bundle" (OuterVolumeSpecName: "bundle") pod "360c9c37-3a55-4736-a6bf-f8af19bc5879" (UID: "360c9c37-3a55-4736-a6bf-f8af19bc5879"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:20:09.335709 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:09.335687    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/360c9c37-3a55-4736-a6bf-f8af19bc5879-kube-api-access-xhzx8" (OuterVolumeSpecName: "kube-api-access-xhzx8") pod "360c9c37-3a55-4736-a6bf-f8af19bc5879" (UID: "360c9c37-3a55-4736-a6bf-f8af19bc5879"). InnerVolumeSpecName "kube-api-access-xhzx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:20:09.337506 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:09.337486    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/360c9c37-3a55-4736-a6bf-f8af19bc5879-util" (OuterVolumeSpecName: "util") pod "360c9c37-3a55-4736-a6bf-f8af19bc5879" (UID: "360c9c37-3a55-4736-a6bf-f8af19bc5879"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:20:09.434487 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:09.434438    2522 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/360c9c37-3a55-4736-a6bf-f8af19bc5879-bundle\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:20:09.434487 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:09.434483    2522 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/360c9c37-3a55-4736-a6bf-f8af19bc5879-util\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:20:09.434487 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:09.434494    2522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhzx8\" (UniqueName: \"kubernetes.io/projected/360c9c37-3a55-4736-a6bf-f8af19bc5879-kube-api-access-xhzx8\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:20:10.095050 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:10.095024    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ab4c7n"
Sep 18 19:20:10.095236 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:10.095020    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ab4c7n" event={"ID":"360c9c37-3a55-4736-a6bf-f8af19bc5879","Type":"ContainerDied","Data":"13406a359c22eaa5433e4578e5a1da6b115cf4f1995a0ab818f9ac51bc164561"}
Sep 18 19:20:10.095236 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:10.095141    2522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13406a359c22eaa5433e4578e5a1da6b115cf4f1995a0ab818f9ac51bc164561"
Sep 18 19:20:18.705682 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:18.705648    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-drcfl"]
Sep 18 19:20:18.706185 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:18.706110    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="360c9c37-3a55-4736-a6bf-f8af19bc5879" containerName="pull"
Sep 18 19:20:18.706185 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:18.706128    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="360c9c37-3a55-4736-a6bf-f8af19bc5879" containerName="pull"
Sep 18 19:20:18.706185 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:18.706141    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="360c9c37-3a55-4736-a6bf-f8af19bc5879" containerName="util"
Sep 18 19:20:18.706185 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:18.706149    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="360c9c37-3a55-4736-a6bf-f8af19bc5879" containerName="util"
Sep 18 19:20:18.706185 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:18.706158    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="360c9c37-3a55-4736-a6bf-f8af19bc5879" containerName="extract"
Sep 18 19:20:18.706185 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:18.706166    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="360c9c37-3a55-4736-a6bf-f8af19bc5879" containerName="extract"
Sep 18 19:20:18.706492 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:18.706242    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="360c9c37-3a55-4736-a6bf-f8af19bc5879" containerName="extract"
Sep 18 19:20:18.709480 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:18.709460    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-drcfl"
Sep 18 19:20:18.727372 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:18.727355    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 18 19:20:18.727489 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:18.727380    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 18 19:20:18.728935 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:18.728920    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 18 19:20:18.733634 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:18.733618    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-k94c5\""
Sep 18 19:20:18.776448 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:18.776423    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-drcfl"]
Sep 18 19:20:18.807025 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:18.806984    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mc6f\" (UniqueName: \"kubernetes.io/projected/029a5bce-3ec1-40db-8202-ee23f290d4a6-kube-api-access-5mc6f\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-drcfl\" (UID: \"029a5bce-3ec1-40db-8202-ee23f290d4a6\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-drcfl"
Sep 18 19:20:18.807136 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:18.807050    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/029a5bce-3ec1-40db-8202-ee23f290d4a6-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-drcfl\" (UID: \"029a5bce-3ec1-40db-8202-ee23f290d4a6\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-drcfl"
Sep 18 19:20:18.908240 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:18.908203    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mc6f\" (UniqueName: \"kubernetes.io/projected/029a5bce-3ec1-40db-8202-ee23f290d4a6-kube-api-access-5mc6f\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-drcfl\" (UID: \"029a5bce-3ec1-40db-8202-ee23f290d4a6\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-drcfl"
Sep 18 19:20:18.908350 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:18.908256    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/029a5bce-3ec1-40db-8202-ee23f290d4a6-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-drcfl\" (UID: \"029a5bce-3ec1-40db-8202-ee23f290d4a6\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-drcfl"
Sep 18 19:20:18.910355 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:18.910331    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/029a5bce-3ec1-40db-8202-ee23f290d4a6-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-drcfl\" (UID: \"029a5bce-3ec1-40db-8202-ee23f290d4a6\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-drcfl"
Sep 18 19:20:18.922224 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:18.922198    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mc6f\" (UniqueName: \"kubernetes.io/projected/029a5bce-3ec1-40db-8202-ee23f290d4a6-kube-api-access-5mc6f\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-drcfl\" (UID: \"029a5bce-3ec1-40db-8202-ee23f290d4a6\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-drcfl"
Sep 18 19:20:19.019621 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:19.019529    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-drcfl"
Sep 18 19:20:19.345070 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:19.345040    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-drcfl"]
Sep 18 19:20:19.374543 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:20:19.374510    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod029a5bce_3ec1_40db_8202_ee23f290d4a6.slice/crio-7dc0b8e7eb057f50af27b216ed389cf94d5b28a0e78330855cef7b02b1fb10cd WatchSource:0}: Error finding container 7dc0b8e7eb057f50af27b216ed389cf94d5b28a0e78330855cef7b02b1fb10cd: Status 404 returned error can't find the container with id 7dc0b8e7eb057f50af27b216ed389cf94d5b28a0e78330855cef7b02b1fb10cd
Sep 18 19:20:20.137915 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:20.137877    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-drcfl" event={"ID":"029a5bce-3ec1-40db-8202-ee23f290d4a6","Type":"ContainerStarted","Data":"7dc0b8e7eb057f50af27b216ed389cf94d5b28a0e78330855cef7b02b1fb10cd"}
Sep 18 19:20:23.155876 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:23.155835    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-drcfl" event={"ID":"029a5bce-3ec1-40db-8202-ee23f290d4a6","Type":"ContainerStarted","Data":"388acb6b4139f3eabee571a5d41746c7a29622fa7742bccbfc18276829f1d4f9"}
Sep 18 19:20:23.156259 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:23.155963    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-drcfl"
Sep 18 19:20:23.198308 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:23.198263    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-drcfl" podStartSLOduration=2.044085125 podStartE2EDuration="5.198248425s" totalImagesPullingTime="3.1541633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:20:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:20:23.196018628 +0000 UTC m=+383.079545090" watchObservedRunningTime="2026-09-18 19:20:23.198248425 +0000 UTC m=+383.081774889"
Sep 18 19:20:23.542829 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:23.542757    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-7qwnj"]
Sep 18 19:20:23.546047 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:23.546029    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-7qwnj"
Sep 18 19:20:23.556327 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:23.556309    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 18 19:20:23.556660 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:23.556644    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 18 19:20:23.556754 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:23.556665    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-mstl6\""
Sep 18 19:20:23.571032 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:23.571009    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-7qwnj"]
Sep 18 19:20:23.647484 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:23.647459    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/97cbc5d7-ec00-42d6-a924-6f58b6c1b7ca-certificates\") pod \"keda-operator-579c8958f7-7qwnj\" (UID: \"97cbc5d7-ec00-42d6-a924-6f58b6c1b7ca\") " pod="openshift-keda/keda-operator-579c8958f7-7qwnj"
Sep 18 19:20:23.647666 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:23.647497    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn6b4\" (UniqueName: \"kubernetes.io/projected/97cbc5d7-ec00-42d6-a924-6f58b6c1b7ca-kube-api-access-wn6b4\") pod \"keda-operator-579c8958f7-7qwnj\" (UID: \"97cbc5d7-ec00-42d6-a924-6f58b6c1b7ca\") " pod="openshift-keda/keda-operator-579c8958f7-7qwnj"
Sep 18 19:20:23.647666 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:23.647540    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/97cbc5d7-ec00-42d6-a924-6f58b6c1b7ca-cabundle0\") pod \"keda-operator-579c8958f7-7qwnj\" (UID: \"97cbc5d7-ec00-42d6-a924-6f58b6c1b7ca\") " pod="openshift-keda/keda-operator-579c8958f7-7qwnj"
Sep 18 19:20:23.748033 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:23.748003    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn6b4\" (UniqueName: \"kubernetes.io/projected/97cbc5d7-ec00-42d6-a924-6f58b6c1b7ca-kube-api-access-wn6b4\") pod \"keda-operator-579c8958f7-7qwnj\" (UID: \"97cbc5d7-ec00-42d6-a924-6f58b6c1b7ca\") " pod="openshift-keda/keda-operator-579c8958f7-7qwnj"
Sep 18 19:20:23.748187 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:23.748053    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/97cbc5d7-ec00-42d6-a924-6f58b6c1b7ca-cabundle0\") pod \"keda-operator-579c8958f7-7qwnj\" (UID: \"97cbc5d7-ec00-42d6-a924-6f58b6c1b7ca\") " pod="openshift-keda/keda-operator-579c8958f7-7qwnj"
Sep 18 19:20:23.748187 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:23.748122    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/97cbc5d7-ec00-42d6-a924-6f58b6c1b7ca-certificates\") pod \"keda-operator-579c8958f7-7qwnj\" (UID: \"97cbc5d7-ec00-42d6-a924-6f58b6c1b7ca\") " pod="openshift-keda/keda-operator-579c8958f7-7qwnj"
Sep 18 19:20:23.748299 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:20:23.748209    2522 secret.go:281] references non-existent secret key: ca.crt
Sep 18 19:20:23.748299 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:20:23.748224    2522 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 18 19:20:23.748299 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:20:23.748235    2522 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-7qwnj: references non-existent secret key: ca.crt
Sep 18 19:20:23.748299 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:20:23.748289    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/97cbc5d7-ec00-42d6-a924-6f58b6c1b7ca-certificates podName:97cbc5d7-ec00-42d6-a924-6f58b6c1b7ca nodeName:}" failed. No retries permitted until 2026-09-18 19:20:24.248272393 +0000 UTC m=+384.131798833 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/97cbc5d7-ec00-42d6-a924-6f58b6c1b7ca-certificates") pod "keda-operator-579c8958f7-7qwnj" (UID: "97cbc5d7-ec00-42d6-a924-6f58b6c1b7ca") : references non-existent secret key: ca.crt
Sep 18 19:20:23.748661 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:23.748642    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/97cbc5d7-ec00-42d6-a924-6f58b6c1b7ca-cabundle0\") pod \"keda-operator-579c8958f7-7qwnj\" (UID: \"97cbc5d7-ec00-42d6-a924-6f58b6c1b7ca\") " pod="openshift-keda/keda-operator-579c8958f7-7qwnj"
Sep 18 19:20:23.765796 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:23.765768    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn6b4\" (UniqueName: \"kubernetes.io/projected/97cbc5d7-ec00-42d6-a924-6f58b6c1b7ca-kube-api-access-wn6b4\") pod \"keda-operator-579c8958f7-7qwnj\" (UID: \"97cbc5d7-ec00-42d6-a924-6f58b6c1b7ca\") " pod="openshift-keda/keda-operator-579c8958f7-7qwnj"
Sep 18 19:20:24.171235 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:24.171200    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-mxr6m"]
Sep 18 19:20:24.175861 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:24.175828    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-mxr6m"
Sep 18 19:20:24.179809 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:24.179786    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 18 19:20:24.203715 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:24.203682    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-mxr6m"]
Sep 18 19:20:24.251735 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:24.251710    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e33c9bc2-b125-4950-8b54-d74e05360575-certificates\") pod \"keda-metrics-apiserver-77857b7b98-mxr6m\" (UID: \"e33c9bc2-b125-4950-8b54-d74e05360575\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-mxr6m"
Sep 18 19:20:24.251853 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:24.251794    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/97cbc5d7-ec00-42d6-a924-6f58b6c1b7ca-certificates\") pod \"keda-operator-579c8958f7-7qwnj\" (UID: \"97cbc5d7-ec00-42d6-a924-6f58b6c1b7ca\") " pod="openshift-keda/keda-operator-579c8958f7-7qwnj"
Sep 18 19:20:24.251853 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:24.251843    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhsts\" (UniqueName: \"kubernetes.io/projected/e33c9bc2-b125-4950-8b54-d74e05360575-kube-api-access-bhsts\") pod \"keda-metrics-apiserver-77857b7b98-mxr6m\" (UID: \"e33c9bc2-b125-4950-8b54-d74e05360575\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-mxr6m"
Sep 18 19:20:24.251969 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:24.251877    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/e33c9bc2-b125-4950-8b54-d74e05360575-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-mxr6m\" (UID: \"e33c9bc2-b125-4950-8b54-d74e05360575\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-mxr6m"
Sep 18 19:20:24.254116 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:24.254098    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/97cbc5d7-ec00-42d6-a924-6f58b6c1b7ca-certificates\") pod \"keda-operator-579c8958f7-7qwnj\" (UID: \"97cbc5d7-ec00-42d6-a924-6f58b6c1b7ca\") " pod="openshift-keda/keda-operator-579c8958f7-7qwnj"
Sep 18 19:20:24.352714 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:24.352684    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhsts\" (UniqueName: \"kubernetes.io/projected/e33c9bc2-b125-4950-8b54-d74e05360575-kube-api-access-bhsts\") pod \"keda-metrics-apiserver-77857b7b98-mxr6m\" (UID: \"e33c9bc2-b125-4950-8b54-d74e05360575\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-mxr6m"
Sep 18 19:20:24.352828 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:24.352719    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/e33c9bc2-b125-4950-8b54-d74e05360575-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-mxr6m\" (UID: \"e33c9bc2-b125-4950-8b54-d74e05360575\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-mxr6m"
Sep 18 19:20:24.352828 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:24.352762    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e33c9bc2-b125-4950-8b54-d74e05360575-certificates\") pod \"keda-metrics-apiserver-77857b7b98-mxr6m\" (UID: \"e33c9bc2-b125-4950-8b54-d74e05360575\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-mxr6m"
Sep 18 19:20:24.352938 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:20:24.352853    2522 projected.go:266] Couldn't get secret openshift-keda/keda-metrics-apiserver-certs: secret "keda-metrics-apiserver-certs" not found
Sep 18 19:20:24.352938 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:20:24.352867    2522 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-77857b7b98-mxr6m: secret "keda-metrics-apiserver-certs" not found
Sep 18 19:20:24.352938 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:20:24.352927    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e33c9bc2-b125-4950-8b54-d74e05360575-certificates podName:e33c9bc2-b125-4950-8b54-d74e05360575 nodeName:}" failed. No retries permitted until 2026-09-18 19:20:24.85291134 +0000 UTC m=+384.736437781 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/e33c9bc2-b125-4950-8b54-d74e05360575-certificates") pod "keda-metrics-apiserver-77857b7b98-mxr6m" (UID: "e33c9bc2-b125-4950-8b54-d74e05360575") : secret "keda-metrics-apiserver-certs" not found
Sep 18 19:20:24.353218 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:24.353195    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/e33c9bc2-b125-4950-8b54-d74e05360575-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-mxr6m\" (UID: \"e33c9bc2-b125-4950-8b54-d74e05360575\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-mxr6m"
Sep 18 19:20:24.365455 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:24.365434    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhsts\" (UniqueName: \"kubernetes.io/projected/e33c9bc2-b125-4950-8b54-d74e05360575-kube-api-access-bhsts\") pod \"keda-metrics-apiserver-77857b7b98-mxr6m\" (UID: \"e33c9bc2-b125-4950-8b54-d74e05360575\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-mxr6m"
Sep 18 19:20:24.439465 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:24.439404    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-zxpfj"]
Sep 18 19:20:24.443091 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:24.443068    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-zxpfj"
Sep 18 19:20:24.448445 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:24.448422    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 18 19:20:24.456056 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:24.456039    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-7qwnj"
Sep 18 19:20:24.464143 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:24.464123    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-zxpfj"]
Sep 18 19:20:24.553969 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:24.553933    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5a61b1c0-df1b-4990-bc0d-f6a34d73055f-certificates\") pod \"keda-admission-6c4d864b86-zxpfj\" (UID: \"5a61b1c0-df1b-4990-bc0d-f6a34d73055f\") " pod="openshift-keda/keda-admission-6c4d864b86-zxpfj"
Sep 18 19:20:24.554082 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:24.553973    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68jq9\" (UniqueName: \"kubernetes.io/projected/5a61b1c0-df1b-4990-bc0d-f6a34d73055f-kube-api-access-68jq9\") pod \"keda-admission-6c4d864b86-zxpfj\" (UID: \"5a61b1c0-df1b-4990-bc0d-f6a34d73055f\") " pod="openshift-keda/keda-admission-6c4d864b86-zxpfj"
Sep 18 19:20:24.654953 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:24.654924    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5a61b1c0-df1b-4990-bc0d-f6a34d73055f-certificates\") pod \"keda-admission-6c4d864b86-zxpfj\" (UID: \"5a61b1c0-df1b-4990-bc0d-f6a34d73055f\") " pod="openshift-keda/keda-admission-6c4d864b86-zxpfj"
Sep 18 19:20:24.654953 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:24.654953    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68jq9\" (UniqueName: \"kubernetes.io/projected/5a61b1c0-df1b-4990-bc0d-f6a34d73055f-kube-api-access-68jq9\") pod \"keda-admission-6c4d864b86-zxpfj\" (UID: \"5a61b1c0-df1b-4990-bc0d-f6a34d73055f\") " pod="openshift-keda/keda-admission-6c4d864b86-zxpfj"
Sep 18 19:20:24.657215 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:24.657196    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5a61b1c0-df1b-4990-bc0d-f6a34d73055f-certificates\") pod \"keda-admission-6c4d864b86-zxpfj\" (UID: \"5a61b1c0-df1b-4990-bc0d-f6a34d73055f\") " pod="openshift-keda/keda-admission-6c4d864b86-zxpfj"
Sep 18 19:20:24.667079 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:24.667056    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68jq9\" (UniqueName: \"kubernetes.io/projected/5a61b1c0-df1b-4990-bc0d-f6a34d73055f-kube-api-access-68jq9\") pod \"keda-admission-6c4d864b86-zxpfj\" (UID: \"5a61b1c0-df1b-4990-bc0d-f6a34d73055f\") " pod="openshift-keda/keda-admission-6c4d864b86-zxpfj"
Sep 18 19:20:24.753591 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:24.753499    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-zxpfj"
Sep 18 19:20:24.758763 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:24.758738    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-7qwnj"]
Sep 18 19:20:24.760054 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:20:24.760019    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97cbc5d7_ec00_42d6_a924_6f58b6c1b7ca.slice/crio-f2e4eedbd080b59ffe3eb603ec5112279ac5412ce7bbf09e7cde2236800e2440 WatchSource:0}: Error finding container f2e4eedbd080b59ffe3eb603ec5112279ac5412ce7bbf09e7cde2236800e2440: Status 404 returned error can't find the container with id f2e4eedbd080b59ffe3eb603ec5112279ac5412ce7bbf09e7cde2236800e2440
Sep 18 19:20:24.856912 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:24.856869    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e33c9bc2-b125-4950-8b54-d74e05360575-certificates\") pod \"keda-metrics-apiserver-77857b7b98-mxr6m\" (UID: \"e33c9bc2-b125-4950-8b54-d74e05360575\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-mxr6m"
Sep 18 19:20:24.859329 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:24.859306    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e33c9bc2-b125-4950-8b54-d74e05360575-certificates\") pod \"keda-metrics-apiserver-77857b7b98-mxr6m\" (UID: \"e33c9bc2-b125-4950-8b54-d74e05360575\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-mxr6m"
Sep 18 19:20:25.050750 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:25.050663    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-zxpfj"]
Sep 18 19:20:25.054919 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:20:25.054891    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a61b1c0_df1b_4990_bc0d_f6a34d73055f.slice/crio-98b5b3fe690df82a1e7a645578009221c3a58472391b3dfc51be7cc161bde9b2 WatchSource:0}: Error finding container 98b5b3fe690df82a1e7a645578009221c3a58472391b3dfc51be7cc161bde9b2: Status 404 returned error can't find the container with id 98b5b3fe690df82a1e7a645578009221c3a58472391b3dfc51be7cc161bde9b2
Sep 18 19:20:25.086907 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:25.086882    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-mxr6m"
Sep 18 19:20:25.167347 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:25.167316    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-7qwnj" event={"ID":"97cbc5d7-ec00-42d6-a924-6f58b6c1b7ca","Type":"ContainerStarted","Data":"f2e4eedbd080b59ffe3eb603ec5112279ac5412ce7bbf09e7cde2236800e2440"}
Sep 18 19:20:25.168518 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:25.168493    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-zxpfj" event={"ID":"5a61b1c0-df1b-4990-bc0d-f6a34d73055f","Type":"ContainerStarted","Data":"98b5b3fe690df82a1e7a645578009221c3a58472391b3dfc51be7cc161bde9b2"}
Sep 18 19:20:25.381999 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:25.381965    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-mxr6m"]
Sep 18 19:20:25.384013 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:20:25.383990    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode33c9bc2_b125_4950_8b54_d74e05360575.slice/crio-f4499c32cac7b9619949c299b92d97bacef0ff3f3799e03a02dfd505fb87e9d4 WatchSource:0}: Error finding container f4499c32cac7b9619949c299b92d97bacef0ff3f3799e03a02dfd505fb87e9d4: Status 404 returned error can't find the container with id f4499c32cac7b9619949c299b92d97bacef0ff3f3799e03a02dfd505fb87e9d4
Sep 18 19:20:26.175195 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:26.175149    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-mxr6m" event={"ID":"e33c9bc2-b125-4950-8b54-d74e05360575","Type":"ContainerStarted","Data":"f4499c32cac7b9619949c299b92d97bacef0ff3f3799e03a02dfd505fb87e9d4"}
Sep 18 19:20:30.195322 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:30.195285    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-7qwnj" event={"ID":"97cbc5d7-ec00-42d6-a924-6f58b6c1b7ca","Type":"ContainerStarted","Data":"07cf2e89560e923f47aa376165f1201de02d7f28933732b37c77c7758f0eba4e"}
Sep 18 19:20:30.195777 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:30.195420    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-7qwnj"
Sep 18 19:20:30.196993 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:30.196966    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-zxpfj" event={"ID":"5a61b1c0-df1b-4990-bc0d-f6a34d73055f","Type":"ContainerStarted","Data":"c7a5eac92ba3eebebad55a1544280182ecebc5bcb5a1f1f79ae7c1dfe16b7265"}
Sep 18 19:20:30.197125 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:30.197017    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-zxpfj"
Sep 18 19:20:30.198529 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:30.198503    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-mxr6m" event={"ID":"e33c9bc2-b125-4950-8b54-d74e05360575","Type":"ContainerStarted","Data":"5194b1d282ee43af2fe1319c08179789d900d3d3318fb09bbf21a463726088d0"}
Sep 18 19:20:30.198670 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:30.198607    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-mxr6m"
Sep 18 19:20:30.242664 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:30.242625    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-7qwnj" podStartSLOduration=2.69838168 podStartE2EDuration="7.242612304s" totalImagesPullingTime="4.544230624s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:20:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:20:30.240683914 +0000 UTC m=+390.124210387" watchObservedRunningTime="2026-09-18 19:20:30.242612304 +0000 UTC m=+390.126138765"
Sep 18 19:20:30.287449 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:30.287407    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-zxpfj" podStartSLOduration=2.037940233 podStartE2EDuration="6.287394242s" totalImagesPullingTime="4.249454009s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:20:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:20:30.281294565 +0000 UTC m=+390.164821027" watchObservedRunningTime="2026-09-18 19:20:30.287394242 +0000 UTC m=+390.170920705"
Sep 18 19:20:41.210233 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:41.210201    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-mxr6m"
Sep 18 19:20:41.229870 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:41.229818    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-mxr6m" podStartSLOduration=13.299968863 podStartE2EDuration="17.229800473s" totalImagesPullingTime="3.92983161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:20:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:20:30.320629077 +0000 UTC m=+390.204155535" watchObservedRunningTime="2026-09-18 19:20:41.229800473 +0000 UTC m=+401.113326935"
Sep 18 19:20:44.163834 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:44.163800    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-drcfl"
Sep 18 19:20:51.205685 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:51.205655    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-zxpfj"
Sep 18 19:20:51.206066 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:20:51.205731    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-7qwnj"
Sep 18 19:21:50.648968 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:21:50.648933    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-m8mlb"]
Sep 18 19:21:50.652198 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:21:50.652180    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-m8mlb"
Sep 18 19:21:50.662416 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:21:50.662398    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 18 19:21:50.663361 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:21:50.663344    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 18 19:21:50.664777 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:21:50.664759    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 18 19:21:50.664855 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:21:50.664763    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-qflql\""
Sep 18 19:21:50.677749 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:21:50.677727    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-m8mlb"]
Sep 18 19:21:50.700695 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:21:50.700669    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-qc86j"]
Sep 18 19:21:50.704027 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:21:50.704006    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-qc86j"
Sep 18 19:21:50.707380 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:21:50.707361    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-8gwl5\""
Sep 18 19:21:50.707470 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:21:50.707391    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 18 19:21:50.722533 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:21:50.722511    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-qc86j"]
Sep 18 19:21:50.778989 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:21:50.778962    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/69e3cc75-b8c2-4777-974f-b017538db378-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-m8mlb\" (UID: \"69e3cc75-b8c2-4777-974f-b017538db378\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-m8mlb"
Sep 18 19:21:50.779141 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:21:50.779034    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbjwc\" (UniqueName: \"kubernetes.io/projected/69e3cc75-b8c2-4777-974f-b017538db378-kube-api-access-pbjwc\") pod \"llmisvc-controller-manager-645f9c8fd6-m8mlb\" (UID: \"69e3cc75-b8c2-4777-974f-b017538db378\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-m8mlb"
Sep 18 19:21:50.880411 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:21:50.880376    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/69e3cc75-b8c2-4777-974f-b017538db378-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-m8mlb\" (UID: \"69e3cc75-b8c2-4777-974f-b017538db378\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-m8mlb"
Sep 18 19:21:50.880411 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:21:50.880409    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46cl8\" (UniqueName: \"kubernetes.io/projected/c05217f4-19ec-45ed-ac0c-9898689234c2-kube-api-access-46cl8\") pod \"seaweedfs-64568bcd49-qc86j\" (UID: \"c05217f4-19ec-45ed-ac0c-9898689234c2\") " pod="kserve/seaweedfs-64568bcd49-qc86j"
Sep 18 19:21:50.880614 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:21:50.880466    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c05217f4-19ec-45ed-ac0c-9898689234c2-data\") pod \"seaweedfs-64568bcd49-qc86j\" (UID: \"c05217f4-19ec-45ed-ac0c-9898689234c2\") " pod="kserve/seaweedfs-64568bcd49-qc86j"
Sep 18 19:21:50.880614 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:21:50.880507    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbjwc\" (UniqueName: \"kubernetes.io/projected/69e3cc75-b8c2-4777-974f-b017538db378-kube-api-access-pbjwc\") pod \"llmisvc-controller-manager-645f9c8fd6-m8mlb\" (UID: \"69e3cc75-b8c2-4777-974f-b017538db378\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-m8mlb"
Sep 18 19:21:50.880614 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:21:50.880523    2522 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 18 19:21:50.880614 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:21:50.880608    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69e3cc75-b8c2-4777-974f-b017538db378-cert podName:69e3cc75-b8c2-4777-974f-b017538db378 nodeName:}" failed. No retries permitted until 2026-09-18 19:21:51.380584741 +0000 UTC m=+471.264111185 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/69e3cc75-b8c2-4777-974f-b017538db378-cert") pod "llmisvc-controller-manager-645f9c8fd6-m8mlb" (UID: "69e3cc75-b8c2-4777-974f-b017538db378") : secret "llmisvc-webhook-server-cert" not found
Sep 18 19:21:50.896191 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:21:50.896164    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbjwc\" (UniqueName: \"kubernetes.io/projected/69e3cc75-b8c2-4777-974f-b017538db378-kube-api-access-pbjwc\") pod \"llmisvc-controller-manager-645f9c8fd6-m8mlb\" (UID: \"69e3cc75-b8c2-4777-974f-b017538db378\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-m8mlb"
Sep 18 19:21:50.981770 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:21:50.981700    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46cl8\" (UniqueName: \"kubernetes.io/projected/c05217f4-19ec-45ed-ac0c-9898689234c2-kube-api-access-46cl8\") pod \"seaweedfs-64568bcd49-qc86j\" (UID: \"c05217f4-19ec-45ed-ac0c-9898689234c2\") " pod="kserve/seaweedfs-64568bcd49-qc86j"
Sep 18 19:21:50.981770 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:21:50.981766    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c05217f4-19ec-45ed-ac0c-9898689234c2-data\") pod \"seaweedfs-64568bcd49-qc86j\" (UID: \"c05217f4-19ec-45ed-ac0c-9898689234c2\") " pod="kserve/seaweedfs-64568bcd49-qc86j"
Sep 18 19:21:50.982138 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:21:50.982119    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c05217f4-19ec-45ed-ac0c-9898689234c2-data\") pod \"seaweedfs-64568bcd49-qc86j\" (UID: \"c05217f4-19ec-45ed-ac0c-9898689234c2\") " pod="kserve/seaweedfs-64568bcd49-qc86j"
Sep 18 19:21:50.993976 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:21:50.993946    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46cl8\" (UniqueName: \"kubernetes.io/projected/c05217f4-19ec-45ed-ac0c-9898689234c2-kube-api-access-46cl8\") pod \"seaweedfs-64568bcd49-qc86j\" (UID: \"c05217f4-19ec-45ed-ac0c-9898689234c2\") " pod="kserve/seaweedfs-64568bcd49-qc86j"
Sep 18 19:21:51.012497 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:21:51.012478    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-qc86j"
Sep 18 19:21:51.285901 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:21:51.285823    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-qc86j"]
Sep 18 19:21:51.288984 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:21:51.288957    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc05217f4_19ec_45ed_ac0c_9898689234c2.slice/crio-3949badac497b5c073fcbe9dd94b784051c3f0a4d26bbb525bfeb5bf9fdde570 WatchSource:0}: Error finding container 3949badac497b5c073fcbe9dd94b784051c3f0a4d26bbb525bfeb5bf9fdde570: Status 404 returned error can't find the container with id 3949badac497b5c073fcbe9dd94b784051c3f0a4d26bbb525bfeb5bf9fdde570
Sep 18 19:21:51.385629 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:21:51.385598    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/69e3cc75-b8c2-4777-974f-b017538db378-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-m8mlb\" (UID: \"69e3cc75-b8c2-4777-974f-b017538db378\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-m8mlb"
Sep 18 19:21:51.388034 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:21:51.388009    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/69e3cc75-b8c2-4777-974f-b017538db378-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-m8mlb\" (UID: \"69e3cc75-b8c2-4777-974f-b017538db378\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-m8mlb"
Sep 18 19:21:51.561421 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:21:51.561337    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-m8mlb"
Sep 18 19:21:51.562031 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:21:51.562007    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-qc86j" event={"ID":"c05217f4-19ec-45ed-ac0c-9898689234c2","Type":"ContainerStarted","Data":"3949badac497b5c073fcbe9dd94b784051c3f0a4d26bbb525bfeb5bf9fdde570"}
Sep 18 19:21:51.833966 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:21:51.833937    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-m8mlb"]
Sep 18 19:21:51.835861 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:21:51.835830    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod69e3cc75_b8c2_4777_974f_b017538db378.slice/crio-6f6d43ad6774b70b279a56338aab0ff6b2fe56767599f2e006a643a57e5c62fe WatchSource:0}: Error finding container 6f6d43ad6774b70b279a56338aab0ff6b2fe56767599f2e006a643a57e5c62fe: Status 404 returned error can't find the container with id 6f6d43ad6774b70b279a56338aab0ff6b2fe56767599f2e006a643a57e5c62fe
Sep 18 19:21:52.570868 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:21:52.570796    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-m8mlb" event={"ID":"69e3cc75-b8c2-4777-974f-b017538db378","Type":"ContainerStarted","Data":"6f6d43ad6774b70b279a56338aab0ff6b2fe56767599f2e006a643a57e5c62fe"}
Sep 18 19:21:55.586801 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:21:55.586767    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-m8mlb" event={"ID":"69e3cc75-b8c2-4777-974f-b017538db378","Type":"ContainerStarted","Data":"eba07cdd49a3cc23fb6840da311b2a09012f86689396b8c33f424fbd602b2b8c"}
Sep 18 19:21:55.587279 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:21:55.586883    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-m8mlb"
Sep 18 19:21:55.588395 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:21:55.588374    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-qc86j" event={"ID":"c05217f4-19ec-45ed-ac0c-9898689234c2","Type":"ContainerStarted","Data":"ca14d55d5aa6f2e8481286e2d2b971105eaf03ece7f575e7b8b99b322036205c"}
Sep 18 19:21:55.588511 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:21:55.588419    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-qc86j"
Sep 18 19:21:55.611441 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:21:55.611381    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-645f9c8fd6-m8mlb" podStartSLOduration=2.143227821 podStartE2EDuration="5.611366895s" totalImagesPullingTime="3.468139074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:21:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:21:55.609214013 +0000 UTC m=+475.492740475" watchObservedRunningTime="2026-09-18 19:21:55.611366895 +0000 UTC m=+475.494893375"
Sep 18 19:21:55.632081 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:21:55.631995    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-qc86j" podStartSLOduration=1.527153929 podStartE2EDuration="5.631979181s" totalImagesPullingTime="4.104825252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:21:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:21:55.631037933 +0000 UTC m=+475.514564396" watchObservedRunningTime="2026-09-18 19:21:55.631979181 +0000 UTC m=+475.515505642"
Sep 18 19:22:01.596168 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:22:01.596129    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-qc86j"
Sep 18 19:22:26.595345 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:22:26.595313    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-m8mlb"
Sep 18 19:22:54.495534 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:22:54.495446    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6bd78c47cf-5hjvg"]
Sep 18 19:23:19.514579 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:23:19.514519    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6bd78c47cf-5hjvg" podUID="e87fd35d-f7a7-429f-a997-0267c01d7405" containerName="console" containerID="cri-o://33aeba817ba7617eb9c185fa049336e631586ee21590bf04b59b210d06306a53" gracePeriod=15
Sep 18 19:23:19.759829 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:23:19.759803    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bd78c47cf-5hjvg"
Sep 18 19:23:19.929713 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:23:19.929663    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e87fd35d-f7a7-429f-a997-0267c01d7405-console-serving-cert\") pod \"e87fd35d-f7a7-429f-a997-0267c01d7405\" (UID: \"e87fd35d-f7a7-429f-a997-0267c01d7405\") "
Sep 18 19:23:19.929713 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:23:19.929719    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e87fd35d-f7a7-429f-a997-0267c01d7405-service-ca\") pod \"e87fd35d-f7a7-429f-a997-0267c01d7405\" (UID: \"e87fd35d-f7a7-429f-a997-0267c01d7405\") "
Sep 18 19:23:19.929941 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:23:19.929785    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5lf4g\" (UniqueName: \"kubernetes.io/projected/e87fd35d-f7a7-429f-a997-0267c01d7405-kube-api-access-5lf4g\") pod \"e87fd35d-f7a7-429f-a997-0267c01d7405\" (UID: \"e87fd35d-f7a7-429f-a997-0267c01d7405\") "
Sep 18 19:23:19.929941 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:23:19.929808    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e87fd35d-f7a7-429f-a997-0267c01d7405-console-oauth-config\") pod \"e87fd35d-f7a7-429f-a997-0267c01d7405\" (UID: \"e87fd35d-f7a7-429f-a997-0267c01d7405\") "
Sep 18 19:23:19.929941 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:23:19.929841    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e87fd35d-f7a7-429f-a997-0267c01d7405-oauth-serving-cert\") pod \"e87fd35d-f7a7-429f-a997-0267c01d7405\" (UID: \"e87fd35d-f7a7-429f-a997-0267c01d7405\") "
Sep 18 19:23:19.929941 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:23:19.929865    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e87fd35d-f7a7-429f-a997-0267c01d7405-trusted-ca-bundle\") pod \"e87fd35d-f7a7-429f-a997-0267c01d7405\" (UID: \"e87fd35d-f7a7-429f-a997-0267c01d7405\") "
Sep 18 19:23:19.929941 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:23:19.929915    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e87fd35d-f7a7-429f-a997-0267c01d7405-console-config\") pod \"e87fd35d-f7a7-429f-a997-0267c01d7405\" (UID: \"e87fd35d-f7a7-429f-a997-0267c01d7405\") "
Sep 18 19:23:19.930192 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:23:19.930165    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e87fd35d-f7a7-429f-a997-0267c01d7405-service-ca" (OuterVolumeSpecName: "service-ca") pod "e87fd35d-f7a7-429f-a997-0267c01d7405" (UID: "e87fd35d-f7a7-429f-a997-0267c01d7405"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:23:19.930367 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:23:19.930292    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e87fd35d-f7a7-429f-a997-0267c01d7405-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e87fd35d-f7a7-429f-a997-0267c01d7405" (UID: "e87fd35d-f7a7-429f-a997-0267c01d7405"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:23:19.930475 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:23:19.930359    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e87fd35d-f7a7-429f-a997-0267c01d7405-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e87fd35d-f7a7-429f-a997-0267c01d7405" (UID: "e87fd35d-f7a7-429f-a997-0267c01d7405"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:23:19.930549 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:23:19.930469    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e87fd35d-f7a7-429f-a997-0267c01d7405-console-config" (OuterVolumeSpecName: "console-config") pod "e87fd35d-f7a7-429f-a997-0267c01d7405" (UID: "e87fd35d-f7a7-429f-a997-0267c01d7405"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:23:19.931984 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:23:19.931957    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e87fd35d-f7a7-429f-a997-0267c01d7405-kube-api-access-5lf4g" (OuterVolumeSpecName: "kube-api-access-5lf4g") pod "e87fd35d-f7a7-429f-a997-0267c01d7405" (UID: "e87fd35d-f7a7-429f-a997-0267c01d7405"). InnerVolumeSpecName "kube-api-access-5lf4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:23:19.932341 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:23:19.932319    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e87fd35d-f7a7-429f-a997-0267c01d7405-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e87fd35d-f7a7-429f-a997-0267c01d7405" (UID: "e87fd35d-f7a7-429f-a997-0267c01d7405"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:23:19.932407 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:23:19.932339    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e87fd35d-f7a7-429f-a997-0267c01d7405-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e87fd35d-f7a7-429f-a997-0267c01d7405" (UID: "e87fd35d-f7a7-429f-a997-0267c01d7405"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:23:19.970191 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:23:19.970162    2522 generic.go:356] "Generic (PLEG): container finished" podID="e87fd35d-f7a7-429f-a997-0267c01d7405" containerID="33aeba817ba7617eb9c185fa049336e631586ee21590bf04b59b210d06306a53" exitCode=0
Sep 18 19:23:19.970335 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:23:19.970206    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bd78c47cf-5hjvg"
Sep 18 19:23:19.970335 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:23:19.970207    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bd78c47cf-5hjvg" event={"ID":"e87fd35d-f7a7-429f-a997-0267c01d7405","Type":"ContainerDied","Data":"33aeba817ba7617eb9c185fa049336e631586ee21590bf04b59b210d06306a53"}
Sep 18 19:23:19.970335 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:23:19.970322    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bd78c47cf-5hjvg" event={"ID":"e87fd35d-f7a7-429f-a997-0267c01d7405","Type":"ContainerDied","Data":"9daa93334f22b8e5a6c15bad9deb9aa5f2add8d11f2c6072290dde9169e5469a"}
Sep 18 19:23:19.970504 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:23:19.970343    2522 scope.go:117] "RemoveContainer" containerID="33aeba817ba7617eb9c185fa049336e631586ee21590bf04b59b210d06306a53"
Sep 18 19:23:19.980412 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:23:19.980391    2522 scope.go:117] "RemoveContainer" containerID="33aeba817ba7617eb9c185fa049336e631586ee21590bf04b59b210d06306a53"
Sep 18 19:23:19.980736 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:23:19.980713    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33aeba817ba7617eb9c185fa049336e631586ee21590bf04b59b210d06306a53\": container with ID starting with 33aeba817ba7617eb9c185fa049336e631586ee21590bf04b59b210d06306a53 not found: ID does not exist" containerID="33aeba817ba7617eb9c185fa049336e631586ee21590bf04b59b210d06306a53"
Sep 18 19:23:19.980814 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:23:19.980748    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33aeba817ba7617eb9c185fa049336e631586ee21590bf04b59b210d06306a53"} err="failed to get container status \"33aeba817ba7617eb9c185fa049336e631586ee21590bf04b59b210d06306a53\": rpc error: code = NotFound desc = could not find container \"33aeba817ba7617eb9c185fa049336e631586ee21590bf04b59b210d06306a53\": container with ID starting with 33aeba817ba7617eb9c185fa049336e631586ee21590bf04b59b210d06306a53 not found: ID does not exist"
Sep 18 19:23:19.994571 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:23:19.994531    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6bd78c47cf-5hjvg"]
Sep 18 19:23:19.997727 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:23:19.997702    2522 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6bd78c47cf-5hjvg"]
Sep 18 19:23:20.031407 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:23:20.031370    2522 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e87fd35d-f7a7-429f-a997-0267c01d7405-oauth-serving-cert\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:23:20.031407 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:23:20.031399    2522 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e87fd35d-f7a7-429f-a997-0267c01d7405-trusted-ca-bundle\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:23:20.031597 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:23:20.031414    2522 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e87fd35d-f7a7-429f-a997-0267c01d7405-console-config\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:23:20.031597 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:23:20.031427    2522 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e87fd35d-f7a7-429f-a997-0267c01d7405-console-serving-cert\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:23:20.031597 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:23:20.031439    2522 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e87fd35d-f7a7-429f-a997-0267c01d7405-service-ca\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:23:20.031597 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:23:20.031450    2522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lf4g\" (UniqueName: \"kubernetes.io/projected/e87fd35d-f7a7-429f-a997-0267c01d7405-kube-api-access-5lf4g\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:23:20.031597 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:23:20.031461    2522 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e87fd35d-f7a7-429f-a997-0267c01d7405-console-oauth-config\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:23:20.721156 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:23:20.721124    2522 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e87fd35d-f7a7-429f-a997-0267c01d7405" path="/var/lib/kubelet/pods/e87fd35d-f7a7-429f-a997-0267c01d7405/volumes"
Sep 18 19:24:50.126695 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:24:50.126659    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-gcrkw"]
Sep 18 19:24:50.127100 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:24:50.127002    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e87fd35d-f7a7-429f-a997-0267c01d7405" containerName="console"
Sep 18 19:24:50.127100 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:24:50.127016    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="e87fd35d-f7a7-429f-a997-0267c01d7405" containerName="console"
Sep 18 19:24:50.127100 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:24:50.127091    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="e87fd35d-f7a7-429f-a997-0267c01d7405" containerName="console"
Sep 18 19:24:50.130155 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:24:50.130137    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-gcrkw"
Sep 18 19:24:50.138354 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:24:50.138331    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 18 19:24:50.155720 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:24:50.155697    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-gcrkw"]
Sep 18 19:24:50.246859 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:24:50.246828    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/410e9f03-2f2a-4ba9-b6bb-07e2f8c2aa35-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-gcrkw\" (UID: \"410e9f03-2f2a-4ba9-b6bb-07e2f8c2aa35\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-gcrkw"
Sep 18 19:24:50.247003 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:24:50.246875    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v9m4\" (UniqueName: \"kubernetes.io/projected/410e9f03-2f2a-4ba9-b6bb-07e2f8c2aa35-kube-api-access-7v9m4\") pod \"seaweedfs-tls-custom-6644bf5d6c-gcrkw\" (UID: \"410e9f03-2f2a-4ba9-b6bb-07e2f8c2aa35\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-gcrkw"
Sep 18 19:24:50.348014 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:24:50.347987    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v9m4\" (UniqueName: \"kubernetes.io/projected/410e9f03-2f2a-4ba9-b6bb-07e2f8c2aa35-kube-api-access-7v9m4\") pod \"seaweedfs-tls-custom-6644bf5d6c-gcrkw\" (UID: \"410e9f03-2f2a-4ba9-b6bb-07e2f8c2aa35\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-gcrkw"
Sep 18 19:24:50.348130 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:24:50.348048    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/410e9f03-2f2a-4ba9-b6bb-07e2f8c2aa35-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-gcrkw\" (UID: \"410e9f03-2f2a-4ba9-b6bb-07e2f8c2aa35\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-gcrkw"
Sep 18 19:24:50.348362 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:24:50.348347    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/410e9f03-2f2a-4ba9-b6bb-07e2f8c2aa35-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-gcrkw\" (UID: \"410e9f03-2f2a-4ba9-b6bb-07e2f8c2aa35\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-gcrkw"
Sep 18 19:24:50.363263 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:24:50.363242    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v9m4\" (UniqueName: \"kubernetes.io/projected/410e9f03-2f2a-4ba9-b6bb-07e2f8c2aa35-kube-api-access-7v9m4\") pod \"seaweedfs-tls-custom-6644bf5d6c-gcrkw\" (UID: \"410e9f03-2f2a-4ba9-b6bb-07e2f8c2aa35\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-gcrkw"
Sep 18 19:24:50.439408 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:24:50.439341    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-gcrkw"
Sep 18 19:24:50.715823 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:24:50.715731    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-gcrkw"]
Sep 18 19:24:50.725804 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:24:50.725773    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod410e9f03_2f2a_4ba9_b6bb_07e2f8c2aa35.slice/crio-c4f1f10c7c364bcd5194555992df4f43abdd2bc5e448ff002686d1ab1e96a66b WatchSource:0}: Error finding container c4f1f10c7c364bcd5194555992df4f43abdd2bc5e448ff002686d1ab1e96a66b: Status 404 returned error can't find the container with id c4f1f10c7c364bcd5194555992df4f43abdd2bc5e448ff002686d1ab1e96a66b
Sep 18 19:24:50.727313 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:24:50.727293    2522 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:24:51.380989 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:24:51.380954    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-gcrkw" event={"ID":"410e9f03-2f2a-4ba9-b6bb-07e2f8c2aa35","Type":"ContainerStarted","Data":"325b1490dc7a99411eb60de20e7f4f8addbed1b7b310365cb540b6d4090adb36"}
Sep 18 19:24:51.381405 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:24:51.381005    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-gcrkw" event={"ID":"410e9f03-2f2a-4ba9-b6bb-07e2f8c2aa35","Type":"ContainerStarted","Data":"c4f1f10c7c364bcd5194555992df4f43abdd2bc5e448ff002686d1ab1e96a66b"}
Sep 18 19:24:51.399181 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:24:51.399131    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-gcrkw" podStartSLOduration=1.110243879 podStartE2EDuration="1.399116753s" totalImagesPullingTime="288.872874ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:24:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:24:51.398794929 +0000 UTC m=+651.282321392" watchObservedRunningTime="2026-09-18 19:24:51.399116753 +0000 UTC m=+651.282643214"
Sep 18 19:24:52.312791 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:24:52.312758    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-gcrkw"]
Sep 18 19:24:53.390125 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:24:53.390087    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-gcrkw" podUID="410e9f03-2f2a-4ba9-b6bb-07e2f8c2aa35" containerName="seaweedfs-tls-custom" containerID="cri-o://325b1490dc7a99411eb60de20e7f4f8addbed1b7b310365cb540b6d4090adb36" gracePeriod=30
Sep 18 19:24:54.631979 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:24:54.631956    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-gcrkw"
Sep 18 19:24:54.685615 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:24:54.685511    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7v9m4\" (UniqueName: \"kubernetes.io/projected/410e9f03-2f2a-4ba9-b6bb-07e2f8c2aa35-kube-api-access-7v9m4\") pod \"410e9f03-2f2a-4ba9-b6bb-07e2f8c2aa35\" (UID: \"410e9f03-2f2a-4ba9-b6bb-07e2f8c2aa35\") "
Sep 18 19:24:54.685615 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:24:54.685581    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/410e9f03-2f2a-4ba9-b6bb-07e2f8c2aa35-data\") pod \"410e9f03-2f2a-4ba9-b6bb-07e2f8c2aa35\" (UID: \"410e9f03-2f2a-4ba9-b6bb-07e2f8c2aa35\") "
Sep 18 19:24:54.686505 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:24:54.686478    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/410e9f03-2f2a-4ba9-b6bb-07e2f8c2aa35-data" (OuterVolumeSpecName: "data") pod "410e9f03-2f2a-4ba9-b6bb-07e2f8c2aa35" (UID: "410e9f03-2f2a-4ba9-b6bb-07e2f8c2aa35"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:24:54.687542 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:24:54.687508    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/410e9f03-2f2a-4ba9-b6bb-07e2f8c2aa35-kube-api-access-7v9m4" (OuterVolumeSpecName: "kube-api-access-7v9m4") pod "410e9f03-2f2a-4ba9-b6bb-07e2f8c2aa35" (UID: "410e9f03-2f2a-4ba9-b6bb-07e2f8c2aa35"). InnerVolumeSpecName "kube-api-access-7v9m4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:24:54.786651 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:24:54.786621    2522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v9m4\" (UniqueName: \"kubernetes.io/projected/410e9f03-2f2a-4ba9-b6bb-07e2f8c2aa35-kube-api-access-7v9m4\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:24:54.786651 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:24:54.786645    2522 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/410e9f03-2f2a-4ba9-b6bb-07e2f8c2aa35-data\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:24:55.399888 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:24:55.399856    2522 generic.go:356] "Generic (PLEG): container finished" podID="410e9f03-2f2a-4ba9-b6bb-07e2f8c2aa35" containerID="325b1490dc7a99411eb60de20e7f4f8addbed1b7b310365cb540b6d4090adb36" exitCode=0
Sep 18 19:24:55.400068 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:24:55.399897    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-gcrkw"
Sep 18 19:24:55.400068 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:24:55.399941    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-gcrkw" event={"ID":"410e9f03-2f2a-4ba9-b6bb-07e2f8c2aa35","Type":"ContainerDied","Data":"325b1490dc7a99411eb60de20e7f4f8addbed1b7b310365cb540b6d4090adb36"}
Sep 18 19:24:55.400068 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:24:55.399988    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-gcrkw" event={"ID":"410e9f03-2f2a-4ba9-b6bb-07e2f8c2aa35","Type":"ContainerDied","Data":"c4f1f10c7c364bcd5194555992df4f43abdd2bc5e448ff002686d1ab1e96a66b"}
Sep 18 19:24:55.400068 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:24:55.400010    2522 scope.go:117] "RemoveContainer" containerID="325b1490dc7a99411eb60de20e7f4f8addbed1b7b310365cb540b6d4090adb36"
Sep 18 19:24:55.409341 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:24:55.409319    2522 scope.go:117] "RemoveContainer" containerID="325b1490dc7a99411eb60de20e7f4f8addbed1b7b310365cb540b6d4090adb36"
Sep 18 19:24:55.409629 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:24:55.409607    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"325b1490dc7a99411eb60de20e7f4f8addbed1b7b310365cb540b6d4090adb36\": container with ID starting with 325b1490dc7a99411eb60de20e7f4f8addbed1b7b310365cb540b6d4090adb36 not found: ID does not exist" containerID="325b1490dc7a99411eb60de20e7f4f8addbed1b7b310365cb540b6d4090adb36"
Sep 18 19:24:55.409682 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:24:55.409638    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"325b1490dc7a99411eb60de20e7f4f8addbed1b7b310365cb540b6d4090adb36"} err="failed to get container status \"325b1490dc7a99411eb60de20e7f4f8addbed1b7b310365cb540b6d4090adb36\": rpc error: code = NotFound desc = could not find container \"325b1490dc7a99411eb60de20e7f4f8addbed1b7b310365cb540b6d4090adb36\": container with ID starting with 325b1490dc7a99411eb60de20e7f4f8addbed1b7b310365cb540b6d4090adb36 not found: ID does not exist"
Sep 18 19:24:55.418461 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:24:55.418442    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-gcrkw"]
Sep 18 19:24:55.422375 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:24:55.422358    2522 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-gcrkw"]
Sep 18 19:24:55.454171 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:24:55.454153    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-l987k"]
Sep 18 19:24:55.454517 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:24:55.454499    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="410e9f03-2f2a-4ba9-b6bb-07e2f8c2aa35" containerName="seaweedfs-tls-custom"
Sep 18 19:24:55.454570 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:24:55.454520    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="410e9f03-2f2a-4ba9-b6bb-07e2f8c2aa35" containerName="seaweedfs-tls-custom"
Sep 18 19:24:55.454620 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:24:55.454610    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="410e9f03-2f2a-4ba9-b6bb-07e2f8c2aa35" containerName="seaweedfs-tls-custom"
Sep 18 19:24:55.458756 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:24:55.458742    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-l987k"
Sep 18 19:24:55.461766 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:24:55.461749    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom\""
Sep 18 19:24:55.461851 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:24:55.461782    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 18 19:24:55.468271 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:24:55.468253    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-l987k"]
Sep 18 19:24:55.593945 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:24:55.593912    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/02dd1836-2988-4996-9a32-07600c3ee746-data\") pod \"seaweedfs-tls-custom-8b44d767d-l987k\" (UID: \"02dd1836-2988-4996-9a32-07600c3ee746\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-l987k"
Sep 18 19:24:55.593945 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:24:55.593946    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/02dd1836-2988-4996-9a32-07600c3ee746-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-l987k\" (UID: \"02dd1836-2988-4996-9a32-07600c3ee746\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-l987k"
Sep 18 19:24:55.594135 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:24:55.593962    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj9cn\" (UniqueName: \"kubernetes.io/projected/02dd1836-2988-4996-9a32-07600c3ee746-kube-api-access-zj9cn\") pod \"seaweedfs-tls-custom-8b44d767d-l987k\" (UID: \"02dd1836-2988-4996-9a32-07600c3ee746\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-l987k"
Sep 18 19:24:55.694414 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:24:55.694349    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/02dd1836-2988-4996-9a32-07600c3ee746-data\") pod \"seaweedfs-tls-custom-8b44d767d-l987k\" (UID: \"02dd1836-2988-4996-9a32-07600c3ee746\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-l987k"
Sep 18 19:24:55.694414 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:24:55.694380    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/02dd1836-2988-4996-9a32-07600c3ee746-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-l987k\" (UID: \"02dd1836-2988-4996-9a32-07600c3ee746\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-l987k"
Sep 18 19:24:55.694414 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:24:55.694400    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj9cn\" (UniqueName: \"kubernetes.io/projected/02dd1836-2988-4996-9a32-07600c3ee746-kube-api-access-zj9cn\") pod \"seaweedfs-tls-custom-8b44d767d-l987k\" (UID: \"02dd1836-2988-4996-9a32-07600c3ee746\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-l987k"
Sep 18 19:24:55.694843 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:24:55.694794    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/02dd1836-2988-4996-9a32-07600c3ee746-data\") pod \"seaweedfs-tls-custom-8b44d767d-l987k\" (UID: \"02dd1836-2988-4996-9a32-07600c3ee746\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-l987k"
Sep 18 19:24:55.696780 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:24:55.696760    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/02dd1836-2988-4996-9a32-07600c3ee746-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-l987k\" (UID: \"02dd1836-2988-4996-9a32-07600c3ee746\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-l987k"
Sep 18 19:24:55.705641 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:24:55.705617    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj9cn\" (UniqueName: \"kubernetes.io/projected/02dd1836-2988-4996-9a32-07600c3ee746-kube-api-access-zj9cn\") pod \"seaweedfs-tls-custom-8b44d767d-l987k\" (UID: \"02dd1836-2988-4996-9a32-07600c3ee746\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-l987k"
Sep 18 19:24:55.768263 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:24:55.768244    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-l987k"
Sep 18 19:24:56.036989 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:24:56.036956    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-l987k"]
Sep 18 19:24:56.038966 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:24:56.038939    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02dd1836_2988_4996_9a32_07600c3ee746.slice/crio-41e4778f680c731538dc47a7045e63b780185e1068b0fc92643da62d5bf89c56 WatchSource:0}: Error finding container 41e4778f680c731538dc47a7045e63b780185e1068b0fc92643da62d5bf89c56: Status 404 returned error can't find the container with id 41e4778f680c731538dc47a7045e63b780185e1068b0fc92643da62d5bf89c56
Sep 18 19:24:56.408924 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:24:56.408890    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-l987k" event={"ID":"02dd1836-2988-4996-9a32-07600c3ee746","Type":"ContainerStarted","Data":"ec0dbac5fa08e5d1f2e0f0b23d7acde1eaedef6d0203a4d1e544ce914b2c4273"}
Sep 18 19:24:56.409280 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:24:56.408944    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-l987k" event={"ID":"02dd1836-2988-4996-9a32-07600c3ee746","Type":"ContainerStarted","Data":"41e4778f680c731538dc47a7045e63b780185e1068b0fc92643da62d5bf89c56"}
Sep 18 19:24:56.427742 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:24:56.427699    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-8b44d767d-l987k" podStartSLOduration=1.147319237 podStartE2EDuration="1.427683678s" totalImagesPullingTime="280.364441ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:24:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:24:56.426409596 +0000 UTC m=+656.309936057" watchObservedRunningTime="2026-09-18 19:24:56.427683678 +0000 UTC m=+656.311210140"
Sep 18 19:24:56.721877 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:24:56.721806    2522 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="410e9f03-2f2a-4ba9-b6bb-07e2f8c2aa35" path="/var/lib/kubelet/pods/410e9f03-2f2a-4ba9-b6bb-07e2f8c2aa35/volumes"
Sep 18 19:25:10.011933 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:25:10.011901    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-b5mp8"]
Sep 18 19:25:10.015275 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:25:10.015258    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-b5mp8"
Sep 18 19:25:10.018284 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:25:10.018260    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving\""
Sep 18 19:25:10.018389 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:25:10.018264    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Sep 18 19:25:10.026104 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:25:10.026083    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-b5mp8"]
Sep 18 19:25:10.116602 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:25:10.116543    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xstv\" (UniqueName: \"kubernetes.io/projected/c83c09a0-1343-4c29-8a40-b5a508d54f60-kube-api-access-2xstv\") pod \"seaweedfs-tls-serving-67b7477c6d-b5mp8\" (UID: \"c83c09a0-1343-4c29-8a40-b5a508d54f60\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-b5mp8"
Sep 18 19:25:10.116754 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:25:10.116611    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c83c09a0-1343-4c29-8a40-b5a508d54f60-data\") pod \"seaweedfs-tls-serving-67b7477c6d-b5mp8\" (UID: \"c83c09a0-1343-4c29-8a40-b5a508d54f60\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-b5mp8"
Sep 18 19:25:10.116754 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:25:10.116701    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/c83c09a0-1343-4c29-8a40-b5a508d54f60-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-b5mp8\" (UID: \"c83c09a0-1343-4c29-8a40-b5a508d54f60\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-b5mp8"
Sep 18 19:25:10.217779 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:25:10.217744    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c83c09a0-1343-4c29-8a40-b5a508d54f60-data\") pod \"seaweedfs-tls-serving-67b7477c6d-b5mp8\" (UID: \"c83c09a0-1343-4c29-8a40-b5a508d54f60\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-b5mp8"
Sep 18 19:25:10.217959 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:25:10.217792    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/c83c09a0-1343-4c29-8a40-b5a508d54f60-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-b5mp8\" (UID: \"c83c09a0-1343-4c29-8a40-b5a508d54f60\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-b5mp8"
Sep 18 19:25:10.217959 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:25:10.217836    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xstv\" (UniqueName: \"kubernetes.io/projected/c83c09a0-1343-4c29-8a40-b5a508d54f60-kube-api-access-2xstv\") pod \"seaweedfs-tls-serving-67b7477c6d-b5mp8\" (UID: \"c83c09a0-1343-4c29-8a40-b5a508d54f60\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-b5mp8"
Sep 18 19:25:10.218204 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:25:10.218178    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c83c09a0-1343-4c29-8a40-b5a508d54f60-data\") pod \"seaweedfs-tls-serving-67b7477c6d-b5mp8\" (UID: \"c83c09a0-1343-4c29-8a40-b5a508d54f60\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-b5mp8"
Sep 18 19:25:10.220295 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:25:10.220274    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/c83c09a0-1343-4c29-8a40-b5a508d54f60-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-b5mp8\" (UID: \"c83c09a0-1343-4c29-8a40-b5a508d54f60\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-b5mp8"
Sep 18 19:25:10.226976 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:25:10.226955    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xstv\" (UniqueName: \"kubernetes.io/projected/c83c09a0-1343-4c29-8a40-b5a508d54f60-kube-api-access-2xstv\") pod \"seaweedfs-tls-serving-67b7477c6d-b5mp8\" (UID: \"c83c09a0-1343-4c29-8a40-b5a508d54f60\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-b5mp8"
Sep 18 19:25:10.324654 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:25:10.324625    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-b5mp8"
Sep 18 19:25:10.599674 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:25:10.599650    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-b5mp8"]
Sep 18 19:25:10.601217 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:25:10.601191    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc83c09a0_1343_4c29_8a40_b5a508d54f60.slice/crio-d262230fb147bf16821e441827826b022f93db16c3c56819d9375a0b8952c5d9 WatchSource:0}: Error finding container d262230fb147bf16821e441827826b022f93db16c3c56819d9375a0b8952c5d9: Status 404 returned error can't find the container with id d262230fb147bf16821e441827826b022f93db16c3c56819d9375a0b8952c5d9
Sep 18 19:25:11.488354 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:25:11.488315    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-b5mp8" event={"ID":"c83c09a0-1343-4c29-8a40-b5a508d54f60","Type":"ContainerStarted","Data":"e0d824a14b70cb210f0dc464fff291fb38a0089319fdaa816719a514dd7dc9e4"}
Sep 18 19:25:11.488741 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:25:11.488374    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-b5mp8" event={"ID":"c83c09a0-1343-4c29-8a40-b5a508d54f60","Type":"ContainerStarted","Data":"d262230fb147bf16821e441827826b022f93db16c3c56819d9375a0b8952c5d9"}
Sep 18 19:25:11.505749 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:25:11.505701    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-serving-67b7477c6d-b5mp8" podStartSLOduration=2.24600862 podStartE2EDuration="2.505678847s" totalImagesPullingTime="259.670227ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:25:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:25:11.504879772 +0000 UTC m=+671.388406236" watchObservedRunningTime="2026-09-18 19:25:11.505678847 +0000 UTC m=+671.389205308"
Sep 18 19:25:37.835456 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:25:37.835414    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8"]
Sep 18 19:25:37.839661 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:25:37.839627    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8"
Sep 18 19:25:37.842579 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:25:37.842539    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5b562\""
Sep 18 19:25:37.842692 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:25:37.842650    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Sep 18 19:25:37.842758 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:25:37.842718    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Sep 18 19:25:37.842843 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:25:37.842829    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 19:25:37.843602 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:25:37.843582    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 19:25:37.843704 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:25:37.843658    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 19:25:37.843704 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:25:37.843661    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 19:25:37.851818 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:25:37.851797    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8"]
Sep 18 19:25:37.937061 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:25:37.937017    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj8h8\" (UniqueName: \"kubernetes.io/projected/30c1e530-23e4-4ed6-bc64-c33f1371cd44-kube-api-access-wj8h8\") pod \"isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8\" (UID: \"30c1e530-23e4-4ed6-bc64-c33f1371cd44\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8"
Sep 18 19:25:37.937061 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:25:37.937078    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30c1e530-23e4-4ed6-bc64-c33f1371cd44-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8\" (UID: \"30c1e530-23e4-4ed6-bc64-c33f1371cd44\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8"
Sep 18 19:25:37.937283 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:25:37.937106    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30c1e530-23e4-4ed6-bc64-c33f1371cd44-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8\" (UID: \"30c1e530-23e4-4ed6-bc64-c33f1371cd44\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8"
Sep 18 19:25:37.937283 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:25:37.937149    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30c1e530-23e4-4ed6-bc64-c33f1371cd44-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8\" (UID: \"30c1e530-23e4-4ed6-bc64-c33f1371cd44\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8"
Sep 18 19:25:37.937283 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:25:37.937210    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30c1e530-23e4-4ed6-bc64-c33f1371cd44-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8\" (UID: \"30c1e530-23e4-4ed6-bc64-c33f1371cd44\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8"
Sep 18 19:25:38.037873 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:25:38.037792    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30c1e530-23e4-4ed6-bc64-c33f1371cd44-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8\" (UID: \"30c1e530-23e4-4ed6-bc64-c33f1371cd44\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8"
Sep 18 19:25:38.037873 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:25:38.037831    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30c1e530-23e4-4ed6-bc64-c33f1371cd44-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8\" (UID: \"30c1e530-23e4-4ed6-bc64-c33f1371cd44\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8"
Sep 18 19:25:38.038037 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:25:38.037902    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30c1e530-23e4-4ed6-bc64-c33f1371cd44-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8\" (UID: \"30c1e530-23e4-4ed6-bc64-c33f1371cd44\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8"
Sep 18 19:25:38.038037 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:25:38.037923    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30c1e530-23e4-4ed6-bc64-c33f1371cd44-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8\" (UID: \"30c1e530-23e4-4ed6-bc64-c33f1371cd44\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8"
Sep 18 19:25:38.038037 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:25:38.037986    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj8h8\" (UniqueName: \"kubernetes.io/projected/30c1e530-23e4-4ed6-bc64-c33f1371cd44-kube-api-access-wj8h8\") pod \"isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8\" (UID: \"30c1e530-23e4-4ed6-bc64-c33f1371cd44\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8"
Sep 18 19:25:38.038219 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:25:38.038198    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30c1e530-23e4-4ed6-bc64-c33f1371cd44-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8\" (UID: \"30c1e530-23e4-4ed6-bc64-c33f1371cd44\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8"
Sep 18 19:25:38.038601 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:25:38.038557    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30c1e530-23e4-4ed6-bc64-c33f1371cd44-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8\" (UID: \"30c1e530-23e4-4ed6-bc64-c33f1371cd44\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8"
Sep 18 19:25:38.038664 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:25:38.038583    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30c1e530-23e4-4ed6-bc64-c33f1371cd44-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8\" (UID: \"30c1e530-23e4-4ed6-bc64-c33f1371cd44\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8"
Sep 18 19:25:38.040399 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:25:38.040382    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30c1e530-23e4-4ed6-bc64-c33f1371cd44-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8\" (UID: \"30c1e530-23e4-4ed6-bc64-c33f1371cd44\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8"
Sep 18 19:25:38.049746 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:25:38.049726    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj8h8\" (UniqueName: \"kubernetes.io/projected/30c1e530-23e4-4ed6-bc64-c33f1371cd44-kube-api-access-wj8h8\") pod \"isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8\" (UID: \"30c1e530-23e4-4ed6-bc64-c33f1371cd44\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8"
Sep 18 19:25:38.150889 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:25:38.150856    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8"
Sep 18 19:25:38.425600 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:25:38.425552    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8"]
Sep 18 19:25:38.427628 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:25:38.427593    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30c1e530_23e4_4ed6_bc64_c33f1371cd44.slice/crio-5a579a63007d053352ff9644fb1df3929ebda4b21ed1aa7909e3e0babe4f3576 WatchSource:0}: Error finding container 5a579a63007d053352ff9644fb1df3929ebda4b21ed1aa7909e3e0babe4f3576: Status 404 returned error can't find the container with id 5a579a63007d053352ff9644fb1df3929ebda4b21ed1aa7909e3e0babe4f3576
Sep 18 19:25:38.616006 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:25:38.615972    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8" event={"ID":"30c1e530-23e4-4ed6-bc64-c33f1371cd44","Type":"ContainerStarted","Data":"5a579a63007d053352ff9644fb1df3929ebda4b21ed1aa7909e3e0babe4f3576"}
Sep 18 19:25:42.637597 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:25:42.637544    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8" event={"ID":"30c1e530-23e4-4ed6-bc64-c33f1371cd44","Type":"ContainerStarted","Data":"89e223b12fcd90a0b98086ab671383ee25dad3d7b44e0fb098258b25d953e2d0"}
Sep 18 19:25:45.653243 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:25:45.653206    2522 generic.go:356] "Generic (PLEG): container finished" podID="30c1e530-23e4-4ed6-bc64-c33f1371cd44" containerID="89e223b12fcd90a0b98086ab671383ee25dad3d7b44e0fb098258b25d953e2d0" exitCode=0
Sep 18 19:25:45.653684 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:25:45.653266    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8" event={"ID":"30c1e530-23e4-4ed6-bc64-c33f1371cd44","Type":"ContainerDied","Data":"89e223b12fcd90a0b98086ab671383ee25dad3d7b44e0fb098258b25d953e2d0"}
Sep 18 19:25:59.737819 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:25:59.737720    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8" event={"ID":"30c1e530-23e4-4ed6-bc64-c33f1371cd44","Type":"ContainerStarted","Data":"d6dfe68b8e8d901b4f74d0948d3f04a6956f60f3d8ece7251299763d894d1f18"}
Sep 18 19:26:02.756701 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:26:02.756665    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8" event={"ID":"30c1e530-23e4-4ed6-bc64-c33f1371cd44","Type":"ContainerStarted","Data":"13805af73a698951ad75f0202951729befcc08c434966dc8508115b948a859d9"}
Sep 18 19:26:05.775363 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:26:05.775322    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8" event={"ID":"30c1e530-23e4-4ed6-bc64-c33f1371cd44","Type":"ContainerStarted","Data":"d1fb575cc993fee079ca5d941cced54c6cb1a19469390ea4f1b99debed50ed57"}
Sep 18 19:26:05.775802 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:26:05.775525    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8"
Sep 18 19:26:05.775802 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:26:05.775689    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8"
Sep 18 19:26:05.777129 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:26:05.777101    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8" podUID="30c1e530-23e4-4ed6-bc64-c33f1371cd44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 18 19:26:05.812893 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:26:05.812841    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8" podStartSLOduration=2.802176351 podStartE2EDuration="28.812825607s" totalImagesPullingTime="22.282871747s" totalInitContainerRuntime="3.727777509s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:25:37 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:26:05.81189579 +0000 UTC m=+725.695422252" watchObservedRunningTime="2026-09-18 19:26:05.812825607 +0000 UTC m=+725.696352069"
Sep 18 19:26:06.780649 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:26:06.780612    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8"
Sep 18 19:26:06.781057 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:26:06.780819    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8" podUID="30c1e530-23e4-4ed6-bc64-c33f1371cd44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 18 19:26:06.782159 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:26:06.782131    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8" podUID="30c1e530-23e4-4ed6-bc64-c33f1371cd44" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:26:07.786661 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:26:07.786620    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8" podUID="30c1e530-23e4-4ed6-bc64-c33f1371cd44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 18 19:26:07.787182 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:26:07.786971    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8" podUID="30c1e530-23e4-4ed6-bc64-c33f1371cd44" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:26:07.790532 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:26:07.790508    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8"
Sep 18 19:26:08.791306 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:26:08.791267    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8" podUID="30c1e530-23e4-4ed6-bc64-c33f1371cd44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 18 19:26:08.791741 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:26:08.791545    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8" podUID="30c1e530-23e4-4ed6-bc64-c33f1371cd44" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:26:18.791918 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:26:18.791869    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8" podUID="30c1e530-23e4-4ed6-bc64-c33f1371cd44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 18 19:26:18.792408 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:26:18.792259    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8" podUID="30c1e530-23e4-4ed6-bc64-c33f1371cd44" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:26:28.791950 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:26:28.791907    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8" podUID="30c1e530-23e4-4ed6-bc64-c33f1371cd44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 18 19:26:28.792428 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:26:28.792274    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8" podUID="30c1e530-23e4-4ed6-bc64-c33f1371cd44" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:26:38.791578 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:26:38.791529    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8" podUID="30c1e530-23e4-4ed6-bc64-c33f1371cd44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 18 19:26:38.791970 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:26:38.791943    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8" podUID="30c1e530-23e4-4ed6-bc64-c33f1371cd44" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:26:48.792111 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:26:48.792059    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8" podUID="30c1e530-23e4-4ed6-bc64-c33f1371cd44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 18 19:26:48.792512 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:26:48.792468    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8" podUID="30c1e530-23e4-4ed6-bc64-c33f1371cd44" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:26:58.792079 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:26:58.792035    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8" podUID="30c1e530-23e4-4ed6-bc64-c33f1371cd44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 18 19:26:58.792588 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:26:58.792419    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8" podUID="30c1e530-23e4-4ed6-bc64-c33f1371cd44" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:27:08.791235 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:27:08.791182    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8" podUID="30c1e530-23e4-4ed6-bc64-c33f1371cd44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 18 19:27:08.791644 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:27:08.791621    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8" podUID="30c1e530-23e4-4ed6-bc64-c33f1371cd44" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:27:11.716381 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:27:11.716337    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8" podUID="30c1e530-23e4-4ed6-bc64-c33f1371cd44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 18 19:27:11.716821 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:27:11.716617    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8" podUID="30c1e530-23e4-4ed6-bc64-c33f1371cd44" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:27:21.717462 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:27:21.717432    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8"
Sep 18 19:27:21.717900 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:27:21.717877    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8"
Sep 18 19:27:36.627665 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:27:36.627585    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8"]
Sep 18 19:27:36.628176 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:27:36.627850    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8" podUID="30c1e530-23e4-4ed6-bc64-c33f1371cd44" containerName="agent" containerID="cri-o://d1fb575cc993fee079ca5d941cced54c6cb1a19469390ea4f1b99debed50ed57" gracePeriod=30
Sep 18 19:27:36.628176 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:27:36.627849    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8" podUID="30c1e530-23e4-4ed6-bc64-c33f1371cd44" containerName="kserve-container" containerID="cri-o://d6dfe68b8e8d901b4f74d0948d3f04a6956f60f3d8ece7251299763d894d1f18" gracePeriod=30
Sep 18 19:27:36.628176 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:27:36.627887    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8" podUID="30c1e530-23e4-4ed6-bc64-c33f1371cd44" containerName="kube-rbac-proxy" containerID="cri-o://13805af73a698951ad75f0202951729befcc08c434966dc8508115b948a859d9" gracePeriod=30
Sep 18 19:27:36.795629 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:27:36.795590    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh"]
Sep 18 19:27:36.799282 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:27:36.799265    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh"
Sep 18 19:27:36.801950 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:27:36.801931    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Sep 18 19:27:36.802020 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:27:36.801976    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Sep 18 19:27:36.810661 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:27:36.810638    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh"]
Sep 18 19:27:36.861966 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:27:36.861929    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7deb9ac2-3d2b-4b0f-a137-f2a862faac04-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh\" (UID: \"7deb9ac2-3d2b-4b0f-a137-f2a862faac04\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh"
Sep 18 19:27:36.862135 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:27:36.861980    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7deb9ac2-3d2b-4b0f-a137-f2a862faac04-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh\" (UID: \"7deb9ac2-3d2b-4b0f-a137-f2a862faac04\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh"
Sep 18 19:27:36.862135 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:27:36.862037    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-759kb\" (UniqueName: \"kubernetes.io/projected/7deb9ac2-3d2b-4b0f-a137-f2a862faac04-kube-api-access-759kb\") pod \"isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh\" (UID: \"7deb9ac2-3d2b-4b0f-a137-f2a862faac04\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh"
Sep 18 19:27:36.862135 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:27:36.862060    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7deb9ac2-3d2b-4b0f-a137-f2a862faac04-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh\" (UID: \"7deb9ac2-3d2b-4b0f-a137-f2a862faac04\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh"
Sep 18 19:27:36.862135 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:27:36.862102    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7deb9ac2-3d2b-4b0f-a137-f2a862faac04-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh\" (UID: \"7deb9ac2-3d2b-4b0f-a137-f2a862faac04\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh"
Sep 18 19:27:36.963115 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:27:36.963023    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7deb9ac2-3d2b-4b0f-a137-f2a862faac04-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh\" (UID: \"7deb9ac2-3d2b-4b0f-a137-f2a862faac04\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh"
Sep 18 19:27:36.963115 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:27:36.963102    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7deb9ac2-3d2b-4b0f-a137-f2a862faac04-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh\" (UID: \"7deb9ac2-3d2b-4b0f-a137-f2a862faac04\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh"
Sep 18 19:27:36.963328 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:27:36.963140    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7deb9ac2-3d2b-4b0f-a137-f2a862faac04-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh\" (UID: \"7deb9ac2-3d2b-4b0f-a137-f2a862faac04\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh"
Sep 18 19:27:36.963328 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:27:36.963196    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-759kb\" (UniqueName: \"kubernetes.io/projected/7deb9ac2-3d2b-4b0f-a137-f2a862faac04-kube-api-access-759kb\") pod \"isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh\" (UID: \"7deb9ac2-3d2b-4b0f-a137-f2a862faac04\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh"
Sep 18 19:27:36.963328 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:27:36.963226    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7deb9ac2-3d2b-4b0f-a137-f2a862faac04-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh\" (UID: \"7deb9ac2-3d2b-4b0f-a137-f2a862faac04\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh"
Sep 18 19:27:36.963651 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:27:36.963628    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7deb9ac2-3d2b-4b0f-a137-f2a862faac04-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh\" (UID: \"7deb9ac2-3d2b-4b0f-a137-f2a862faac04\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh"
Sep 18 19:27:36.963930 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:27:36.963909    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7deb9ac2-3d2b-4b0f-a137-f2a862faac04-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh\" (UID: \"7deb9ac2-3d2b-4b0f-a137-f2a862faac04\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh"
Sep 18 19:27:36.963978 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:27:36.963909    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7deb9ac2-3d2b-4b0f-a137-f2a862faac04-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh\" (UID: \"7deb9ac2-3d2b-4b0f-a137-f2a862faac04\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh"
Sep 18 19:27:36.965785 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:27:36.965758    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7deb9ac2-3d2b-4b0f-a137-f2a862faac04-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh\" (UID: \"7deb9ac2-3d2b-4b0f-a137-f2a862faac04\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh"
Sep 18 19:27:36.972982 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:27:36.972958    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-759kb\" (UniqueName: \"kubernetes.io/projected/7deb9ac2-3d2b-4b0f-a137-f2a862faac04-kube-api-access-759kb\") pod \"isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh\" (UID: \"7deb9ac2-3d2b-4b0f-a137-f2a862faac04\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh"
Sep 18 19:27:37.110007 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:27:37.109970    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh"
Sep 18 19:27:37.203977 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:27:37.203940    2522 generic.go:356] "Generic (PLEG): container finished" podID="30c1e530-23e4-4ed6-bc64-c33f1371cd44" containerID="13805af73a698951ad75f0202951729befcc08c434966dc8508115b948a859d9" exitCode=2
Sep 18 19:27:37.204152 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:27:37.204020    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8" event={"ID":"30c1e530-23e4-4ed6-bc64-c33f1371cd44","Type":"ContainerDied","Data":"13805af73a698951ad75f0202951729befcc08c434966dc8508115b948a859d9"}
Sep 18 19:27:37.386092 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:27:37.386060    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh"]
Sep 18 19:27:37.391735 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:27:37.389470    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7deb9ac2_3d2b_4b0f_a137_f2a862faac04.slice/crio-ecaf535f16945792302382d96d673acf40bf74ee988f588607e7e7bef2b2e146 WatchSource:0}: Error finding container ecaf535f16945792302382d96d673acf40bf74ee988f588607e7e7bef2b2e146: Status 404 returned error can't find the container with id ecaf535f16945792302382d96d673acf40bf74ee988f588607e7e7bef2b2e146
Sep 18 19:27:37.787789 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:27:37.787741    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8" podUID="30c1e530-23e4-4ed6-bc64-c33f1371cd44" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Sep 18 19:27:38.209978 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:27:38.209935    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh" event={"ID":"7deb9ac2-3d2b-4b0f-a137-f2a862faac04","Type":"ContainerStarted","Data":"4b261efdf5ea29e3ffde93c1c07d3bd5867f03d8fe46095f9c9ce3bd249c6ba2"}
Sep 18 19:27:38.210150 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:27:38.210053    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh" event={"ID":"7deb9ac2-3d2b-4b0f-a137-f2a862faac04","Type":"ContainerStarted","Data":"ecaf535f16945792302382d96d673acf40bf74ee988f588607e7e7bef2b2e146"}
Sep 18 19:27:41.717152 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:27:41.717114    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8" podUID="30c1e530-23e4-4ed6-bc64-c33f1371cd44" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:27:41.718392 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:27:41.718363    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8" podUID="30c1e530-23e4-4ed6-bc64-c33f1371cd44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 18 19:27:42.229718 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:27:42.229685    2522 generic.go:356] "Generic (PLEG): container finished" podID="7deb9ac2-3d2b-4b0f-a137-f2a862faac04" containerID="4b261efdf5ea29e3ffde93c1c07d3bd5867f03d8fe46095f9c9ce3bd249c6ba2" exitCode=0
Sep 18 19:27:42.229887 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:27:42.229757    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh" event={"ID":"7deb9ac2-3d2b-4b0f-a137-f2a862faac04","Type":"ContainerDied","Data":"4b261efdf5ea29e3ffde93c1c07d3bd5867f03d8fe46095f9c9ce3bd249c6ba2"}
Sep 18 19:27:42.232539 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:27:42.232515    2522 generic.go:356] "Generic (PLEG): container finished" podID="30c1e530-23e4-4ed6-bc64-c33f1371cd44" containerID="d6dfe68b8e8d901b4f74d0948d3f04a6956f60f3d8ece7251299763d894d1f18" exitCode=0
Sep 18 19:27:42.232631 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:27:42.232574    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8" event={"ID":"30c1e530-23e4-4ed6-bc64-c33f1371cd44","Type":"ContainerDied","Data":"d6dfe68b8e8d901b4f74d0948d3f04a6956f60f3d8ece7251299763d894d1f18"}
Sep 18 19:27:42.787739 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:27:42.787699    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8" podUID="30c1e530-23e4-4ed6-bc64-c33f1371cd44" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Sep 18 19:27:43.239787 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:27:43.239754    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh" event={"ID":"7deb9ac2-3d2b-4b0f-a137-f2a862faac04","Type":"ContainerStarted","Data":"114d960f0d66de80cf01d1dbc37d975df8a64d8f93480468c2ebbf880b0deb53"}
Sep 18 19:27:43.239962 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:27:43.239822    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh" event={"ID":"7deb9ac2-3d2b-4b0f-a137-f2a862faac04","Type":"ContainerStarted","Data":"e6c037b1727d03151ebad40663aa622b16f90bc64a3ee0c70799648a3399ca15"}
Sep 18 19:27:43.239962 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:27:43.239848    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh" event={"ID":"7deb9ac2-3d2b-4b0f-a137-f2a862faac04","Type":"ContainerStarted","Data":"6d3dfda860638178f2d0d6580b8570c40bcf0fafdb04f85a2b51d66950cf9c36"}
Sep 18 19:27:43.240138 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:27:43.240110    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh"
Sep 18 19:27:43.264761 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:27:43.264708    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh" podStartSLOduration=3.334527792 podStartE2EDuration="7.264695411s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.930167619s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:27:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:27:43.26355502 +0000 UTC m=+823.147081484" watchObservedRunningTime="2026-09-18 19:27:43.264695411 +0000 UTC m=+823.148221874"
Sep 18 19:27:44.244790 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:27:44.244748    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh"
Sep 18 19:27:44.245136 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:27:44.244824    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh"
Sep 18 19:27:44.246343 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:27:44.246307    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh" podUID="7deb9ac2-3d2b-4b0f-a137-f2a862faac04" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:5000: connect: connection refused"
Sep 18 19:27:44.246970 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:27:44.246943    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh" podUID="7deb9ac2-3d2b-4b0f-a137-f2a862faac04" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:27:45.249655 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:27:45.249619    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh" podUID="7deb9ac2-3d2b-4b0f-a137-f2a862faac04" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:5000: connect: connection refused"
Sep 18 19:27:45.250145 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:27:45.250021    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh" podUID="7deb9ac2-3d2b-4b0f-a137-f2a862faac04" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:27:47.786903 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:27:47.786862    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8" podUID="30c1e530-23e4-4ed6-bc64-c33f1371cd44" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Sep 18 19:27:47.787325 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:27:47.787004    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8"
Sep 18 19:27:50.254520 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:27:50.254487    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh"
Sep 18 19:27:50.255169 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:27:50.255140    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh" podUID="7deb9ac2-3d2b-4b0f-a137-f2a862faac04" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:5000: connect: connection refused"
Sep 18 19:27:50.255320 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:27:50.255296    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh" podUID="7deb9ac2-3d2b-4b0f-a137-f2a862faac04" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:27:51.717021 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:27:51.716974    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8" podUID="30c1e530-23e4-4ed6-bc64-c33f1371cd44" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:27:51.718394 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:27:51.718369    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8" podUID="30c1e530-23e4-4ed6-bc64-c33f1371cd44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 18 19:27:52.787019 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:27:52.786978    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8" podUID="30c1e530-23e4-4ed6-bc64-c33f1371cd44" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Sep 18 19:27:57.787246 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:27:57.787199    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8" podUID="30c1e530-23e4-4ed6-bc64-c33f1371cd44" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Sep 18 19:28:00.255602 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:28:00.255547    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh" podUID="7deb9ac2-3d2b-4b0f-a137-f2a862faac04" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:5000: connect: connection refused"
Sep 18 19:28:00.256070 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:28:00.256025    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh" podUID="7deb9ac2-3d2b-4b0f-a137-f2a862faac04" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:28:01.716930 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:28:01.716887    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8" podUID="30c1e530-23e4-4ed6-bc64-c33f1371cd44" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:28:01.717313 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:28:01.717026    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8"
Sep 18 19:28:01.718117 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:28:01.718094    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8" podUID="30c1e530-23e4-4ed6-bc64-c33f1371cd44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 18 19:28:01.718197 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:28:01.718185    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8"
Sep 18 19:28:02.786978 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:28:02.786933    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8" podUID="30c1e530-23e4-4ed6-bc64-c33f1371cd44" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Sep 18 19:28:06.779542 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:28:06.779515    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8"
Sep 18 19:28:06.823587 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:28:06.823549    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30c1e530-23e4-4ed6-bc64-c33f1371cd44-kserve-provision-location\") pod \"30c1e530-23e4-4ed6-bc64-c33f1371cd44\" (UID: \"30c1e530-23e4-4ed6-bc64-c33f1371cd44\") "
Sep 18 19:28:06.823728 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:28:06.823615    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30c1e530-23e4-4ed6-bc64-c33f1371cd44-proxy-tls\") pod \"30c1e530-23e4-4ed6-bc64-c33f1371cd44\" (UID: \"30c1e530-23e4-4ed6-bc64-c33f1371cd44\") "
Sep 18 19:28:06.823728 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:28:06.823639    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wj8h8\" (UniqueName: \"kubernetes.io/projected/30c1e530-23e4-4ed6-bc64-c33f1371cd44-kube-api-access-wj8h8\") pod \"30c1e530-23e4-4ed6-bc64-c33f1371cd44\" (UID: \"30c1e530-23e4-4ed6-bc64-c33f1371cd44\") "
Sep 18 19:28:06.823728 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:28:06.823662    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30c1e530-23e4-4ed6-bc64-c33f1371cd44-cabundle-cert\") pod \"30c1e530-23e4-4ed6-bc64-c33f1371cd44\" (UID: \"30c1e530-23e4-4ed6-bc64-c33f1371cd44\") "
Sep 18 19:28:06.823728 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:28:06.823688    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30c1e530-23e4-4ed6-bc64-c33f1371cd44-kube-rbac-proxy-sar-config\") pod \"30c1e530-23e4-4ed6-bc64-c33f1371cd44\" (UID: \"30c1e530-23e4-4ed6-bc64-c33f1371cd44\") "
Sep 18 19:28:06.824010 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:28:06.823865    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30c1e530-23e4-4ed6-bc64-c33f1371cd44-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "30c1e530-23e4-4ed6-bc64-c33f1371cd44" (UID: "30c1e530-23e4-4ed6-bc64-c33f1371cd44"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:28:06.824073 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:28:06.824049    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30c1e530-23e4-4ed6-bc64-c33f1371cd44-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "30c1e530-23e4-4ed6-bc64-c33f1371cd44" (UID: "30c1e530-23e4-4ed6-bc64-c33f1371cd44"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:28:06.824143 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:28:06.824113    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30c1e530-23e4-4ed6-bc64-c33f1371cd44-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "30c1e530-23e4-4ed6-bc64-c33f1371cd44" (UID: "30c1e530-23e4-4ed6-bc64-c33f1371cd44"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:28:06.825921 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:28:06.825899    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30c1e530-23e4-4ed6-bc64-c33f1371cd44-kube-api-access-wj8h8" (OuterVolumeSpecName: "kube-api-access-wj8h8") pod "30c1e530-23e4-4ed6-bc64-c33f1371cd44" (UID: "30c1e530-23e4-4ed6-bc64-c33f1371cd44"). InnerVolumeSpecName "kube-api-access-wj8h8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:28:06.825921 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:28:06.825907    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30c1e530-23e4-4ed6-bc64-c33f1371cd44-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "30c1e530-23e4-4ed6-bc64-c33f1371cd44" (UID: "30c1e530-23e4-4ed6-bc64-c33f1371cd44"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:28:06.924453 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:28:06.924354    2522 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30c1e530-23e4-4ed6-bc64-c33f1371cd44-kserve-provision-location\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:28:06.924453 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:28:06.924388    2522 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30c1e530-23e4-4ed6-bc64-c33f1371cd44-proxy-tls\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:28:06.924453 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:28:06.924403    2522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wj8h8\" (UniqueName: \"kubernetes.io/projected/30c1e530-23e4-4ed6-bc64-c33f1371cd44-kube-api-access-wj8h8\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:28:06.924453 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:28:06.924415    2522 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30c1e530-23e4-4ed6-bc64-c33f1371cd44-cabundle-cert\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:28:06.924453 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:28:06.924426    2522 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30c1e530-23e4-4ed6-bc64-c33f1371cd44-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:28:07.359805 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:28:07.359764    2522 generic.go:356] "Generic (PLEG): container finished" podID="30c1e530-23e4-4ed6-bc64-c33f1371cd44" containerID="d1fb575cc993fee079ca5d941cced54c6cb1a19469390ea4f1b99debed50ed57" exitCode=0
Sep 18 19:28:07.360002 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:28:07.359828    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8"
Sep 18 19:28:07.360002 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:28:07.359841    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8" event={"ID":"30c1e530-23e4-4ed6-bc64-c33f1371cd44","Type":"ContainerDied","Data":"d1fb575cc993fee079ca5d941cced54c6cb1a19469390ea4f1b99debed50ed57"}
Sep 18 19:28:07.360002 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:28:07.359915    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8" event={"ID":"30c1e530-23e4-4ed6-bc64-c33f1371cd44","Type":"ContainerDied","Data":"5a579a63007d053352ff9644fb1df3929ebda4b21ed1aa7909e3e0babe4f3576"}
Sep 18 19:28:07.360002 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:28:07.359947    2522 scope.go:117] "RemoveContainer" containerID="d1fb575cc993fee079ca5d941cced54c6cb1a19469390ea4f1b99debed50ed57"
Sep 18 19:28:07.368792 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:28:07.368775    2522 scope.go:117] "RemoveContainer" containerID="13805af73a698951ad75f0202951729befcc08c434966dc8508115b948a859d9"
Sep 18 19:28:07.376467 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:28:07.376447    2522 scope.go:117] "RemoveContainer" containerID="d6dfe68b8e8d901b4f74d0948d3f04a6956f60f3d8ece7251299763d894d1f18"
Sep 18 19:28:07.383624 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:28:07.383608    2522 scope.go:117] "RemoveContainer" containerID="89e223b12fcd90a0b98086ab671383ee25dad3d7b44e0fb098258b25d953e2d0"
Sep 18 19:28:07.387284 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:28:07.387259    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8"]
Sep 18 19:28:07.391440 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:28:07.391423    2522 scope.go:117] "RemoveContainer" containerID="d1fb575cc993fee079ca5d941cced54c6cb1a19469390ea4f1b99debed50ed57"
Sep 18 19:28:07.391762 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:28:07.391743    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1fb575cc993fee079ca5d941cced54c6cb1a19469390ea4f1b99debed50ed57\": container with ID starting with d1fb575cc993fee079ca5d941cced54c6cb1a19469390ea4f1b99debed50ed57 not found: ID does not exist" containerID="d1fb575cc993fee079ca5d941cced54c6cb1a19469390ea4f1b99debed50ed57"
Sep 18 19:28:07.391829 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:28:07.391768    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1fb575cc993fee079ca5d941cced54c6cb1a19469390ea4f1b99debed50ed57"} err="failed to get container status \"d1fb575cc993fee079ca5d941cced54c6cb1a19469390ea4f1b99debed50ed57\": rpc error: code = NotFound desc = could not find container \"d1fb575cc993fee079ca5d941cced54c6cb1a19469390ea4f1b99debed50ed57\": container with ID starting with d1fb575cc993fee079ca5d941cced54c6cb1a19469390ea4f1b99debed50ed57 not found: ID does not exist"
Sep 18 19:28:07.391829 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:28:07.391786    2522 scope.go:117] "RemoveContainer" containerID="13805af73a698951ad75f0202951729befcc08c434966dc8508115b948a859d9"
Sep 18 19:28:07.392031 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:28:07.392013    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13805af73a698951ad75f0202951729befcc08c434966dc8508115b948a859d9\": container with ID starting with 13805af73a698951ad75f0202951729befcc08c434966dc8508115b948a859d9 not found: ID does not exist" containerID="13805af73a698951ad75f0202951729befcc08c434966dc8508115b948a859d9"
Sep 18 19:28:07.392092 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:28:07.392038    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13805af73a698951ad75f0202951729befcc08c434966dc8508115b948a859d9"} err="failed to get container status \"13805af73a698951ad75f0202951729befcc08c434966dc8508115b948a859d9\": rpc error: code = NotFound desc = could not find container \"13805af73a698951ad75f0202951729befcc08c434966dc8508115b948a859d9\": container with ID starting with 13805af73a698951ad75f0202951729befcc08c434966dc8508115b948a859d9 not found: ID does not exist"
Sep 18 19:28:07.392092 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:28:07.392056    2522 scope.go:117] "RemoveContainer" containerID="d6dfe68b8e8d901b4f74d0948d3f04a6956f60f3d8ece7251299763d894d1f18"
Sep 18 19:28:07.392330 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:28:07.392316    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6dfe68b8e8d901b4f74d0948d3f04a6956f60f3d8ece7251299763d894d1f18\": container with ID starting with d6dfe68b8e8d901b4f74d0948d3f04a6956f60f3d8ece7251299763d894d1f18 not found: ID does not exist" containerID="d6dfe68b8e8d901b4f74d0948d3f04a6956f60f3d8ece7251299763d894d1f18"
Sep 18 19:28:07.392367 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:28:07.392335    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6dfe68b8e8d901b4f74d0948d3f04a6956f60f3d8ece7251299763d894d1f18"} err="failed to get container status \"d6dfe68b8e8d901b4f74d0948d3f04a6956f60f3d8ece7251299763d894d1f18\": rpc error: code = NotFound desc = could not find container \"d6dfe68b8e8d901b4f74d0948d3f04a6956f60f3d8ece7251299763d894d1f18\": container with ID starting with d6dfe68b8e8d901b4f74d0948d3f04a6956f60f3d8ece7251299763d894d1f18 not found: ID does not exist"
Sep 18 19:28:07.392367 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:28:07.392346    2522 scope.go:117] "RemoveContainer" containerID="89e223b12fcd90a0b98086ab671383ee25dad3d7b44e0fb098258b25d953e2d0"
Sep 18 19:28:07.392670 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:28:07.392615    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89e223b12fcd90a0b98086ab671383ee25dad3d7b44e0fb098258b25d953e2d0\": container with ID starting with 89e223b12fcd90a0b98086ab671383ee25dad3d7b44e0fb098258b25d953e2d0 not found: ID does not exist" containerID="89e223b12fcd90a0b98086ab671383ee25dad3d7b44e0fb098258b25d953e2d0"
Sep 18 19:28:07.392670 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:28:07.392645    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89e223b12fcd90a0b98086ab671383ee25dad3d7b44e0fb098258b25d953e2d0"} err="failed to get container status \"89e223b12fcd90a0b98086ab671383ee25dad3d7b44e0fb098258b25d953e2d0\": rpc error: code = NotFound desc = could not find container \"89e223b12fcd90a0b98086ab671383ee25dad3d7b44e0fb098258b25d953e2d0\": container with ID starting with 89e223b12fcd90a0b98086ab671383ee25dad3d7b44e0fb098258b25d953e2d0 not found: ID does not exist"
Sep 18 19:28:07.395435 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:28:07.395415    2522 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7b5b55bfb6-zftm8"]
Sep 18 19:28:08.721497 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:28:08.721461    2522 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30c1e530-23e4-4ed6-bc64-c33f1371cd44" path="/var/lib/kubelet/pods/30c1e530-23e4-4ed6-bc64-c33f1371cd44/volumes"
Sep 18 19:28:10.255024 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:28:10.254986    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh" podUID="7deb9ac2-3d2b-4b0f-a137-f2a862faac04" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:5000: connect: connection refused"
Sep 18 19:28:10.255482 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:28:10.255457    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh" podUID="7deb9ac2-3d2b-4b0f-a137-f2a862faac04" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:28:20.255098 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:28:20.255051    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh" podUID="7deb9ac2-3d2b-4b0f-a137-f2a862faac04" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:5000: connect: connection refused"
Sep 18 19:28:20.255588 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:28:20.255546    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh" podUID="7deb9ac2-3d2b-4b0f-a137-f2a862faac04" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:28:30.255690 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:28:30.255650    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh" podUID="7deb9ac2-3d2b-4b0f-a137-f2a862faac04" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:5000: connect: connection refused"
Sep 18 19:28:30.256165 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:28:30.256142    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh" podUID="7deb9ac2-3d2b-4b0f-a137-f2a862faac04" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:28:40.255732 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:28:40.255686    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh" podUID="7deb9ac2-3d2b-4b0f-a137-f2a862faac04" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:5000: connect: connection refused"
Sep 18 19:28:40.256271 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:28:40.256098    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh" podUID="7deb9ac2-3d2b-4b0f-a137-f2a862faac04" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:28:50.255791 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:28:50.255751    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh" podUID="7deb9ac2-3d2b-4b0f-a137-f2a862faac04" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:5000: connect: connection refused"
Sep 18 19:28:50.256256 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:28:50.256234    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh" podUID="7deb9ac2-3d2b-4b0f-a137-f2a862faac04" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:29:00.255610 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:00.255506    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh"
Sep 18 19:29:00.256067 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:00.255896    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh"
Sep 18 19:29:12.262154 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:12.262122    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh"]
Sep 18 19:29:12.262747 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:12.262454    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh" podUID="7deb9ac2-3d2b-4b0f-a137-f2a862faac04" containerName="kserve-container" containerID="cri-o://6d3dfda860638178f2d0d6580b8570c40bcf0fafdb04f85a2b51d66950cf9c36" gracePeriod=30
Sep 18 19:29:12.262747 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:12.262536    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh" podUID="7deb9ac2-3d2b-4b0f-a137-f2a862faac04" containerName="agent" containerID="cri-o://114d960f0d66de80cf01d1dbc37d975df8a64d8f93480468c2ebbf880b0deb53" gracePeriod=30
Sep 18 19:29:12.262747 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:12.262575    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh" podUID="7deb9ac2-3d2b-4b0f-a137-f2a862faac04" containerName="kube-rbac-proxy" containerID="cri-o://e6c037b1727d03151ebad40663aa622b16f90bc64a3ee0c70799648a3399ca15" gracePeriod=30
Sep 18 19:29:12.672209 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:12.672175    2522 generic.go:356] "Generic (PLEG): container finished" podID="7deb9ac2-3d2b-4b0f-a137-f2a862faac04" containerID="e6c037b1727d03151ebad40663aa622b16f90bc64a3ee0c70799648a3399ca15" exitCode=2
Sep 18 19:29:12.672403 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:12.672243    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh" event={"ID":"7deb9ac2-3d2b-4b0f-a137-f2a862faac04","Type":"ContainerDied","Data":"e6c037b1727d03151ebad40663aa622b16f90bc64a3ee0c70799648a3399ca15"}
Sep 18 19:29:15.250788 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:15.250743    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh" podUID="7deb9ac2-3d2b-4b0f-a137-f2a862faac04" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Sep 18 19:29:17.699798 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:17.699762    2522 generic.go:356] "Generic (PLEG): container finished" podID="7deb9ac2-3d2b-4b0f-a137-f2a862faac04" containerID="6d3dfda860638178f2d0d6580b8570c40bcf0fafdb04f85a2b51d66950cf9c36" exitCode=0
Sep 18 19:29:17.700231 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:17.699828    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh" event={"ID":"7deb9ac2-3d2b-4b0f-a137-f2a862faac04","Type":"ContainerDied","Data":"6d3dfda860638178f2d0d6580b8570c40bcf0fafdb04f85a2b51d66950cf9c36"}
Sep 18 19:29:20.250296 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:20.250260    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh" podUID="7deb9ac2-3d2b-4b0f-a137-f2a862faac04" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Sep 18 19:29:20.255004 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:20.254964    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh" podUID="7deb9ac2-3d2b-4b0f-a137-f2a862faac04" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:5000: connect: connection refused"
Sep 18 19:29:20.255336 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:20.255309    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh" podUID="7deb9ac2-3d2b-4b0f-a137-f2a862faac04" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:29:25.250034 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:25.249992    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh" podUID="7deb9ac2-3d2b-4b0f-a137-f2a862faac04" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Sep 18 19:29:25.250439 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:25.250118    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh"
Sep 18 19:29:30.250382 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:30.250338    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh" podUID="7deb9ac2-3d2b-4b0f-a137-f2a862faac04" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Sep 18 19:29:30.255056 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:30.255020    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh" podUID="7deb9ac2-3d2b-4b0f-a137-f2a862faac04" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:5000: connect: connection refused"
Sep 18 19:29:30.255419 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:30.255399    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh" podUID="7deb9ac2-3d2b-4b0f-a137-f2a862faac04" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:29:32.582151 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:32.582119    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq"]
Sep 18 19:29:32.582539 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:32.582483    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30c1e530-23e4-4ed6-bc64-c33f1371cd44" containerName="storage-initializer"
Sep 18 19:29:32.582539 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:32.582495    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="30c1e530-23e4-4ed6-bc64-c33f1371cd44" containerName="storage-initializer"
Sep 18 19:29:32.582539 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:32.582503    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30c1e530-23e4-4ed6-bc64-c33f1371cd44" containerName="kube-rbac-proxy"
Sep 18 19:29:32.582539 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:32.582515    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="30c1e530-23e4-4ed6-bc64-c33f1371cd44" containerName="kube-rbac-proxy"
Sep 18 19:29:32.582539 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:32.582525    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30c1e530-23e4-4ed6-bc64-c33f1371cd44" containerName="kserve-container"
Sep 18 19:29:32.582539 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:32.582532    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="30c1e530-23e4-4ed6-bc64-c33f1371cd44" containerName="kserve-container"
Sep 18 19:29:32.582539 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:32.582540    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30c1e530-23e4-4ed6-bc64-c33f1371cd44" containerName="agent"
Sep 18 19:29:32.582790 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:32.582545    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="30c1e530-23e4-4ed6-bc64-c33f1371cd44" containerName="agent"
Sep 18 19:29:32.582790 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:32.582642    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="30c1e530-23e4-4ed6-bc64-c33f1371cd44" containerName="agent"
Sep 18 19:29:32.582790 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:32.582651    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="30c1e530-23e4-4ed6-bc64-c33f1371cd44" containerName="kserve-container"
Sep 18 19:29:32.582790 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:32.582659    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="30c1e530-23e4-4ed6-bc64-c33f1371cd44" containerName="kube-rbac-proxy"
Sep 18 19:29:32.586015 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:32.585996    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq"
Sep 18 19:29:32.588688 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:32.588667    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Sep 18 19:29:32.588790 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:32.588728    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Sep 18 19:29:32.603118 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:32.603094    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq"]
Sep 18 19:29:32.727346 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:32.727312    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca87c0a8-7e84-4da0-82a4-c65a4b99879a-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-6b98ddf9c7-hkjkq\" (UID: \"ca87c0a8-7e84-4da0-82a4-c65a4b99879a\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq"
Sep 18 19:29:32.727527 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:32.727394    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca87c0a8-7e84-4da0-82a4-c65a4b99879a-cabundle-cert\") pod \"isvc-logger-predictor-6b98ddf9c7-hkjkq\" (UID: \"ca87c0a8-7e84-4da0-82a4-c65a4b99879a\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq"
Sep 18 19:29:32.727527 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:32.727440    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd6nx\" (UniqueName: \"kubernetes.io/projected/ca87c0a8-7e84-4da0-82a4-c65a4b99879a-kube-api-access-gd6nx\") pod \"isvc-logger-predictor-6b98ddf9c7-hkjkq\" (UID: \"ca87c0a8-7e84-4da0-82a4-c65a4b99879a\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq"
Sep 18 19:29:32.727527 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:32.727475    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca87c0a8-7e84-4da0-82a4-c65a4b99879a-kserve-provision-location\") pod \"isvc-logger-predictor-6b98ddf9c7-hkjkq\" (UID: \"ca87c0a8-7e84-4da0-82a4-c65a4b99879a\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq"
Sep 18 19:29:32.727527 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:32.727511    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca87c0a8-7e84-4da0-82a4-c65a4b99879a-proxy-tls\") pod \"isvc-logger-predictor-6b98ddf9c7-hkjkq\" (UID: \"ca87c0a8-7e84-4da0-82a4-c65a4b99879a\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq"
Sep 18 19:29:32.828502 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:32.828466    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca87c0a8-7e84-4da0-82a4-c65a4b99879a-cabundle-cert\") pod \"isvc-logger-predictor-6b98ddf9c7-hkjkq\" (UID: \"ca87c0a8-7e84-4da0-82a4-c65a4b99879a\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq"
Sep 18 19:29:32.828502 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:32.828507    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd6nx\" (UniqueName: \"kubernetes.io/projected/ca87c0a8-7e84-4da0-82a4-c65a4b99879a-kube-api-access-gd6nx\") pod \"isvc-logger-predictor-6b98ddf9c7-hkjkq\" (UID: \"ca87c0a8-7e84-4da0-82a4-c65a4b99879a\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq"
Sep 18 19:29:32.828775 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:32.828529    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca87c0a8-7e84-4da0-82a4-c65a4b99879a-kserve-provision-location\") pod \"isvc-logger-predictor-6b98ddf9c7-hkjkq\" (UID: \"ca87c0a8-7e84-4da0-82a4-c65a4b99879a\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq"
Sep 18 19:29:32.828775 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:32.828553    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca87c0a8-7e84-4da0-82a4-c65a4b99879a-proxy-tls\") pod \"isvc-logger-predictor-6b98ddf9c7-hkjkq\" (UID: \"ca87c0a8-7e84-4da0-82a4-c65a4b99879a\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq"
Sep 18 19:29:32.828775 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:32.828591    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca87c0a8-7e84-4da0-82a4-c65a4b99879a-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-6b98ddf9c7-hkjkq\" (UID: \"ca87c0a8-7e84-4da0-82a4-c65a4b99879a\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq"
Sep 18 19:29:32.829024 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:32.829003    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca87c0a8-7e84-4da0-82a4-c65a4b99879a-kserve-provision-location\") pod \"isvc-logger-predictor-6b98ddf9c7-hkjkq\" (UID: \"ca87c0a8-7e84-4da0-82a4-c65a4b99879a\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq"
Sep 18 19:29:32.829262 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:32.829241    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca87c0a8-7e84-4da0-82a4-c65a4b99879a-cabundle-cert\") pod \"isvc-logger-predictor-6b98ddf9c7-hkjkq\" (UID: \"ca87c0a8-7e84-4da0-82a4-c65a4b99879a\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq"
Sep 18 19:29:32.829346 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:32.829329    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca87c0a8-7e84-4da0-82a4-c65a4b99879a-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-6b98ddf9c7-hkjkq\" (UID: \"ca87c0a8-7e84-4da0-82a4-c65a4b99879a\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq"
Sep 18 19:29:32.831131 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:32.831104    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca87c0a8-7e84-4da0-82a4-c65a4b99879a-proxy-tls\") pod \"isvc-logger-predictor-6b98ddf9c7-hkjkq\" (UID: \"ca87c0a8-7e84-4da0-82a4-c65a4b99879a\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq"
Sep 18 19:29:32.838421 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:32.838367    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd6nx\" (UniqueName: \"kubernetes.io/projected/ca87c0a8-7e84-4da0-82a4-c65a4b99879a-kube-api-access-gd6nx\") pod \"isvc-logger-predictor-6b98ddf9c7-hkjkq\" (UID: \"ca87c0a8-7e84-4da0-82a4-c65a4b99879a\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq"
Sep 18 19:29:32.896003 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:32.895949    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq"
Sep 18 19:29:33.174295 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:33.174265    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq"]
Sep 18 19:29:33.175933 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:29:33.175902    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca87c0a8_7e84_4da0_82a4_c65a4b99879a.slice/crio-19a779109d50f7d286c7865f3bf65097b592dcaf544382bf9ec9c5aa07854a21 WatchSource:0}: Error finding container 19a779109d50f7d286c7865f3bf65097b592dcaf544382bf9ec9c5aa07854a21: Status 404 returned error can't find the container with id 19a779109d50f7d286c7865f3bf65097b592dcaf544382bf9ec9c5aa07854a21
Sep 18 19:29:33.775388 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:33.775349    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq" event={"ID":"ca87c0a8-7e84-4da0-82a4-c65a4b99879a","Type":"ContainerStarted","Data":"4f0d59112b314a608fd66af6aebf8562d1924149b25cc7fdfd853f71f6b9c74b"}
Sep 18 19:29:33.775804 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:33.775429    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq" event={"ID":"ca87c0a8-7e84-4da0-82a4-c65a4b99879a","Type":"ContainerStarted","Data":"19a779109d50f7d286c7865f3bf65097b592dcaf544382bf9ec9c5aa07854a21"}
Sep 18 19:29:35.249958 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:35.249913    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh" podUID="7deb9ac2-3d2b-4b0f-a137-f2a862faac04" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Sep 18 19:29:37.797146 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:37.797114    2522 generic.go:356] "Generic (PLEG): container finished" podID="ca87c0a8-7e84-4da0-82a4-c65a4b99879a" containerID="4f0d59112b314a608fd66af6aebf8562d1924149b25cc7fdfd853f71f6b9c74b" exitCode=0
Sep 18 19:29:37.797596 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:37.797163    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq" event={"ID":"ca87c0a8-7e84-4da0-82a4-c65a4b99879a","Type":"ContainerDied","Data":"4f0d59112b314a608fd66af6aebf8562d1924149b25cc7fdfd853f71f6b9c74b"}
Sep 18 19:29:38.804783 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:38.804747    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq" event={"ID":"ca87c0a8-7e84-4da0-82a4-c65a4b99879a","Type":"ContainerStarted","Data":"96b445e457b4337299db0d8036a8039462f37a42dd3fadcfd1cedfb855393990"}
Sep 18 19:29:38.805173 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:38.804812    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq" event={"ID":"ca87c0a8-7e84-4da0-82a4-c65a4b99879a","Type":"ContainerStarted","Data":"9bd138c9ce7494e76e42928f061f30ed64de009db386f1e6276f19de480618ec"}
Sep 18 19:29:38.805173 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:38.804837    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq" event={"ID":"ca87c0a8-7e84-4da0-82a4-c65a4b99879a","Type":"ContainerStarted","Data":"4f00a10a61ac7b882abaed2195fa89b29213235be59d6603a76ec89bd12e2c0a"}
Sep 18 19:29:38.805173 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:38.805167    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq"
Sep 18 19:29:38.805331 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:38.805218    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq"
Sep 18 19:29:38.806876 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:38.806840    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq" podUID="ca87c0a8-7e84-4da0-82a4-c65a4b99879a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 18 19:29:38.831918 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:38.831869    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq" podStartSLOduration=3.041486503 podStartE2EDuration="6.831854909s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.790368406s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:29:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:29:38.829273736 +0000 UTC m=+938.712800212" watchObservedRunningTime="2026-09-18 19:29:38.831854909 +0000 UTC m=+938.715381415"
Sep 18 19:29:39.810133 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:39.810095    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq"
Sep 18 19:29:39.810527 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:39.810213    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq" podUID="ca87c0a8-7e84-4da0-82a4-c65a4b99879a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 18 19:29:39.811369 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:39.811342    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq" podUID="ca87c0a8-7e84-4da0-82a4-c65a4b99879a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:29:40.249977 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:40.249937    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh" podUID="7deb9ac2-3d2b-4b0f-a137-f2a862faac04" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Sep 18 19:29:40.255336 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:40.255304    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh" podUID="7deb9ac2-3d2b-4b0f-a137-f2a862faac04" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:5000: connect: connection refused"
Sep 18 19:29:40.255461 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:40.255440    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh"
Sep 18 19:29:40.255793 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:40.255776    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh" podUID="7deb9ac2-3d2b-4b0f-a137-f2a862faac04" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:29:40.255861 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:40.255850    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh"
Sep 18 19:29:40.814376 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:40.814333    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq" podUID="ca87c0a8-7e84-4da0-82a4-c65a4b99879a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 18 19:29:40.814887 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:40.814831    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq" podUID="ca87c0a8-7e84-4da0-82a4-c65a4b99879a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:29:42.414007 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:42.413977    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh"
Sep 18 19:29:42.512577 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:42.512523    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7deb9ac2-3d2b-4b0f-a137-f2a862faac04-kserve-provision-location\") pod \"7deb9ac2-3d2b-4b0f-a137-f2a862faac04\" (UID: \"7deb9ac2-3d2b-4b0f-a137-f2a862faac04\") "
Sep 18 19:29:42.512759 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:42.512593    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-759kb\" (UniqueName: \"kubernetes.io/projected/7deb9ac2-3d2b-4b0f-a137-f2a862faac04-kube-api-access-759kb\") pod \"7deb9ac2-3d2b-4b0f-a137-f2a862faac04\" (UID: \"7deb9ac2-3d2b-4b0f-a137-f2a862faac04\") "
Sep 18 19:29:42.512759 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:42.512670    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7deb9ac2-3d2b-4b0f-a137-f2a862faac04-cabundle-cert\") pod \"7deb9ac2-3d2b-4b0f-a137-f2a862faac04\" (UID: \"7deb9ac2-3d2b-4b0f-a137-f2a862faac04\") "
Sep 18 19:29:42.512759 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:42.512696    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7deb9ac2-3d2b-4b0f-a137-f2a862faac04-proxy-tls\") pod \"7deb9ac2-3d2b-4b0f-a137-f2a862faac04\" (UID: \"7deb9ac2-3d2b-4b0f-a137-f2a862faac04\") "
Sep 18 19:29:42.512759 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:42.512717    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7deb9ac2-3d2b-4b0f-a137-f2a862faac04-kube-rbac-proxy-sar-config\") pod \"7deb9ac2-3d2b-4b0f-a137-f2a862faac04\" (UID: \"7deb9ac2-3d2b-4b0f-a137-f2a862faac04\") "
Sep 18 19:29:42.512947 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:42.512892    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7deb9ac2-3d2b-4b0f-a137-f2a862faac04-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7deb9ac2-3d2b-4b0f-a137-f2a862faac04" (UID: "7deb9ac2-3d2b-4b0f-a137-f2a862faac04"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:29:42.513129 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:42.513107    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7deb9ac2-3d2b-4b0f-a137-f2a862faac04-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7deb9ac2-3d2b-4b0f-a137-f2a862faac04" (UID: "7deb9ac2-3d2b-4b0f-a137-f2a862faac04"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:29:42.513198 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:42.513111    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7deb9ac2-3d2b-4b0f-a137-f2a862faac04-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7deb9ac2-3d2b-4b0f-a137-f2a862faac04" (UID: "7deb9ac2-3d2b-4b0f-a137-f2a862faac04"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:29:42.514856 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:42.514833    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7deb9ac2-3d2b-4b0f-a137-f2a862faac04-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7deb9ac2-3d2b-4b0f-a137-f2a862faac04" (UID: "7deb9ac2-3d2b-4b0f-a137-f2a862faac04"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:29:42.514932 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:42.514863    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7deb9ac2-3d2b-4b0f-a137-f2a862faac04-kube-api-access-759kb" (OuterVolumeSpecName: "kube-api-access-759kb") pod "7deb9ac2-3d2b-4b0f-a137-f2a862faac04" (UID: "7deb9ac2-3d2b-4b0f-a137-f2a862faac04"). InnerVolumeSpecName "kube-api-access-759kb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:29:42.613743 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:42.613704    2522 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7deb9ac2-3d2b-4b0f-a137-f2a862faac04-cabundle-cert\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:29:42.613743 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:42.613732    2522 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7deb9ac2-3d2b-4b0f-a137-f2a862faac04-proxy-tls\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:29:42.613743 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:42.613741    2522 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7deb9ac2-3d2b-4b0f-a137-f2a862faac04-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:29:42.613743 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:42.613750    2522 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7deb9ac2-3d2b-4b0f-a137-f2a862faac04-kserve-provision-location\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:29:42.614010 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:42.613760    2522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-759kb\" (UniqueName: \"kubernetes.io/projected/7deb9ac2-3d2b-4b0f-a137-f2a862faac04-kube-api-access-759kb\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:29:42.825117 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:42.825088    2522 generic.go:356] "Generic (PLEG): container finished" podID="7deb9ac2-3d2b-4b0f-a137-f2a862faac04" containerID="114d960f0d66de80cf01d1dbc37d975df8a64d8f93480468c2ebbf880b0deb53" exitCode=0
Sep 18 19:29:42.825280 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:42.825149    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh"
Sep 18 19:29:42.825280 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:42.825165    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh" event={"ID":"7deb9ac2-3d2b-4b0f-a137-f2a862faac04","Type":"ContainerDied","Data":"114d960f0d66de80cf01d1dbc37d975df8a64d8f93480468c2ebbf880b0deb53"}
Sep 18 19:29:42.825280 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:42.825227    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh" event={"ID":"7deb9ac2-3d2b-4b0f-a137-f2a862faac04","Type":"ContainerDied","Data":"ecaf535f16945792302382d96d673acf40bf74ee988f588607e7e7bef2b2e146"}
Sep 18 19:29:42.825280 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:42.825258    2522 scope.go:117] "RemoveContainer" containerID="114d960f0d66de80cf01d1dbc37d975df8a64d8f93480468c2ebbf880b0deb53"
Sep 18 19:29:42.835271 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:42.835249    2522 scope.go:117] "RemoveContainer" containerID="e6c037b1727d03151ebad40663aa622b16f90bc64a3ee0c70799648a3399ca15"
Sep 18 19:29:42.843020 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:42.843001    2522 scope.go:117] "RemoveContainer" containerID="6d3dfda860638178f2d0d6580b8570c40bcf0fafdb04f85a2b51d66950cf9c36"
Sep 18 19:29:42.845902 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:42.845875    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh"]
Sep 18 19:29:42.847527 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:42.847506    2522 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-86bd8b8bfb-5v5rh"]
Sep 18 19:29:42.851379 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:42.851363    2522 scope.go:117] "RemoveContainer" containerID="4b261efdf5ea29e3ffde93c1c07d3bd5867f03d8fe46095f9c9ce3bd249c6ba2"
Sep 18 19:29:42.858966 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:42.858939    2522 scope.go:117] "RemoveContainer" containerID="114d960f0d66de80cf01d1dbc37d975df8a64d8f93480468c2ebbf880b0deb53"
Sep 18 19:29:42.859237 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:29:42.859220    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"114d960f0d66de80cf01d1dbc37d975df8a64d8f93480468c2ebbf880b0deb53\": container with ID starting with 114d960f0d66de80cf01d1dbc37d975df8a64d8f93480468c2ebbf880b0deb53 not found: ID does not exist" containerID="114d960f0d66de80cf01d1dbc37d975df8a64d8f93480468c2ebbf880b0deb53"
Sep 18 19:29:42.859288 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:42.859247    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"114d960f0d66de80cf01d1dbc37d975df8a64d8f93480468c2ebbf880b0deb53"} err="failed to get container status \"114d960f0d66de80cf01d1dbc37d975df8a64d8f93480468c2ebbf880b0deb53\": rpc error: code = NotFound desc = could not find container \"114d960f0d66de80cf01d1dbc37d975df8a64d8f93480468c2ebbf880b0deb53\": container with ID starting with 114d960f0d66de80cf01d1dbc37d975df8a64d8f93480468c2ebbf880b0deb53 not found: ID does not exist"
Sep 18 19:29:42.859288 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:42.859267    2522 scope.go:117] "RemoveContainer" containerID="e6c037b1727d03151ebad40663aa622b16f90bc64a3ee0c70799648a3399ca15"
Sep 18 19:29:42.859521 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:29:42.859503    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6c037b1727d03151ebad40663aa622b16f90bc64a3ee0c70799648a3399ca15\": container with ID starting with e6c037b1727d03151ebad40663aa622b16f90bc64a3ee0c70799648a3399ca15 not found: ID does not exist" containerID="e6c037b1727d03151ebad40663aa622b16f90bc64a3ee0c70799648a3399ca15"
Sep 18 19:29:42.859585 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:42.859530    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6c037b1727d03151ebad40663aa622b16f90bc64a3ee0c70799648a3399ca15"} err="failed to get container status \"e6c037b1727d03151ebad40663aa622b16f90bc64a3ee0c70799648a3399ca15\": rpc error: code = NotFound desc = could not find container \"e6c037b1727d03151ebad40663aa622b16f90bc64a3ee0c70799648a3399ca15\": container with ID starting with e6c037b1727d03151ebad40663aa622b16f90bc64a3ee0c70799648a3399ca15 not found: ID does not exist"
Sep 18 19:29:42.859585 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:42.859549    2522 scope.go:117] "RemoveContainer" containerID="6d3dfda860638178f2d0d6580b8570c40bcf0fafdb04f85a2b51d66950cf9c36"
Sep 18 19:29:42.859795 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:29:42.859779    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d3dfda860638178f2d0d6580b8570c40bcf0fafdb04f85a2b51d66950cf9c36\": container with ID starting with 6d3dfda860638178f2d0d6580b8570c40bcf0fafdb04f85a2b51d66950cf9c36 not found: ID does not exist" containerID="6d3dfda860638178f2d0d6580b8570c40bcf0fafdb04f85a2b51d66950cf9c36"
Sep 18 19:29:42.859835 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:42.859799    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d3dfda860638178f2d0d6580b8570c40bcf0fafdb04f85a2b51d66950cf9c36"} err="failed to get container status \"6d3dfda860638178f2d0d6580b8570c40bcf0fafdb04f85a2b51d66950cf9c36\": rpc error: code = NotFound desc = could not find container \"6d3dfda860638178f2d0d6580b8570c40bcf0fafdb04f85a2b51d66950cf9c36\": container with ID starting with 6d3dfda860638178f2d0d6580b8570c40bcf0fafdb04f85a2b51d66950cf9c36 not found: ID does not exist"
Sep 18 19:29:42.859835 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:42.859812    2522 scope.go:117] "RemoveContainer" containerID="4b261efdf5ea29e3ffde93c1c07d3bd5867f03d8fe46095f9c9ce3bd249c6ba2"
Sep 18 19:29:42.860056 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:29:42.860041    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b261efdf5ea29e3ffde93c1c07d3bd5867f03d8fe46095f9c9ce3bd249c6ba2\": container with ID starting with 4b261efdf5ea29e3ffde93c1c07d3bd5867f03d8fe46095f9c9ce3bd249c6ba2 not found: ID does not exist" containerID="4b261efdf5ea29e3ffde93c1c07d3bd5867f03d8fe46095f9c9ce3bd249c6ba2"
Sep 18 19:29:42.860108 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:42.860057    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b261efdf5ea29e3ffde93c1c07d3bd5867f03d8fe46095f9c9ce3bd249c6ba2"} err="failed to get container status \"4b261efdf5ea29e3ffde93c1c07d3bd5867f03d8fe46095f9c9ce3bd249c6ba2\": rpc error: code = NotFound desc = could not find container \"4b261efdf5ea29e3ffde93c1c07d3bd5867f03d8fe46095f9c9ce3bd249c6ba2\": container with ID starting with 4b261efdf5ea29e3ffde93c1c07d3bd5867f03d8fe46095f9c9ce3bd249c6ba2 not found: ID does not exist"
Sep 18 19:29:44.722406 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:44.722376    2522 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7deb9ac2-3d2b-4b0f-a137-f2a862faac04" path="/var/lib/kubelet/pods/7deb9ac2-3d2b-4b0f-a137-f2a862faac04/volumes"
Sep 18 19:29:45.818260 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:45.818235    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq"
Sep 18 19:29:45.818708 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:45.818682    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq" podUID="ca87c0a8-7e84-4da0-82a4-c65a4b99879a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 18 19:29:45.818985 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:45.818964    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq" podUID="ca87c0a8-7e84-4da0-82a4-c65a4b99879a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:29:55.819619 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:55.819553    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq" podUID="ca87c0a8-7e84-4da0-82a4-c65a4b99879a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 18 19:29:55.820125 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:29:55.820030    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq" podUID="ca87c0a8-7e84-4da0-82a4-c65a4b99879a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:30:05.818782 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:30:05.818739    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq" podUID="ca87c0a8-7e84-4da0-82a4-c65a4b99879a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 18 19:30:05.819290 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:30:05.819231    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq" podUID="ca87c0a8-7e84-4da0-82a4-c65a4b99879a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:30:15.818863 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:30:15.818825    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq" podUID="ca87c0a8-7e84-4da0-82a4-c65a4b99879a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 18 19:30:15.819307 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:30:15.819216    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq" podUID="ca87c0a8-7e84-4da0-82a4-c65a4b99879a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:30:25.819023 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:30:25.818928    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq" podUID="ca87c0a8-7e84-4da0-82a4-c65a4b99879a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 18 19:30:25.819476 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:30:25.819443    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq" podUID="ca87c0a8-7e84-4da0-82a4-c65a4b99879a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:30:35.818904 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:30:35.818862    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq" podUID="ca87c0a8-7e84-4da0-82a4-c65a4b99879a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 18 19:30:35.819406 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:30:35.819278    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq" podUID="ca87c0a8-7e84-4da0-82a4-c65a4b99879a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:30:45.819049 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:30:45.819000    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq" podUID="ca87c0a8-7e84-4da0-82a4-c65a4b99879a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 18 19:30:45.819510 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:30:45.819347    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq" podUID="ca87c0a8-7e84-4da0-82a4-c65a4b99879a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:30:55.819515 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:30:55.819484    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq"
Sep 18 19:30:55.820050 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:30:55.819818    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq"
Sep 18 19:31:08.154632 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:08.154597    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq"]
Sep 18 19:31:08.155194 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:08.154844    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq" podUID="ca87c0a8-7e84-4da0-82a4-c65a4b99879a" containerName="kserve-container" containerID="cri-o://4f00a10a61ac7b882abaed2195fa89b29213235be59d6603a76ec89bd12e2c0a" gracePeriod=30
Sep 18 19:31:08.155194 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:08.154883    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq" podUID="ca87c0a8-7e84-4da0-82a4-c65a4b99879a" containerName="agent" containerID="cri-o://96b445e457b4337299db0d8036a8039462f37a42dd3fadcfd1cedfb855393990" gracePeriod=30
Sep 18 19:31:08.155194 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:08.154893    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq" podUID="ca87c0a8-7e84-4da0-82a4-c65a4b99879a" containerName="kube-rbac-proxy" containerID="cri-o://9bd138c9ce7494e76e42928f061f30ed64de009db386f1e6276f19de480618ec" gracePeriod=30
Sep 18 19:31:08.393514 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:08.393473    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7458d6ddd-6dqs7"]
Sep 18 19:31:08.394551 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:08.394523    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7deb9ac2-3d2b-4b0f-a137-f2a862faac04" containerName="kserve-container"
Sep 18 19:31:08.394730 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:08.394718    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="7deb9ac2-3d2b-4b0f-a137-f2a862faac04" containerName="kserve-container"
Sep 18 19:31:08.394843 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:08.394831    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7deb9ac2-3d2b-4b0f-a137-f2a862faac04" containerName="kube-rbac-proxy"
Sep 18 19:31:08.394921 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:08.394910    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="7deb9ac2-3d2b-4b0f-a137-f2a862faac04" containerName="kube-rbac-proxy"
Sep 18 19:31:08.395011 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:08.395002    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7deb9ac2-3d2b-4b0f-a137-f2a862faac04" containerName="storage-initializer"
Sep 18 19:31:08.395078 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:08.395070    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="7deb9ac2-3d2b-4b0f-a137-f2a862faac04" containerName="storage-initializer"
Sep 18 19:31:08.395154 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:08.395145    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7deb9ac2-3d2b-4b0f-a137-f2a862faac04" containerName="agent"
Sep 18 19:31:08.395216 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:08.395209    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="7deb9ac2-3d2b-4b0f-a137-f2a862faac04" containerName="agent"
Sep 18 19:31:08.395458 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:08.395446    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="7deb9ac2-3d2b-4b0f-a137-f2a862faac04" containerName="agent"
Sep 18 19:31:08.395746 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:08.395731    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="7deb9ac2-3d2b-4b0f-a137-f2a862faac04" containerName="kserve-container"
Sep 18 19:31:08.395842 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:08.395833    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="7deb9ac2-3d2b-4b0f-a137-f2a862faac04" containerName="kube-rbac-proxy"
Sep 18 19:31:08.400538 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:08.400513    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7458d6ddd-6dqs7"
Sep 18 19:31:08.403382 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:08.403362    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Sep 18 19:31:08.403735 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:08.403719    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Sep 18 19:31:08.410985 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:08.410966    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7458d6ddd-6dqs7"]
Sep 18 19:31:08.531174 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:08.531133    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8cff6a49-d996-4754-b0f5-ed4125173c9b-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7458d6ddd-6dqs7\" (UID: \"8cff6a49-d996-4754-b0f5-ed4125173c9b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7458d6ddd-6dqs7"
Sep 18 19:31:08.531174 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:08.531174    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bkdn\" (UniqueName: \"kubernetes.io/projected/8cff6a49-d996-4754-b0f5-ed4125173c9b-kube-api-access-6bkdn\") pod \"isvc-autogluon-v1-predictor-7458d6ddd-6dqs7\" (UID: \"8cff6a49-d996-4754-b0f5-ed4125173c9b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7458d6ddd-6dqs7"
Sep 18 19:31:08.531413 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:08.531286    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8cff6a49-d996-4754-b0f5-ed4125173c9b-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7458d6ddd-6dqs7\" (UID: \"8cff6a49-d996-4754-b0f5-ed4125173c9b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7458d6ddd-6dqs7"
Sep 18 19:31:08.531413 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:08.531324    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8cff6a49-d996-4754-b0f5-ed4125173c9b-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7458d6ddd-6dqs7\" (UID: \"8cff6a49-d996-4754-b0f5-ed4125173c9b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7458d6ddd-6dqs7"
Sep 18 19:31:08.531413 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:08.531353    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8cff6a49-d996-4754-b0f5-ed4125173c9b-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7458d6ddd-6dqs7\" (UID: \"8cff6a49-d996-4754-b0f5-ed4125173c9b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7458d6ddd-6dqs7"
Sep 18 19:31:08.632649 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:08.632602    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8cff6a49-d996-4754-b0f5-ed4125173c9b-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7458d6ddd-6dqs7\" (UID: \"8cff6a49-d996-4754-b0f5-ed4125173c9b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7458d6ddd-6dqs7"
Sep 18 19:31:08.632649 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:08.632651    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8cff6a49-d996-4754-b0f5-ed4125173c9b-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7458d6ddd-6dqs7\" (UID: \"8cff6a49-d996-4754-b0f5-ed4125173c9b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7458d6ddd-6dqs7"
Sep 18 19:31:08.632908 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:08.632691    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8cff6a49-d996-4754-b0f5-ed4125173c9b-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7458d6ddd-6dqs7\" (UID: \"8cff6a49-d996-4754-b0f5-ed4125173c9b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7458d6ddd-6dqs7"
Sep 18 19:31:08.632908 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:08.632748    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8cff6a49-d996-4754-b0f5-ed4125173c9b-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7458d6ddd-6dqs7\" (UID: \"8cff6a49-d996-4754-b0f5-ed4125173c9b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7458d6ddd-6dqs7"
Sep 18 19:31:08.632908 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:08.632768    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bkdn\" (UniqueName: \"kubernetes.io/projected/8cff6a49-d996-4754-b0f5-ed4125173c9b-kube-api-access-6bkdn\") pod \"isvc-autogluon-v1-predictor-7458d6ddd-6dqs7\" (UID: \"8cff6a49-d996-4754-b0f5-ed4125173c9b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7458d6ddd-6dqs7"
Sep 18 19:31:08.633125 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:08.633103    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8cff6a49-d996-4754-b0f5-ed4125173c9b-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7458d6ddd-6dqs7\" (UID: \"8cff6a49-d996-4754-b0f5-ed4125173c9b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7458d6ddd-6dqs7"
Sep 18 19:31:08.633416 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:08.633395    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8cff6a49-d996-4754-b0f5-ed4125173c9b-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7458d6ddd-6dqs7\" (UID: \"8cff6a49-d996-4754-b0f5-ed4125173c9b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7458d6ddd-6dqs7"
Sep 18 19:31:08.633665 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:08.633434    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8cff6a49-d996-4754-b0f5-ed4125173c9b-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7458d6ddd-6dqs7\" (UID: \"8cff6a49-d996-4754-b0f5-ed4125173c9b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7458d6ddd-6dqs7"
Sep 18 19:31:08.635473 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:08.635454    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8cff6a49-d996-4754-b0f5-ed4125173c9b-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7458d6ddd-6dqs7\" (UID: \"8cff6a49-d996-4754-b0f5-ed4125173c9b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7458d6ddd-6dqs7"
Sep 18 19:31:08.642247 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:08.642226    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bkdn\" (UniqueName: \"kubernetes.io/projected/8cff6a49-d996-4754-b0f5-ed4125173c9b-kube-api-access-6bkdn\") pod \"isvc-autogluon-v1-predictor-7458d6ddd-6dqs7\" (UID: \"8cff6a49-d996-4754-b0f5-ed4125173c9b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7458d6ddd-6dqs7"
Sep 18 19:31:08.711384 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:08.711293    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7458d6ddd-6dqs7"
Sep 18 19:31:08.992034 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:08.992003    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7458d6ddd-6dqs7"]
Sep 18 19:31:08.993779 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:31:08.993753    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cff6a49_d996_4754_b0f5_ed4125173c9b.slice/crio-5e72d13066221fb57372e01fcc84ec80a85b229a6268807afa373fa4b8cb0f0c WatchSource:0}: Error finding container 5e72d13066221fb57372e01fcc84ec80a85b229a6268807afa373fa4b8cb0f0c: Status 404 returned error can't find the container with id 5e72d13066221fb57372e01fcc84ec80a85b229a6268807afa373fa4b8cb0f0c
Sep 18 19:31:09.235330 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:09.235290    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7458d6ddd-6dqs7" event={"ID":"8cff6a49-d996-4754-b0f5-ed4125173c9b","Type":"ContainerStarted","Data":"466bfeb1e52cfbbabe8a17a5c3ad643f9b8e59abe28409272e64e82efe3918bd"}
Sep 18 19:31:09.235843 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:09.235351    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7458d6ddd-6dqs7" event={"ID":"8cff6a49-d996-4754-b0f5-ed4125173c9b","Type":"ContainerStarted","Data":"5e72d13066221fb57372e01fcc84ec80a85b229a6268807afa373fa4b8cb0f0c"}
Sep 18 19:31:09.238363 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:09.238329    2522 generic.go:356] "Generic (PLEG): container finished" podID="ca87c0a8-7e84-4da0-82a4-c65a4b99879a" containerID="9bd138c9ce7494e76e42928f061f30ed64de009db386f1e6276f19de480618ec" exitCode=2
Sep 18 19:31:09.238507 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:09.238391    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq" event={"ID":"ca87c0a8-7e84-4da0-82a4-c65a4b99879a","Type":"ContainerDied","Data":"9bd138c9ce7494e76e42928f061f30ed64de009db386f1e6276f19de480618ec"}
Sep 18 19:31:10.814629 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:10.814588    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq" podUID="ca87c0a8-7e84-4da0-82a4-c65a4b99879a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
Sep 18 19:31:13.258751 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:13.258671    2522 generic.go:356] "Generic (PLEG): container finished" podID="ca87c0a8-7e84-4da0-82a4-c65a4b99879a" containerID="4f00a10a61ac7b882abaed2195fa89b29213235be59d6603a76ec89bd12e2c0a" exitCode=0
Sep 18 19:31:13.259084 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:13.258741    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq" event={"ID":"ca87c0a8-7e84-4da0-82a4-c65a4b99879a","Type":"ContainerDied","Data":"4f00a10a61ac7b882abaed2195fa89b29213235be59d6603a76ec89bd12e2c0a"}
Sep 18 19:31:15.814856 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:15.814811    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq" podUID="ca87c0a8-7e84-4da0-82a4-c65a4b99879a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
Sep 18 19:31:15.819152 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:15.819126    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq" podUID="ca87c0a8-7e84-4da0-82a4-c65a4b99879a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 18 19:31:15.819490 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:15.819468    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq" podUID="ca87c0a8-7e84-4da0-82a4-c65a4b99879a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:31:20.815131 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:20.815073    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq" podUID="ca87c0a8-7e84-4da0-82a4-c65a4b99879a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
Sep 18 19:31:20.815599 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:20.815217    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq"
Sep 18 19:31:25.814871 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:25.814833    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq" podUID="ca87c0a8-7e84-4da0-82a4-c65a4b99879a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
Sep 18 19:31:25.819180 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:25.819156    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq" podUID="ca87c0a8-7e84-4da0-82a4-c65a4b99879a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 18 19:31:25.819551 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:25.819529    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq" podUID="ca87c0a8-7e84-4da0-82a4-c65a4b99879a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:31:30.351963 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:30.351929    2522 generic.go:356] "Generic (PLEG): container finished" podID="8cff6a49-d996-4754-b0f5-ed4125173c9b" containerID="466bfeb1e52cfbbabe8a17a5c3ad643f9b8e59abe28409272e64e82efe3918bd" exitCode=0
Sep 18 19:31:30.352410 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:30.352009    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7458d6ddd-6dqs7" event={"ID":"8cff6a49-d996-4754-b0f5-ed4125173c9b","Type":"ContainerDied","Data":"466bfeb1e52cfbbabe8a17a5c3ad643f9b8e59abe28409272e64e82efe3918bd"}
Sep 18 19:31:30.353354 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:30.353341    2522 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:31:30.815625 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:30.815532    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq" podUID="ca87c0a8-7e84-4da0-82a4-c65a4b99879a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
Sep 18 19:31:35.814871 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:35.814825    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq" podUID="ca87c0a8-7e84-4da0-82a4-c65a4b99879a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
Sep 18 19:31:35.819114 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:35.819087    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq" podUID="ca87c0a8-7e84-4da0-82a4-c65a4b99879a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 18 19:31:35.819252 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:35.819190    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq"
Sep 18 19:31:35.819545 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:35.819517    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq" podUID="ca87c0a8-7e84-4da0-82a4-c65a4b99879a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:31:35.819695 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:35.819645    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq"
Sep 18 19:31:38.331951 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:38.331925    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq"
Sep 18 19:31:38.381299 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:38.381265    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca87c0a8-7e84-4da0-82a4-c65a4b99879a-proxy-tls\") pod \"ca87c0a8-7e84-4da0-82a4-c65a4b99879a\" (UID: \"ca87c0a8-7e84-4da0-82a4-c65a4b99879a\") "
Sep 18 19:31:38.381458 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:38.381327    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca87c0a8-7e84-4da0-82a4-c65a4b99879a-cabundle-cert\") pod \"ca87c0a8-7e84-4da0-82a4-c65a4b99879a\" (UID: \"ca87c0a8-7e84-4da0-82a4-c65a4b99879a\") "
Sep 18 19:31:38.381458 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:38.381376    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca87c0a8-7e84-4da0-82a4-c65a4b99879a-kube-rbac-proxy-sar-config\") pod \"ca87c0a8-7e84-4da0-82a4-c65a4b99879a\" (UID: \"ca87c0a8-7e84-4da0-82a4-c65a4b99879a\") "
Sep 18 19:31:38.381458 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:38.381446    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gd6nx\" (UniqueName: \"kubernetes.io/projected/ca87c0a8-7e84-4da0-82a4-c65a4b99879a-kube-api-access-gd6nx\") pod \"ca87c0a8-7e84-4da0-82a4-c65a4b99879a\" (UID: \"ca87c0a8-7e84-4da0-82a4-c65a4b99879a\") "
Sep 18 19:31:38.381620 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:38.381477    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca87c0a8-7e84-4da0-82a4-c65a4b99879a-kserve-provision-location\") pod \"ca87c0a8-7e84-4da0-82a4-c65a4b99879a\" (UID: \"ca87c0a8-7e84-4da0-82a4-c65a4b99879a\") "
Sep 18 19:31:38.381849 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:38.381816    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca87c0a8-7e84-4da0-82a4-c65a4b99879a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ca87c0a8-7e84-4da0-82a4-c65a4b99879a" (UID: "ca87c0a8-7e84-4da0-82a4-c65a4b99879a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:31:38.381849 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:38.381821    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca87c0a8-7e84-4da0-82a4-c65a4b99879a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ca87c0a8-7e84-4da0-82a4-c65a4b99879a" (UID: "ca87c0a8-7e84-4da0-82a4-c65a4b99879a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:31:38.381849 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:38.381820    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca87c0a8-7e84-4da0-82a4-c65a4b99879a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ca87c0a8-7e84-4da0-82a4-c65a4b99879a" (UID: "ca87c0a8-7e84-4da0-82a4-c65a4b99879a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:31:38.383395 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:38.383373    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca87c0a8-7e84-4da0-82a4-c65a4b99879a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ca87c0a8-7e84-4da0-82a4-c65a4b99879a" (UID: "ca87c0a8-7e84-4da0-82a4-c65a4b99879a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:31:38.383498 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:38.383476    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca87c0a8-7e84-4da0-82a4-c65a4b99879a-kube-api-access-gd6nx" (OuterVolumeSpecName: "kube-api-access-gd6nx") pod "ca87c0a8-7e84-4da0-82a4-c65a4b99879a" (UID: "ca87c0a8-7e84-4da0-82a4-c65a4b99879a"). InnerVolumeSpecName "kube-api-access-gd6nx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:31:38.401953 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:38.401927    2522 generic.go:356] "Generic (PLEG): container finished" podID="ca87c0a8-7e84-4da0-82a4-c65a4b99879a" containerID="96b445e457b4337299db0d8036a8039462f37a42dd3fadcfd1cedfb855393990" exitCode=0
Sep 18 19:31:38.402097 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:38.401985    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq" event={"ID":"ca87c0a8-7e84-4da0-82a4-c65a4b99879a","Type":"ContainerDied","Data":"96b445e457b4337299db0d8036a8039462f37a42dd3fadcfd1cedfb855393990"}
Sep 18 19:31:38.402097 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:38.401991    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq"
Sep 18 19:31:38.402097 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:38.402055    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq" event={"ID":"ca87c0a8-7e84-4da0-82a4-c65a4b99879a","Type":"ContainerDied","Data":"19a779109d50f7d286c7865f3bf65097b592dcaf544382bf9ec9c5aa07854a21"}
Sep 18 19:31:38.402830 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:38.402100    2522 scope.go:117] "RemoveContainer" containerID="96b445e457b4337299db0d8036a8039462f37a42dd3fadcfd1cedfb855393990"
Sep 18 19:31:38.411284 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:38.411265    2522 scope.go:117] "RemoveContainer" containerID="9bd138c9ce7494e76e42928f061f30ed64de009db386f1e6276f19de480618ec"
Sep 18 19:31:38.419249 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:38.419233    2522 scope.go:117] "RemoveContainer" containerID="4f00a10a61ac7b882abaed2195fa89b29213235be59d6603a76ec89bd12e2c0a"
Sep 18 19:31:38.426119 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:38.426083    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq"]
Sep 18 19:31:38.428977 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:38.428957    2522 scope.go:117] "RemoveContainer" containerID="4f0d59112b314a608fd66af6aebf8562d1924149b25cc7fdfd853f71f6b9c74b"
Sep 18 19:31:38.433584 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:38.430598    2522 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6b98ddf9c7-hkjkq"]
Sep 18 19:31:38.442748 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:38.442722    2522 scope.go:117] "RemoveContainer" containerID="96b445e457b4337299db0d8036a8039462f37a42dd3fadcfd1cedfb855393990"
Sep 18 19:31:38.443123 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:31:38.443087    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96b445e457b4337299db0d8036a8039462f37a42dd3fadcfd1cedfb855393990\": container with ID starting with 96b445e457b4337299db0d8036a8039462f37a42dd3fadcfd1cedfb855393990 not found: ID does not exist" containerID="96b445e457b4337299db0d8036a8039462f37a42dd3fadcfd1cedfb855393990"
Sep 18 19:31:38.443212 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:38.443134    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96b445e457b4337299db0d8036a8039462f37a42dd3fadcfd1cedfb855393990"} err="failed to get container status \"96b445e457b4337299db0d8036a8039462f37a42dd3fadcfd1cedfb855393990\": rpc error: code = NotFound desc = could not find container \"96b445e457b4337299db0d8036a8039462f37a42dd3fadcfd1cedfb855393990\": container with ID starting with 96b445e457b4337299db0d8036a8039462f37a42dd3fadcfd1cedfb855393990 not found: ID does not exist"
Sep 18 19:31:38.443212 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:38.443162    2522 scope.go:117] "RemoveContainer" containerID="9bd138c9ce7494e76e42928f061f30ed64de009db386f1e6276f19de480618ec"
Sep 18 19:31:38.443511 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:31:38.443491    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bd138c9ce7494e76e42928f061f30ed64de009db386f1e6276f19de480618ec\": container with ID starting with 9bd138c9ce7494e76e42928f061f30ed64de009db386f1e6276f19de480618ec not found: ID does not exist" containerID="9bd138c9ce7494e76e42928f061f30ed64de009db386f1e6276f19de480618ec"
Sep 18 19:31:38.443618 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:38.443517    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bd138c9ce7494e76e42928f061f30ed64de009db386f1e6276f19de480618ec"} err="failed to get container status \"9bd138c9ce7494e76e42928f061f30ed64de009db386f1e6276f19de480618ec\": rpc error: code = NotFound desc = could not find container \"9bd138c9ce7494e76e42928f061f30ed64de009db386f1e6276f19de480618ec\": container with ID starting with 9bd138c9ce7494e76e42928f061f30ed64de009db386f1e6276f19de480618ec not found: ID does not exist"
Sep 18 19:31:38.443618 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:38.443532    2522 scope.go:117] "RemoveContainer" containerID="4f00a10a61ac7b882abaed2195fa89b29213235be59d6603a76ec89bd12e2c0a"
Sep 18 19:31:38.443854 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:31:38.443831    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f00a10a61ac7b882abaed2195fa89b29213235be59d6603a76ec89bd12e2c0a\": container with ID starting with 4f00a10a61ac7b882abaed2195fa89b29213235be59d6603a76ec89bd12e2c0a not found: ID does not exist" containerID="4f00a10a61ac7b882abaed2195fa89b29213235be59d6603a76ec89bd12e2c0a"
Sep 18 19:31:38.443928 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:38.443861    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f00a10a61ac7b882abaed2195fa89b29213235be59d6603a76ec89bd12e2c0a"} err="failed to get container status \"4f00a10a61ac7b882abaed2195fa89b29213235be59d6603a76ec89bd12e2c0a\": rpc error: code = NotFound desc = could not find container \"4f00a10a61ac7b882abaed2195fa89b29213235be59d6603a76ec89bd12e2c0a\": container with ID starting with 4f00a10a61ac7b882abaed2195fa89b29213235be59d6603a76ec89bd12e2c0a not found: ID does not exist"
Sep 18 19:31:38.443928 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:38.443881    2522 scope.go:117] "RemoveContainer" containerID="4f0d59112b314a608fd66af6aebf8562d1924149b25cc7fdfd853f71f6b9c74b"
Sep 18 19:31:38.444167 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:31:38.444147    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f0d59112b314a608fd66af6aebf8562d1924149b25cc7fdfd853f71f6b9c74b\": container with ID starting with 4f0d59112b314a608fd66af6aebf8562d1924149b25cc7fdfd853f71f6b9c74b not found: ID does not exist" containerID="4f0d59112b314a608fd66af6aebf8562d1924149b25cc7fdfd853f71f6b9c74b"
Sep 18 19:31:38.444277 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:38.444167    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f0d59112b314a608fd66af6aebf8562d1924149b25cc7fdfd853f71f6b9c74b"} err="failed to get container status \"4f0d59112b314a608fd66af6aebf8562d1924149b25cc7fdfd853f71f6b9c74b\": rpc error: code = NotFound desc = could not find container \"4f0d59112b314a608fd66af6aebf8562d1924149b25cc7fdfd853f71f6b9c74b\": container with ID starting with 4f0d59112b314a608fd66af6aebf8562d1924149b25cc7fdfd853f71f6b9c74b not found: ID does not exist"
Sep 18 19:31:38.482443 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:38.482403    2522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd6nx\" (UniqueName: \"kubernetes.io/projected/ca87c0a8-7e84-4da0-82a4-c65a4b99879a-kube-api-access-gd6nx\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:31:38.482443 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:38.482441    2522 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca87c0a8-7e84-4da0-82a4-c65a4b99879a-kserve-provision-location\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:31:38.482678 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:38.482457    2522 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca87c0a8-7e84-4da0-82a4-c65a4b99879a-proxy-tls\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:31:38.482678 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:38.482472    2522 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca87c0a8-7e84-4da0-82a4-c65a4b99879a-cabundle-cert\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:31:38.482678 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:38.482486    2522 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca87c0a8-7e84-4da0-82a4-c65a4b99879a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:31:38.722104 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:31:38.722015    2522 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca87c0a8-7e84-4da0-82a4-c65a4b99879a" path="/var/lib/kubelet/pods/ca87c0a8-7e84-4da0-82a4-c65a4b99879a/volumes"
Sep 18 19:33:26.980800 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:33:26.980751    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7458d6ddd-6dqs7" event={"ID":"8cff6a49-d996-4754-b0f5-ed4125173c9b","Type":"ContainerStarted","Data":"f2e90d18ea6f9bad9a07137043861e2467d4f6f4995a9674cf03772684dfe7c3"}
Sep 18 19:33:26.981212 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:33:26.980822    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7458d6ddd-6dqs7" event={"ID":"8cff6a49-d996-4754-b0f5-ed4125173c9b","Type":"ContainerStarted","Data":"dddecb2a47573cc78f66076c6c4bb2c681faa9761bbf111c2c2dc098042fc1cf"}
Sep 18 19:33:26.981212 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:33:26.980933    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7458d6ddd-6dqs7"
Sep 18 19:33:27.016431 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:33:27.016358    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7458d6ddd-6dqs7" podStartSLOduration=2.631303269 podStartE2EDuration="2m19.016343094s" totalImagesPullingTime="1m55.996118355s" totalInitContainerRuntime="20.38892147s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:31:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:33:27.015160451 +0000 UTC m=+1166.898686952" watchObservedRunningTime="2026-09-18 19:33:27.016343094 +0000 UTC m=+1166.899869598"
Sep 18 19:33:27.985624 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:33:27.985589    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7458d6ddd-6dqs7"
Sep 18 19:33:27.987181 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:33:27.987153    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7458d6ddd-6dqs7" podUID="8cff6a49-d996-4754-b0f5-ed4125173c9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 18 19:33:28.989893 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:33:28.989854    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7458d6ddd-6dqs7" podUID="8cff6a49-d996-4754-b0f5-ed4125173c9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 18 19:33:33.995054 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:33:33.995024    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7458d6ddd-6dqs7"
Sep 18 19:33:33.995663 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:33:33.995617    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7458d6ddd-6dqs7" podUID="8cff6a49-d996-4754-b0f5-ed4125173c9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 18 19:33:43.996545 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:33:43.996515    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7458d6ddd-6dqs7"
Sep 18 19:33:54.722135 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:33:54.722103    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7458d6ddd-6dqs7"]
Sep 18 19:33:54.722547 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:33:54.722347    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7458d6ddd-6dqs7" podUID="8cff6a49-d996-4754-b0f5-ed4125173c9b" containerName="kserve-container" containerID="cri-o://dddecb2a47573cc78f66076c6c4bb2c681faa9761bbf111c2c2dc098042fc1cf" gracePeriod=30
Sep 18 19:33:54.722547 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:33:54.722378    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7458d6ddd-6dqs7" podUID="8cff6a49-d996-4754-b0f5-ed4125173c9b" containerName="kube-rbac-proxy" containerID="cri-o://f2e90d18ea6f9bad9a07137043861e2467d4f6f4995a9674cf03772684dfe7c3" gracePeriod=30
Sep 18 19:33:55.122298 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:33:55.122261    2522 generic.go:356] "Generic (PLEG): container finished" podID="8cff6a49-d996-4754-b0f5-ed4125173c9b" containerID="f2e90d18ea6f9bad9a07137043861e2467d4f6f4995a9674cf03772684dfe7c3" exitCode=2
Sep 18 19:33:55.122485 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:33:55.122332    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7458d6ddd-6dqs7" event={"ID":"8cff6a49-d996-4754-b0f5-ed4125173c9b","Type":"ContainerDied","Data":"f2e90d18ea6f9bad9a07137043861e2467d4f6f4995a9674cf03772684dfe7c3"}
Sep 18 19:33:56.130572 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:33:56.130521    2522 generic.go:356] "Generic (PLEG): container finished" podID="8cff6a49-d996-4754-b0f5-ed4125173c9b" containerID="dddecb2a47573cc78f66076c6c4bb2c681faa9761bbf111c2c2dc098042fc1cf" exitCode=0
Sep 18 19:33:56.130941 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:33:56.130601    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7458d6ddd-6dqs7" event={"ID":"8cff6a49-d996-4754-b0f5-ed4125173c9b","Type":"ContainerDied","Data":"dddecb2a47573cc78f66076c6c4bb2c681faa9761bbf111c2c2dc098042fc1cf"}
Sep 18 19:33:56.203899 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:33:56.203871    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7458d6ddd-6dqs7"
Sep 18 19:33:56.220453 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:33:56.220387    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6bkdn\" (UniqueName: \"kubernetes.io/projected/8cff6a49-d996-4754-b0f5-ed4125173c9b-kube-api-access-6bkdn\") pod \"8cff6a49-d996-4754-b0f5-ed4125173c9b\" (UID: \"8cff6a49-d996-4754-b0f5-ed4125173c9b\") "
Sep 18 19:33:56.220453 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:33:56.220417    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8cff6a49-d996-4754-b0f5-ed4125173c9b-cabundle-cert\") pod \"8cff6a49-d996-4754-b0f5-ed4125173c9b\" (UID: \"8cff6a49-d996-4754-b0f5-ed4125173c9b\") "
Sep 18 19:33:56.220633 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:33:56.220463    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8cff6a49-d996-4754-b0f5-ed4125173c9b-proxy-tls\") pod \"8cff6a49-d996-4754-b0f5-ed4125173c9b\" (UID: \"8cff6a49-d996-4754-b0f5-ed4125173c9b\") "
Sep 18 19:33:56.220633 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:33:56.220496    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8cff6a49-d996-4754-b0f5-ed4125173c9b-kserve-provision-location\") pod \"8cff6a49-d996-4754-b0f5-ed4125173c9b\" (UID: \"8cff6a49-d996-4754-b0f5-ed4125173c9b\") "
Sep 18 19:33:56.220633 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:33:56.220524    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8cff6a49-d996-4754-b0f5-ed4125173c9b-kube-rbac-proxy-sar-config\") pod \"8cff6a49-d996-4754-b0f5-ed4125173c9b\" (UID: \"8cff6a49-d996-4754-b0f5-ed4125173c9b\") "
Sep 18 19:33:56.220908 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:33:56.220879    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cff6a49-d996-4754-b0f5-ed4125173c9b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8cff6a49-d996-4754-b0f5-ed4125173c9b" (UID: "8cff6a49-d996-4754-b0f5-ed4125173c9b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:33:56.220984 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:33:56.220938    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cff6a49-d996-4754-b0f5-ed4125173c9b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8cff6a49-d996-4754-b0f5-ed4125173c9b" (UID: "8cff6a49-d996-4754-b0f5-ed4125173c9b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:33:56.222512 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:33:56.222485    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cff6a49-d996-4754-b0f5-ed4125173c9b-kube-api-access-6bkdn" (OuterVolumeSpecName: "kube-api-access-6bkdn") pod "8cff6a49-d996-4754-b0f5-ed4125173c9b" (UID: "8cff6a49-d996-4754-b0f5-ed4125173c9b"). InnerVolumeSpecName "kube-api-access-6bkdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:33:56.222512 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:33:56.222499    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cff6a49-d996-4754-b0f5-ed4125173c9b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8cff6a49-d996-4754-b0f5-ed4125173c9b" (UID: "8cff6a49-d996-4754-b0f5-ed4125173c9b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:33:56.223812 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:33:56.223772    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cff6a49-d996-4754-b0f5-ed4125173c9b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8cff6a49-d996-4754-b0f5-ed4125173c9b" (UID: "8cff6a49-d996-4754-b0f5-ed4125173c9b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:33:56.321155 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:33:56.321116    2522 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8cff6a49-d996-4754-b0f5-ed4125173c9b-proxy-tls\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:33:56.321155 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:33:56.321149    2522 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8cff6a49-d996-4754-b0f5-ed4125173c9b-kserve-provision-location\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:33:56.321155 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:33:56.321158    2522 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8cff6a49-d996-4754-b0f5-ed4125173c9b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:33:56.321399 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:33:56.321168    2522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bkdn\" (UniqueName: \"kubernetes.io/projected/8cff6a49-d996-4754-b0f5-ed4125173c9b-kube-api-access-6bkdn\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:33:56.321399 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:33:56.321178    2522 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8cff6a49-d996-4754-b0f5-ed4125173c9b-cabundle-cert\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:33:57.137087 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:33:57.137055    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7458d6ddd-6dqs7"
Sep 18 19:33:57.137482 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:33:57.137054    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7458d6ddd-6dqs7" event={"ID":"8cff6a49-d996-4754-b0f5-ed4125173c9b","Type":"ContainerDied","Data":"5e72d13066221fb57372e01fcc84ec80a85b229a6268807afa373fa4b8cb0f0c"}
Sep 18 19:33:57.137482 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:33:57.137206    2522 scope.go:117] "RemoveContainer" containerID="f2e90d18ea6f9bad9a07137043861e2467d4f6f4995a9674cf03772684dfe7c3"
Sep 18 19:33:57.145703 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:33:57.145677    2522 scope.go:117] "RemoveContainer" containerID="dddecb2a47573cc78f66076c6c4bb2c681faa9761bbf111c2c2dc098042fc1cf"
Sep 18 19:33:57.153830 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:33:57.153811    2522 scope.go:117] "RemoveContainer" containerID="466bfeb1e52cfbbabe8a17a5c3ad643f9b8e59abe28409272e64e82efe3918bd"
Sep 18 19:33:57.158326 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:33:57.158304    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7458d6ddd-6dqs7"]
Sep 18 19:33:57.163609 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:33:57.163591    2522 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7458d6ddd-6dqs7"]
Sep 18 19:33:58.722783 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:33:58.722745    2522 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cff6a49-d996-4754-b0f5-ed4125173c9b" path="/var/lib/kubelet/pods/8cff6a49-d996-4754-b0f5-ed4125173c9b/volumes"
Sep 18 19:33:59.980766 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:33:59.980729    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67fdc6896b-skxsr"]
Sep 18 19:33:59.981126 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:33:59.981078    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca87c0a8-7e84-4da0-82a4-c65a4b99879a" containerName="agent"
Sep 18 19:33:59.981126 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:33:59.981090    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca87c0a8-7e84-4da0-82a4-c65a4b99879a" containerName="agent"
Sep 18 19:33:59.981126 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:33:59.981104    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca87c0a8-7e84-4da0-82a4-c65a4b99879a" containerName="kserve-container"
Sep 18 19:33:59.981126 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:33:59.981110    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca87c0a8-7e84-4da0-82a4-c65a4b99879a" containerName="kserve-container"
Sep 18 19:33:59.981126 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:33:59.981117    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cff6a49-d996-4754-b0f5-ed4125173c9b" containerName="kserve-container"
Sep 18 19:33:59.981126 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:33:59.981123    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cff6a49-d996-4754-b0f5-ed4125173c9b" containerName="kserve-container"
Sep 18 19:33:59.981315 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:33:59.981130    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca87c0a8-7e84-4da0-82a4-c65a4b99879a" containerName="storage-initializer"
Sep 18 19:33:59.981315 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:33:59.981135    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca87c0a8-7e84-4da0-82a4-c65a4b99879a" containerName="storage-initializer"
Sep 18 19:33:59.981315 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:33:59.981143    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cff6a49-d996-4754-b0f5-ed4125173c9b" containerName="storage-initializer"
Sep 18 19:33:59.981315 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:33:59.981150    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cff6a49-d996-4754-b0f5-ed4125173c9b" containerName="storage-initializer"
Sep 18 19:33:59.981315 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:33:59.981158    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cff6a49-d996-4754-b0f5-ed4125173c9b" containerName="kube-rbac-proxy"
Sep 18 19:33:59.981315 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:33:59.981163    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cff6a49-d996-4754-b0f5-ed4125173c9b" containerName="kube-rbac-proxy"
Sep 18 19:33:59.981315 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:33:59.981172    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca87c0a8-7e84-4da0-82a4-c65a4b99879a" containerName="kube-rbac-proxy"
Sep 18 19:33:59.981315 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:33:59.981177    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca87c0a8-7e84-4da0-82a4-c65a4b99879a" containerName="kube-rbac-proxy"
Sep 18 19:33:59.981315 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:33:59.981230    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca87c0a8-7e84-4da0-82a4-c65a4b99879a" containerName="agent"
Sep 18 19:33:59.981315 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:33:59.981239    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cff6a49-d996-4754-b0f5-ed4125173c9b" containerName="kserve-container"
Sep 18 19:33:59.981315 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:33:59.981245    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca87c0a8-7e84-4da0-82a4-c65a4b99879a" containerName="kserve-container"
Sep 18 19:33:59.981315 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:33:59.981250    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cff6a49-d996-4754-b0f5-ed4125173c9b" containerName="kube-rbac-proxy"
Sep 18 19:33:59.981315 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:33:59.981256    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca87c0a8-7e84-4da0-82a4-c65a4b99879a" containerName="kube-rbac-proxy"
Sep 18 19:34:00.008889 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:34:00.008866    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67fdc6896b-skxsr"
Sep 18 19:34:00.013398 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:34:00.013378    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 19:34:00.013592 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:34:00.013382    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 19:34:00.013592 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:34:00.013578    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Sep 18 19:34:00.013689 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:34:00.013623    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 19:34:00.013735 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:34:00.013714    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Sep 18 19:34:00.014342 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:34:00.014326    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 19:34:00.015527 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:34:00.015510    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5b562\""
Sep 18 19:34:00.027094 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:34:00.027076    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67fdc6896b-skxsr"]
Sep 18 19:34:00.048257 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:34:00.048235    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bcf4a7a-cd35-4ca3-a007-a9179a25e820-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-67fdc6896b-skxsr\" (UID: \"7bcf4a7a-cd35-4ca3-a007-a9179a25e820\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67fdc6896b-skxsr"
Sep 18 19:34:00.048348 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:34:00.048265    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bcf4a7a-cd35-4ca3-a007-a9179a25e820-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67fdc6896b-skxsr\" (UID: \"7bcf4a7a-cd35-4ca3-a007-a9179a25e820\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67fdc6896b-skxsr"
Sep 18 19:34:00.048348 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:34:00.048298    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bcf4a7a-cd35-4ca3-a007-a9179a25e820-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-67fdc6896b-skxsr\" (UID: \"7bcf4a7a-cd35-4ca3-a007-a9179a25e820\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67fdc6896b-skxsr"
Sep 18 19:34:00.048348 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:34:00.048341    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7bcf4a7a-cd35-4ca3-a007-a9179a25e820-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-67fdc6896b-skxsr\" (UID: \"7bcf4a7a-cd35-4ca3-a007-a9179a25e820\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67fdc6896b-skxsr"
Sep 18 19:34:00.048474 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:34:00.048356    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrmgl\" (UniqueName: \"kubernetes.io/projected/7bcf4a7a-cd35-4ca3-a007-a9179a25e820-kube-api-access-wrmgl\") pod \"isvc-autogluon-v2-predictor-67fdc6896b-skxsr\" (UID: \"7bcf4a7a-cd35-4ca3-a007-a9179a25e820\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67fdc6896b-skxsr"
Sep 18 19:34:00.149116 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:34:00.149089    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bcf4a7a-cd35-4ca3-a007-a9179a25e820-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-67fdc6896b-skxsr\" (UID: \"7bcf4a7a-cd35-4ca3-a007-a9179a25e820\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67fdc6896b-skxsr"
Sep 18 19:34:00.149246 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:34:00.149140    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7bcf4a7a-cd35-4ca3-a007-a9179a25e820-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-67fdc6896b-skxsr\" (UID: \"7bcf4a7a-cd35-4ca3-a007-a9179a25e820\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67fdc6896b-skxsr"
Sep 18 19:34:00.149246 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:34:00.149163    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrmgl\" (UniqueName: \"kubernetes.io/projected/7bcf4a7a-cd35-4ca3-a007-a9179a25e820-kube-api-access-wrmgl\") pod \"isvc-autogluon-v2-predictor-67fdc6896b-skxsr\" (UID: \"7bcf4a7a-cd35-4ca3-a007-a9179a25e820\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67fdc6896b-skxsr"
Sep 18 19:34:00.149246 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:34:00.149208    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bcf4a7a-cd35-4ca3-a007-a9179a25e820-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-67fdc6896b-skxsr\" (UID: \"7bcf4a7a-cd35-4ca3-a007-a9179a25e820\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67fdc6896b-skxsr"
Sep 18 19:34:00.149246 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:34:00.149238    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bcf4a7a-cd35-4ca3-a007-a9179a25e820-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67fdc6896b-skxsr\" (UID: \"7bcf4a7a-cd35-4ca3-a007-a9179a25e820\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67fdc6896b-skxsr"
Sep 18 19:34:00.149528 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:34:00.149510    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bcf4a7a-cd35-4ca3-a007-a9179a25e820-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-67fdc6896b-skxsr\" (UID: \"7bcf4a7a-cd35-4ca3-a007-a9179a25e820\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67fdc6896b-skxsr"
Sep 18 19:34:00.149930 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:34:00.149907    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7bcf4a7a-cd35-4ca3-a007-a9179a25e820-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-67fdc6896b-skxsr\" (UID: \"7bcf4a7a-cd35-4ca3-a007-a9179a25e820\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67fdc6896b-skxsr"
Sep 18 19:34:00.150030 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:34:00.149909    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bcf4a7a-cd35-4ca3-a007-a9179a25e820-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-67fdc6896b-skxsr\" (UID: \"7bcf4a7a-cd35-4ca3-a007-a9179a25e820\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67fdc6896b-skxsr"
Sep 18 19:34:00.151806 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:34:00.151786    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bcf4a7a-cd35-4ca3-a007-a9179a25e820-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67fdc6896b-skxsr\" (UID: \"7bcf4a7a-cd35-4ca3-a007-a9179a25e820\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67fdc6896b-skxsr"
Sep 18 19:34:00.163761 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:34:00.163737    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrmgl\" (UniqueName: \"kubernetes.io/projected/7bcf4a7a-cd35-4ca3-a007-a9179a25e820-kube-api-access-wrmgl\") pod \"isvc-autogluon-v2-predictor-67fdc6896b-skxsr\" (UID: \"7bcf4a7a-cd35-4ca3-a007-a9179a25e820\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67fdc6896b-skxsr"
Sep 18 19:34:00.319186 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:34:00.319161    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67fdc6896b-skxsr"
Sep 18 19:34:00.607721 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:34:00.607693    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67fdc6896b-skxsr"]
Sep 18 19:34:00.609986 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:34:00.609960    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bcf4a7a_cd35_4ca3_a007_a9179a25e820.slice/crio-67bad4b35b952033602b66ebf0daa818e50cc4505b5b43552163184bd3411eef WatchSource:0}: Error finding container 67bad4b35b952033602b66ebf0daa818e50cc4505b5b43552163184bd3411eef: Status 404 returned error can't find the container with id 67bad4b35b952033602b66ebf0daa818e50cc4505b5b43552163184bd3411eef
Sep 18 19:34:00.614608 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:34:00.614585    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 19:34:01.164751 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:34:01.164718    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67fdc6896b-skxsr" event={"ID":"7bcf4a7a-cd35-4ca3-a007-a9179a25e820","Type":"ContainerStarted","Data":"a33709badba0a0850d83719919459331f8022c7bed0891befe537b8c1b423537"}
Sep 18 19:34:01.165159 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:34:01.164786    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67fdc6896b-skxsr" event={"ID":"7bcf4a7a-cd35-4ca3-a007-a9179a25e820","Type":"ContainerStarted","Data":"67bad4b35b952033602b66ebf0daa818e50cc4505b5b43552163184bd3411eef"}
Sep 18 19:34:30.310510 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:34:30.310408    2522 generic.go:356] "Generic (PLEG): container finished" podID="7bcf4a7a-cd35-4ca3-a007-a9179a25e820" containerID="a33709badba0a0850d83719919459331f8022c7bed0891befe537b8c1b423537" exitCode=0
Sep 18 19:34:30.310510 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:34:30.310433    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67fdc6896b-skxsr" event={"ID":"7bcf4a7a-cd35-4ca3-a007-a9179a25e820","Type":"ContainerDied","Data":"a33709badba0a0850d83719919459331f8022c7bed0891befe537b8c1b423537"}
Sep 18 19:34:31.318003 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:34:31.317971    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67fdc6896b-skxsr" event={"ID":"7bcf4a7a-cd35-4ca3-a007-a9179a25e820","Type":"ContainerStarted","Data":"5da56df90346d9996ab984f6c775a9ce1d8fb9ac9929226bdd92df77ae4da709"}
Sep 18 19:34:31.318388 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:34:31.318027    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67fdc6896b-skxsr" event={"ID":"7bcf4a7a-cd35-4ca3-a007-a9179a25e820","Type":"ContainerStarted","Data":"2aae0e927df2219b57b6614655c882d686cf88ee1f8ed3074649578b10d6d68d"}
Sep 18 19:34:31.318388 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:34:31.318210    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67fdc6896b-skxsr"
Sep 18 19:34:31.344465 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:34:31.344422    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67fdc6896b-skxsr" podStartSLOduration=3.052256831 podStartE2EDuration="32.34440403s" totalImagesPullingTime="0s" totalInitContainerRuntime="29.292147199s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:33:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:34:31.343373862 +0000 UTC m=+1231.226900331" watchObservedRunningTime="2026-09-18 19:34:31.34440403 +0000 UTC m=+1231.227930491"
Sep 18 19:34:32.323522 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:34:32.323485    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67fdc6896b-skxsr"
Sep 18 19:34:38.333547 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:34:38.333522    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67fdc6896b-skxsr"
Sep 18 19:36:02.750034 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:02.750002    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67fdc6896b-skxsr"
Sep 18 19:36:13.758751 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:13.758700    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67fdc6896b-skxsr" podUID="7bcf4a7a-cd35-4ca3-a007-a9179a25e820" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.39:8080/v2/models/isvc-autogluon-v2/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 18 19:36:16.110269 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:16.110238    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67fdc6896b-skxsr"]
Sep 18 19:36:16.110705 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:16.110469    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67fdc6896b-skxsr" podUID="7bcf4a7a-cd35-4ca3-a007-a9179a25e820" containerName="kserve-container" containerID="cri-o://2aae0e927df2219b57b6614655c882d686cf88ee1f8ed3074649578b10d6d68d" gracePeriod=30
Sep 18 19:36:16.110705 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:16.110531    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67fdc6896b-skxsr" podUID="7bcf4a7a-cd35-4ca3-a007-a9179a25e820" containerName="kube-rbac-proxy" containerID="cri-o://5da56df90346d9996ab984f6c775a9ce1d8fb9ac9929226bdd92df77ae4da709" gracePeriod=30
Sep 18 19:36:16.801159 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:16.801120    2522 generic.go:356] "Generic (PLEG): container finished" podID="7bcf4a7a-cd35-4ca3-a007-a9179a25e820" containerID="5da56df90346d9996ab984f6c775a9ce1d8fb9ac9929226bdd92df77ae4da709" exitCode=2
Sep 18 19:36:16.801333 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:16.801170    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67fdc6896b-skxsr" event={"ID":"7bcf4a7a-cd35-4ca3-a007-a9179a25e820","Type":"ContainerDied","Data":"5da56df90346d9996ab984f6c775a9ce1d8fb9ac9929226bdd92df77ae4da709"}
Sep 18 19:36:17.556865 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:17.556841    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67fdc6896b-skxsr"
Sep 18 19:36:17.675516 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:17.675426    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bcf4a7a-cd35-4ca3-a007-a9179a25e820-kserve-provision-location\") pod \"7bcf4a7a-cd35-4ca3-a007-a9179a25e820\" (UID: \"7bcf4a7a-cd35-4ca3-a007-a9179a25e820\") "
Sep 18 19:36:17.675516 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:17.675482    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bcf4a7a-cd35-4ca3-a007-a9179a25e820-kube-rbac-proxy-sar-config\") pod \"7bcf4a7a-cd35-4ca3-a007-a9179a25e820\" (UID: \"7bcf4a7a-cd35-4ca3-a007-a9179a25e820\") "
Sep 18 19:36:17.675516 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:17.675515    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrmgl\" (UniqueName: \"kubernetes.io/projected/7bcf4a7a-cd35-4ca3-a007-a9179a25e820-kube-api-access-wrmgl\") pod \"7bcf4a7a-cd35-4ca3-a007-a9179a25e820\" (UID: \"7bcf4a7a-cd35-4ca3-a007-a9179a25e820\") "
Sep 18 19:36:17.675816 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:17.675545    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7bcf4a7a-cd35-4ca3-a007-a9179a25e820-cabundle-cert\") pod \"7bcf4a7a-cd35-4ca3-a007-a9179a25e820\" (UID: \"7bcf4a7a-cd35-4ca3-a007-a9179a25e820\") "
Sep 18 19:36:17.675816 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:17.675586    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bcf4a7a-cd35-4ca3-a007-a9179a25e820-proxy-tls\") pod \"7bcf4a7a-cd35-4ca3-a007-a9179a25e820\" (UID: \"7bcf4a7a-cd35-4ca3-a007-a9179a25e820\") "
Sep 18 19:36:17.675935 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:17.675894    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bcf4a7a-cd35-4ca3-a007-a9179a25e820-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7bcf4a7a-cd35-4ca3-a007-a9179a25e820" (UID: "7bcf4a7a-cd35-4ca3-a007-a9179a25e820"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:36:17.675992 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:17.675943    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bcf4a7a-cd35-4ca3-a007-a9179a25e820-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7bcf4a7a-cd35-4ca3-a007-a9179a25e820" (UID: "7bcf4a7a-cd35-4ca3-a007-a9179a25e820"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:36:17.677824 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:17.677792    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bcf4a7a-cd35-4ca3-a007-a9179a25e820-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7bcf4a7a-cd35-4ca3-a007-a9179a25e820" (UID: "7bcf4a7a-cd35-4ca3-a007-a9179a25e820"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:36:17.677947 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:17.677893    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bcf4a7a-cd35-4ca3-a007-a9179a25e820-kube-api-access-wrmgl" (OuterVolumeSpecName: "kube-api-access-wrmgl") pod "7bcf4a7a-cd35-4ca3-a007-a9179a25e820" (UID: "7bcf4a7a-cd35-4ca3-a007-a9179a25e820"). InnerVolumeSpecName "kube-api-access-wrmgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:36:17.679931 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:17.679909    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bcf4a7a-cd35-4ca3-a007-a9179a25e820-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7bcf4a7a-cd35-4ca3-a007-a9179a25e820" (UID: "7bcf4a7a-cd35-4ca3-a007-a9179a25e820"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:36:17.776200 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:17.776161    2522 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bcf4a7a-cd35-4ca3-a007-a9179a25e820-kserve-provision-location\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:36:17.776200 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:17.776191    2522 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bcf4a7a-cd35-4ca3-a007-a9179a25e820-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:36:17.776200 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:17.776204    2522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrmgl\" (UniqueName: \"kubernetes.io/projected/7bcf4a7a-cd35-4ca3-a007-a9179a25e820-kube-api-access-wrmgl\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:36:17.776461 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:17.776215    2522 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7bcf4a7a-cd35-4ca3-a007-a9179a25e820-cabundle-cert\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:36:17.776461 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:17.776224    2522 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bcf4a7a-cd35-4ca3-a007-a9179a25e820-proxy-tls\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:36:17.807949 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:17.807914    2522 generic.go:356] "Generic (PLEG): container finished" podID="7bcf4a7a-cd35-4ca3-a007-a9179a25e820" containerID="2aae0e927df2219b57b6614655c882d686cf88ee1f8ed3074649578b10d6d68d" exitCode=0
Sep 18 19:36:17.808117 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:17.807983    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67fdc6896b-skxsr"
Sep 18 19:36:17.808117 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:17.808001    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67fdc6896b-skxsr" event={"ID":"7bcf4a7a-cd35-4ca3-a007-a9179a25e820","Type":"ContainerDied","Data":"2aae0e927df2219b57b6614655c882d686cf88ee1f8ed3074649578b10d6d68d"}
Sep 18 19:36:17.808117 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:17.808075    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67fdc6896b-skxsr" event={"ID":"7bcf4a7a-cd35-4ca3-a007-a9179a25e820","Type":"ContainerDied","Data":"67bad4b35b952033602b66ebf0daa818e50cc4505b5b43552163184bd3411eef"}
Sep 18 19:36:17.808117 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:17.808114    2522 scope.go:117] "RemoveContainer" containerID="5da56df90346d9996ab984f6c775a9ce1d8fb9ac9929226bdd92df77ae4da709"
Sep 18 19:36:17.816996 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:17.816973    2522 scope.go:117] "RemoveContainer" containerID="2aae0e927df2219b57b6614655c882d686cf88ee1f8ed3074649578b10d6d68d"
Sep 18 19:36:17.824739 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:17.824707    2522 scope.go:117] "RemoveContainer" containerID="a33709badba0a0850d83719919459331f8022c7bed0891befe537b8c1b423537"
Sep 18 19:36:17.831706 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:17.831682    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67fdc6896b-skxsr"]
Sep 18 19:36:17.834419 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:17.834397    2522 scope.go:117] "RemoveContainer" containerID="5da56df90346d9996ab984f6c775a9ce1d8fb9ac9929226bdd92df77ae4da709"
Sep 18 19:36:17.834861 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:36:17.834822    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5da56df90346d9996ab984f6c775a9ce1d8fb9ac9929226bdd92df77ae4da709\": container with ID starting with 5da56df90346d9996ab984f6c775a9ce1d8fb9ac9929226bdd92df77ae4da709 not found: ID does not exist" containerID="5da56df90346d9996ab984f6c775a9ce1d8fb9ac9929226bdd92df77ae4da709"
Sep 18 19:36:17.834968 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:17.834872    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5da56df90346d9996ab984f6c775a9ce1d8fb9ac9929226bdd92df77ae4da709"} err="failed to get container status \"5da56df90346d9996ab984f6c775a9ce1d8fb9ac9929226bdd92df77ae4da709\": rpc error: code = NotFound desc = could not find container \"5da56df90346d9996ab984f6c775a9ce1d8fb9ac9929226bdd92df77ae4da709\": container with ID starting with 5da56df90346d9996ab984f6c775a9ce1d8fb9ac9929226bdd92df77ae4da709 not found: ID does not exist"
Sep 18 19:36:17.834968 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:17.834898    2522 scope.go:117] "RemoveContainer" containerID="2aae0e927df2219b57b6614655c882d686cf88ee1f8ed3074649578b10d6d68d"
Sep 18 19:36:17.835280 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:36:17.835249    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2aae0e927df2219b57b6614655c882d686cf88ee1f8ed3074649578b10d6d68d\": container with ID starting with 2aae0e927df2219b57b6614655c882d686cf88ee1f8ed3074649578b10d6d68d not found: ID does not exist" containerID="2aae0e927df2219b57b6614655c882d686cf88ee1f8ed3074649578b10d6d68d"
Sep 18 19:36:17.835370 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:17.835289    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2aae0e927df2219b57b6614655c882d686cf88ee1f8ed3074649578b10d6d68d"} err="failed to get container status \"2aae0e927df2219b57b6614655c882d686cf88ee1f8ed3074649578b10d6d68d\": rpc error: code = NotFound desc = could not find container \"2aae0e927df2219b57b6614655c882d686cf88ee1f8ed3074649578b10d6d68d\": container with ID starting with 2aae0e927df2219b57b6614655c882d686cf88ee1f8ed3074649578b10d6d68d not found: ID does not exist"
Sep 18 19:36:17.835370 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:17.835313    2522 scope.go:117] "RemoveContainer" containerID="a33709badba0a0850d83719919459331f8022c7bed0891befe537b8c1b423537"
Sep 18 19:36:17.836023 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:36:17.835995    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a33709badba0a0850d83719919459331f8022c7bed0891befe537b8c1b423537\": container with ID starting with a33709badba0a0850d83719919459331f8022c7bed0891befe537b8c1b423537 not found: ID does not exist" containerID="a33709badba0a0850d83719919459331f8022c7bed0891befe537b8c1b423537"
Sep 18 19:36:17.836144 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:17.836030    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a33709badba0a0850d83719919459331f8022c7bed0891befe537b8c1b423537"} err="failed to get container status \"a33709badba0a0850d83719919459331f8022c7bed0891befe537b8c1b423537\": rpc error: code = NotFound desc = could not find container \"a33709badba0a0850d83719919459331f8022c7bed0891befe537b8c1b423537\": container with ID starting with a33709badba0a0850d83719919459331f8022c7bed0891befe537b8c1b423537 not found: ID does not exist"
Sep 18 19:36:17.836220 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:17.836196    2522 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67fdc6896b-skxsr"]
Sep 18 19:36:18.722180 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:18.722142    2522 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bcf4a7a-cd35-4ca3-a007-a9179a25e820" path="/var/lib/kubelet/pods/7bcf4a7a-cd35-4ca3-a007-a9179a25e820/volumes"
Sep 18 19:36:21.373996 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:21.373960    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m"]
Sep 18 19:36:21.374435 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:21.374353    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bcf4a7a-cd35-4ca3-a007-a9179a25e820" containerName="kube-rbac-proxy"
Sep 18 19:36:21.374435 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:21.374366    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bcf4a7a-cd35-4ca3-a007-a9179a25e820" containerName="kube-rbac-proxy"
Sep 18 19:36:21.374435 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:21.374387    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bcf4a7a-cd35-4ca3-a007-a9179a25e820" containerName="storage-initializer"
Sep 18 19:36:21.374435 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:21.374393    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bcf4a7a-cd35-4ca3-a007-a9179a25e820" containerName="storage-initializer"
Sep 18 19:36:21.374435 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:21.374400    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bcf4a7a-cd35-4ca3-a007-a9179a25e820" containerName="kserve-container"
Sep 18 19:36:21.374435 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:21.374405    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bcf4a7a-cd35-4ca3-a007-a9179a25e820" containerName="kserve-container"
Sep 18 19:36:21.374796 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:21.374457    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bcf4a7a-cd35-4ca3-a007-a9179a25e820" containerName="kserve-container"
Sep 18 19:36:21.374796 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:21.374470    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bcf4a7a-cd35-4ca3-a007-a9179a25e820" containerName="kube-rbac-proxy"
Sep 18 19:36:21.379271 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:21.379250    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m"
Sep 18 19:36:21.383764 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:21.383741    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 19:36:21.383885 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:21.383786    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 19:36:21.383885 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:21.383822    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Sep 18 19:36:21.383885 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:21.383850    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 19:36:21.384034 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:21.384021    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 19:36:21.384151 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:21.384140    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5b562\""
Sep 18 19:36:21.384193 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:21.384152    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Sep 18 19:36:21.396109 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:21.396084    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m"]
Sep 18 19:36:21.509714 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:21.509678    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e03af1eb-6e02-41ed-b61c-0d7611e815b2-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m\" (UID: \"e03af1eb-6e02-41ed-b61c-0d7611e815b2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m"
Sep 18 19:36:21.509892 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:21.509797    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e03af1eb-6e02-41ed-b61c-0d7611e815b2-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m\" (UID: \"e03af1eb-6e02-41ed-b61c-0d7611e815b2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m"
Sep 18 19:36:21.509892 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:21.509836    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e03af1eb-6e02-41ed-b61c-0d7611e815b2-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m\" (UID: \"e03af1eb-6e02-41ed-b61c-0d7611e815b2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m"
Sep 18 19:36:21.509892 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:21.509855    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbdc8\" (UniqueName: \"kubernetes.io/projected/e03af1eb-6e02-41ed-b61c-0d7611e815b2-kube-api-access-dbdc8\") pod \"isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m\" (UID: \"e03af1eb-6e02-41ed-b61c-0d7611e815b2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m"
Sep 18 19:36:21.509892 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:21.509877    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e03af1eb-6e02-41ed-b61c-0d7611e815b2-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m\" (UID: \"e03af1eb-6e02-41ed-b61c-0d7611e815b2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m"
Sep 18 19:36:21.610459 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:21.610425    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e03af1eb-6e02-41ed-b61c-0d7611e815b2-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m\" (UID: \"e03af1eb-6e02-41ed-b61c-0d7611e815b2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m"
Sep 18 19:36:21.610684 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:21.610612    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e03af1eb-6e02-41ed-b61c-0d7611e815b2-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m\" (UID: \"e03af1eb-6e02-41ed-b61c-0d7611e815b2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m"
Sep 18 19:36:21.610684 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:21.610656    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e03af1eb-6e02-41ed-b61c-0d7611e815b2-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m\" (UID: \"e03af1eb-6e02-41ed-b61c-0d7611e815b2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m"
Sep 18 19:36:21.610684 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:21.610674    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbdc8\" (UniqueName: \"kubernetes.io/projected/e03af1eb-6e02-41ed-b61c-0d7611e815b2-kube-api-access-dbdc8\") pod \"isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m\" (UID: \"e03af1eb-6e02-41ed-b61c-0d7611e815b2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m"
Sep 18 19:36:21.610871 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:21.610694    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e03af1eb-6e02-41ed-b61c-0d7611e815b2-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m\" (UID: \"e03af1eb-6e02-41ed-b61c-0d7611e815b2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m"
Sep 18 19:36:21.610948 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:21.610925    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e03af1eb-6e02-41ed-b61c-0d7611e815b2-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m\" (UID: \"e03af1eb-6e02-41ed-b61c-0d7611e815b2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m"
Sep 18 19:36:21.611224 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:21.611199    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e03af1eb-6e02-41ed-b61c-0d7611e815b2-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m\" (UID: \"e03af1eb-6e02-41ed-b61c-0d7611e815b2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m"
Sep 18 19:36:21.611327 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:21.611233    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e03af1eb-6e02-41ed-b61c-0d7611e815b2-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m\" (UID: \"e03af1eb-6e02-41ed-b61c-0d7611e815b2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m"
Sep 18 19:36:21.613080 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:21.613059    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e03af1eb-6e02-41ed-b61c-0d7611e815b2-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m\" (UID: \"e03af1eb-6e02-41ed-b61c-0d7611e815b2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m"
Sep 18 19:36:21.620295 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:21.620276    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbdc8\" (UniqueName: \"kubernetes.io/projected/e03af1eb-6e02-41ed-b61c-0d7611e815b2-kube-api-access-dbdc8\") pod \"isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m\" (UID: \"e03af1eb-6e02-41ed-b61c-0d7611e815b2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m"
Sep 18 19:36:21.689493 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:21.689406    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m"
Sep 18 19:36:21.962399 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:21.962325    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m"]
Sep 18 19:36:21.965478 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:36:21.965449    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode03af1eb_6e02_41ed_b61c_0d7611e815b2.slice/crio-76d63ca9938520161f750e7cd408b540e9911961c717b8a833da0fa1fd9eb3ea WatchSource:0}: Error finding container 76d63ca9938520161f750e7cd408b540e9911961c717b8a833da0fa1fd9eb3ea: Status 404 returned error can't find the container with id 76d63ca9938520161f750e7cd408b540e9911961c717b8a833da0fa1fd9eb3ea
Sep 18 19:36:22.834221 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:22.834179    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m" event={"ID":"e03af1eb-6e02-41ed-b61c-0d7611e815b2","Type":"ContainerStarted","Data":"cc04caacde256610f55c8b87f3604bb92e53150f66c6ba05d384d454d5a19d00"}
Sep 18 19:36:22.834675 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:22.834244    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m" event={"ID":"e03af1eb-6e02-41ed-b61c-0d7611e815b2","Type":"ContainerStarted","Data":"76d63ca9938520161f750e7cd408b540e9911961c717b8a833da0fa1fd9eb3ea"}
Sep 18 19:36:27.858805 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:27.858773    2522 generic.go:356] "Generic (PLEG): container finished" podID="e03af1eb-6e02-41ed-b61c-0d7611e815b2" containerID="cc04caacde256610f55c8b87f3604bb92e53150f66c6ba05d384d454d5a19d00" exitCode=0
Sep 18 19:36:27.859141 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:27.858848    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m" event={"ID":"e03af1eb-6e02-41ed-b61c-0d7611e815b2","Type":"ContainerDied","Data":"cc04caacde256610f55c8b87f3604bb92e53150f66c6ba05d384d454d5a19d00"}
Sep 18 19:36:28.865420 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:28.865385    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m" event={"ID":"e03af1eb-6e02-41ed-b61c-0d7611e815b2","Type":"ContainerStarted","Data":"702a2860feaf19b213e97fba35532ffe086d5e04ed7844e5a47d7ef69e2d30fa"}
Sep 18 19:36:28.865904 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:28.865457    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m" event={"ID":"e03af1eb-6e02-41ed-b61c-0d7611e815b2","Type":"ContainerStarted","Data":"4cdfeb3a5deb24cf03216fa0d0adbe1050355d8b5aa6d11d79814c8151be01cf"}
Sep 18 19:36:28.865904 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:28.865692    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m"
Sep 18 19:36:28.899515 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:28.899466    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m" podStartSLOduration=2.176495493 podStartE2EDuration="7.89945127s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.722955777s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:36:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:36:28.899024655 +0000 UTC m=+1348.782551154" watchObservedRunningTime="2026-09-18 19:36:28.89945127 +0000 UTC m=+1348.782977732"
Sep 18 19:36:29.870969 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:29.870937    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m"
Sep 18 19:36:35.882383 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:35.882349    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m"
Sep 18 19:36:50.064591 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:50.064538    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-7857ffc886-cghqq"]
Sep 18 19:36:50.068253 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:50.068238    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7857ffc886-cghqq"
Sep 18 19:36:50.071526 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:50.071503    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-predictor-serving-cert\""
Sep 18 19:36:50.071942 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:50.071925    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
Sep 18 19:36:50.083398 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:50.083372    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-7857ffc886-cghqq"]
Sep 18 19:36:50.139697 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:50.139676    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e95f8be-691b-4d5f-b83a-032e1bef58ce-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-7857ffc886-cghqq\" (UID: \"2e95f8be-691b-4d5f-b83a-032e1bef58ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7857ffc886-cghqq"
Sep 18 19:36:50.139815 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:50.139722    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e95f8be-691b-4d5f-b83a-032e1bef58ce-proxy-tls\") pod \"isvc-sklearn-predictor-7857ffc886-cghqq\" (UID: \"2e95f8be-691b-4d5f-b83a-032e1bef58ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7857ffc886-cghqq"
Sep 18 19:36:50.139815 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:50.139744    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e95f8be-691b-4d5f-b83a-032e1bef58ce-cabundle-cert\") pod \"isvc-sklearn-predictor-7857ffc886-cghqq\" (UID: \"2e95f8be-691b-4d5f-b83a-032e1bef58ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7857ffc886-cghqq"
Sep 18 19:36:50.139815 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:50.139793    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e95f8be-691b-4d5f-b83a-032e1bef58ce-kserve-provision-location\") pod \"isvc-sklearn-predictor-7857ffc886-cghqq\" (UID: \"2e95f8be-691b-4d5f-b83a-032e1bef58ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7857ffc886-cghqq"
Sep 18 19:36:50.139975 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:50.139861    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzjh9\" (UniqueName: \"kubernetes.io/projected/2e95f8be-691b-4d5f-b83a-032e1bef58ce-kube-api-access-nzjh9\") pod \"isvc-sklearn-predictor-7857ffc886-cghqq\" (UID: \"2e95f8be-691b-4d5f-b83a-032e1bef58ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7857ffc886-cghqq"
Sep 18 19:36:50.240304 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:50.240279    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzjh9\" (UniqueName: \"kubernetes.io/projected/2e95f8be-691b-4d5f-b83a-032e1bef58ce-kube-api-access-nzjh9\") pod \"isvc-sklearn-predictor-7857ffc886-cghqq\" (UID: \"2e95f8be-691b-4d5f-b83a-032e1bef58ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7857ffc886-cghqq"
Sep 18 19:36:50.240382 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:50.240318    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e95f8be-691b-4d5f-b83a-032e1bef58ce-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-7857ffc886-cghqq\" (UID: \"2e95f8be-691b-4d5f-b83a-032e1bef58ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7857ffc886-cghqq"
Sep 18 19:36:50.240382 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:50.240361    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e95f8be-691b-4d5f-b83a-032e1bef58ce-proxy-tls\") pod \"isvc-sklearn-predictor-7857ffc886-cghqq\" (UID: \"2e95f8be-691b-4d5f-b83a-032e1bef58ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7857ffc886-cghqq"
Sep 18 19:36:50.240461 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:50.240382    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e95f8be-691b-4d5f-b83a-032e1bef58ce-cabundle-cert\") pod \"isvc-sklearn-predictor-7857ffc886-cghqq\" (UID: \"2e95f8be-691b-4d5f-b83a-032e1bef58ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7857ffc886-cghqq"
Sep 18 19:36:50.240461 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:50.240399    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e95f8be-691b-4d5f-b83a-032e1bef58ce-kserve-provision-location\") pod \"isvc-sklearn-predictor-7857ffc886-cghqq\" (UID: \"2e95f8be-691b-4d5f-b83a-032e1bef58ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7857ffc886-cghqq"
Sep 18 19:36:50.240586 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:36:50.240508    2522 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-predictor-serving-cert: secret "isvc-sklearn-predictor-serving-cert" not found
Sep 18 19:36:50.240586 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:36:50.240575    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e95f8be-691b-4d5f-b83a-032e1bef58ce-proxy-tls podName:2e95f8be-691b-4d5f-b83a-032e1bef58ce nodeName:}" failed. No retries permitted until 2026-09-18 19:36:50.740542986 +0000 UTC m=+1370.624069437 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2e95f8be-691b-4d5f-b83a-032e1bef58ce-proxy-tls") pod "isvc-sklearn-predictor-7857ffc886-cghqq" (UID: "2e95f8be-691b-4d5f-b83a-032e1bef58ce") : secret "isvc-sklearn-predictor-serving-cert" not found
Sep 18 19:36:50.240861 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:50.240837    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e95f8be-691b-4d5f-b83a-032e1bef58ce-kserve-provision-location\") pod \"isvc-sklearn-predictor-7857ffc886-cghqq\" (UID: \"2e95f8be-691b-4d5f-b83a-032e1bef58ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7857ffc886-cghqq"
Sep 18 19:36:50.241104 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:50.241085    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e95f8be-691b-4d5f-b83a-032e1bef58ce-cabundle-cert\") pod \"isvc-sklearn-predictor-7857ffc886-cghqq\" (UID: \"2e95f8be-691b-4d5f-b83a-032e1bef58ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7857ffc886-cghqq"
Sep 18 19:36:50.241216 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:50.241167    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e95f8be-691b-4d5f-b83a-032e1bef58ce-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-7857ffc886-cghqq\" (UID: \"2e95f8be-691b-4d5f-b83a-032e1bef58ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7857ffc886-cghqq"
Sep 18 19:36:50.251782 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:50.251764    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzjh9\" (UniqueName: \"kubernetes.io/projected/2e95f8be-691b-4d5f-b83a-032e1bef58ce-kube-api-access-nzjh9\") pod \"isvc-sklearn-predictor-7857ffc886-cghqq\" (UID: \"2e95f8be-691b-4d5f-b83a-032e1bef58ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7857ffc886-cghqq"
Sep 18 19:36:50.743822 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:50.743782    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e95f8be-691b-4d5f-b83a-032e1bef58ce-proxy-tls\") pod \"isvc-sklearn-predictor-7857ffc886-cghqq\" (UID: \"2e95f8be-691b-4d5f-b83a-032e1bef58ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7857ffc886-cghqq"
Sep 18 19:36:50.746178 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:50.746155    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e95f8be-691b-4d5f-b83a-032e1bef58ce-proxy-tls\") pod \"isvc-sklearn-predictor-7857ffc886-cghqq\" (UID: \"2e95f8be-691b-4d5f-b83a-032e1bef58ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7857ffc886-cghqq"
Sep 18 19:36:50.978732 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:50.978700    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7857ffc886-cghqq"
Sep 18 19:36:51.267223 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:51.267200    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-7857ffc886-cghqq"]
Sep 18 19:36:51.269769 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:36:51.269739    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e95f8be_691b_4d5f_b83a_032e1bef58ce.slice/crio-d9ded66a18772c1c3b1aa8cf332089c0986a893bfe8d1930673638515503a472 WatchSource:0}: Error finding container d9ded66a18772c1c3b1aa8cf332089c0986a893bfe8d1930673638515503a472: Status 404 returned error can't find the container with id d9ded66a18772c1c3b1aa8cf332089c0986a893bfe8d1930673638515503a472
Sep 18 19:36:51.976472 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:51.976438    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7857ffc886-cghqq" event={"ID":"2e95f8be-691b-4d5f-b83a-032e1bef58ce","Type":"ContainerStarted","Data":"3d8fa5d7ecc293a1845e73098a1f11d2e21cdb0409a9311197c0f24be4549045"}
Sep 18 19:36:51.976660 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:51.976506    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7857ffc886-cghqq" event={"ID":"2e95f8be-691b-4d5f-b83a-032e1bef58ce","Type":"ContainerStarted","Data":"d9ded66a18772c1c3b1aa8cf332089c0986a893bfe8d1930673638515503a472"}
Sep 18 19:36:55.995486 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:55.995446    2522 generic.go:356] "Generic (PLEG): container finished" podID="2e95f8be-691b-4d5f-b83a-032e1bef58ce" containerID="3d8fa5d7ecc293a1845e73098a1f11d2e21cdb0409a9311197c0f24be4549045" exitCode=0
Sep 18 19:36:55.996077 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:55.995530    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7857ffc886-cghqq" event={"ID":"2e95f8be-691b-4d5f-b83a-032e1bef58ce","Type":"ContainerDied","Data":"3d8fa5d7ecc293a1845e73098a1f11d2e21cdb0409a9311197c0f24be4549045"}
Sep 18 19:36:57.003117 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:57.003083    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7857ffc886-cghqq" event={"ID":"2e95f8be-691b-4d5f-b83a-032e1bef58ce","Type":"ContainerStarted","Data":"84740521da22ccc4492b2a7379ca27c8d06eb6ceb3a4e5f089e82f0197422082"}
Sep 18 19:36:57.003495 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:57.003151    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7857ffc886-cghqq" event={"ID":"2e95f8be-691b-4d5f-b83a-032e1bef58ce","Type":"ContainerStarted","Data":"78cfc2f758aec185e761f59c0ce9af38b6c5f99df3fa574d5a4b977a3a7f8592"}
Sep 18 19:36:57.003495 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:57.003339    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7857ffc886-cghqq"
Sep 18 19:36:57.037915 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:57.037867    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7857ffc886-cghqq" podStartSLOduration=3.303483217 podStartE2EDuration="7.037851281s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.734368064s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:36:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:36:57.036221212 +0000 UTC m=+1376.919747700" watchObservedRunningTime="2026-09-18 19:36:57.037851281 +0000 UTC m=+1376.921377769"
Sep 18 19:36:58.008602 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:58.008549    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7857ffc886-cghqq"
Sep 18 19:36:58.009970 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:58.009940    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7857ffc886-cghqq" podUID="2e95f8be-691b-4d5f-b83a-032e1bef58ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 18 19:36:59.013916 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:36:59.013875    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7857ffc886-cghqq" podUID="2e95f8be-691b-4d5f-b83a-032e1bef58ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 18 19:37:04.018870 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:37:04.018838    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7857ffc886-cghqq"
Sep 18 19:37:04.019358 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:37:04.019328    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7857ffc886-cghqq" podUID="2e95f8be-691b-4d5f-b83a-032e1bef58ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 18 19:37:14.019250 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:37:14.019204    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7857ffc886-cghqq" podUID="2e95f8be-691b-4d5f-b83a-032e1bef58ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 18 19:37:24.019657 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:37:24.019607    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7857ffc886-cghqq" podUID="2e95f8be-691b-4d5f-b83a-032e1bef58ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 18 19:37:34.020032 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:37:34.019990    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7857ffc886-cghqq" podUID="2e95f8be-691b-4d5f-b83a-032e1bef58ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 18 19:37:44.019262 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:37:44.019217    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7857ffc886-cghqq" podUID="2e95f8be-691b-4d5f-b83a-032e1bef58ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 18 19:37:54.019736 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:37:54.019641    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7857ffc886-cghqq" podUID="2e95f8be-691b-4d5f-b83a-032e1bef58ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 18 19:38:01.749833 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:01.749802    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m"
Sep 18 19:38:04.019306 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:04.019269    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7857ffc886-cghqq" podUID="2e95f8be-691b-4d5f-b83a-032e1bef58ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 18 19:38:12.758762 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:12.758715    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m" podUID="e03af1eb-6e02-41ed-b61c-0d7611e815b2" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.40:8080/v2/models/isvc-autogluon-v2-variants/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 18 19:38:14.021388 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:14.021355    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7857ffc886-cghqq"
Sep 18 19:38:17.916107 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:17.916073    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m"]
Sep 18 19:38:17.916617 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:17.916315    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m" podUID="e03af1eb-6e02-41ed-b61c-0d7611e815b2" containerName="kserve-container" containerID="cri-o://4cdfeb3a5deb24cf03216fa0d0adbe1050355d8b5aa6d11d79814c8151be01cf" gracePeriod=30
Sep 18 19:38:17.916617 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:17.916369    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m" podUID="e03af1eb-6e02-41ed-b61c-0d7611e815b2" containerName="kube-rbac-proxy" containerID="cri-o://702a2860feaf19b213e97fba35532ffe086d5e04ed7844e5a47d7ef69e2d30fa" gracePeriod=30
Sep 18 19:38:18.391856 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:18.391821    2522 generic.go:356] "Generic (PLEG): container finished" podID="e03af1eb-6e02-41ed-b61c-0d7611e815b2" containerID="702a2860feaf19b213e97fba35532ffe086d5e04ed7844e5a47d7ef69e2d30fa" exitCode=2
Sep 18 19:38:18.392034 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:18.391896    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m" event={"ID":"e03af1eb-6e02-41ed-b61c-0d7611e815b2","Type":"ContainerDied","Data":"702a2860feaf19b213e97fba35532ffe086d5e04ed7844e5a47d7ef69e2d30fa"}
Sep 18 19:38:19.355541 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:19.355518    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m"
Sep 18 19:38:19.399551 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:19.399523    2522 generic.go:356] "Generic (PLEG): container finished" podID="e03af1eb-6e02-41ed-b61c-0d7611e815b2" containerID="4cdfeb3a5deb24cf03216fa0d0adbe1050355d8b5aa6d11d79814c8151be01cf" exitCode=0
Sep 18 19:38:19.399725 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:19.399601    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m"
Sep 18 19:38:19.399725 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:19.399603    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m" event={"ID":"e03af1eb-6e02-41ed-b61c-0d7611e815b2","Type":"ContainerDied","Data":"4cdfeb3a5deb24cf03216fa0d0adbe1050355d8b5aa6d11d79814c8151be01cf"}
Sep 18 19:38:19.399725 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:19.399666    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m" event={"ID":"e03af1eb-6e02-41ed-b61c-0d7611e815b2","Type":"ContainerDied","Data":"76d63ca9938520161f750e7cd408b540e9911961c717b8a833da0fa1fd9eb3ea"}
Sep 18 19:38:19.399725 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:19.399693    2522 scope.go:117] "RemoveContainer" containerID="702a2860feaf19b213e97fba35532ffe086d5e04ed7844e5a47d7ef69e2d30fa"
Sep 18 19:38:19.407959 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:19.407945    2522 scope.go:117] "RemoveContainer" containerID="4cdfeb3a5deb24cf03216fa0d0adbe1050355d8b5aa6d11d79814c8151be01cf"
Sep 18 19:38:19.415211 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:19.415196    2522 scope.go:117] "RemoveContainer" containerID="cc04caacde256610f55c8b87f3604bb92e53150f66c6ba05d384d454d5a19d00"
Sep 18 19:38:19.422152 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:19.422138    2522 scope.go:117] "RemoveContainer" containerID="702a2860feaf19b213e97fba35532ffe086d5e04ed7844e5a47d7ef69e2d30fa"
Sep 18 19:38:19.422377 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:38:19.422359    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"702a2860feaf19b213e97fba35532ffe086d5e04ed7844e5a47d7ef69e2d30fa\": container with ID starting with 702a2860feaf19b213e97fba35532ffe086d5e04ed7844e5a47d7ef69e2d30fa not found: ID does not exist" containerID="702a2860feaf19b213e97fba35532ffe086d5e04ed7844e5a47d7ef69e2d30fa"
Sep 18 19:38:19.422422 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:19.422386    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"702a2860feaf19b213e97fba35532ffe086d5e04ed7844e5a47d7ef69e2d30fa"} err="failed to get container status \"702a2860feaf19b213e97fba35532ffe086d5e04ed7844e5a47d7ef69e2d30fa\": rpc error: code = NotFound desc = could not find container \"702a2860feaf19b213e97fba35532ffe086d5e04ed7844e5a47d7ef69e2d30fa\": container with ID starting with 702a2860feaf19b213e97fba35532ffe086d5e04ed7844e5a47d7ef69e2d30fa not found: ID does not exist"
Sep 18 19:38:19.422422 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:19.422402    2522 scope.go:117] "RemoveContainer" containerID="4cdfeb3a5deb24cf03216fa0d0adbe1050355d8b5aa6d11d79814c8151be01cf"
Sep 18 19:38:19.422639 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:38:19.422623    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cdfeb3a5deb24cf03216fa0d0adbe1050355d8b5aa6d11d79814c8151be01cf\": container with ID starting with 4cdfeb3a5deb24cf03216fa0d0adbe1050355d8b5aa6d11d79814c8151be01cf not found: ID does not exist" containerID="4cdfeb3a5deb24cf03216fa0d0adbe1050355d8b5aa6d11d79814c8151be01cf"
Sep 18 19:38:19.422684 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:19.422643    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cdfeb3a5deb24cf03216fa0d0adbe1050355d8b5aa6d11d79814c8151be01cf"} err="failed to get container status \"4cdfeb3a5deb24cf03216fa0d0adbe1050355d8b5aa6d11d79814c8151be01cf\": rpc error: code = NotFound desc = could not find container \"4cdfeb3a5deb24cf03216fa0d0adbe1050355d8b5aa6d11d79814c8151be01cf\": container with ID starting with 4cdfeb3a5deb24cf03216fa0d0adbe1050355d8b5aa6d11d79814c8151be01cf not found: ID does not exist"
Sep 18 19:38:19.422684 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:19.422657    2522 scope.go:117] "RemoveContainer" containerID="cc04caacde256610f55c8b87f3604bb92e53150f66c6ba05d384d454d5a19d00"
Sep 18 19:38:19.422882 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:38:19.422862    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc04caacde256610f55c8b87f3604bb92e53150f66c6ba05d384d454d5a19d00\": container with ID starting with cc04caacde256610f55c8b87f3604bb92e53150f66c6ba05d384d454d5a19d00 not found: ID does not exist" containerID="cc04caacde256610f55c8b87f3604bb92e53150f66c6ba05d384d454d5a19d00"
Sep 18 19:38:19.422989 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:19.422884    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc04caacde256610f55c8b87f3604bb92e53150f66c6ba05d384d454d5a19d00"} err="failed to get container status \"cc04caacde256610f55c8b87f3604bb92e53150f66c6ba05d384d454d5a19d00\": rpc error: code = NotFound desc = could not find container \"cc04caacde256610f55c8b87f3604bb92e53150f66c6ba05d384d454d5a19d00\": container with ID starting with cc04caacde256610f55c8b87f3604bb92e53150f66c6ba05d384d454d5a19d00 not found: ID does not exist"
Sep 18 19:38:19.461298 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:19.461246    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbdc8\" (UniqueName: \"kubernetes.io/projected/e03af1eb-6e02-41ed-b61c-0d7611e815b2-kube-api-access-dbdc8\") pod \"e03af1eb-6e02-41ed-b61c-0d7611e815b2\" (UID: \"e03af1eb-6e02-41ed-b61c-0d7611e815b2\") "
Sep 18 19:38:19.461298 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:19.461274    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e03af1eb-6e02-41ed-b61c-0d7611e815b2-cabundle-cert\") pod \"e03af1eb-6e02-41ed-b61c-0d7611e815b2\" (UID: \"e03af1eb-6e02-41ed-b61c-0d7611e815b2\") "
Sep 18 19:38:19.461298 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:19.461295    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e03af1eb-6e02-41ed-b61c-0d7611e815b2-proxy-tls\") pod \"e03af1eb-6e02-41ed-b61c-0d7611e815b2\" (UID: \"e03af1eb-6e02-41ed-b61c-0d7611e815b2\") "
Sep 18 19:38:19.461446 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:19.461322    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e03af1eb-6e02-41ed-b61c-0d7611e815b2-kserve-provision-location\") pod \"e03af1eb-6e02-41ed-b61c-0d7611e815b2\" (UID: \"e03af1eb-6e02-41ed-b61c-0d7611e815b2\") "
Sep 18 19:38:19.461446 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:19.461366    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e03af1eb-6e02-41ed-b61c-0d7611e815b2-kube-rbac-proxy-sar-config\") pod \"e03af1eb-6e02-41ed-b61c-0d7611e815b2\" (UID: \"e03af1eb-6e02-41ed-b61c-0d7611e815b2\") "
Sep 18 19:38:19.461818 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:19.461783    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e03af1eb-6e02-41ed-b61c-0d7611e815b2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e03af1eb-6e02-41ed-b61c-0d7611e815b2" (UID: "e03af1eb-6e02-41ed-b61c-0d7611e815b2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:38:19.461818 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:19.461800    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e03af1eb-6e02-41ed-b61c-0d7611e815b2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e03af1eb-6e02-41ed-b61c-0d7611e815b2" (UID: "e03af1eb-6e02-41ed-b61c-0d7611e815b2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:38:19.463240 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:19.463220    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e03af1eb-6e02-41ed-b61c-0d7611e815b2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e03af1eb-6e02-41ed-b61c-0d7611e815b2" (UID: "e03af1eb-6e02-41ed-b61c-0d7611e815b2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:38:19.463385 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:19.463363    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e03af1eb-6e02-41ed-b61c-0d7611e815b2-kube-api-access-dbdc8" (OuterVolumeSpecName: "kube-api-access-dbdc8") pod "e03af1eb-6e02-41ed-b61c-0d7611e815b2" (UID: "e03af1eb-6e02-41ed-b61c-0d7611e815b2"). InnerVolumeSpecName "kube-api-access-dbdc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:38:19.465834 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:19.465809    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e03af1eb-6e02-41ed-b61c-0d7611e815b2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e03af1eb-6e02-41ed-b61c-0d7611e815b2" (UID: "e03af1eb-6e02-41ed-b61c-0d7611e815b2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:38:19.562198 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:19.562174    2522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbdc8\" (UniqueName: \"kubernetes.io/projected/e03af1eb-6e02-41ed-b61c-0d7611e815b2-kube-api-access-dbdc8\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:38:19.562198 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:19.562195    2522 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e03af1eb-6e02-41ed-b61c-0d7611e815b2-cabundle-cert\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:38:19.562352 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:19.562205    2522 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e03af1eb-6e02-41ed-b61c-0d7611e815b2-proxy-tls\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:38:19.562352 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:19.562213    2522 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e03af1eb-6e02-41ed-b61c-0d7611e815b2-kserve-provision-location\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:38:19.562352 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:19.562222    2522 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e03af1eb-6e02-41ed-b61c-0d7611e815b2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:38:19.729725 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:19.729657    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m"]
Sep 18 19:38:19.738823 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:19.738799    2522 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-9f89ddc5c-kt25m"]
Sep 18 19:38:20.505717 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:20.505688    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-7857ffc886-cghqq_2e95f8be-691b-4d5f-b83a-032e1bef58ce/kserve-container/0.log"
Sep 18 19:38:20.721705 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:20.721677    2522 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e03af1eb-6e02-41ed-b61c-0d7611e815b2" path="/var/lib/kubelet/pods/e03af1eb-6e02-41ed-b61c-0d7611e815b2/volumes"
Sep 18 19:38:22.521759 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:22.521731    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-7857ffc886-cghqq_2e95f8be-691b-4d5f-b83a-032e1bef58ce/kserve-container/0.log"
Sep 18 19:38:23.247133 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:23.247098    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f"]
Sep 18 19:38:23.247504 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:23.247491    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e03af1eb-6e02-41ed-b61c-0d7611e815b2" containerName="kserve-container"
Sep 18 19:38:23.247550 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:23.247506    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="e03af1eb-6e02-41ed-b61c-0d7611e815b2" containerName="kserve-container"
Sep 18 19:38:23.247550 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:23.247522    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e03af1eb-6e02-41ed-b61c-0d7611e815b2" containerName="storage-initializer"
Sep 18 19:38:23.247550 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:23.247528    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="e03af1eb-6e02-41ed-b61c-0d7611e815b2" containerName="storage-initializer"
Sep 18 19:38:23.247550 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:23.247539    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e03af1eb-6e02-41ed-b61c-0d7611e815b2" containerName="kube-rbac-proxy"
Sep 18 19:38:23.247550 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:23.247545    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="e03af1eb-6e02-41ed-b61c-0d7611e815b2" containerName="kube-rbac-proxy"
Sep 18 19:38:23.247741 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:23.247626    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="e03af1eb-6e02-41ed-b61c-0d7611e815b2" containerName="kube-rbac-proxy"
Sep 18 19:38:23.247741 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:23.247636    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="e03af1eb-6e02-41ed-b61c-0d7611e815b2" containerName="kserve-container"
Sep 18 19:38:23.252297 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:23.252281    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f"
Sep 18 19:38:23.255127 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:23.255109    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Sep 18 19:38:23.255243 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:23.255114    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Sep 18 19:38:23.275113 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:23.275085    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f"]
Sep 18 19:38:23.393617 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:23.393581    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae77b228-c4fb-4bfd-9af7-3e2b265b90fb-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f\" (UID: \"ae77b228-c4fb-4bfd-9af7-3e2b265b90fb\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f"
Sep 18 19:38:23.393816 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:23.393631    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae77b228-c4fb-4bfd-9af7-3e2b265b90fb-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f\" (UID: \"ae77b228-c4fb-4bfd-9af7-3e2b265b90fb\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f"
Sep 18 19:38:23.393816 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:23.393661    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae77b228-c4fb-4bfd-9af7-3e2b265b90fb-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f\" (UID: \"ae77b228-c4fb-4bfd-9af7-3e2b265b90fb\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f"
Sep 18 19:38:23.393816 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:23.393683    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d89qj\" (UniqueName: \"kubernetes.io/projected/ae77b228-c4fb-4bfd-9af7-3e2b265b90fb-kube-api-access-d89qj\") pod \"isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f\" (UID: \"ae77b228-c4fb-4bfd-9af7-3e2b265b90fb\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f"
Sep 18 19:38:23.393816 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:23.393713    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae77b228-c4fb-4bfd-9af7-3e2b265b90fb-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f\" (UID: \"ae77b228-c4fb-4bfd-9af7-3e2b265b90fb\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f"
Sep 18 19:38:23.494339 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:23.494291    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae77b228-c4fb-4bfd-9af7-3e2b265b90fb-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f\" (UID: \"ae77b228-c4fb-4bfd-9af7-3e2b265b90fb\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f"
Sep 18 19:38:23.494339 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:23.494345    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae77b228-c4fb-4bfd-9af7-3e2b265b90fb-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f\" (UID: \"ae77b228-c4fb-4bfd-9af7-3e2b265b90fb\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f"
Sep 18 19:38:23.494613 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:23.494367    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae77b228-c4fb-4bfd-9af7-3e2b265b90fb-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f\" (UID: \"ae77b228-c4fb-4bfd-9af7-3e2b265b90fb\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f"
Sep 18 19:38:23.494613 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:23.494386    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d89qj\" (UniqueName: \"kubernetes.io/projected/ae77b228-c4fb-4bfd-9af7-3e2b265b90fb-kube-api-access-d89qj\") pod \"isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f\" (UID: \"ae77b228-c4fb-4bfd-9af7-3e2b265b90fb\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f"
Sep 18 19:38:23.494613 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:23.494407    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae77b228-c4fb-4bfd-9af7-3e2b265b90fb-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f\" (UID: \"ae77b228-c4fb-4bfd-9af7-3e2b265b90fb\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f"
Sep 18 19:38:23.494801 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:23.494779    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae77b228-c4fb-4bfd-9af7-3e2b265b90fb-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f\" (UID: \"ae77b228-c4fb-4bfd-9af7-3e2b265b90fb\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f"
Sep 18 19:38:23.495188 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:23.495166    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae77b228-c4fb-4bfd-9af7-3e2b265b90fb-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f\" (UID: \"ae77b228-c4fb-4bfd-9af7-3e2b265b90fb\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f"
Sep 18 19:38:23.495225 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:23.495166    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae77b228-c4fb-4bfd-9af7-3e2b265b90fb-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f\" (UID: \"ae77b228-c4fb-4bfd-9af7-3e2b265b90fb\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f"
Sep 18 19:38:23.496831 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:23.496810    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae77b228-c4fb-4bfd-9af7-3e2b265b90fb-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f\" (UID: \"ae77b228-c4fb-4bfd-9af7-3e2b265b90fb\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f"
Sep 18 19:38:23.504894 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:23.504838    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d89qj\" (UniqueName: \"kubernetes.io/projected/ae77b228-c4fb-4bfd-9af7-3e2b265b90fb-kube-api-access-d89qj\") pod \"isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f\" (UID: \"ae77b228-c4fb-4bfd-9af7-3e2b265b90fb\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f"
Sep 18 19:38:23.562723 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:23.562686    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f"
Sep 18 19:38:23.841343 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:23.841316    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f"]
Sep 18 19:38:23.842728 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:38:23.842702    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae77b228_c4fb_4bfd_9af7_3e2b265b90fb.slice/crio-1d2338eb6a8b8c42f588d147774e72c5b43d1e88d70110a57b3a17d054f49f35 WatchSource:0}: Error finding container 1d2338eb6a8b8c42f588d147774e72c5b43d1e88d70110a57b3a17d054f49f35: Status 404 returned error can't find the container with id 1d2338eb6a8b8c42f588d147774e72c5b43d1e88d70110a57b3a17d054f49f35
Sep 18 19:38:24.427725 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:24.427685    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f" event={"ID":"ae77b228-c4fb-4bfd-9af7-3e2b265b90fb","Type":"ContainerStarted","Data":"f35cfeb61d9422c2cec5fa47d7175be782490487ede696e8053d848ba9092840"}
Sep 18 19:38:24.427966 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:24.427761    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f" event={"ID":"ae77b228-c4fb-4bfd-9af7-3e2b265b90fb","Type":"ContainerStarted","Data":"1d2338eb6a8b8c42f588d147774e72c5b43d1e88d70110a57b3a17d054f49f35"}
Sep 18 19:38:24.531831 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:24.531802    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-7857ffc886-cghqq_2e95f8be-691b-4d5f-b83a-032e1bef58ce/kserve-container/0.log"
Sep 18 19:38:26.542150 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:26.542118    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-7857ffc886-cghqq_2e95f8be-691b-4d5f-b83a-032e1bef58ce/kserve-container/0.log"
Sep 18 19:38:28.552573 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:28.552535    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-7857ffc886-cghqq_2e95f8be-691b-4d5f-b83a-032e1bef58ce/kserve-container/0.log"
Sep 18 19:38:29.452968 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:29.452933    2522 generic.go:356] "Generic (PLEG): container finished" podID="ae77b228-c4fb-4bfd-9af7-3e2b265b90fb" containerID="f35cfeb61d9422c2cec5fa47d7175be782490487ede696e8053d848ba9092840" exitCode=0
Sep 18 19:38:29.453127 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:29.453005    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f" event={"ID":"ae77b228-c4fb-4bfd-9af7-3e2b265b90fb","Type":"ContainerDied","Data":"f35cfeb61d9422c2cec5fa47d7175be782490487ede696e8053d848ba9092840"}
Sep 18 19:38:30.460633 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:30.460597    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f" event={"ID":"ae77b228-c4fb-4bfd-9af7-3e2b265b90fb","Type":"ContainerStarted","Data":"bcd32bb932ecd6754cae059a095f5e1ba297651d58b63f29d247891c6863b319"}
Sep 18 19:38:30.461056 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:30.460656    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f" event={"ID":"ae77b228-c4fb-4bfd-9af7-3e2b265b90fb","Type":"ContainerStarted","Data":"357fb084c67285ff804aa022f63eaf185eaea9aad7e07c3d1eb4f1a0b9633dbd"}
Sep 18 19:38:30.461056 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:30.460877    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f"
Sep 18 19:38:30.461056 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:30.461028    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f"
Sep 18 19:38:30.483495 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:30.483440    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f" podStartSLOduration=2.769662227 podStartE2EDuration="7.483421858s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.713759631s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:38:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:30.481196426 +0000 UTC m=+1470.364722911" watchObservedRunningTime="2026-09-18 19:38:30.483421858 +0000 UTC m=+1470.366948321"
Sep 18 19:38:30.563457 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:30.563431    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-7857ffc886-cghqq_2e95f8be-691b-4d5f-b83a-032e1bef58ce/kserve-container/0.log"
Sep 18 19:38:32.576137 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:32.576105    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-7857ffc886-cghqq_2e95f8be-691b-4d5f-b83a-032e1bef58ce/kserve-container/0.log"
Sep 18 19:38:34.587805 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:34.587780    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-7857ffc886-cghqq_2e95f8be-691b-4d5f-b83a-032e1bef58ce/kserve-container/0.log"
Sep 18 19:38:36.471882 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:36.471853    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f"
Sep 18 19:38:36.599937 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:36.599903    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-7857ffc886-cghqq_2e95f8be-691b-4d5f-b83a-032e1bef58ce/kserve-container/0.log"
Sep 18 19:38:38.613179 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:38.613146    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-7857ffc886-cghqq_2e95f8be-691b-4d5f-b83a-032e1bef58ce/kserve-container/0.log"
Sep 18 19:38:40.625629 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:40.625596    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-7857ffc886-cghqq_2e95f8be-691b-4d5f-b83a-032e1bef58ce/kserve-container/0.log"
Sep 18 19:38:42.635412 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:42.635386    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-7857ffc886-cghqq_2e95f8be-691b-4d5f-b83a-032e1bef58ce/kserve-container/0.log"
Sep 18 19:38:44.645659 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:44.645632    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-7857ffc886-cghqq_2e95f8be-691b-4d5f-b83a-032e1bef58ce/kserve-container/0.log"
Sep 18 19:38:46.655811 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:46.655778    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-7857ffc886-cghqq_2e95f8be-691b-4d5f-b83a-032e1bef58ce/kserve-container/0.log"
Sep 18 19:38:48.667061 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:48.667035    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-7857ffc886-cghqq_2e95f8be-691b-4d5f-b83a-032e1bef58ce/kserve-container/0.log"
Sep 18 19:38:50.678925 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:50.678891    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-7857ffc886-cghqq_2e95f8be-691b-4d5f-b83a-032e1bef58ce/kserve-container/0.log"
Sep 18 19:38:50.844895 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:50.844862    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-7857ffc886-cghqq"]
Sep 18 19:38:50.845186 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:50.845158    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7857ffc886-cghqq" podUID="2e95f8be-691b-4d5f-b83a-032e1bef58ce" containerName="kserve-container" containerID="cri-o://78cfc2f758aec185e761f59c0ce9af38b6c5f99df3fa574d5a4b977a3a7f8592" gracePeriod=30
Sep 18 19:38:50.845278 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:50.845250    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7857ffc886-cghqq" podUID="2e95f8be-691b-4d5f-b83a-032e1bef58ce" containerName="kube-rbac-proxy" containerID="cri-o://84740521da22ccc4492b2a7379ca27c8d06eb6ceb3a4e5f089e82f0197422082" gracePeriod=30
Sep 18 19:38:51.561773 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:51.561740    2522 generic.go:356] "Generic (PLEG): container finished" podID="2e95f8be-691b-4d5f-b83a-032e1bef58ce" containerID="84740521da22ccc4492b2a7379ca27c8d06eb6ceb3a4e5f089e82f0197422082" exitCode=2
Sep 18 19:38:51.561963 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:51.561814    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7857ffc886-cghqq" event={"ID":"2e95f8be-691b-4d5f-b83a-032e1bef58ce","Type":"ContainerDied","Data":"84740521da22ccc4492b2a7379ca27c8d06eb6ceb3a4e5f089e82f0197422082"}
Sep 18 19:38:54.014494 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:54.014451    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7857ffc886-cghqq" podUID="2e95f8be-691b-4d5f-b83a-032e1bef58ce" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.41:8643/healthz\": dial tcp 10.132.0.41:8643: connect: connection refused"
Sep 18 19:38:54.019755 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:54.019732    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7857ffc886-cghqq" podUID="2e95f8be-691b-4d5f-b83a-032e1bef58ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 18 19:38:55.787007 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:55.786981    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7857ffc886-cghqq"
Sep 18 19:38:55.879198 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:55.879165    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nzjh9\" (UniqueName: \"kubernetes.io/projected/2e95f8be-691b-4d5f-b83a-032e1bef58ce-kube-api-access-nzjh9\") pod \"2e95f8be-691b-4d5f-b83a-032e1bef58ce\" (UID: \"2e95f8be-691b-4d5f-b83a-032e1bef58ce\") "
Sep 18 19:38:55.879382 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:55.879236    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e95f8be-691b-4d5f-b83a-032e1bef58ce-proxy-tls\") pod \"2e95f8be-691b-4d5f-b83a-032e1bef58ce\" (UID: \"2e95f8be-691b-4d5f-b83a-032e1bef58ce\") "
Sep 18 19:38:55.879382 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:55.879294    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e95f8be-691b-4d5f-b83a-032e1bef58ce-kube-rbac-proxy-sar-config\") pod \"2e95f8be-691b-4d5f-b83a-032e1bef58ce\" (UID: \"2e95f8be-691b-4d5f-b83a-032e1bef58ce\") "
Sep 18 19:38:55.879382 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:55.879325    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e95f8be-691b-4d5f-b83a-032e1bef58ce-cabundle-cert\") pod \"2e95f8be-691b-4d5f-b83a-032e1bef58ce\" (UID: \"2e95f8be-691b-4d5f-b83a-032e1bef58ce\") "
Sep 18 19:38:55.879382 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:55.879367    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e95f8be-691b-4d5f-b83a-032e1bef58ce-kserve-provision-location\") pod \"2e95f8be-691b-4d5f-b83a-032e1bef58ce\" (UID: \"2e95f8be-691b-4d5f-b83a-032e1bef58ce\") "
Sep 18 19:38:55.879758 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:55.879734    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e95f8be-691b-4d5f-b83a-032e1bef58ce-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2e95f8be-691b-4d5f-b83a-032e1bef58ce" (UID: "2e95f8be-691b-4d5f-b83a-032e1bef58ce"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:38:55.879884 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:55.879852    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e95f8be-691b-4d5f-b83a-032e1bef58ce-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2e95f8be-691b-4d5f-b83a-032e1bef58ce" (UID: "2e95f8be-691b-4d5f-b83a-032e1bef58ce"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:38:55.880098 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:55.879924    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e95f8be-691b-4d5f-b83a-032e1bef58ce-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2e95f8be-691b-4d5f-b83a-032e1bef58ce" (UID: "2e95f8be-691b-4d5f-b83a-032e1bef58ce"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:38:55.881534 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:55.881482    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e95f8be-691b-4d5f-b83a-032e1bef58ce-kube-api-access-nzjh9" (OuterVolumeSpecName: "kube-api-access-nzjh9") pod "2e95f8be-691b-4d5f-b83a-032e1bef58ce" (UID: "2e95f8be-691b-4d5f-b83a-032e1bef58ce"). InnerVolumeSpecName "kube-api-access-nzjh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:38:55.881534 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:55.881482    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e95f8be-691b-4d5f-b83a-032e1bef58ce-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2e95f8be-691b-4d5f-b83a-032e1bef58ce" (UID: "2e95f8be-691b-4d5f-b83a-032e1bef58ce"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:38:55.980072 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:55.980039    2522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzjh9\" (UniqueName: \"kubernetes.io/projected/2e95f8be-691b-4d5f-b83a-032e1bef58ce-kube-api-access-nzjh9\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:38:55.980251 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:55.980090    2522 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e95f8be-691b-4d5f-b83a-032e1bef58ce-proxy-tls\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:38:55.980251 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:55.980109    2522 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e95f8be-691b-4d5f-b83a-032e1bef58ce-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:38:55.980251 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:55.980122    2522 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e95f8be-691b-4d5f-b83a-032e1bef58ce-cabundle-cert\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:38:55.980251 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:55.980144    2522 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e95f8be-691b-4d5f-b83a-032e1bef58ce-kserve-provision-location\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:38:56.586850 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:56.586811    2522 generic.go:356] "Generic (PLEG): container finished" podID="2e95f8be-691b-4d5f-b83a-032e1bef58ce" containerID="78cfc2f758aec185e761f59c0ce9af38b6c5f99df3fa574d5a4b977a3a7f8592" exitCode=0
Sep 18 19:38:56.587026 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:56.586891    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7857ffc886-cghqq"
Sep 18 19:38:56.587026 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:56.586900    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7857ffc886-cghqq" event={"ID":"2e95f8be-691b-4d5f-b83a-032e1bef58ce","Type":"ContainerDied","Data":"78cfc2f758aec185e761f59c0ce9af38b6c5f99df3fa574d5a4b977a3a7f8592"}
Sep 18 19:38:56.587026 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:56.587001    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7857ffc886-cghqq" event={"ID":"2e95f8be-691b-4d5f-b83a-032e1bef58ce","Type":"ContainerDied","Data":"d9ded66a18772c1c3b1aa8cf332089c0986a893bfe8d1930673638515503a472"}
Sep 18 19:38:56.587149 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:56.587029    2522 scope.go:117] "RemoveContainer" containerID="84740521da22ccc4492b2a7379ca27c8d06eb6ceb3a4e5f089e82f0197422082"
Sep 18 19:38:56.597390 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:56.597372    2522 scope.go:117] "RemoveContainer" containerID="78cfc2f758aec185e761f59c0ce9af38b6c5f99df3fa574d5a4b977a3a7f8592"
Sep 18 19:38:56.605808 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:56.605781    2522 scope.go:117] "RemoveContainer" containerID="3d8fa5d7ecc293a1845e73098a1f11d2e21cdb0409a9311197c0f24be4549045"
Sep 18 19:38:56.615423 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:56.615405    2522 scope.go:117] "RemoveContainer" containerID="84740521da22ccc4492b2a7379ca27c8d06eb6ceb3a4e5f089e82f0197422082"
Sep 18 19:38:56.615795 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:38:56.615759    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84740521da22ccc4492b2a7379ca27c8d06eb6ceb3a4e5f089e82f0197422082\": container with ID starting with 84740521da22ccc4492b2a7379ca27c8d06eb6ceb3a4e5f089e82f0197422082 not found: ID does not exist" containerID="84740521da22ccc4492b2a7379ca27c8d06eb6ceb3a4e5f089e82f0197422082"
Sep 18 19:38:56.615885 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:56.615806    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84740521da22ccc4492b2a7379ca27c8d06eb6ceb3a4e5f089e82f0197422082"} err="failed to get container status \"84740521da22ccc4492b2a7379ca27c8d06eb6ceb3a4e5f089e82f0197422082\": rpc error: code = NotFound desc = could not find container \"84740521da22ccc4492b2a7379ca27c8d06eb6ceb3a4e5f089e82f0197422082\": container with ID starting with 84740521da22ccc4492b2a7379ca27c8d06eb6ceb3a4e5f089e82f0197422082 not found: ID does not exist"
Sep 18 19:38:56.615885 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:56.615831    2522 scope.go:117] "RemoveContainer" containerID="78cfc2f758aec185e761f59c0ce9af38b6c5f99df3fa574d5a4b977a3a7f8592"
Sep 18 19:38:56.616665 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:38:56.616637    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78cfc2f758aec185e761f59c0ce9af38b6c5f99df3fa574d5a4b977a3a7f8592\": container with ID starting with 78cfc2f758aec185e761f59c0ce9af38b6c5f99df3fa574d5a4b977a3a7f8592 not found: ID does not exist" containerID="78cfc2f758aec185e761f59c0ce9af38b6c5f99df3fa574d5a4b977a3a7f8592"
Sep 18 19:38:56.616767 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:56.616671    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78cfc2f758aec185e761f59c0ce9af38b6c5f99df3fa574d5a4b977a3a7f8592"} err="failed to get container status \"78cfc2f758aec185e761f59c0ce9af38b6c5f99df3fa574d5a4b977a3a7f8592\": rpc error: code = NotFound desc = could not find container \"78cfc2f758aec185e761f59c0ce9af38b6c5f99df3fa574d5a4b977a3a7f8592\": container with ID starting with 78cfc2f758aec185e761f59c0ce9af38b6c5f99df3fa574d5a4b977a3a7f8592 not found: ID does not exist"
Sep 18 19:38:56.616767 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:56.616693    2522 scope.go:117] "RemoveContainer" containerID="3d8fa5d7ecc293a1845e73098a1f11d2e21cdb0409a9311197c0f24be4549045"
Sep 18 19:38:56.617172 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:38:56.617132    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d8fa5d7ecc293a1845e73098a1f11d2e21cdb0409a9311197c0f24be4549045\": container with ID starting with 3d8fa5d7ecc293a1845e73098a1f11d2e21cdb0409a9311197c0f24be4549045 not found: ID does not exist" containerID="3d8fa5d7ecc293a1845e73098a1f11d2e21cdb0409a9311197c0f24be4549045"
Sep 18 19:38:56.617793 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:56.617163    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d8fa5d7ecc293a1845e73098a1f11d2e21cdb0409a9311197c0f24be4549045"} err="failed to get container status \"3d8fa5d7ecc293a1845e73098a1f11d2e21cdb0409a9311197c0f24be4549045\": rpc error: code = NotFound desc = could not find container \"3d8fa5d7ecc293a1845e73098a1f11d2e21cdb0409a9311197c0f24be4549045\": container with ID starting with 3d8fa5d7ecc293a1845e73098a1f11d2e21cdb0409a9311197c0f24be4549045 not found: ID does not exist"
Sep 18 19:38:56.619371 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:56.619351    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-7857ffc886-cghqq"]
Sep 18 19:38:56.630897 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:56.630124    2522 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-7857ffc886-cghqq"]
Sep 18 19:38:56.723448 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:38:56.723413    2522 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e95f8be-691b-4d5f-b83a-032e1bef58ce" path="/var/lib/kubelet/pods/2e95f8be-691b-4d5f-b83a-032e1bef58ce/volumes"
Sep 18 19:40:03.750457 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:03.750413    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f"
Sep 18 19:40:14.758732 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:14.758681    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f" podUID="ae77b228-c4fb-4bfd-9af7-3e2b265b90fb" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.42:8080/v2/models/isvc-autogluon-v2-noslash/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 18 19:40:19.806124 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:19.806091    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f"]
Sep 18 19:40:19.806611 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:19.806380    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f" podUID="ae77b228-c4fb-4bfd-9af7-3e2b265b90fb" containerName="kserve-container" containerID="cri-o://357fb084c67285ff804aa022f63eaf185eaea9aad7e07c3d1eb4f1a0b9633dbd" gracePeriod=30
Sep 18 19:40:19.806611 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:19.806434    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f" podUID="ae77b228-c4fb-4bfd-9af7-3e2b265b90fb" containerName="kube-rbac-proxy" containerID="cri-o://bcd32bb932ecd6754cae059a095f5e1ba297651d58b63f29d247891c6863b319" gracePeriod=30
Sep 18 19:40:19.975502 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:19.975464    2522 generic.go:356] "Generic (PLEG): container finished" podID="ae77b228-c4fb-4bfd-9af7-3e2b265b90fb" containerID="bcd32bb932ecd6754cae059a095f5e1ba297651d58b63f29d247891c6863b319" exitCode=2
Sep 18 19:40:19.975690 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:19.975537    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f" event={"ID":"ae77b228-c4fb-4bfd-9af7-3e2b265b90fb","Type":"ContainerDied","Data":"bcd32bb932ecd6754cae059a095f5e1ba297651d58b63f29d247891c6863b319"}
Sep 18 19:40:21.050068 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:40:21.050037    2522 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae77b228_c4fb_4bfd_9af7_3e2b265b90fb.slice/crio-conmon-357fb084c67285ff804aa022f63eaf185eaea9aad7e07c3d1eb4f1a0b9633dbd.scope\": RecentStats: unable to find data in memory cache]"
Sep 18 19:40:21.178152 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:21.178128    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f"
Sep 18 19:40:21.304305 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:21.304216    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d89qj\" (UniqueName: \"kubernetes.io/projected/ae77b228-c4fb-4bfd-9af7-3e2b265b90fb-kube-api-access-d89qj\") pod \"ae77b228-c4fb-4bfd-9af7-3e2b265b90fb\" (UID: \"ae77b228-c4fb-4bfd-9af7-3e2b265b90fb\") "
Sep 18 19:40:21.304305 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:21.304264    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae77b228-c4fb-4bfd-9af7-3e2b265b90fb-proxy-tls\") pod \"ae77b228-c4fb-4bfd-9af7-3e2b265b90fb\" (UID: \"ae77b228-c4fb-4bfd-9af7-3e2b265b90fb\") "
Sep 18 19:40:21.304305 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:21.304293    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae77b228-c4fb-4bfd-9af7-3e2b265b90fb-kube-rbac-proxy-sar-config\") pod \"ae77b228-c4fb-4bfd-9af7-3e2b265b90fb\" (UID: \"ae77b228-c4fb-4bfd-9af7-3e2b265b90fb\") "
Sep 18 19:40:21.304540 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:21.304346    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae77b228-c4fb-4bfd-9af7-3e2b265b90fb-cabundle-cert\") pod \"ae77b228-c4fb-4bfd-9af7-3e2b265b90fb\" (UID: \"ae77b228-c4fb-4bfd-9af7-3e2b265b90fb\") "
Sep 18 19:40:21.304540 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:21.304400    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae77b228-c4fb-4bfd-9af7-3e2b265b90fb-kserve-provision-location\") pod \"ae77b228-c4fb-4bfd-9af7-3e2b265b90fb\" (UID: \"ae77b228-c4fb-4bfd-9af7-3e2b265b90fb\") "
Sep 18 19:40:21.304742 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:21.304709    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae77b228-c4fb-4bfd-9af7-3e2b265b90fb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ae77b228-c4fb-4bfd-9af7-3e2b265b90fb" (UID: "ae77b228-c4fb-4bfd-9af7-3e2b265b90fb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:40:21.304802 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:21.304771    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae77b228-c4fb-4bfd-9af7-3e2b265b90fb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ae77b228-c4fb-4bfd-9af7-3e2b265b90fb" (UID: "ae77b228-c4fb-4bfd-9af7-3e2b265b90fb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:40:21.306528 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:21.306494    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae77b228-c4fb-4bfd-9af7-3e2b265b90fb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ae77b228-c4fb-4bfd-9af7-3e2b265b90fb" (UID: "ae77b228-c4fb-4bfd-9af7-3e2b265b90fb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:40:21.306528 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:21.306502    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae77b228-c4fb-4bfd-9af7-3e2b265b90fb-kube-api-access-d89qj" (OuterVolumeSpecName: "kube-api-access-d89qj") pod "ae77b228-c4fb-4bfd-9af7-3e2b265b90fb" (UID: "ae77b228-c4fb-4bfd-9af7-3e2b265b90fb"). InnerVolumeSpecName "kube-api-access-d89qj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:40:21.308995 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:21.308973    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae77b228-c4fb-4bfd-9af7-3e2b265b90fb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ae77b228-c4fb-4bfd-9af7-3e2b265b90fb" (UID: "ae77b228-c4fb-4bfd-9af7-3e2b265b90fb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:40:21.405098 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:21.405057    2522 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae77b228-c4fb-4bfd-9af7-3e2b265b90fb-proxy-tls\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:40:21.405098 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:21.405093    2522 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae77b228-c4fb-4bfd-9af7-3e2b265b90fb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:40:21.405098 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:21.405106    2522 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae77b228-c4fb-4bfd-9af7-3e2b265b90fb-cabundle-cert\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:40:21.405336 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:21.405119    2522 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae77b228-c4fb-4bfd-9af7-3e2b265b90fb-kserve-provision-location\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:40:21.405336 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:21.405131    2522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d89qj\" (UniqueName: \"kubernetes.io/projected/ae77b228-c4fb-4bfd-9af7-3e2b265b90fb-kube-api-access-d89qj\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:40:21.987980 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:21.987946    2522 generic.go:356] "Generic (PLEG): container finished" podID="ae77b228-c4fb-4bfd-9af7-3e2b265b90fb" containerID="357fb084c67285ff804aa022f63eaf185eaea9aad7e07c3d1eb4f1a0b9633dbd" exitCode=0
Sep 18 19:40:21.988154 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:21.987989    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f" event={"ID":"ae77b228-c4fb-4bfd-9af7-3e2b265b90fb","Type":"ContainerDied","Data":"357fb084c67285ff804aa022f63eaf185eaea9aad7e07c3d1eb4f1a0b9633dbd"}
Sep 18 19:40:21.988154 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:21.988000    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f"
Sep 18 19:40:21.988154 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:21.988036    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f" event={"ID":"ae77b228-c4fb-4bfd-9af7-3e2b265b90fb","Type":"ContainerDied","Data":"1d2338eb6a8b8c42f588d147774e72c5b43d1e88d70110a57b3a17d054f49f35"}
Sep 18 19:40:21.988154 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:21.988057    2522 scope.go:117] "RemoveContainer" containerID="bcd32bb932ecd6754cae059a095f5e1ba297651d58b63f29d247891c6863b319"
Sep 18 19:40:21.997447 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:21.997432    2522 scope.go:117] "RemoveContainer" containerID="357fb084c67285ff804aa022f63eaf185eaea9aad7e07c3d1eb4f1a0b9633dbd"
Sep 18 19:40:22.004990 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:22.004972    2522 scope.go:117] "RemoveContainer" containerID="f35cfeb61d9422c2cec5fa47d7175be782490487ede696e8053d848ba9092840"
Sep 18 19:40:22.012209 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:22.012192    2522 scope.go:117] "RemoveContainer" containerID="bcd32bb932ecd6754cae059a095f5e1ba297651d58b63f29d247891c6863b319"
Sep 18 19:40:22.012498 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:40:22.012476    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcd32bb932ecd6754cae059a095f5e1ba297651d58b63f29d247891c6863b319\": container with ID starting with bcd32bb932ecd6754cae059a095f5e1ba297651d58b63f29d247891c6863b319 not found: ID does not exist" containerID="bcd32bb932ecd6754cae059a095f5e1ba297651d58b63f29d247891c6863b319"
Sep 18 19:40:22.012610 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:22.012509    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcd32bb932ecd6754cae059a095f5e1ba297651d58b63f29d247891c6863b319"} err="failed to get container status \"bcd32bb932ecd6754cae059a095f5e1ba297651d58b63f29d247891c6863b319\": rpc error: code = NotFound desc = could not find container \"bcd32bb932ecd6754cae059a095f5e1ba297651d58b63f29d247891c6863b319\": container with ID starting with bcd32bb932ecd6754cae059a095f5e1ba297651d58b63f29d247891c6863b319 not found: ID does not exist"
Sep 18 19:40:22.012610 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:22.012527    2522 scope.go:117] "RemoveContainer" containerID="357fb084c67285ff804aa022f63eaf185eaea9aad7e07c3d1eb4f1a0b9633dbd"
Sep 18 19:40:22.012805 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:40:22.012787    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"357fb084c67285ff804aa022f63eaf185eaea9aad7e07c3d1eb4f1a0b9633dbd\": container with ID starting with 357fb084c67285ff804aa022f63eaf185eaea9aad7e07c3d1eb4f1a0b9633dbd not found: ID does not exist" containerID="357fb084c67285ff804aa022f63eaf185eaea9aad7e07c3d1eb4f1a0b9633dbd"
Sep 18 19:40:22.012868 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:22.012813    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"357fb084c67285ff804aa022f63eaf185eaea9aad7e07c3d1eb4f1a0b9633dbd"} err="failed to get container status \"357fb084c67285ff804aa022f63eaf185eaea9aad7e07c3d1eb4f1a0b9633dbd\": rpc error: code = NotFound desc = could not find container \"357fb084c67285ff804aa022f63eaf185eaea9aad7e07c3d1eb4f1a0b9633dbd\": container with ID starting with 357fb084c67285ff804aa022f63eaf185eaea9aad7e07c3d1eb4f1a0b9633dbd not found: ID does not exist"
Sep 18 19:40:22.012868 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:22.012836    2522 scope.go:117] "RemoveContainer" containerID="f35cfeb61d9422c2cec5fa47d7175be782490487ede696e8053d848ba9092840"
Sep 18 19:40:22.013096 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:40:22.013072    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f35cfeb61d9422c2cec5fa47d7175be782490487ede696e8053d848ba9092840\": container with ID starting with f35cfeb61d9422c2cec5fa47d7175be782490487ede696e8053d848ba9092840 not found: ID does not exist" containerID="f35cfeb61d9422c2cec5fa47d7175be782490487ede696e8053d848ba9092840"
Sep 18 19:40:22.013142 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:22.013093    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f35cfeb61d9422c2cec5fa47d7175be782490487ede696e8053d848ba9092840"} err="failed to get container status \"f35cfeb61d9422c2cec5fa47d7175be782490487ede696e8053d848ba9092840\": rpc error: code = NotFound desc = could not find container \"f35cfeb61d9422c2cec5fa47d7175be782490487ede696e8053d848ba9092840\": container with ID starting with f35cfeb61d9422c2cec5fa47d7175be782490487ede696e8053d848ba9092840 not found: ID does not exist"
Sep 18 19:40:22.015740 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:22.015719    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f"]
Sep 18 19:40:22.022481 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:22.022457    2522 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5cdb8fd688-h9b9f"]
Sep 18 19:40:22.722008 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:22.721972    2522 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae77b228-c4fb-4bfd-9af7-3e2b265b90fb" path="/var/lib/kubelet/pods/ae77b228-c4fb-4bfd-9af7-3e2b265b90fb/volumes"
Sep 18 19:40:25.065086 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:25.065044    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-68594f479c-52l8w"]
Sep 18 19:40:25.065614 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:25.065595    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae77b228-c4fb-4bfd-9af7-3e2b265b90fb" containerName="kserve-container"
Sep 18 19:40:25.065720 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:25.065615    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae77b228-c4fb-4bfd-9af7-3e2b265b90fb" containerName="kserve-container"
Sep 18 19:40:25.065720 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:25.065629    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e95f8be-691b-4d5f-b83a-032e1bef58ce" containerName="kserve-container"
Sep 18 19:40:25.065720 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:25.065638    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e95f8be-691b-4d5f-b83a-032e1bef58ce" containerName="kserve-container"
Sep 18 19:40:25.065720 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:25.065648    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae77b228-c4fb-4bfd-9af7-3e2b265b90fb" containerName="storage-initializer"
Sep 18 19:40:25.065720 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:25.065657    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae77b228-c4fb-4bfd-9af7-3e2b265b90fb" containerName="storage-initializer"
Sep 18 19:40:25.065720 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:25.065672    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae77b228-c4fb-4bfd-9af7-3e2b265b90fb" containerName="kube-rbac-proxy"
Sep 18 19:40:25.065720 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:25.065680    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae77b228-c4fb-4bfd-9af7-3e2b265b90fb" containerName="kube-rbac-proxy"
Sep 18 19:40:25.065720 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:25.065689    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e95f8be-691b-4d5f-b83a-032e1bef58ce" containerName="storage-initializer"
Sep 18 19:40:25.065720 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:25.065698    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e95f8be-691b-4d5f-b83a-032e1bef58ce" containerName="storage-initializer"
Sep 18 19:40:25.065720 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:25.065708    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e95f8be-691b-4d5f-b83a-032e1bef58ce" containerName="kube-rbac-proxy"
Sep 18 19:40:25.065720 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:25.065716    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e95f8be-691b-4d5f-b83a-032e1bef58ce" containerName="kube-rbac-proxy"
Sep 18 19:40:25.066256 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:25.065805    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e95f8be-691b-4d5f-b83a-032e1bef58ce" containerName="kserve-container"
Sep 18 19:40:25.066256 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:25.065817    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae77b228-c4fb-4bfd-9af7-3e2b265b90fb" containerName="kube-rbac-proxy"
Sep 18 19:40:25.066256 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:25.065830    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae77b228-c4fb-4bfd-9af7-3e2b265b90fb" containerName="kserve-container"
Sep 18 19:40:25.066256 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:25.065842    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e95f8be-691b-4d5f-b83a-032e1bef58ce" containerName="kube-rbac-proxy"
Sep 18 19:40:25.071125 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:25.071105    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-68594f479c-52l8w"
Sep 18 19:40:25.075926 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:25.075903    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 19:40:25.076046 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:25.075906    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 19:40:25.076046 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:25.075907    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 19:40:25.077069 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:25.077051    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Sep 18 19:40:25.077987 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:25.077967    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5b562\""
Sep 18 19:40:25.078078 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:25.077967    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Sep 18 19:40:25.080724 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:25.080708    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 19:40:25.092955 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:25.092928    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-68594f479c-52l8w"]
Sep 18 19:40:25.139457 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:25.139422    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ee79c7b-926e-445c-a927-69bdc38cda3d-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-68594f479c-52l8w\" (UID: \"7ee79c7b-926e-445c-a927-69bdc38cda3d\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-68594f479c-52l8w"
Sep 18 19:40:25.139662 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:25.139468    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwn4z\" (UniqueName: \"kubernetes.io/projected/7ee79c7b-926e-445c-a927-69bdc38cda3d-kube-api-access-bwn4z\") pod \"isvc-autogluon-ts-v1-predictor-68594f479c-52l8w\" (UID: \"7ee79c7b-926e-445c-a927-69bdc38cda3d\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-68594f479c-52l8w"
Sep 18 19:40:25.139662 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:25.139502    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ee79c7b-926e-445c-a927-69bdc38cda3d-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-68594f479c-52l8w\" (UID: \"7ee79c7b-926e-445c-a927-69bdc38cda3d\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-68594f479c-52l8w"
Sep 18 19:40:25.139662 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:25.139533    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ee79c7b-926e-445c-a927-69bdc38cda3d-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-68594f479c-52l8w\" (UID: \"7ee79c7b-926e-445c-a927-69bdc38cda3d\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-68594f479c-52l8w"
Sep 18 19:40:25.139662 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:25.139608    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ee79c7b-926e-445c-a927-69bdc38cda3d-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-68594f479c-52l8w\" (UID: \"7ee79c7b-926e-445c-a927-69bdc38cda3d\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-68594f479c-52l8w"
Sep 18 19:40:25.240886 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:25.240847    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwn4z\" (UniqueName: \"kubernetes.io/projected/7ee79c7b-926e-445c-a927-69bdc38cda3d-kube-api-access-bwn4z\") pod \"isvc-autogluon-ts-v1-predictor-68594f479c-52l8w\" (UID: \"7ee79c7b-926e-445c-a927-69bdc38cda3d\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-68594f479c-52l8w"
Sep 18 19:40:25.240886 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:25.240887    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ee79c7b-926e-445c-a927-69bdc38cda3d-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-68594f479c-52l8w\" (UID: \"7ee79c7b-926e-445c-a927-69bdc38cda3d\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-68594f479c-52l8w"
Sep 18 19:40:25.241114 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:25.240909    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ee79c7b-926e-445c-a927-69bdc38cda3d-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-68594f479c-52l8w\" (UID: \"7ee79c7b-926e-445c-a927-69bdc38cda3d\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-68594f479c-52l8w"
Sep 18 19:40:25.241114 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:25.240935    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ee79c7b-926e-445c-a927-69bdc38cda3d-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-68594f479c-52l8w\" (UID: \"7ee79c7b-926e-445c-a927-69bdc38cda3d\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-68594f479c-52l8w"
Sep 18 19:40:25.241114 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:25.240997    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ee79c7b-926e-445c-a927-69bdc38cda3d-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-68594f479c-52l8w\" (UID: \"7ee79c7b-926e-445c-a927-69bdc38cda3d\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-68594f479c-52l8w"
Sep 18 19:40:25.241533 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:25.241503    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ee79c7b-926e-445c-a927-69bdc38cda3d-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-68594f479c-52l8w\" (UID: \"7ee79c7b-926e-445c-a927-69bdc38cda3d\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-68594f479c-52l8w"
Sep 18 19:40:25.241756 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:25.241739    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ee79c7b-926e-445c-a927-69bdc38cda3d-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-68594f479c-52l8w\" (UID: \"7ee79c7b-926e-445c-a927-69bdc38cda3d\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-68594f479c-52l8w"
Sep 18 19:40:25.241839 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:25.241758    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ee79c7b-926e-445c-a927-69bdc38cda3d-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-68594f479c-52l8w\" (UID: \"7ee79c7b-926e-445c-a927-69bdc38cda3d\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-68594f479c-52l8w"
Sep 18 19:40:25.243325 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:25.243307    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ee79c7b-926e-445c-a927-69bdc38cda3d-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-68594f479c-52l8w\" (UID: \"7ee79c7b-926e-445c-a927-69bdc38cda3d\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-68594f479c-52l8w"
Sep 18 19:40:25.251104 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:25.251083    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwn4z\" (UniqueName: \"kubernetes.io/projected/7ee79c7b-926e-445c-a927-69bdc38cda3d-kube-api-access-bwn4z\") pod \"isvc-autogluon-ts-v1-predictor-68594f479c-52l8w\" (UID: \"7ee79c7b-926e-445c-a927-69bdc38cda3d\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-68594f479c-52l8w"
Sep 18 19:40:25.381527 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:25.381423    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-68594f479c-52l8w"
Sep 18 19:40:25.675986 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:25.675784    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-68594f479c-52l8w"]
Sep 18 19:40:25.678711 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:40:25.678684    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ee79c7b_926e_445c_a927_69bdc38cda3d.slice/crio-651136c6e8dcdfab677491817a57c67e60bd78a23b6accd55ba58ccb29867f6d WatchSource:0}: Error finding container 651136c6e8dcdfab677491817a57c67e60bd78a23b6accd55ba58ccb29867f6d: Status 404 returned error can't find the container with id 651136c6e8dcdfab677491817a57c67e60bd78a23b6accd55ba58ccb29867f6d
Sep 18 19:40:26.010342 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:26.010247    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-68594f479c-52l8w" event={"ID":"7ee79c7b-926e-445c-a927-69bdc38cda3d","Type":"ContainerStarted","Data":"3754951c77ef714ce6c605ac8ad68649987119a8a0ca849a3dcb50dd56bb7e41"}
Sep 18 19:40:26.010342 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:26.010328    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-68594f479c-52l8w" event={"ID":"7ee79c7b-926e-445c-a927-69bdc38cda3d","Type":"ContainerStarted","Data":"651136c6e8dcdfab677491817a57c67e60bd78a23b6accd55ba58ccb29867f6d"}
Sep 18 19:40:31.033790 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:31.033756    2522 generic.go:356] "Generic (PLEG): container finished" podID="7ee79c7b-926e-445c-a927-69bdc38cda3d" containerID="3754951c77ef714ce6c605ac8ad68649987119a8a0ca849a3dcb50dd56bb7e41" exitCode=0
Sep 18 19:40:31.034198 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:31.033838    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-68594f479c-52l8w" event={"ID":"7ee79c7b-926e-445c-a927-69bdc38cda3d","Type":"ContainerDied","Data":"3754951c77ef714ce6c605ac8ad68649987119a8a0ca849a3dcb50dd56bb7e41"}
Sep 18 19:40:32.041123 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:32.041088    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-68594f479c-52l8w" event={"ID":"7ee79c7b-926e-445c-a927-69bdc38cda3d","Type":"ContainerStarted","Data":"f0853cf194373979e90b80c024f061f75e5393c621ce4090599fad9d4bab3389"}
Sep 18 19:40:32.041551 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:32.041183    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-68594f479c-52l8w" event={"ID":"7ee79c7b-926e-445c-a927-69bdc38cda3d","Type":"ContainerStarted","Data":"1d5991d141f3528e2f11e5d68d0b2ba329fd6f80212d5daa9b983f8f22da114c"}
Sep 18 19:40:32.041551 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:32.041387    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-68594f479c-52l8w"
Sep 18 19:40:32.073571 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:32.073492    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-68594f479c-52l8w" podStartSLOduration=2.321679987 podStartE2EDuration="7.073470651s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.751790664s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:40:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:40:32.070786762 +0000 UTC m=+1591.954313225" watchObservedRunningTime="2026-09-18 19:40:32.073470651 +0000 UTC m=+1591.956997114"
Sep 18 19:40:33.045743 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:33.045710    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-68594f479c-52l8w"
Sep 18 19:40:33.047073 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:33.047034    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-68594f479c-52l8w" podUID="7ee79c7b-926e-445c-a927-69bdc38cda3d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 18 19:40:34.050818 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:34.050776    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-68594f479c-52l8w" podUID="7ee79c7b-926e-445c-a927-69bdc38cda3d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 18 19:40:39.055684 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:39.055653    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-68594f479c-52l8w"
Sep 18 19:40:39.056311 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:39.056283    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-68594f479c-52l8w" podUID="7ee79c7b-926e-445c-a927-69bdc38cda3d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 18 19:40:49.057090 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:40:49.057052    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-68594f479c-52l8w"
Sep 18 19:41:07.693384 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:07.693351    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-68594f479c-52l8w"]
Sep 18 19:41:07.693844 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:07.693593    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-68594f479c-52l8w" podUID="7ee79c7b-926e-445c-a927-69bdc38cda3d" containerName="kserve-container" containerID="cri-o://1d5991d141f3528e2f11e5d68d0b2ba329fd6f80212d5daa9b983f8f22da114c" gracePeriod=30
Sep 18 19:41:07.693844 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:07.693619    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-68594f479c-52l8w" podUID="7ee79c7b-926e-445c-a927-69bdc38cda3d" containerName="kube-rbac-proxy" containerID="cri-o://f0853cf194373979e90b80c024f061f75e5393c621ce4090599fad9d4bab3389" gracePeriod=30
Sep 18 19:41:08.219027 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:08.218985    2522 generic.go:356] "Generic (PLEG): container finished" podID="7ee79c7b-926e-445c-a927-69bdc38cda3d" containerID="f0853cf194373979e90b80c024f061f75e5393c621ce4090599fad9d4bab3389" exitCode=2
Sep 18 19:41:08.219202 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:08.219052    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-68594f479c-52l8w" event={"ID":"7ee79c7b-926e-445c-a927-69bdc38cda3d","Type":"ContainerDied","Data":"f0853cf194373979e90b80c024f061f75e5393c621ce4090599fad9d4bab3389"}
Sep 18 19:41:09.051694 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:09.051640    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-68594f479c-52l8w" podUID="7ee79c7b-926e-445c-a927-69bdc38cda3d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused"
Sep 18 19:41:09.056793 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:09.056765    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-68594f479c-52l8w" podUID="7ee79c7b-926e-445c-a927-69bdc38cda3d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 18 19:41:09.225922 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:09.225875    2522 generic.go:356] "Generic (PLEG): container finished" podID="7ee79c7b-926e-445c-a927-69bdc38cda3d" containerID="1d5991d141f3528e2f11e5d68d0b2ba329fd6f80212d5daa9b983f8f22da114c" exitCode=0
Sep 18 19:41:09.225922 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:09.225921    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-68594f479c-52l8w" event={"ID":"7ee79c7b-926e-445c-a927-69bdc38cda3d","Type":"ContainerDied","Data":"1d5991d141f3528e2f11e5d68d0b2ba329fd6f80212d5daa9b983f8f22da114c"}
Sep 18 19:41:09.343575 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:09.343542    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-68594f479c-52l8w"
Sep 18 19:41:09.411147 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:09.411118    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ee79c7b-926e-445c-a927-69bdc38cda3d-kserve-provision-location\") pod \"7ee79c7b-926e-445c-a927-69bdc38cda3d\" (UID: \"7ee79c7b-926e-445c-a927-69bdc38cda3d\") "
Sep 18 19:41:09.411306 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:09.411175    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bwn4z\" (UniqueName: \"kubernetes.io/projected/7ee79c7b-926e-445c-a927-69bdc38cda3d-kube-api-access-bwn4z\") pod \"7ee79c7b-926e-445c-a927-69bdc38cda3d\" (UID: \"7ee79c7b-926e-445c-a927-69bdc38cda3d\") "
Sep 18 19:41:09.411306 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:09.411204    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ee79c7b-926e-445c-a927-69bdc38cda3d-proxy-tls\") pod \"7ee79c7b-926e-445c-a927-69bdc38cda3d\" (UID: \"7ee79c7b-926e-445c-a927-69bdc38cda3d\") "
Sep 18 19:41:09.411306 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:09.411226    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ee79c7b-926e-445c-a927-69bdc38cda3d-cabundle-cert\") pod \"7ee79c7b-926e-445c-a927-69bdc38cda3d\" (UID: \"7ee79c7b-926e-445c-a927-69bdc38cda3d\") "
Sep 18 19:41:09.411306 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:09.411253    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ee79c7b-926e-445c-a927-69bdc38cda3d-kube-rbac-proxy-sar-config\") pod \"7ee79c7b-926e-445c-a927-69bdc38cda3d\" (UID: \"7ee79c7b-926e-445c-a927-69bdc38cda3d\") "
Sep 18 19:41:09.411653 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:09.411609    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ee79c7b-926e-445c-a927-69bdc38cda3d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7ee79c7b-926e-445c-a927-69bdc38cda3d" (UID: "7ee79c7b-926e-445c-a927-69bdc38cda3d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:41:09.411773 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:09.411692    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ee79c7b-926e-445c-a927-69bdc38cda3d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7ee79c7b-926e-445c-a927-69bdc38cda3d" (UID: "7ee79c7b-926e-445c-a927-69bdc38cda3d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:41:09.412156 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:09.412134    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee79c7b-926e-445c-a927-69bdc38cda3d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7ee79c7b-926e-445c-a927-69bdc38cda3d" (UID: "7ee79c7b-926e-445c-a927-69bdc38cda3d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:41:09.413258 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:09.413243    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ee79c7b-926e-445c-a927-69bdc38cda3d-kube-api-access-bwn4z" (OuterVolumeSpecName: "kube-api-access-bwn4z") pod "7ee79c7b-926e-445c-a927-69bdc38cda3d" (UID: "7ee79c7b-926e-445c-a927-69bdc38cda3d"). InnerVolumeSpecName "kube-api-access-bwn4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:41:09.413386 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:09.413367    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ee79c7b-926e-445c-a927-69bdc38cda3d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7ee79c7b-926e-445c-a927-69bdc38cda3d" (UID: "7ee79c7b-926e-445c-a927-69bdc38cda3d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:41:09.512439 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:09.512407    2522 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ee79c7b-926e-445c-a927-69bdc38cda3d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:41:09.512439 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:09.512432    2522 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ee79c7b-926e-445c-a927-69bdc38cda3d-kserve-provision-location\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:41:09.512439 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:09.512443    2522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwn4z\" (UniqueName: \"kubernetes.io/projected/7ee79c7b-926e-445c-a927-69bdc38cda3d-kube-api-access-bwn4z\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:41:09.512439 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:09.512452    2522 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ee79c7b-926e-445c-a927-69bdc38cda3d-proxy-tls\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:41:09.512733 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:09.512460    2522 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ee79c7b-926e-445c-a927-69bdc38cda3d-cabundle-cert\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:41:10.232669 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:10.232636    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-68594f479c-52l8w" event={"ID":"7ee79c7b-926e-445c-a927-69bdc38cda3d","Type":"ContainerDied","Data":"651136c6e8dcdfab677491817a57c67e60bd78a23b6accd55ba58ccb29867f6d"}
Sep 18 19:41:10.233112 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:10.232681    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-68594f479c-52l8w"
Sep 18 19:41:10.233112 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:10.232704    2522 scope.go:117] "RemoveContainer" containerID="f0853cf194373979e90b80c024f061f75e5393c621ce4090599fad9d4bab3389"
Sep 18 19:41:10.241320 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:10.241300    2522 scope.go:117] "RemoveContainer" containerID="1d5991d141f3528e2f11e5d68d0b2ba329fd6f80212d5daa9b983f8f22da114c"
Sep 18 19:41:10.249146 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:10.249123    2522 scope.go:117] "RemoveContainer" containerID="3754951c77ef714ce6c605ac8ad68649987119a8a0ca849a3dcb50dd56bb7e41"
Sep 18 19:41:10.258835 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:10.258809    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-68594f479c-52l8w"]
Sep 18 19:41:10.264645 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:10.264624    2522 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-68594f479c-52l8w"]
Sep 18 19:41:10.722282 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:10.722248    2522 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ee79c7b-926e-445c-a927-69bdc38cda3d" path="/var/lib/kubelet/pods/7ee79c7b-926e-445c-a927-69bdc38cda3d/volumes"
Sep 18 19:41:12.769504 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:12.769470    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv"]
Sep 18 19:41:12.769918 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:12.769848    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ee79c7b-926e-445c-a927-69bdc38cda3d" containerName="kube-rbac-proxy"
Sep 18 19:41:12.769918 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:12.769863    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ee79c7b-926e-445c-a927-69bdc38cda3d" containerName="kube-rbac-proxy"
Sep 18 19:41:12.769918 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:12.769880    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ee79c7b-926e-445c-a927-69bdc38cda3d" containerName="kserve-container"
Sep 18 19:41:12.769918 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:12.769887    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ee79c7b-926e-445c-a927-69bdc38cda3d" containerName="kserve-container"
Sep 18 19:41:12.769918 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:12.769900    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ee79c7b-926e-445c-a927-69bdc38cda3d" containerName="storage-initializer"
Sep 18 19:41:12.769918 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:12.769906    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ee79c7b-926e-445c-a927-69bdc38cda3d" containerName="storage-initializer"
Sep 18 19:41:12.770117 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:12.769969    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ee79c7b-926e-445c-a927-69bdc38cda3d" containerName="kube-rbac-proxy"
Sep 18 19:41:12.770117 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:12.769978    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ee79c7b-926e-445c-a927-69bdc38cda3d" containerName="kserve-container"
Sep 18 19:41:12.774912 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:12.774890    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv"
Sep 18 19:41:12.778537 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:12.778516    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-kube-rbac-proxy-sar-config\""
Sep 18 19:41:12.779073 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:12.779053    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 19:41:12.779167 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:12.779073    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5b562\""
Sep 18 19:41:12.779167 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:12.779058    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-predictor-serving-cert\""
Sep 18 19:41:12.779167 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:12.779057    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 19:41:12.779698 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:12.779679    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 19:41:12.779807 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:12.779722    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 19:41:12.785397 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:12.785376    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv"]
Sep 18 19:41:12.839115 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:12.839078    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea80a852-9b00-41cc-bc5b-8579ed054ffd-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv\" (UID: \"ea80a852-9b00-41cc-bc5b-8579ed054ffd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv"
Sep 18 19:41:12.839277 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:12.839157    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea80a852-9b00-41cc-bc5b-8579ed054ffd-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv\" (UID: \"ea80a852-9b00-41cc-bc5b-8579ed054ffd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv"
Sep 18 19:41:12.839277 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:12.839182    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea80a852-9b00-41cc-bc5b-8579ed054ffd-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv\" (UID: \"ea80a852-9b00-41cc-bc5b-8579ed054ffd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv"
Sep 18 19:41:12.839277 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:12.839227    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea80a852-9b00-41cc-bc5b-8579ed054ffd-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv\" (UID: \"ea80a852-9b00-41cc-bc5b-8579ed054ffd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv"
Sep 18 19:41:12.839418 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:12.839332    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c89r\" (UniqueName: \"kubernetes.io/projected/ea80a852-9b00-41cc-bc5b-8579ed054ffd-kube-api-access-5c89r\") pod \"isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv\" (UID: \"ea80a852-9b00-41cc-bc5b-8579ed054ffd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv"
Sep 18 19:41:12.940061 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:12.940018    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea80a852-9b00-41cc-bc5b-8579ed054ffd-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv\" (UID: \"ea80a852-9b00-41cc-bc5b-8579ed054ffd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv"
Sep 18 19:41:12.940277 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:12.940074    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c89r\" (UniqueName: \"kubernetes.io/projected/ea80a852-9b00-41cc-bc5b-8579ed054ffd-kube-api-access-5c89r\") pod \"isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv\" (UID: \"ea80a852-9b00-41cc-bc5b-8579ed054ffd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv"
Sep 18 19:41:12.940277 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:12.940111    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea80a852-9b00-41cc-bc5b-8579ed054ffd-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv\" (UID: \"ea80a852-9b00-41cc-bc5b-8579ed054ffd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv"
Sep 18 19:41:12.940277 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:12.940138    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea80a852-9b00-41cc-bc5b-8579ed054ffd-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv\" (UID: \"ea80a852-9b00-41cc-bc5b-8579ed054ffd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv"
Sep 18 19:41:12.940277 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:12.940161    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea80a852-9b00-41cc-bc5b-8579ed054ffd-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv\" (UID: \"ea80a852-9b00-41cc-bc5b-8579ed054ffd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv"
Sep 18 19:41:12.940753 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:12.940729    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea80a852-9b00-41cc-bc5b-8579ed054ffd-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv\" (UID: \"ea80a852-9b00-41cc-bc5b-8579ed054ffd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv"
Sep 18 19:41:12.940955 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:12.940937    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea80a852-9b00-41cc-bc5b-8579ed054ffd-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv\" (UID: \"ea80a852-9b00-41cc-bc5b-8579ed054ffd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv"
Sep 18 19:41:12.940995 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:12.940966    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea80a852-9b00-41cc-bc5b-8579ed054ffd-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv\" (UID: \"ea80a852-9b00-41cc-bc5b-8579ed054ffd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv"
Sep 18 19:41:12.942638 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:12.942614    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea80a852-9b00-41cc-bc5b-8579ed054ffd-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv\" (UID: \"ea80a852-9b00-41cc-bc5b-8579ed054ffd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv"
Sep 18 19:41:12.948406 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:12.948374    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c89r\" (UniqueName: \"kubernetes.io/projected/ea80a852-9b00-41cc-bc5b-8579ed054ffd-kube-api-access-5c89r\") pod \"isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv\" (UID: \"ea80a852-9b00-41cc-bc5b-8579ed054ffd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv"
Sep 18 19:41:13.085740 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:13.085713    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv"
Sep 18 19:41:13.360397 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:13.360368    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv"]
Sep 18 19:41:13.362066 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:41:13.362034    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea80a852_9b00_41cc_bc5b_8579ed054ffd.slice/crio-b29e4145058bae37bf0bef4866dd9e73ee035cb79ede86b8ff00a70951746ecd WatchSource:0}: Error finding container b29e4145058bae37bf0bef4866dd9e73ee035cb79ede86b8ff00a70951746ecd: Status 404 returned error can't find the container with id b29e4145058bae37bf0bef4866dd9e73ee035cb79ede86b8ff00a70951746ecd
Sep 18 19:41:14.256940 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:14.256900    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv" event={"ID":"ea80a852-9b00-41cc-bc5b-8579ed054ffd","Type":"ContainerStarted","Data":"c957a42031b8ee528d2bac879a8f8d659fb21b669385695554965d918155b8b6"}
Sep 18 19:41:14.257308 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:14.256975    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv" event={"ID":"ea80a852-9b00-41cc-bc5b-8579ed054ffd","Type":"ContainerStarted","Data":"b29e4145058bae37bf0bef4866dd9e73ee035cb79ede86b8ff00a70951746ecd"}
Sep 18 19:41:19.280577 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:19.280526    2522 generic.go:356] "Generic (PLEG): container finished" podID="ea80a852-9b00-41cc-bc5b-8579ed054ffd" containerID="c957a42031b8ee528d2bac879a8f8d659fb21b669385695554965d918155b8b6" exitCode=0
Sep 18 19:41:19.280981 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:19.280602    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv" event={"ID":"ea80a852-9b00-41cc-bc5b-8579ed054ffd","Type":"ContainerDied","Data":"c957a42031b8ee528d2bac879a8f8d659fb21b669385695554965d918155b8b6"}
Sep 18 19:41:20.288477 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:20.288436    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv" event={"ID":"ea80a852-9b00-41cc-bc5b-8579ed054ffd","Type":"ContainerStarted","Data":"6e50f64aeb72747cfca787382aea38912314ebf9d189d7f4bf5b2b17b98527bd"}
Sep 18 19:41:20.289032 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:20.288497    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv" event={"ID":"ea80a852-9b00-41cc-bc5b-8579ed054ffd","Type":"ContainerStarted","Data":"226a6129b08cd34395adad3da893f69046a385e85ef4f2c428bda986752717c9"}
Sep 18 19:41:20.289032 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:20.288778    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv"
Sep 18 19:41:20.289032 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:20.288923    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv"
Sep 18 19:41:20.290706 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:20.290670    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv" podUID="ea80a852-9b00-41cc-bc5b-8579ed054ffd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 18 19:41:20.311528 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:20.311481    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv" podStartSLOduration=2.928279773 podStartE2EDuration="8.31146297s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.383183197s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:41:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:41:20.310804776 +0000 UTC m=+1640.194331233" watchObservedRunningTime="2026-09-18 19:41:20.31146297 +0000 UTC m=+1640.194989432"
Sep 18 19:41:21.294141 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:21.294103    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv" podUID="ea80a852-9b00-41cc-bc5b-8579ed054ffd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 18 19:41:26.299016 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:26.298986    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv"
Sep 18 19:41:26.299486 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:26.299460    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv" podUID="ea80a852-9b00-41cc-bc5b-8579ed054ffd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 18 19:41:36.300736 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:36.300706    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv"
Sep 18 19:41:48.202758 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:48.202724    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv"]
Sep 18 19:41:48.203189 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:48.202941    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv" podUID="ea80a852-9b00-41cc-bc5b-8579ed054ffd" containerName="kserve-container" containerID="cri-o://226a6129b08cd34395adad3da893f69046a385e85ef4f2c428bda986752717c9" gracePeriod=30
Sep 18 19:41:48.203189 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:48.202999    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv" podUID="ea80a852-9b00-41cc-bc5b-8579ed054ffd" containerName="kube-rbac-proxy" containerID="cri-o://6e50f64aeb72747cfca787382aea38912314ebf9d189d7f4bf5b2b17b98527bd" gracePeriod=30
Sep 18 19:41:48.430009 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:48.429968    2522 generic.go:356] "Generic (PLEG): container finished" podID="ea80a852-9b00-41cc-bc5b-8579ed054ffd" containerID="6e50f64aeb72747cfca787382aea38912314ebf9d189d7f4bf5b2b17b98527bd" exitCode=2
Sep 18 19:41:48.430187 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:48.430022    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv" event={"ID":"ea80a852-9b00-41cc-bc5b-8579ed054ffd","Type":"ContainerDied","Data":"6e50f64aeb72747cfca787382aea38912314ebf9d189d7f4bf5b2b17b98527bd"}
Sep 18 19:41:49.650522 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:49.650490    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv"
Sep 18 19:41:49.759868 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:49.759786    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea80a852-9b00-41cc-bc5b-8579ed054ffd-kube-rbac-proxy-sar-config\") pod \"ea80a852-9b00-41cc-bc5b-8579ed054ffd\" (UID: \"ea80a852-9b00-41cc-bc5b-8579ed054ffd\") "
Sep 18 19:41:49.759868 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:49.759848    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea80a852-9b00-41cc-bc5b-8579ed054ffd-cabundle-cert\") pod \"ea80a852-9b00-41cc-bc5b-8579ed054ffd\" (UID: \"ea80a852-9b00-41cc-bc5b-8579ed054ffd\") "
Sep 18 19:41:49.760059 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:49.759979    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea80a852-9b00-41cc-bc5b-8579ed054ffd-kserve-provision-location\") pod \"ea80a852-9b00-41cc-bc5b-8579ed054ffd\" (UID: \"ea80a852-9b00-41cc-bc5b-8579ed054ffd\") "
Sep 18 19:41:49.760059 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:49.760018    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5c89r\" (UniqueName: \"kubernetes.io/projected/ea80a852-9b00-41cc-bc5b-8579ed054ffd-kube-api-access-5c89r\") pod \"ea80a852-9b00-41cc-bc5b-8579ed054ffd\" (UID: \"ea80a852-9b00-41cc-bc5b-8579ed054ffd\") "
Sep 18 19:41:49.760059 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:49.760054    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea80a852-9b00-41cc-bc5b-8579ed054ffd-proxy-tls\") pod \"ea80a852-9b00-41cc-bc5b-8579ed054ffd\" (UID: \"ea80a852-9b00-41cc-bc5b-8579ed054ffd\") "
Sep 18 19:41:49.760275 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:49.760251    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea80a852-9b00-41cc-bc5b-8579ed054ffd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ea80a852-9b00-41cc-bc5b-8579ed054ffd" (UID: "ea80a852-9b00-41cc-bc5b-8579ed054ffd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:41:49.760275 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:49.760265    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea80a852-9b00-41cc-bc5b-8579ed054ffd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ea80a852-9b00-41cc-bc5b-8579ed054ffd" (UID: "ea80a852-9b00-41cc-bc5b-8579ed054ffd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:41:49.760413 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:49.760346    2522 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea80a852-9b00-41cc-bc5b-8579ed054ffd-cabundle-cert\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:41:49.760413 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:49.760363    2522 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea80a852-9b00-41cc-bc5b-8579ed054ffd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:41:49.760892 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:49.760872    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea80a852-9b00-41cc-bc5b-8579ed054ffd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ea80a852-9b00-41cc-bc5b-8579ed054ffd" (UID: "ea80a852-9b00-41cc-bc5b-8579ed054ffd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:41:49.762170 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:49.762138    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea80a852-9b00-41cc-bc5b-8579ed054ffd-kube-api-access-5c89r" (OuterVolumeSpecName: "kube-api-access-5c89r") pod "ea80a852-9b00-41cc-bc5b-8579ed054ffd" (UID: "ea80a852-9b00-41cc-bc5b-8579ed054ffd"). InnerVolumeSpecName "kube-api-access-5c89r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:41:49.762170 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:49.762147    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea80a852-9b00-41cc-bc5b-8579ed054ffd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ea80a852-9b00-41cc-bc5b-8579ed054ffd" (UID: "ea80a852-9b00-41cc-bc5b-8579ed054ffd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:41:49.861449 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:49.861410    2522 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea80a852-9b00-41cc-bc5b-8579ed054ffd-kserve-provision-location\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:41:49.861449 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:49.861440    2522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5c89r\" (UniqueName: \"kubernetes.io/projected/ea80a852-9b00-41cc-bc5b-8579ed054ffd-kube-api-access-5c89r\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:41:49.861449 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:49.861450    2522 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea80a852-9b00-41cc-bc5b-8579ed054ffd-proxy-tls\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:41:50.448112 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:50.448076    2522 generic.go:356] "Generic (PLEG): container finished" podID="ea80a852-9b00-41cc-bc5b-8579ed054ffd" containerID="226a6129b08cd34395adad3da893f69046a385e85ef4f2c428bda986752717c9" exitCode=0
Sep 18 19:41:50.448299 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:50.448142    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv"
Sep 18 19:41:50.448299 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:50.448144    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv" event={"ID":"ea80a852-9b00-41cc-bc5b-8579ed054ffd","Type":"ContainerDied","Data":"226a6129b08cd34395adad3da893f69046a385e85ef4f2c428bda986752717c9"}
Sep 18 19:41:50.448299 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:50.448288    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv" event={"ID":"ea80a852-9b00-41cc-bc5b-8579ed054ffd","Type":"ContainerDied","Data":"b29e4145058bae37bf0bef4866dd9e73ee035cb79ede86b8ff00a70951746ecd"}
Sep 18 19:41:50.448405 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:50.448343    2522 scope.go:117] "RemoveContainer" containerID="6e50f64aeb72747cfca787382aea38912314ebf9d189d7f4bf5b2b17b98527bd"
Sep 18 19:41:50.457970 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:50.457948    2522 scope.go:117] "RemoveContainer" containerID="226a6129b08cd34395adad3da893f69046a385e85ef4f2c428bda986752717c9"
Sep 18 19:41:50.465343 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:50.465327    2522 scope.go:117] "RemoveContainer" containerID="c957a42031b8ee528d2bac879a8f8d659fb21b669385695554965d918155b8b6"
Sep 18 19:41:50.472535 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:50.472516    2522 scope.go:117] "RemoveContainer" containerID="6e50f64aeb72747cfca787382aea38912314ebf9d189d7f4bf5b2b17b98527bd"
Sep 18 19:41:50.472829 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:41:50.472799    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e50f64aeb72747cfca787382aea38912314ebf9d189d7f4bf5b2b17b98527bd\": container with ID starting with 6e50f64aeb72747cfca787382aea38912314ebf9d189d7f4bf5b2b17b98527bd not found: ID does not exist" containerID="6e50f64aeb72747cfca787382aea38912314ebf9d189d7f4bf5b2b17b98527bd"
Sep 18 19:41:50.472920 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:50.472828    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e50f64aeb72747cfca787382aea38912314ebf9d189d7f4bf5b2b17b98527bd"} err="failed to get container status \"6e50f64aeb72747cfca787382aea38912314ebf9d189d7f4bf5b2b17b98527bd\": rpc error: code = NotFound desc = could not find container \"6e50f64aeb72747cfca787382aea38912314ebf9d189d7f4bf5b2b17b98527bd\": container with ID starting with 6e50f64aeb72747cfca787382aea38912314ebf9d189d7f4bf5b2b17b98527bd not found: ID does not exist"
Sep 18 19:41:50.472920 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:50.472846    2522 scope.go:117] "RemoveContainer" containerID="226a6129b08cd34395adad3da893f69046a385e85ef4f2c428bda986752717c9"
Sep 18 19:41:50.473087 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:41:50.473071    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"226a6129b08cd34395adad3da893f69046a385e85ef4f2c428bda986752717c9\": container with ID starting with 226a6129b08cd34395adad3da893f69046a385e85ef4f2c428bda986752717c9 not found: ID does not exist" containerID="226a6129b08cd34395adad3da893f69046a385e85ef4f2c428bda986752717c9"
Sep 18 19:41:50.473149 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:50.473090    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"226a6129b08cd34395adad3da893f69046a385e85ef4f2c428bda986752717c9"} err="failed to get container status \"226a6129b08cd34395adad3da893f69046a385e85ef4f2c428bda986752717c9\": rpc error: code = NotFound desc = could not find container \"226a6129b08cd34395adad3da893f69046a385e85ef4f2c428bda986752717c9\": container with ID starting with 226a6129b08cd34395adad3da893f69046a385e85ef4f2c428bda986752717c9 not found: ID does not exist"
Sep 18 19:41:50.473149 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:50.473102    2522 scope.go:117] "RemoveContainer" containerID="c957a42031b8ee528d2bac879a8f8d659fb21b669385695554965d918155b8b6"
Sep 18 19:41:50.473377 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:41:50.473358    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c957a42031b8ee528d2bac879a8f8d659fb21b669385695554965d918155b8b6\": container with ID starting with c957a42031b8ee528d2bac879a8f8d659fb21b669385695554965d918155b8b6 not found: ID does not exist" containerID="c957a42031b8ee528d2bac879a8f8d659fb21b669385695554965d918155b8b6"
Sep 18 19:41:50.473428 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:50.473381    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c957a42031b8ee528d2bac879a8f8d659fb21b669385695554965d918155b8b6"} err="failed to get container status \"c957a42031b8ee528d2bac879a8f8d659fb21b669385695554965d918155b8b6\": rpc error: code = NotFound desc = could not find container \"c957a42031b8ee528d2bac879a8f8d659fb21b669385695554965d918155b8b6\": container with ID starting with c957a42031b8ee528d2bac879a8f8d659fb21b669385695554965d918155b8b6 not found: ID does not exist"
Sep 18 19:41:50.477758 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:50.477736    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv"]
Sep 18 19:41:50.480777 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:50.480756    2522 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-f77fddfc8-p4nzv"]
Sep 18 19:41:50.723335 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:50.723262    2522 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea80a852-9b00-41cc-bc5b-8579ed054ffd" path="/var/lib/kubelet/pods/ea80a852-9b00-41cc-bc5b-8579ed054ffd/volumes"
Sep 18 19:41:53.675307 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:53.675271    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7"]
Sep 18 19:41:53.675840 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:53.675658    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea80a852-9b00-41cc-bc5b-8579ed054ffd" containerName="storage-initializer"
Sep 18 19:41:53.675840 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:53.675671    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea80a852-9b00-41cc-bc5b-8579ed054ffd" containerName="storage-initializer"
Sep 18 19:41:53.675840 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:53.675679    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea80a852-9b00-41cc-bc5b-8579ed054ffd" containerName="kserve-container"
Sep 18 19:41:53.675840 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:53.675687    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea80a852-9b00-41cc-bc5b-8579ed054ffd" containerName="kserve-container"
Sep 18 19:41:53.675840 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:53.675703    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea80a852-9b00-41cc-bc5b-8579ed054ffd" containerName="kube-rbac-proxy"
Sep 18 19:41:53.675840 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:53.675713    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea80a852-9b00-41cc-bc5b-8579ed054ffd" containerName="kube-rbac-proxy"
Sep 18 19:41:53.675840 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:53.675777    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea80a852-9b00-41cc-bc5b-8579ed054ffd" containerName="kube-rbac-proxy"
Sep 18 19:41:53.675840 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:53.675784    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea80a852-9b00-41cc-bc5b-8579ed054ffd" containerName="kserve-container"
Sep 18 19:41:53.680422 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:53.680392    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7"
Sep 18 19:41:53.690154 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:53.690133    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12da7015-ac39-4fe1-9d4b-75142f6f496a-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7\" (UID: \"12da7015-ac39-4fe1-9d4b-75142f6f496a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7"
Sep 18 19:41:53.690265 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:53.690162    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgq7w\" (UniqueName: \"kubernetes.io/projected/12da7015-ac39-4fe1-9d4b-75142f6f496a-kube-api-access-qgq7w\") pod \"isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7\" (UID: \"12da7015-ac39-4fe1-9d4b-75142f6f496a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7"
Sep 18 19:41:53.690265 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:53.690188    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12da7015-ac39-4fe1-9d4b-75142f6f496a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7\" (UID: \"12da7015-ac39-4fe1-9d4b-75142f6f496a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7"
Sep 18 19:41:53.690265 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:53.690248    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/12da7015-ac39-4fe1-9d4b-75142f6f496a-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7\" (UID: \"12da7015-ac39-4fe1-9d4b-75142f6f496a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7"
Sep 18 19:41:53.690380 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:53.690326    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12da7015-ac39-4fe1-9d4b-75142f6f496a-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7\" (UID: \"12da7015-ac39-4fe1-9d4b-75142f6f496a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7"
Sep 18 19:41:53.696800 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:53.696777    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 19:41:53.696942 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:53.696923    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5b562\""
Sep 18 19:41:53.698337 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:53.698318    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-bzxwz"]
Sep 18 19:41:53.699027 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:53.699009    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Sep 18 19:41:53.700122 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:53.700093    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 19:41:53.700122 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:53.700093    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 19:41:53.700386 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:53.700371    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 19:41:53.700679 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:53.700662    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 18 19:41:53.701989 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:53.701972    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-bzxwz"
Sep 18 19:41:53.709950 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:53.709933    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 18 19:41:53.727640 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:53.727613    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7"]
Sep 18 19:41:53.748308 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:53.748272    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-bzxwz"]
Sep 18 19:41:53.791642 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:53.791610    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9df2f2da-c098-407d-b943-a581bee7af4f-kserve-provision-location\") pod \"isvc-canary-create-predictor-7b568f75-bzxwz\" (UID: \"9df2f2da-c098-407d-b943-a581bee7af4f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-bzxwz"
Sep 18 19:41:53.791836 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:53.791649    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12da7015-ac39-4fe1-9d4b-75142f6f496a-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7\" (UID: \"12da7015-ac39-4fe1-9d4b-75142f6f496a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7"
Sep 18 19:41:53.791836 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:53.791675    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9df2f2da-c098-407d-b943-a581bee7af4f-proxy-tls\") pod \"isvc-canary-create-predictor-7b568f75-bzxwz\" (UID: \"9df2f2da-c098-407d-b943-a581bee7af4f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-bzxwz"
Sep 18 19:41:53.791836 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:53.791700    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9df2f2da-c098-407d-b943-a581bee7af4f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-7b568f75-bzxwz\" (UID: \"9df2f2da-c098-407d-b943-a581bee7af4f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-bzxwz"
Sep 18 19:41:53.791836 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:53.791722    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgq7w\" (UniqueName: \"kubernetes.io/projected/12da7015-ac39-4fe1-9d4b-75142f6f496a-kube-api-access-qgq7w\") pod \"isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7\" (UID: \"12da7015-ac39-4fe1-9d4b-75142f6f496a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7"
Sep 18 19:41:53.791836 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:53.791747    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9df2f2da-c098-407d-b943-a581bee7af4f-cabundle-cert\") pod \"isvc-canary-create-predictor-7b568f75-bzxwz\" (UID: \"9df2f2da-c098-407d-b943-a581bee7af4f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-bzxwz"
Sep 18 19:41:53.791836 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:41:53.791769    2522 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-v2-predictor-serving-cert: secret "isvc-canary-create-v2-predictor-serving-cert" not found
Sep 18 19:41:53.792476 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:41:53.791844    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12da7015-ac39-4fe1-9d4b-75142f6f496a-proxy-tls podName:12da7015-ac39-4fe1-9d4b-75142f6f496a nodeName:}" failed. No retries permitted until 2026-09-18 19:41:54.291826536 +0000 UTC m=+1674.175352975 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/12da7015-ac39-4fe1-9d4b-75142f6f496a-proxy-tls") pod "isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7" (UID: "12da7015-ac39-4fe1-9d4b-75142f6f496a") : secret "isvc-canary-create-v2-predictor-serving-cert" not found
Sep 18 19:41:53.792476 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:53.791772    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12da7015-ac39-4fe1-9d4b-75142f6f496a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7\" (UID: \"12da7015-ac39-4fe1-9d4b-75142f6f496a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7"
Sep 18 19:41:53.792476 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:53.791897    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkh22\" (UniqueName: \"kubernetes.io/projected/9df2f2da-c098-407d-b943-a581bee7af4f-kube-api-access-bkh22\") pod \"isvc-canary-create-predictor-7b568f75-bzxwz\" (UID: \"9df2f2da-c098-407d-b943-a581bee7af4f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-bzxwz"
Sep 18 19:41:53.792476 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:53.791933    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/12da7015-ac39-4fe1-9d4b-75142f6f496a-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7\" (UID: \"12da7015-ac39-4fe1-9d4b-75142f6f496a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7"
Sep 18 19:41:53.792476 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:53.792037    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12da7015-ac39-4fe1-9d4b-75142f6f496a-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7\" (UID: \"12da7015-ac39-4fe1-9d4b-75142f6f496a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7"
Sep 18 19:41:53.792476 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:53.792353    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12da7015-ac39-4fe1-9d4b-75142f6f496a-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7\" (UID: \"12da7015-ac39-4fe1-9d4b-75142f6f496a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7"
Sep 18 19:41:53.792771 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:53.792508    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12da7015-ac39-4fe1-9d4b-75142f6f496a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7\" (UID: \"12da7015-ac39-4fe1-9d4b-75142f6f496a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7"
Sep 18 19:41:53.792771 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:53.792624    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/12da7015-ac39-4fe1-9d4b-75142f6f496a-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7\" (UID: \"12da7015-ac39-4fe1-9d4b-75142f6f496a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7"
Sep 18 19:41:53.803672 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:53.803649    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgq7w\" (UniqueName: \"kubernetes.io/projected/12da7015-ac39-4fe1-9d4b-75142f6f496a-kube-api-access-qgq7w\") pod \"isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7\" (UID: \"12da7015-ac39-4fe1-9d4b-75142f6f496a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7"
Sep 18 19:41:53.892989 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:53.892954    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9df2f2da-c098-407d-b943-a581bee7af4f-kserve-provision-location\") pod \"isvc-canary-create-predictor-7b568f75-bzxwz\" (UID: \"9df2f2da-c098-407d-b943-a581bee7af4f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-bzxwz"
Sep 18 19:41:53.893177 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:53.892997    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9df2f2da-c098-407d-b943-a581bee7af4f-proxy-tls\") pod \"isvc-canary-create-predictor-7b568f75-bzxwz\" (UID: \"9df2f2da-c098-407d-b943-a581bee7af4f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-bzxwz"
Sep 18 19:41:53.893177 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:53.893019    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9df2f2da-c098-407d-b943-a581bee7af4f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-7b568f75-bzxwz\" (UID: \"9df2f2da-c098-407d-b943-a581bee7af4f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-bzxwz"
Sep 18 19:41:53.893177 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:53.893041    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9df2f2da-c098-407d-b943-a581bee7af4f-cabundle-cert\") pod \"isvc-canary-create-predictor-7b568f75-bzxwz\" (UID: \"9df2f2da-c098-407d-b943-a581bee7af4f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-bzxwz"
Sep 18 19:41:53.893177 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:53.893068    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkh22\" (UniqueName: \"kubernetes.io/projected/9df2f2da-c098-407d-b943-a581bee7af4f-kube-api-access-bkh22\") pod \"isvc-canary-create-predictor-7b568f75-bzxwz\" (UID: \"9df2f2da-c098-407d-b943-a581bee7af4f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-bzxwz"
Sep 18 19:41:53.893447 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:53.893422    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9df2f2da-c098-407d-b943-a581bee7af4f-kserve-provision-location\") pod \"isvc-canary-create-predictor-7b568f75-bzxwz\" (UID: \"9df2f2da-c098-407d-b943-a581bee7af4f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-bzxwz"
Sep 18 19:41:53.893788 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:53.893765    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9df2f2da-c098-407d-b943-a581bee7af4f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-7b568f75-bzxwz\" (UID: \"9df2f2da-c098-407d-b943-a581bee7af4f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-bzxwz"
Sep 18 19:41:53.893788 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:53.893784    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9df2f2da-c098-407d-b943-a581bee7af4f-cabundle-cert\") pod \"isvc-canary-create-predictor-7b568f75-bzxwz\" (UID: \"9df2f2da-c098-407d-b943-a581bee7af4f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-bzxwz"
Sep 18 19:41:53.895579 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:53.895547    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9df2f2da-c098-407d-b943-a581bee7af4f-proxy-tls\") pod \"isvc-canary-create-predictor-7b568f75-bzxwz\" (UID: \"9df2f2da-c098-407d-b943-a581bee7af4f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-bzxwz"
Sep 18 19:41:53.908045 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:53.908018    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkh22\" (UniqueName: \"kubernetes.io/projected/9df2f2da-c098-407d-b943-a581bee7af4f-kube-api-access-bkh22\") pod \"isvc-canary-create-predictor-7b568f75-bzxwz\" (UID: \"9df2f2da-c098-407d-b943-a581bee7af4f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-bzxwz"
Sep 18 19:41:54.011777 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:54.011662    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-bzxwz"
Sep 18 19:41:54.290324 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:54.290293    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-bzxwz"]
Sep 18 19:41:54.295108 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:41:54.295081    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9df2f2da_c098_407d_b943_a581bee7af4f.slice/crio-fef30f7267bc52aaa13aa9872942b9bcf7676ff99b83723116fc2e8246a78157 WatchSource:0}: Error finding container fef30f7267bc52aaa13aa9872942b9bcf7676ff99b83723116fc2e8246a78157: Status 404 returned error can't find the container with id fef30f7267bc52aaa13aa9872942b9bcf7676ff99b83723116fc2e8246a78157
Sep 18 19:41:54.296776 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:54.296756    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12da7015-ac39-4fe1-9d4b-75142f6f496a-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7\" (UID: \"12da7015-ac39-4fe1-9d4b-75142f6f496a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7"
Sep 18 19:41:54.299169 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:54.299149    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12da7015-ac39-4fe1-9d4b-75142f6f496a-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7\" (UID: \"12da7015-ac39-4fe1-9d4b-75142f6f496a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7"
Sep 18 19:41:54.468128 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:54.468095    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-bzxwz" event={"ID":"9df2f2da-c098-407d-b943-a581bee7af4f","Type":"ContainerStarted","Data":"5b44d5e943327ef3781a7e2bfae916d55989eeac314a34bd4356c3649bd96925"}
Sep 18 19:41:54.468294 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:54.468163    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-bzxwz" event={"ID":"9df2f2da-c098-407d-b943-a581bee7af4f","Type":"ContainerStarted","Data":"fef30f7267bc52aaa13aa9872942b9bcf7676ff99b83723116fc2e8246a78157"}
Sep 18 19:41:54.590616 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:54.590574    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7"
Sep 18 19:41:54.884028 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:54.884000    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7"]
Sep 18 19:41:54.885458 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:41:54.885430    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12da7015_ac39_4fe1_9d4b_75142f6f496a.slice/crio-18d22c530556a0a2e6b66c00d4a2a50b6eaab40b8d04bea9785992c61301d902 WatchSource:0}: Error finding container 18d22c530556a0a2e6b66c00d4a2a50b6eaab40b8d04bea9785992c61301d902: Status 404 returned error can't find the container with id 18d22c530556a0a2e6b66c00d4a2a50b6eaab40b8d04bea9785992c61301d902
Sep 18 19:41:55.475787 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:55.475744    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7" event={"ID":"12da7015-ac39-4fe1-9d4b-75142f6f496a","Type":"ContainerStarted","Data":"5a068d47cf0709e2ac179729e6562b543b3aeadbd37ac88e02469ed75c7b477f"}
Sep 18 19:41:55.475996 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:55.475811    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7" event={"ID":"12da7015-ac39-4fe1-9d4b-75142f6f496a","Type":"ContainerStarted","Data":"18d22c530556a0a2e6b66c00d4a2a50b6eaab40b8d04bea9785992c61301d902"}
Sep 18 19:41:58.491837 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:58.491749    2522 generic.go:356] "Generic (PLEG): container finished" podID="9df2f2da-c098-407d-b943-a581bee7af4f" containerID="5b44d5e943327ef3781a7e2bfae916d55989eeac314a34bd4356c3649bd96925" exitCode=0
Sep 18 19:41:58.491837 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:58.491826    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-bzxwz" event={"ID":"9df2f2da-c098-407d-b943-a581bee7af4f","Type":"ContainerDied","Data":"5b44d5e943327ef3781a7e2bfae916d55989eeac314a34bd4356c3649bd96925"}
Sep 18 19:41:59.498530 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:59.498495    2522 generic.go:356] "Generic (PLEG): container finished" podID="12da7015-ac39-4fe1-9d4b-75142f6f496a" containerID="5a068d47cf0709e2ac179729e6562b543b3aeadbd37ac88e02469ed75c7b477f" exitCode=0
Sep 18 19:41:59.498974 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:59.498590    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7" event={"ID":"12da7015-ac39-4fe1-9d4b-75142f6f496a","Type":"ContainerDied","Data":"5a068d47cf0709e2ac179729e6562b543b3aeadbd37ac88e02469ed75c7b477f"}
Sep 18 19:41:59.500960 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:59.500936    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-bzxwz" event={"ID":"9df2f2da-c098-407d-b943-a581bee7af4f","Type":"ContainerStarted","Data":"c16074ebe032c91ea43848d36ce57e90e6de7c2b12380ddf7458b80c388246ed"}
Sep 18 19:41:59.501066 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:59.500995    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-bzxwz" event={"ID":"9df2f2da-c098-407d-b943-a581bee7af4f","Type":"ContainerStarted","Data":"718115c47b0ac13353cc8e0c1e8a0f313f19c55dc75d8ca81e13791bbffd8c7b"}
Sep 18 19:41:59.501346 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:59.501323    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-bzxwz"
Sep 18 19:41:59.501384 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:59.501372    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-bzxwz"
Sep 18 19:41:59.502913 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:59.502888    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-bzxwz" podUID="9df2f2da-c098-407d-b943-a581bee7af4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 18 19:41:59.545102 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:41:59.545049    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-bzxwz" podStartSLOduration=2.716880828 podStartE2EDuration="6.545033538s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.82815271s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:41:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:41:59.544146625 +0000 UTC m=+1679.427673089" watchObservedRunningTime="2026-09-18 19:41:59.545033538 +0000 UTC m=+1679.428560000"
Sep 18 19:42:00.507988 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:00.507951    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7" event={"ID":"12da7015-ac39-4fe1-9d4b-75142f6f496a","Type":"ContainerStarted","Data":"ecff1d3b590205ad7cf301bf5f93e30a7a1e69b2d758e4bcc7d101dd1c8f524e"}
Sep 18 19:42:00.508433 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:00.508015    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7" event={"ID":"12da7015-ac39-4fe1-9d4b-75142f6f496a","Type":"ContainerStarted","Data":"dc6421446d58eb3b702d1dba652f469f9d8fd453d1a63cf6fec7c85a955b5411"}
Sep 18 19:42:00.511741 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:00.511679    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-bzxwz" podUID="9df2f2da-c098-407d-b943-a581bee7af4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 18 19:42:00.549873 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:00.549815    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7" podStartSLOduration=3.71071739 podStartE2EDuration="7.549798637s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.839081247s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:41:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:42:00.547659877 +0000 UTC m=+1680.431186329" watchObservedRunningTime="2026-09-18 19:42:00.549798637 +0000 UTC m=+1680.433325098"
Sep 18 19:42:01.365197 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:01.365153    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26"]
Sep 18 19:42:01.369824 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:01.369805    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26"
Sep 18 19:42:01.372512 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:01.372486    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-predictor-serving-cert\""
Sep 18 19:42:01.372612 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:01.372582    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\""
Sep 18 19:42:01.383175 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:01.383148    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26"]
Sep 18 19:42:01.459925 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:01.459884    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/876ba837-626c-401e-91c7-384d2d45a796-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26\" (UID: \"876ba837-626c-401e-91c7-384d2d45a796\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26"
Sep 18 19:42:01.460123 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:01.459937    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/876ba837-626c-401e-91c7-384d2d45a796-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26\" (UID: \"876ba837-626c-401e-91c7-384d2d45a796\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26"
Sep 18 19:42:01.460123 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:01.460002    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/876ba837-626c-401e-91c7-384d2d45a796-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26\" (UID: \"876ba837-626c-401e-91c7-384d2d45a796\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26"
Sep 18 19:42:01.460123 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:01.460073    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/876ba837-626c-401e-91c7-384d2d45a796-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26\" (UID: \"876ba837-626c-401e-91c7-384d2d45a796\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26"
Sep 18 19:42:01.460123 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:01.460091    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d4kn\" (UniqueName: \"kubernetes.io/projected/876ba837-626c-401e-91c7-384d2d45a796-kube-api-access-9d4kn\") pod \"isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26\" (UID: \"876ba837-626c-401e-91c7-384d2d45a796\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26"
Sep 18 19:42:01.561256 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:01.561212    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/876ba837-626c-401e-91c7-384d2d45a796-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26\" (UID: \"876ba837-626c-401e-91c7-384d2d45a796\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26"
Sep 18 19:42:01.561747 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:01.561275    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/876ba837-626c-401e-91c7-384d2d45a796-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26\" (UID: \"876ba837-626c-401e-91c7-384d2d45a796\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26"
Sep 18 19:42:01.561747 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:01.561313    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/876ba837-626c-401e-91c7-384d2d45a796-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26\" (UID: \"876ba837-626c-401e-91c7-384d2d45a796\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26"
Sep 18 19:42:01.561747 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:01.561391    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/876ba837-626c-401e-91c7-384d2d45a796-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26\" (UID: \"876ba837-626c-401e-91c7-384d2d45a796\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26"
Sep 18 19:42:01.561747 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:01.561416    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d4kn\" (UniqueName: \"kubernetes.io/projected/876ba837-626c-401e-91c7-384d2d45a796-kube-api-access-9d4kn\") pod \"isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26\" (UID: \"876ba837-626c-401e-91c7-384d2d45a796\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26"
Sep 18 19:42:01.561747 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:42:01.561715    2522 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-serving-cert: secret "isvc-sklearn-runtime-predictor-serving-cert" not found
Sep 18 19:42:01.561747 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:01.561731    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/876ba837-626c-401e-91c7-384d2d45a796-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26\" (UID: \"876ba837-626c-401e-91c7-384d2d45a796\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26"
Sep 18 19:42:01.562074 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:42:01.561786    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/876ba837-626c-401e-91c7-384d2d45a796-proxy-tls podName:876ba837-626c-401e-91c7-384d2d45a796 nodeName:}" failed. No retries permitted until 2026-09-18 19:42:02.06176572 +0000 UTC m=+1681.945292161 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/876ba837-626c-401e-91c7-384d2d45a796-proxy-tls") pod "isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26" (UID: "876ba837-626c-401e-91c7-384d2d45a796") : secret "isvc-sklearn-runtime-predictor-serving-cert" not found
Sep 18 19:42:01.562156 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:01.562135    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/876ba837-626c-401e-91c7-384d2d45a796-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26\" (UID: \"876ba837-626c-401e-91c7-384d2d45a796\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26"
Sep 18 19:42:01.562216 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:01.562150    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/876ba837-626c-401e-91c7-384d2d45a796-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26\" (UID: \"876ba837-626c-401e-91c7-384d2d45a796\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26"
Sep 18 19:42:01.570277 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:01.570247    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d4kn\" (UniqueName: \"kubernetes.io/projected/876ba837-626c-401e-91c7-384d2d45a796-kube-api-access-9d4kn\") pod \"isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26\" (UID: \"876ba837-626c-401e-91c7-384d2d45a796\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26"
Sep 18 19:42:02.067512 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:02.067466    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/876ba837-626c-401e-91c7-384d2d45a796-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26\" (UID: \"876ba837-626c-401e-91c7-384d2d45a796\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26"
Sep 18 19:42:02.070019 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:02.069987    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/876ba837-626c-401e-91c7-384d2d45a796-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26\" (UID: \"876ba837-626c-401e-91c7-384d2d45a796\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26"
Sep 18 19:42:02.279866 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:02.279823    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26"
Sep 18 19:42:02.583180 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:02.583103    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26"]
Sep 18 19:42:02.586175 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:42:02.586150    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod876ba837_626c_401e_91c7_384d2d45a796.slice/crio-d4f29709a73edd3c824c698c44b124619a8cdebb2473636fce207a0283afaff4 WatchSource:0}: Error finding container d4f29709a73edd3c824c698c44b124619a8cdebb2473636fce207a0283afaff4: Status 404 returned error can't find the container with id d4f29709a73edd3c824c698c44b124619a8cdebb2473636fce207a0283afaff4
Sep 18 19:42:03.523914 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:03.523874    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26" event={"ID":"876ba837-626c-401e-91c7-384d2d45a796","Type":"ContainerStarted","Data":"31febaae204bf72dc89fbf72bb4f8aa3be423e911b235d44a5b098f2fb11d2d5"}
Sep 18 19:42:03.524106 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:03.523942    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26" event={"ID":"876ba837-626c-401e-91c7-384d2d45a796","Type":"ContainerStarted","Data":"d4f29709a73edd3c824c698c44b124619a8cdebb2473636fce207a0283afaff4"}
Sep 18 19:42:05.509743 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:05.509704    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7"
Sep 18 19:42:05.510228 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:05.510123    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7"
Sep 18 19:42:05.511212 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:05.511168    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7" podUID="12da7015-ac39-4fe1-9d4b-75142f6f496a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 18 19:42:05.514144 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:05.514117    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-bzxwz"
Sep 18 19:42:05.514646 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:05.514608    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-bzxwz" podUID="9df2f2da-c098-407d-b943-a581bee7af4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 18 19:42:05.514749 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:05.514730    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7"
Sep 18 19:42:05.538958 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:05.538919    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7" podUID="12da7015-ac39-4fe1-9d4b-75142f6f496a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 18 19:42:06.543687 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:06.543648    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7" podUID="12da7015-ac39-4fe1-9d4b-75142f6f496a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 18 19:42:08.554875 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:08.554788    2522 generic.go:356] "Generic (PLEG): container finished" podID="876ba837-626c-401e-91c7-384d2d45a796" containerID="31febaae204bf72dc89fbf72bb4f8aa3be423e911b235d44a5b098f2fb11d2d5" exitCode=0
Sep 18 19:42:08.554875 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:08.554864    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26" event={"ID":"876ba837-626c-401e-91c7-384d2d45a796","Type":"ContainerDied","Data":"31febaae204bf72dc89fbf72bb4f8aa3be423e911b235d44a5b098f2fb11d2d5"}
Sep 18 19:42:09.562641 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:09.562595    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26" event={"ID":"876ba837-626c-401e-91c7-384d2d45a796","Type":"ContainerStarted","Data":"8639fad70ed7d607dd9c1f9ca7390438ff20fa5a549e590341e6125ab210e802"}
Sep 18 19:42:09.563018 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:09.562686    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26" event={"ID":"876ba837-626c-401e-91c7-384d2d45a796","Type":"ContainerStarted","Data":"67321f22092fa36e1f76c51205c74f3a5d5ea694d53f4315595c0fffa585e04a"}
Sep 18 19:42:09.563128 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:09.563099    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26"
Sep 18 19:42:09.563254 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:09.563239    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26"
Sep 18 19:42:09.564420 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:09.564387    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26" podUID="876ba837-626c-401e-91c7-384d2d45a796" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 18 19:42:09.591968 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:09.591921    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26" podStartSLOduration=3.196483712 podStartE2EDuration="8.591906311s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.395422599s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:42:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:42:09.588937986 +0000 UTC m=+1689.472464449" watchObservedRunningTime="2026-09-18 19:42:09.591906311 +0000 UTC m=+1689.475432773"
Sep 18 19:42:10.568280 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:10.568234    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26" podUID="876ba837-626c-401e-91c7-384d2d45a796" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 18 19:42:15.515584 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:15.515516    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-bzxwz" podUID="9df2f2da-c098-407d-b943-a581bee7af4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 18 19:42:15.573407 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:15.573376    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26"
Sep 18 19:42:15.573874 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:15.573847    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26" podUID="876ba837-626c-401e-91c7-384d2d45a796" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 18 19:42:16.543756 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:16.543701    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7" podUID="12da7015-ac39-4fe1-9d4b-75142f6f496a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 18 19:42:25.514840 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:25.514734    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-bzxwz" podUID="9df2f2da-c098-407d-b943-a581bee7af4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 18 19:42:25.575399 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:25.575363    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26"
Sep 18 19:42:26.544059 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:26.544016    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7" podUID="12da7015-ac39-4fe1-9d4b-75142f6f496a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 18 19:42:33.795867 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:33.795835    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26"]
Sep 18 19:42:33.796298 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:33.796087    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26" podUID="876ba837-626c-401e-91c7-384d2d45a796" containerName="kserve-container" containerID="cri-o://67321f22092fa36e1f76c51205c74f3a5d5ea694d53f4315595c0fffa585e04a" gracePeriod=30
Sep 18 19:42:33.796298 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:33.796137    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26" podUID="876ba837-626c-401e-91c7-384d2d45a796" containerName="kube-rbac-proxy" containerID="cri-o://8639fad70ed7d607dd9c1f9ca7390438ff20fa5a549e590341e6125ab210e802" gracePeriod=30
Sep 18 19:42:34.699049 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:34.699010    2522 generic.go:356] "Generic (PLEG): container finished" podID="876ba837-626c-401e-91c7-384d2d45a796" containerID="8639fad70ed7d607dd9c1f9ca7390438ff20fa5a549e590341e6125ab210e802" exitCode=2
Sep 18 19:42:34.699237 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:34.699075    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26" event={"ID":"876ba837-626c-401e-91c7-384d2d45a796","Type":"ContainerDied","Data":"8639fad70ed7d607dd9c1f9ca7390438ff20fa5a549e590341e6125ab210e802"}
Sep 18 19:42:35.052917 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:35.052893    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26"
Sep 18 19:42:35.162918 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:35.162875    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/876ba837-626c-401e-91c7-384d2d45a796-cabundle-cert\") pod \"876ba837-626c-401e-91c7-384d2d45a796\" (UID: \"876ba837-626c-401e-91c7-384d2d45a796\") "
Sep 18 19:42:35.162918 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:35.162922    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/876ba837-626c-401e-91c7-384d2d45a796-kserve-provision-location\") pod \"876ba837-626c-401e-91c7-384d2d45a796\" (UID: \"876ba837-626c-401e-91c7-384d2d45a796\") "
Sep 18 19:42:35.163136 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:35.162965    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/876ba837-626c-401e-91c7-384d2d45a796-kube-rbac-proxy-sar-config\") pod \"876ba837-626c-401e-91c7-384d2d45a796\" (UID: \"876ba837-626c-401e-91c7-384d2d45a796\") "
Sep 18 19:42:35.163136 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:35.163018    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/876ba837-626c-401e-91c7-384d2d45a796-proxy-tls\") pod \"876ba837-626c-401e-91c7-384d2d45a796\" (UID: \"876ba837-626c-401e-91c7-384d2d45a796\") "
Sep 18 19:42:35.163136 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:35.163061    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9d4kn\" (UniqueName: \"kubernetes.io/projected/876ba837-626c-401e-91c7-384d2d45a796-kube-api-access-9d4kn\") pod \"876ba837-626c-401e-91c7-384d2d45a796\" (UID: \"876ba837-626c-401e-91c7-384d2d45a796\") "
Sep 18 19:42:35.163405 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:35.163374    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/876ba837-626c-401e-91c7-384d2d45a796-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "876ba837-626c-401e-91c7-384d2d45a796" (UID: "876ba837-626c-401e-91c7-384d2d45a796"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:42:35.163488 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:35.163383    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/876ba837-626c-401e-91c7-384d2d45a796-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "876ba837-626c-401e-91c7-384d2d45a796" (UID: "876ba837-626c-401e-91c7-384d2d45a796"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:42:35.165348 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:35.165314    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/876ba837-626c-401e-91c7-384d2d45a796-kube-api-access-9d4kn" (OuterVolumeSpecName: "kube-api-access-9d4kn") pod "876ba837-626c-401e-91c7-384d2d45a796" (UID: "876ba837-626c-401e-91c7-384d2d45a796"). InnerVolumeSpecName "kube-api-access-9d4kn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:42:35.165348 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:35.165304    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/876ba837-626c-401e-91c7-384d2d45a796-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "876ba837-626c-401e-91c7-384d2d45a796" (UID: "876ba837-626c-401e-91c7-384d2d45a796"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:42:35.187352 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:35.187308    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/876ba837-626c-401e-91c7-384d2d45a796-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "876ba837-626c-401e-91c7-384d2d45a796" (UID: "876ba837-626c-401e-91c7-384d2d45a796"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:42:35.264844 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:35.264760    2522 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/876ba837-626c-401e-91c7-384d2d45a796-proxy-tls\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:42:35.264844 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:35.264790    2522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d4kn\" (UniqueName: \"kubernetes.io/projected/876ba837-626c-401e-91c7-384d2d45a796-kube-api-access-9d4kn\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:42:35.264844 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:35.264800    2522 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/876ba837-626c-401e-91c7-384d2d45a796-cabundle-cert\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:42:35.264844 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:35.264809    2522 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/876ba837-626c-401e-91c7-384d2d45a796-kserve-provision-location\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:42:35.264844 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:35.264817    2522 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/876ba837-626c-401e-91c7-384d2d45a796-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:42:35.514645 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:35.514595    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-bzxwz" podUID="9df2f2da-c098-407d-b943-a581bee7af4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 18 19:42:35.706104 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:35.706067    2522 generic.go:356] "Generic (PLEG): container finished" podID="876ba837-626c-401e-91c7-384d2d45a796" containerID="67321f22092fa36e1f76c51205c74f3a5d5ea694d53f4315595c0fffa585e04a" exitCode=0
Sep 18 19:42:35.706333 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:35.706124    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26" event={"ID":"876ba837-626c-401e-91c7-384d2d45a796","Type":"ContainerDied","Data":"67321f22092fa36e1f76c51205c74f3a5d5ea694d53f4315595c0fffa585e04a"}
Sep 18 19:42:35.706333 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:35.706150    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26"
Sep 18 19:42:35.706333 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:35.706197    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26" event={"ID":"876ba837-626c-401e-91c7-384d2d45a796","Type":"ContainerDied","Data":"d4f29709a73edd3c824c698c44b124619a8cdebb2473636fce207a0283afaff4"}
Sep 18 19:42:35.706333 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:35.706223    2522 scope.go:117] "RemoveContainer" containerID="8639fad70ed7d607dd9c1f9ca7390438ff20fa5a549e590341e6125ab210e802"
Sep 18 19:42:35.717721 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:35.717703    2522 scope.go:117] "RemoveContainer" containerID="67321f22092fa36e1f76c51205c74f3a5d5ea694d53f4315595c0fffa585e04a"
Sep 18 19:42:35.725695 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:35.725673    2522 scope.go:117] "RemoveContainer" containerID="31febaae204bf72dc89fbf72bb4f8aa3be423e911b235d44a5b098f2fb11d2d5"
Sep 18 19:42:35.734115 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:35.734072    2522 scope.go:117] "RemoveContainer" containerID="8639fad70ed7d607dd9c1f9ca7390438ff20fa5a549e590341e6125ab210e802"
Sep 18 19:42:35.734769 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:42:35.734747    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8639fad70ed7d607dd9c1f9ca7390438ff20fa5a549e590341e6125ab210e802\": container with ID starting with 8639fad70ed7d607dd9c1f9ca7390438ff20fa5a549e590341e6125ab210e802 not found: ID does not exist" containerID="8639fad70ed7d607dd9c1f9ca7390438ff20fa5a549e590341e6125ab210e802"
Sep 18 19:42:35.734907 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:35.734776    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8639fad70ed7d607dd9c1f9ca7390438ff20fa5a549e590341e6125ab210e802"} err="failed to get container status \"8639fad70ed7d607dd9c1f9ca7390438ff20fa5a549e590341e6125ab210e802\": rpc error: code = NotFound desc = could not find container \"8639fad70ed7d607dd9c1f9ca7390438ff20fa5a549e590341e6125ab210e802\": container with ID starting with 8639fad70ed7d607dd9c1f9ca7390438ff20fa5a549e590341e6125ab210e802 not found: ID does not exist"
Sep 18 19:42:35.734907 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:35.734800    2522 scope.go:117] "RemoveContainer" containerID="67321f22092fa36e1f76c51205c74f3a5d5ea694d53f4315595c0fffa585e04a"
Sep 18 19:42:35.735111 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:42:35.735085    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67321f22092fa36e1f76c51205c74f3a5d5ea694d53f4315595c0fffa585e04a\": container with ID starting with 67321f22092fa36e1f76c51205c74f3a5d5ea694d53f4315595c0fffa585e04a not found: ID does not exist" containerID="67321f22092fa36e1f76c51205c74f3a5d5ea694d53f4315595c0fffa585e04a"
Sep 18 19:42:35.735166 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:35.735122    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67321f22092fa36e1f76c51205c74f3a5d5ea694d53f4315595c0fffa585e04a"} err="failed to get container status \"67321f22092fa36e1f76c51205c74f3a5d5ea694d53f4315595c0fffa585e04a\": rpc error: code = NotFound desc = could not find container \"67321f22092fa36e1f76c51205c74f3a5d5ea694d53f4315595c0fffa585e04a\": container with ID starting with 67321f22092fa36e1f76c51205c74f3a5d5ea694d53f4315595c0fffa585e04a not found: ID does not exist"
Sep 18 19:42:35.735166 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:35.735145    2522 scope.go:117] "RemoveContainer" containerID="31febaae204bf72dc89fbf72bb4f8aa3be423e911b235d44a5b098f2fb11d2d5"
Sep 18 19:42:35.735455 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:42:35.735434    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31febaae204bf72dc89fbf72bb4f8aa3be423e911b235d44a5b098f2fb11d2d5\": container with ID starting with 31febaae204bf72dc89fbf72bb4f8aa3be423e911b235d44a5b098f2fb11d2d5 not found: ID does not exist" containerID="31febaae204bf72dc89fbf72bb4f8aa3be423e911b235d44a5b098f2fb11d2d5"
Sep 18 19:42:35.735515 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:35.735463    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31febaae204bf72dc89fbf72bb4f8aa3be423e911b235d44a5b098f2fb11d2d5"} err="failed to get container status \"31febaae204bf72dc89fbf72bb4f8aa3be423e911b235d44a5b098f2fb11d2d5\": rpc error: code = NotFound desc = could not find container \"31febaae204bf72dc89fbf72bb4f8aa3be423e911b235d44a5b098f2fb11d2d5\": container with ID starting with 31febaae204bf72dc89fbf72bb4f8aa3be423e911b235d44a5b098f2fb11d2d5 not found: ID does not exist"
Sep 18 19:42:35.737198 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:35.737176    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26"]
Sep 18 19:42:35.740837 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:35.740817    2522 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-99bbb4b5d-qgj26"]
Sep 18 19:42:36.543643 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:36.543600    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7" podUID="12da7015-ac39-4fe1-9d4b-75142f6f496a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 18 19:42:36.722030 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:36.721987    2522 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="876ba837-626c-401e-91c7-384d2d45a796" path="/var/lib/kubelet/pods/876ba837-626c-401e-91c7-384d2d45a796/volumes"
Sep 18 19:42:45.515291 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:45.515253    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-bzxwz" podUID="9df2f2da-c098-407d-b943-a581bee7af4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 18 19:42:46.544349 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:46.544303    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7" podUID="12da7015-ac39-4fe1-9d4b-75142f6f496a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 18 19:42:55.514810 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:55.514769    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-bzxwz" podUID="9df2f2da-c098-407d-b943-a581bee7af4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 18 19:42:56.544043 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:42:56.543997    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7" podUID="12da7015-ac39-4fe1-9d4b-75142f6f496a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 18 19:43:05.515346 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:05.515300    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-bzxwz" podUID="9df2f2da-c098-407d-b943-a581bee7af4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 18 19:43:06.544142 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:06.544100    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7" podUID="12da7015-ac39-4fe1-9d4b-75142f6f496a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 18 19:43:15.515263 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:15.515216    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-bzxwz" podUID="9df2f2da-c098-407d-b943-a581bee7af4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 18 19:43:16.543742 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:16.543688    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7" podUID="12da7015-ac39-4fe1-9d4b-75142f6f496a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 18 19:43:25.516528 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:25.516497    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-bzxwz"
Sep 18 19:43:26.544779 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:26.544743    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7"
Sep 18 19:43:33.612157 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:33.612119    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-bzxwz"]
Sep 18 19:43:33.612618 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:33.612360    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-bzxwz" podUID="9df2f2da-c098-407d-b943-a581bee7af4f" containerName="kserve-container" containerID="cri-o://718115c47b0ac13353cc8e0c1e8a0f313f19c55dc75d8ca81e13791bbffd8c7b" gracePeriod=30
Sep 18 19:43:33.612618 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:33.612396    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-bzxwz" podUID="9df2f2da-c098-407d-b943-a581bee7af4f" containerName="kube-rbac-proxy" containerID="cri-o://c16074ebe032c91ea43848d36ce57e90e6de7c2b12380ddf7458b80c388246ed" gracePeriod=30
Sep 18 19:43:33.624369 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:33.624345    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7"]
Sep 18 19:43:33.624599 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:33.624552    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7" podUID="12da7015-ac39-4fe1-9d4b-75142f6f496a" containerName="kserve-container" containerID="cri-o://dc6421446d58eb3b702d1dba652f469f9d8fd453d1a63cf6fec7c85a955b5411" gracePeriod=30
Sep 18 19:43:33.624697 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:33.624625    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7" podUID="12da7015-ac39-4fe1-9d4b-75142f6f496a" containerName="kube-rbac-proxy" containerID="cri-o://ecff1d3b590205ad7cf301bf5f93e30a7a1e69b2d758e4bcc7d101dd1c8f524e" gracePeriod=30
Sep 18 19:43:33.986902 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:33.986818    2522 generic.go:356] "Generic (PLEG): container finished" podID="12da7015-ac39-4fe1-9d4b-75142f6f496a" containerID="ecff1d3b590205ad7cf301bf5f93e30a7a1e69b2d758e4bcc7d101dd1c8f524e" exitCode=2
Sep 18 19:43:33.986902 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:33.986891    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7" event={"ID":"12da7015-ac39-4fe1-9d4b-75142f6f496a","Type":"ContainerDied","Data":"ecff1d3b590205ad7cf301bf5f93e30a7a1e69b2d758e4bcc7d101dd1c8f524e"}
Sep 18 19:43:33.989069 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:33.989045    2522 generic.go:356] "Generic (PLEG): container finished" podID="9df2f2da-c098-407d-b943-a581bee7af4f" containerID="c16074ebe032c91ea43848d36ce57e90e6de7c2b12380ddf7458b80c388246ed" exitCode=2
Sep 18 19:43:33.989186 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:33.989097    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-bzxwz" event={"ID":"9df2f2da-c098-407d-b943-a581bee7af4f","Type":"ContainerDied","Data":"c16074ebe032c91ea43848d36ce57e90e6de7c2b12380ddf7458b80c388246ed"}
Sep 18 19:43:35.508980 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:35.508937    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-bzxwz" podUID="9df2f2da-c098-407d-b943-a581bee7af4f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.46:8643/healthz\": dial tcp 10.132.0.46:8643: connect: connection refused"
Sep 18 19:43:35.515184 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:35.515146    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-bzxwz" podUID="9df2f2da-c098-407d-b943-a581bee7af4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 18 19:43:35.515303 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:35.515150    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7" podUID="12da7015-ac39-4fe1-9d4b-75142f6f496a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.45:8643/healthz\": dial tcp 10.132.0.45:8643: connect: connection refused"
Sep 18 19:43:36.544183 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:36.544139    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7" podUID="12da7015-ac39-4fe1-9d4b-75142f6f496a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 18 19:43:38.847168 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:38.847053    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-5rffs"]
Sep 18 19:43:38.847704 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:38.847680    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="876ba837-626c-401e-91c7-384d2d45a796" containerName="kserve-container"
Sep 18 19:43:38.847704 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:38.847700    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="876ba837-626c-401e-91c7-384d2d45a796" containerName="kserve-container"
Sep 18 19:43:38.847860 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:38.847725    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="876ba837-626c-401e-91c7-384d2d45a796" containerName="storage-initializer"
Sep 18 19:43:38.847860 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:38.847734    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="876ba837-626c-401e-91c7-384d2d45a796" containerName="storage-initializer"
Sep 18 19:43:38.847860 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:38.847747    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="876ba837-626c-401e-91c7-384d2d45a796" containerName="kube-rbac-proxy"
Sep 18 19:43:38.847860 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:38.847755    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="876ba837-626c-401e-91c7-384d2d45a796" containerName="kube-rbac-proxy"
Sep 18 19:43:38.847860 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:38.847861    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="876ba837-626c-401e-91c7-384d2d45a796" containerName="kube-rbac-proxy"
Sep 18 19:43:38.848161 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:38.847872    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="876ba837-626c-401e-91c7-384d2d45a796" containerName="kserve-container"
Sep 18 19:43:38.851576 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:38.851541    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-5rffs"
Sep 18 19:43:38.854407 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:38.854385    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 18 19:43:38.854582 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:38.854538    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 18 19:43:38.869160 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:38.869136    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-5rffs"]
Sep 18 19:43:38.899911 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:38.899889    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-bzxwz"
Sep 18 19:43:38.912823 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:38.912803    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7"
Sep 18 19:43:38.932093 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:38.932067    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9df2f2da-c098-407d-b943-a581bee7af4f-cabundle-cert\") pod \"9df2f2da-c098-407d-b943-a581bee7af4f\" (UID: \"9df2f2da-c098-407d-b943-a581bee7af4f\") "
Sep 18 19:43:38.932255 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:38.932111    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9df2f2da-c098-407d-b943-a581bee7af4f-kserve-provision-location\") pod \"9df2f2da-c098-407d-b943-a581bee7af4f\" (UID: \"9df2f2da-c098-407d-b943-a581bee7af4f\") "
Sep 18 19:43:38.932255 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:38.932151    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9df2f2da-c098-407d-b943-a581bee7af4f-proxy-tls\") pod \"9df2f2da-c098-407d-b943-a581bee7af4f\" (UID: \"9df2f2da-c098-407d-b943-a581bee7af4f\") "
Sep 18 19:43:38.932255 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:38.932179    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bkh22\" (UniqueName: \"kubernetes.io/projected/9df2f2da-c098-407d-b943-a581bee7af4f-kube-api-access-bkh22\") pod \"9df2f2da-c098-407d-b943-a581bee7af4f\" (UID: \"9df2f2da-c098-407d-b943-a581bee7af4f\") "
Sep 18 19:43:38.932255 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:38.932235    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9df2f2da-c098-407d-b943-a581bee7af4f-kube-rbac-proxy-sar-config\") pod \"9df2f2da-c098-407d-b943-a581bee7af4f\" (UID: \"9df2f2da-c098-407d-b943-a581bee7af4f\") "
Sep 18 19:43:38.932505 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:38.932352    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z75jf\" (UniqueName: \"kubernetes.io/projected/61e90e37-8b27-4d0b-b6c0-7272ae55e207-kube-api-access-z75jf\") pod \"isvc-canary-promote-predictor-75f64ff6c9-5rffs\" (UID: \"61e90e37-8b27-4d0b-b6c0-7272ae55e207\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-5rffs"
Sep 18 19:43:38.932505 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:38.932408    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61e90e37-8b27-4d0b-b6c0-7272ae55e207-proxy-tls\") pod \"isvc-canary-promote-predictor-75f64ff6c9-5rffs\" (UID: \"61e90e37-8b27-4d0b-b6c0-7272ae55e207\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-5rffs"
Sep 18 19:43:38.932505 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:38.932452    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61e90e37-8b27-4d0b-b6c0-7272ae55e207-kserve-provision-location\") pod \"isvc-canary-promote-predictor-75f64ff6c9-5rffs\" (UID: \"61e90e37-8b27-4d0b-b6c0-7272ae55e207\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-5rffs"
Sep 18 19:43:38.932676 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:38.932513    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/61e90e37-8b27-4d0b-b6c0-7272ae55e207-cabundle-cert\") pod \"isvc-canary-promote-predictor-75f64ff6c9-5rffs\" (UID: \"61e90e37-8b27-4d0b-b6c0-7272ae55e207\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-5rffs"
Sep 18 19:43:38.932676 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:38.932576    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/61e90e37-8b27-4d0b-b6c0-7272ae55e207-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-75f64ff6c9-5rffs\" (UID: \"61e90e37-8b27-4d0b-b6c0-7272ae55e207\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-5rffs"
Sep 18 19:43:38.932946 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:38.932920    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9df2f2da-c098-407d-b943-a581bee7af4f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9df2f2da-c098-407d-b943-a581bee7af4f" (UID: "9df2f2da-c098-407d-b943-a581bee7af4f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:43:38.933546 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:38.933231    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9df2f2da-c098-407d-b943-a581bee7af4f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9df2f2da-c098-407d-b943-a581bee7af4f" (UID: "9df2f2da-c098-407d-b943-a581bee7af4f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:43:38.933546 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:38.933341    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9df2f2da-c098-407d-b943-a581bee7af4f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9df2f2da-c098-407d-b943-a581bee7af4f" (UID: "9df2f2da-c098-407d-b943-a581bee7af4f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:43:38.935610 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:38.935587    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9df2f2da-c098-407d-b943-a581bee7af4f-kube-api-access-bkh22" (OuterVolumeSpecName: "kube-api-access-bkh22") pod "9df2f2da-c098-407d-b943-a581bee7af4f" (UID: "9df2f2da-c098-407d-b943-a581bee7af4f"). InnerVolumeSpecName "kube-api-access-bkh22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:43:38.935899 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:38.935873    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9df2f2da-c098-407d-b943-a581bee7af4f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9df2f2da-c098-407d-b943-a581bee7af4f" (UID: "9df2f2da-c098-407d-b943-a581bee7af4f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:43:39.015858 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:39.015822    2522 generic.go:356] "Generic (PLEG): container finished" podID="12da7015-ac39-4fe1-9d4b-75142f6f496a" containerID="dc6421446d58eb3b702d1dba652f469f9d8fd453d1a63cf6fec7c85a955b5411" exitCode=0
Sep 18 19:43:39.016023 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:39.015876    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7"
Sep 18 19:43:39.016023 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:39.015899    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7" event={"ID":"12da7015-ac39-4fe1-9d4b-75142f6f496a","Type":"ContainerDied","Data":"dc6421446d58eb3b702d1dba652f469f9d8fd453d1a63cf6fec7c85a955b5411"}
Sep 18 19:43:39.016023 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:39.015970    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7" event={"ID":"12da7015-ac39-4fe1-9d4b-75142f6f496a","Type":"ContainerDied","Data":"18d22c530556a0a2e6b66c00d4a2a50b6eaab40b8d04bea9785992c61301d902"}
Sep 18 19:43:39.016023 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:39.016007    2522 scope.go:117] "RemoveContainer" containerID="ecff1d3b590205ad7cf301bf5f93e30a7a1e69b2d758e4bcc7d101dd1c8f524e"
Sep 18 19:43:39.018066 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:39.018042    2522 generic.go:356] "Generic (PLEG): container finished" podID="9df2f2da-c098-407d-b943-a581bee7af4f" containerID="718115c47b0ac13353cc8e0c1e8a0f313f19c55dc75d8ca81e13791bbffd8c7b" exitCode=0
Sep 18 19:43:39.018174 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:39.018091    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-bzxwz"
Sep 18 19:43:39.018174 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:39.018115    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-bzxwz" event={"ID":"9df2f2da-c098-407d-b943-a581bee7af4f","Type":"ContainerDied","Data":"718115c47b0ac13353cc8e0c1e8a0f313f19c55dc75d8ca81e13791bbffd8c7b"}
Sep 18 19:43:39.018174 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:39.018162    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-bzxwz" event={"ID":"9df2f2da-c098-407d-b943-a581bee7af4f","Type":"ContainerDied","Data":"fef30f7267bc52aaa13aa9872942b9bcf7676ff99b83723116fc2e8246a78157"}
Sep 18 19:43:39.027388 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:39.027031    2522 scope.go:117] "RemoveContainer" containerID="dc6421446d58eb3b702d1dba652f469f9d8fd453d1a63cf6fec7c85a955b5411"
Sep 18 19:43:39.032970 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:39.032950    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qgq7w\" (UniqueName: \"kubernetes.io/projected/12da7015-ac39-4fe1-9d4b-75142f6f496a-kube-api-access-qgq7w\") pod \"12da7015-ac39-4fe1-9d4b-75142f6f496a\" (UID: \"12da7015-ac39-4fe1-9d4b-75142f6f496a\") "
Sep 18 19:43:39.033061 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:39.032988    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/12da7015-ac39-4fe1-9d4b-75142f6f496a-cabundle-cert\") pod \"12da7015-ac39-4fe1-9d4b-75142f6f496a\" (UID: \"12da7015-ac39-4fe1-9d4b-75142f6f496a\") "
Sep 18 19:43:39.033061 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:39.033046    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12da7015-ac39-4fe1-9d4b-75142f6f496a-kserve-provision-location\") pod \"12da7015-ac39-4fe1-9d4b-75142f6f496a\" (UID: \"12da7015-ac39-4fe1-9d4b-75142f6f496a\") "
Sep 18 19:43:39.033146 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:39.033088    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12da7015-ac39-4fe1-9d4b-75142f6f496a-kube-rbac-proxy-sar-config\") pod \"12da7015-ac39-4fe1-9d4b-75142f6f496a\" (UID: \"12da7015-ac39-4fe1-9d4b-75142f6f496a\") "
Sep 18 19:43:39.033146 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:39.033127    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12da7015-ac39-4fe1-9d4b-75142f6f496a-proxy-tls\") pod \"12da7015-ac39-4fe1-9d4b-75142f6f496a\" (UID: \"12da7015-ac39-4fe1-9d4b-75142f6f496a\") "
Sep 18 19:43:39.033272 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:39.033254    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z75jf\" (UniqueName: \"kubernetes.io/projected/61e90e37-8b27-4d0b-b6c0-7272ae55e207-kube-api-access-z75jf\") pod \"isvc-canary-promote-predictor-75f64ff6c9-5rffs\" (UID: \"61e90e37-8b27-4d0b-b6c0-7272ae55e207\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-5rffs"
Sep 18 19:43:39.034417 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:39.033317    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61e90e37-8b27-4d0b-b6c0-7272ae55e207-proxy-tls\") pod \"isvc-canary-promote-predictor-75f64ff6c9-5rffs\" (UID: \"61e90e37-8b27-4d0b-b6c0-7272ae55e207\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-5rffs"
Sep 18 19:43:39.034417 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:39.033373    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61e90e37-8b27-4d0b-b6c0-7272ae55e207-kserve-provision-location\") pod \"isvc-canary-promote-predictor-75f64ff6c9-5rffs\" (UID: \"61e90e37-8b27-4d0b-b6c0-7272ae55e207\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-5rffs"
Sep 18 19:43:39.034417 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:39.033407    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12da7015-ac39-4fe1-9d4b-75142f6f496a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "12da7015-ac39-4fe1-9d4b-75142f6f496a" (UID: "12da7015-ac39-4fe1-9d4b-75142f6f496a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:43:39.034417 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:39.033433    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/61e90e37-8b27-4d0b-b6c0-7272ae55e207-cabundle-cert\") pod \"isvc-canary-promote-predictor-75f64ff6c9-5rffs\" (UID: \"61e90e37-8b27-4d0b-b6c0-7272ae55e207\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-5rffs"
Sep 18 19:43:39.034417 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:39.033479    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/61e90e37-8b27-4d0b-b6c0-7272ae55e207-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-75f64ff6c9-5rffs\" (UID: \"61e90e37-8b27-4d0b-b6c0-7272ae55e207\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-5rffs"
Sep 18 19:43:39.034417 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:39.033576    2522 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/12da7015-ac39-4fe1-9d4b-75142f6f496a-cabundle-cert\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:43:39.034417 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:39.033593    2522 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9df2f2da-c098-407d-b943-a581bee7af4f-cabundle-cert\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:43:39.034417 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:39.033608    2522 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9df2f2da-c098-407d-b943-a581bee7af4f-kserve-provision-location\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:43:39.034417 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:39.033621    2522 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9df2f2da-c098-407d-b943-a581bee7af4f-proxy-tls\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:43:39.034417 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:39.033634    2522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkh22\" (UniqueName: \"kubernetes.io/projected/9df2f2da-c098-407d-b943-a581bee7af4f-kube-api-access-bkh22\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:43:39.034417 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:39.033646    2522 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9df2f2da-c098-407d-b943-a581bee7af4f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:43:39.034417 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:39.033472    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12da7015-ac39-4fe1-9d4b-75142f6f496a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "12da7015-ac39-4fe1-9d4b-75142f6f496a" (UID: "12da7015-ac39-4fe1-9d4b-75142f6f496a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:43:39.034417 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:39.034030    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61e90e37-8b27-4d0b-b6c0-7272ae55e207-kserve-provision-location\") pod \"isvc-canary-promote-predictor-75f64ff6c9-5rffs\" (UID: \"61e90e37-8b27-4d0b-b6c0-7272ae55e207\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-5rffs"
Sep 18 19:43:39.034417 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:39.034324    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/61e90e37-8b27-4d0b-b6c0-7272ae55e207-cabundle-cert\") pod \"isvc-canary-promote-predictor-75f64ff6c9-5rffs\" (UID: \"61e90e37-8b27-4d0b-b6c0-7272ae55e207\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-5rffs"
Sep 18 19:43:39.034417 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:39.034320    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/61e90e37-8b27-4d0b-b6c0-7272ae55e207-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-75f64ff6c9-5rffs\" (UID: \"61e90e37-8b27-4d0b-b6c0-7272ae55e207\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-5rffs"
Sep 18 19:43:39.035582 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:39.034603    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12da7015-ac39-4fe1-9d4b-75142f6f496a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "12da7015-ac39-4fe1-9d4b-75142f6f496a" (UID: "12da7015-ac39-4fe1-9d4b-75142f6f496a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:43:39.035582 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:39.035276    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12da7015-ac39-4fe1-9d4b-75142f6f496a-kube-api-access-qgq7w" (OuterVolumeSpecName: "kube-api-access-qgq7w") pod "12da7015-ac39-4fe1-9d4b-75142f6f496a" (UID: "12da7015-ac39-4fe1-9d4b-75142f6f496a"). InnerVolumeSpecName "kube-api-access-qgq7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:43:39.035738 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:39.035712    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12da7015-ac39-4fe1-9d4b-75142f6f496a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "12da7015-ac39-4fe1-9d4b-75142f6f496a" (UID: "12da7015-ac39-4fe1-9d4b-75142f6f496a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:43:39.036200 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:39.036164    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61e90e37-8b27-4d0b-b6c0-7272ae55e207-proxy-tls\") pod \"isvc-canary-promote-predictor-75f64ff6c9-5rffs\" (UID: \"61e90e37-8b27-4d0b-b6c0-7272ae55e207\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-5rffs"
Sep 18 19:43:39.036287 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:39.036272    2522 scope.go:117] "RemoveContainer" containerID="5a068d47cf0709e2ac179729e6562b543b3aeadbd37ac88e02469ed75c7b477f"
Sep 18 19:43:39.044070 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:39.044053    2522 scope.go:117] "RemoveContainer" containerID="ecff1d3b590205ad7cf301bf5f93e30a7a1e69b2d758e4bcc7d101dd1c8f524e"
Sep 18 19:43:39.044326 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:43:39.044308    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecff1d3b590205ad7cf301bf5f93e30a7a1e69b2d758e4bcc7d101dd1c8f524e\": container with ID starting with ecff1d3b590205ad7cf301bf5f93e30a7a1e69b2d758e4bcc7d101dd1c8f524e not found: ID does not exist" containerID="ecff1d3b590205ad7cf301bf5f93e30a7a1e69b2d758e4bcc7d101dd1c8f524e"
Sep 18 19:43:39.044375 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:39.044333    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecff1d3b590205ad7cf301bf5f93e30a7a1e69b2d758e4bcc7d101dd1c8f524e"} err="failed to get container status \"ecff1d3b590205ad7cf301bf5f93e30a7a1e69b2d758e4bcc7d101dd1c8f524e\": rpc error: code = NotFound desc = could not find container \"ecff1d3b590205ad7cf301bf5f93e30a7a1e69b2d758e4bcc7d101dd1c8f524e\": container with ID starting with ecff1d3b590205ad7cf301bf5f93e30a7a1e69b2d758e4bcc7d101dd1c8f524e not found: ID does not exist"
Sep 18 19:43:39.044375 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:39.044350    2522 scope.go:117] "RemoveContainer" containerID="dc6421446d58eb3b702d1dba652f469f9d8fd453d1a63cf6fec7c85a955b5411"
Sep 18 19:43:39.044760 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:43:39.044741    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc6421446d58eb3b702d1dba652f469f9d8fd453d1a63cf6fec7c85a955b5411\": container with ID starting with dc6421446d58eb3b702d1dba652f469f9d8fd453d1a63cf6fec7c85a955b5411 not found: ID does not exist" containerID="dc6421446d58eb3b702d1dba652f469f9d8fd453d1a63cf6fec7c85a955b5411"
Sep 18 19:43:39.044827 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:39.044767    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc6421446d58eb3b702d1dba652f469f9d8fd453d1a63cf6fec7c85a955b5411"} err="failed to get container status \"dc6421446d58eb3b702d1dba652f469f9d8fd453d1a63cf6fec7c85a955b5411\": rpc error: code = NotFound desc = could not find container \"dc6421446d58eb3b702d1dba652f469f9d8fd453d1a63cf6fec7c85a955b5411\": container with ID starting with dc6421446d58eb3b702d1dba652f469f9d8fd453d1a63cf6fec7c85a955b5411 not found: ID does not exist"
Sep 18 19:43:39.044827 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:39.044788    2522 scope.go:117] "RemoveContainer" containerID="5a068d47cf0709e2ac179729e6562b543b3aeadbd37ac88e02469ed75c7b477f"
Sep 18 19:43:39.045066 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:43:39.045045    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a068d47cf0709e2ac179729e6562b543b3aeadbd37ac88e02469ed75c7b477f\": container with ID starting with 5a068d47cf0709e2ac179729e6562b543b3aeadbd37ac88e02469ed75c7b477f not found: ID does not exist" containerID="5a068d47cf0709e2ac179729e6562b543b3aeadbd37ac88e02469ed75c7b477f"
Sep 18 19:43:39.045119 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:39.045073    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a068d47cf0709e2ac179729e6562b543b3aeadbd37ac88e02469ed75c7b477f"} err="failed to get container status \"5a068d47cf0709e2ac179729e6562b543b3aeadbd37ac88e02469ed75c7b477f\": rpc error: code = NotFound desc = could not find container \"5a068d47cf0709e2ac179729e6562b543b3aeadbd37ac88e02469ed75c7b477f\": container with ID starting with 5a068d47cf0709e2ac179729e6562b543b3aeadbd37ac88e02469ed75c7b477f not found: ID does not exist"
Sep 18 19:43:39.045119 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:39.045091    2522 scope.go:117] "RemoveContainer" containerID="c16074ebe032c91ea43848d36ce57e90e6de7c2b12380ddf7458b80c388246ed"
Sep 18 19:43:39.047000 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:39.046972    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z75jf\" (UniqueName: \"kubernetes.io/projected/61e90e37-8b27-4d0b-b6c0-7272ae55e207-kube-api-access-z75jf\") pod \"isvc-canary-promote-predictor-75f64ff6c9-5rffs\" (UID: \"61e90e37-8b27-4d0b-b6c0-7272ae55e207\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-5rffs"
Sep 18 19:43:39.052368 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:39.052352    2522 scope.go:117] "RemoveContainer" containerID="718115c47b0ac13353cc8e0c1e8a0f313f19c55dc75d8ca81e13791bbffd8c7b"
Sep 18 19:43:39.059636 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:39.059622    2522 scope.go:117] "RemoveContainer" containerID="5b44d5e943327ef3781a7e2bfae916d55989eeac314a34bd4356c3649bd96925"
Sep 18 19:43:39.064437 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:39.064416    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-bzxwz"]
Sep 18 19:43:39.067251 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:39.067233    2522 scope.go:117] "RemoveContainer" containerID="c16074ebe032c91ea43848d36ce57e90e6de7c2b12380ddf7458b80c388246ed"
Sep 18 19:43:39.067503 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:43:39.067485    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c16074ebe032c91ea43848d36ce57e90e6de7c2b12380ddf7458b80c388246ed\": container with ID starting with c16074ebe032c91ea43848d36ce57e90e6de7c2b12380ddf7458b80c388246ed not found: ID does not exist" containerID="c16074ebe032c91ea43848d36ce57e90e6de7c2b12380ddf7458b80c388246ed"
Sep 18 19:43:39.067583 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:39.067509    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c16074ebe032c91ea43848d36ce57e90e6de7c2b12380ddf7458b80c388246ed"} err="failed to get container status \"c16074ebe032c91ea43848d36ce57e90e6de7c2b12380ddf7458b80c388246ed\": rpc error: code = NotFound desc = could not find container \"c16074ebe032c91ea43848d36ce57e90e6de7c2b12380ddf7458b80c388246ed\": container with ID starting with c16074ebe032c91ea43848d36ce57e90e6de7c2b12380ddf7458b80c388246ed not found: ID does not exist"
Sep 18 19:43:39.067583 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:39.067527    2522 scope.go:117] "RemoveContainer" containerID="718115c47b0ac13353cc8e0c1e8a0f313f19c55dc75d8ca81e13791bbffd8c7b"
Sep 18 19:43:39.067797 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:43:39.067777    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"718115c47b0ac13353cc8e0c1e8a0f313f19c55dc75d8ca81e13791bbffd8c7b\": container with ID starting with 718115c47b0ac13353cc8e0c1e8a0f313f19c55dc75d8ca81e13791bbffd8c7b not found: ID does not exist" containerID="718115c47b0ac13353cc8e0c1e8a0f313f19c55dc75d8ca81e13791bbffd8c7b"
Sep 18 19:43:39.067874 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:39.067804    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"718115c47b0ac13353cc8e0c1e8a0f313f19c55dc75d8ca81e13791bbffd8c7b"} err="failed to get container status \"718115c47b0ac13353cc8e0c1e8a0f313f19c55dc75d8ca81e13791bbffd8c7b\": rpc error: code = NotFound desc = could not find container \"718115c47b0ac13353cc8e0c1e8a0f313f19c55dc75d8ca81e13791bbffd8c7b\": container with ID starting with 718115c47b0ac13353cc8e0c1e8a0f313f19c55dc75d8ca81e13791bbffd8c7b not found: ID does not exist"
Sep 18 19:43:39.067874 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:39.067821    2522 scope.go:117] "RemoveContainer" containerID="5b44d5e943327ef3781a7e2bfae916d55989eeac314a34bd4356c3649bd96925"
Sep 18 19:43:39.068100 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:43:39.068084    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b44d5e943327ef3781a7e2bfae916d55989eeac314a34bd4356c3649bd96925\": container with ID starting with 5b44d5e943327ef3781a7e2bfae916d55989eeac314a34bd4356c3649bd96925 not found: ID does not exist" containerID="5b44d5e943327ef3781a7e2bfae916d55989eeac314a34bd4356c3649bd96925"
Sep 18 19:43:39.068148 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:39.068109    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b44d5e943327ef3781a7e2bfae916d55989eeac314a34bd4356c3649bd96925"} err="failed to get container status \"5b44d5e943327ef3781a7e2bfae916d55989eeac314a34bd4356c3649bd96925\": rpc error: code = NotFound desc = could not find container \"5b44d5e943327ef3781a7e2bfae916d55989eeac314a34bd4356c3649bd96925\": container with ID starting with 5b44d5e943327ef3781a7e2bfae916d55989eeac314a34bd4356c3649bd96925 not found: ID does not exist"
Sep 18 19:43:39.071633 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:39.071615    2522 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-bzxwz"]
Sep 18 19:43:39.134725 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:39.134697    2522 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12da7015-ac39-4fe1-9d4b-75142f6f496a-kserve-provision-location\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:43:39.134725 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:39.134722    2522 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12da7015-ac39-4fe1-9d4b-75142f6f496a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:43:39.134898 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:39.134733    2522 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12da7015-ac39-4fe1-9d4b-75142f6f496a-proxy-tls\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:43:39.134898 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:39.134742    2522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgq7w\" (UniqueName: \"kubernetes.io/projected/12da7015-ac39-4fe1-9d4b-75142f6f496a-kube-api-access-qgq7w\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:43:39.164007 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:39.163926    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-5rffs"
Sep 18 19:43:39.343772 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:39.343743    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7"]
Sep 18 19:43:39.348388 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:39.348364    2522 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5c6467bc9d-wdpb7"]
Sep 18 19:43:39.444134 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:39.444087    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-5rffs"]
Sep 18 19:43:39.446405 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:43:39.446378    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61e90e37_8b27_4d0b_b6c0_7272ae55e207.slice/crio-e00fe9ec4bfe845c7f59905468ff4424e69502c3b3dacfb1eb4763de603fc4d4 WatchSource:0}: Error finding container e00fe9ec4bfe845c7f59905468ff4424e69502c3b3dacfb1eb4763de603fc4d4: Status 404 returned error can't find the container with id e00fe9ec4bfe845c7f59905468ff4424e69502c3b3dacfb1eb4763de603fc4d4
Sep 18 19:43:40.033436 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:40.033390    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-5rffs" event={"ID":"61e90e37-8b27-4d0b-b6c0-7272ae55e207","Type":"ContainerStarted","Data":"82f8516454de95fd02c2bdc7e8db236ed43381a81b6cfaec6ebc5a26a41c50bb"}
Sep 18 19:43:40.033908 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:40.033504    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-5rffs" event={"ID":"61e90e37-8b27-4d0b-b6c0-7272ae55e207","Type":"ContainerStarted","Data":"e00fe9ec4bfe845c7f59905468ff4424e69502c3b3dacfb1eb4763de603fc4d4"}
Sep 18 19:43:40.721304 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:40.721268    2522 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12da7015-ac39-4fe1-9d4b-75142f6f496a" path="/var/lib/kubelet/pods/12da7015-ac39-4fe1-9d4b-75142f6f496a/volumes"
Sep 18 19:43:40.721808 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:40.721794    2522 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9df2f2da-c098-407d-b943-a581bee7af4f" path="/var/lib/kubelet/pods/9df2f2da-c098-407d-b943-a581bee7af4f/volumes"
Sep 18 19:43:44.055288 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:44.055248    2522 generic.go:356] "Generic (PLEG): container finished" podID="61e90e37-8b27-4d0b-b6c0-7272ae55e207" containerID="82f8516454de95fd02c2bdc7e8db236ed43381a81b6cfaec6ebc5a26a41c50bb" exitCode=0
Sep 18 19:43:44.055288 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:44.055292    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-5rffs" event={"ID":"61e90e37-8b27-4d0b-b6c0-7272ae55e207","Type":"ContainerDied","Data":"82f8516454de95fd02c2bdc7e8db236ed43381a81b6cfaec6ebc5a26a41c50bb"}
Sep 18 19:43:44.257329 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:44.257294    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-587f57f874-krptn"]
Sep 18 19:43:44.257684 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:44.257671    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9df2f2da-c098-407d-b943-a581bee7af4f" containerName="storage-initializer"
Sep 18 19:43:44.257684 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:44.257685    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="9df2f2da-c098-407d-b943-a581bee7af4f" containerName="storage-initializer"
Sep 18 19:43:44.257773 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:44.257695    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12da7015-ac39-4fe1-9d4b-75142f6f496a" containerName="storage-initializer"
Sep 18 19:43:44.257773 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:44.257701    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="12da7015-ac39-4fe1-9d4b-75142f6f496a" containerName="storage-initializer"
Sep 18 19:43:44.257773 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:44.257712    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12da7015-ac39-4fe1-9d4b-75142f6f496a" containerName="kserve-container"
Sep 18 19:43:44.257773 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:44.257719    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="12da7015-ac39-4fe1-9d4b-75142f6f496a" containerName="kserve-container"
Sep 18 19:43:44.257773 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:44.257729    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12da7015-ac39-4fe1-9d4b-75142f6f496a" containerName="kube-rbac-proxy"
Sep 18 19:43:44.257773 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:44.257736    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="12da7015-ac39-4fe1-9d4b-75142f6f496a" containerName="kube-rbac-proxy"
Sep 18 19:43:44.257773 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:44.257748    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9df2f2da-c098-407d-b943-a581bee7af4f" containerName="kserve-container"
Sep 18 19:43:44.257773 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:44.257754    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="9df2f2da-c098-407d-b943-a581bee7af4f" containerName="kserve-container"
Sep 18 19:43:44.257773 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:44.257766    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9df2f2da-c098-407d-b943-a581bee7af4f" containerName="kube-rbac-proxy"
Sep 18 19:43:44.257773 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:44.257771    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="9df2f2da-c098-407d-b943-a581bee7af4f" containerName="kube-rbac-proxy"
Sep 18 19:43:44.258051 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:44.257825    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="12da7015-ac39-4fe1-9d4b-75142f6f496a" containerName="kserve-container"
Sep 18 19:43:44.258051 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:44.257835    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="9df2f2da-c098-407d-b943-a581bee7af4f" containerName="kserve-container"
Sep 18 19:43:44.258051 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:44.257843    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="12da7015-ac39-4fe1-9d4b-75142f6f496a" containerName="kube-rbac-proxy"
Sep 18 19:43:44.258051 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:44.257849    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="9df2f2da-c098-407d-b943-a581bee7af4f" containerName="kube-rbac-proxy"
Sep 18 19:43:44.261095 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:44.261077    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-587f57f874-krptn"
Sep 18 19:43:44.265306 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:44.265285    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
Sep 18 19:43:44.265612 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:44.265593    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-predictor-serving-cert\""
Sep 18 19:43:44.279522 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:44.279495    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-587f57f874-krptn"]
Sep 18 19:43:44.377986 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:44.377944    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/664889b8-50a0-4ef8-9b4e-b4ca6f67ba35-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-587f57f874-krptn\" (UID: \"664889b8-50a0-4ef8-9b4e-b4ca6f67ba35\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-587f57f874-krptn"
Sep 18 19:43:44.378164 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:44.377990    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/664889b8-50a0-4ef8-9b4e-b4ca6f67ba35-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-587f57f874-krptn\" (UID: \"664889b8-50a0-4ef8-9b4e-b4ca6f67ba35\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-587f57f874-krptn"
Sep 18 19:43:44.378164 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:44.378080    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/664889b8-50a0-4ef8-9b4e-b4ca6f67ba35-proxy-tls\") pod \"isvc-sklearn-v2-predictor-587f57f874-krptn\" (UID: \"664889b8-50a0-4ef8-9b4e-b4ca6f67ba35\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-587f57f874-krptn"
Sep 18 19:43:44.378164 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:44.378119    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/664889b8-50a0-4ef8-9b4e-b4ca6f67ba35-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-587f57f874-krptn\" (UID: \"664889b8-50a0-4ef8-9b4e-b4ca6f67ba35\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-587f57f874-krptn"
Sep 18 19:43:44.378164 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:44.378144    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb96h\" (UniqueName: \"kubernetes.io/projected/664889b8-50a0-4ef8-9b4e-b4ca6f67ba35-kube-api-access-zb96h\") pod \"isvc-sklearn-v2-predictor-587f57f874-krptn\" (UID: \"664889b8-50a0-4ef8-9b4e-b4ca6f67ba35\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-587f57f874-krptn"
Sep 18 19:43:44.479066 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:44.479030    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/664889b8-50a0-4ef8-9b4e-b4ca6f67ba35-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-587f57f874-krptn\" (UID: \"664889b8-50a0-4ef8-9b4e-b4ca6f67ba35\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-587f57f874-krptn"
Sep 18 19:43:44.479066 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:44.479069    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb96h\" (UniqueName: \"kubernetes.io/projected/664889b8-50a0-4ef8-9b4e-b4ca6f67ba35-kube-api-access-zb96h\") pod \"isvc-sklearn-v2-predictor-587f57f874-krptn\" (UID: \"664889b8-50a0-4ef8-9b4e-b4ca6f67ba35\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-587f57f874-krptn"
Sep 18 19:43:44.479329 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:44.479121    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/664889b8-50a0-4ef8-9b4e-b4ca6f67ba35-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-587f57f874-krptn\" (UID: \"664889b8-50a0-4ef8-9b4e-b4ca6f67ba35\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-587f57f874-krptn"
Sep 18 19:43:44.479329 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:44.479250    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/664889b8-50a0-4ef8-9b4e-b4ca6f67ba35-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-587f57f874-krptn\" (UID: \"664889b8-50a0-4ef8-9b4e-b4ca6f67ba35\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-587f57f874-krptn"
Sep 18 19:43:44.479446 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:44.479351    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/664889b8-50a0-4ef8-9b4e-b4ca6f67ba35-proxy-tls\") pod \"isvc-sklearn-v2-predictor-587f57f874-krptn\" (UID: \"664889b8-50a0-4ef8-9b4e-b4ca6f67ba35\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-587f57f874-krptn"
Sep 18 19:43:44.479504 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:43:44.479476    2522 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-predictor-serving-cert: secret "isvc-sklearn-v2-predictor-serving-cert" not found
Sep 18 19:43:44.479594 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:43:44.479542    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/664889b8-50a0-4ef8-9b4e-b4ca6f67ba35-proxy-tls podName:664889b8-50a0-4ef8-9b4e-b4ca6f67ba35 nodeName:}" failed. No retries permitted until 2026-09-18 19:43:44.979524045 +0000 UTC m=+1784.863050490 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/664889b8-50a0-4ef8-9b4e-b4ca6f67ba35-proxy-tls") pod "isvc-sklearn-v2-predictor-587f57f874-krptn" (UID: "664889b8-50a0-4ef8-9b4e-b4ca6f67ba35") : secret "isvc-sklearn-v2-predictor-serving-cert" not found
Sep 18 19:43:44.479671 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:44.479604    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/664889b8-50a0-4ef8-9b4e-b4ca6f67ba35-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-587f57f874-krptn\" (UID: \"664889b8-50a0-4ef8-9b4e-b4ca6f67ba35\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-587f57f874-krptn"
Sep 18 19:43:44.479849 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:44.479825    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/664889b8-50a0-4ef8-9b4e-b4ca6f67ba35-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-587f57f874-krptn\" (UID: \"664889b8-50a0-4ef8-9b4e-b4ca6f67ba35\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-587f57f874-krptn"
Sep 18 19:43:44.479918 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:44.479875    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/664889b8-50a0-4ef8-9b4e-b4ca6f67ba35-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-587f57f874-krptn\" (UID: \"664889b8-50a0-4ef8-9b4e-b4ca6f67ba35\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-587f57f874-krptn"
Sep 18 19:43:44.496354 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:44.496322    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb96h\" (UniqueName: \"kubernetes.io/projected/664889b8-50a0-4ef8-9b4e-b4ca6f67ba35-kube-api-access-zb96h\") pod \"isvc-sklearn-v2-predictor-587f57f874-krptn\" (UID: \"664889b8-50a0-4ef8-9b4e-b4ca6f67ba35\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-587f57f874-krptn"
Sep 18 19:43:44.982960 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:44.982922    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/664889b8-50a0-4ef8-9b4e-b4ca6f67ba35-proxy-tls\") pod \"isvc-sklearn-v2-predictor-587f57f874-krptn\" (UID: \"664889b8-50a0-4ef8-9b4e-b4ca6f67ba35\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-587f57f874-krptn"
Sep 18 19:43:44.985433 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:44.985411    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/664889b8-50a0-4ef8-9b4e-b4ca6f67ba35-proxy-tls\") pod \"isvc-sklearn-v2-predictor-587f57f874-krptn\" (UID: \"664889b8-50a0-4ef8-9b4e-b4ca6f67ba35\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-587f57f874-krptn"
Sep 18 19:43:45.062168 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:45.062127    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-5rffs" event={"ID":"61e90e37-8b27-4d0b-b6c0-7272ae55e207","Type":"ContainerStarted","Data":"d46a023c0aecb80426f15db107aef1504a63a4e428f11d1e6be954f4a25a9645"}
Sep 18 19:43:45.062667 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:45.062200    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-5rffs" event={"ID":"61e90e37-8b27-4d0b-b6c0-7272ae55e207","Type":"ContainerStarted","Data":"7ae046cf013af3289abb4d8c80d9b7c0562012bbceaa2dd93fab4b932f6f3292"}
Sep 18 19:43:45.062667 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:45.062417    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-5rffs"
Sep 18 19:43:45.096184 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:45.096133    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-5rffs" podStartSLOduration=3.247299345 podStartE2EDuration="7.096119411s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.848820066s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:43:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:43:45.09499781 +0000 UTC m=+1784.978524295" watchObservedRunningTime="2026-09-18 19:43:45.096119411 +0000 UTC m=+1784.979645873"
Sep 18 19:43:45.170897 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:45.170853    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-587f57f874-krptn"
Sep 18 19:43:45.450536 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:45.450509    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-587f57f874-krptn"]
Sep 18 19:43:45.452654 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:43:45.452618    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod664889b8_50a0_4ef8_9b4e_b4ca6f67ba35.slice/crio-2259e39cfbfbe36b0957447dcea2429ad1a84d6c3975da05ffb37fcdf204333f WatchSource:0}: Error finding container 2259e39cfbfbe36b0957447dcea2429ad1a84d6c3975da05ffb37fcdf204333f: Status 404 returned error can't find the container with id 2259e39cfbfbe36b0957447dcea2429ad1a84d6c3975da05ffb37fcdf204333f
Sep 18 19:43:46.069525 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:46.069485    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-587f57f874-krptn" event={"ID":"664889b8-50a0-4ef8-9b4e-b4ca6f67ba35","Type":"ContainerStarted","Data":"aa05a28d436f5564f07fe6dc8d7c139d5cf6519838ea9235b65d9ff182b28f55"}
Sep 18 19:43:46.069980 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:46.069580    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-587f57f874-krptn" event={"ID":"664889b8-50a0-4ef8-9b4e-b4ca6f67ba35","Type":"ContainerStarted","Data":"2259e39cfbfbe36b0957447dcea2429ad1a84d6c3975da05ffb37fcdf204333f"}
Sep 18 19:43:46.069980 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:46.069849    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-5rffs"
Sep 18 19:43:46.071261 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:46.071240    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-5rffs" podUID="61e90e37-8b27-4d0b-b6c0-7272ae55e207" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 18 19:43:47.074788 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:47.074741    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-5rffs" podUID="61e90e37-8b27-4d0b-b6c0-7272ae55e207" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 18 19:43:50.090027 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:50.089986    2522 generic.go:356] "Generic (PLEG): container finished" podID="664889b8-50a0-4ef8-9b4e-b4ca6f67ba35" containerID="aa05a28d436f5564f07fe6dc8d7c139d5cf6519838ea9235b65d9ff182b28f55" exitCode=0
Sep 18 19:43:50.090529 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:50.090034    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-587f57f874-krptn" event={"ID":"664889b8-50a0-4ef8-9b4e-b4ca6f67ba35","Type":"ContainerDied","Data":"aa05a28d436f5564f07fe6dc8d7c139d5cf6519838ea9235b65d9ff182b28f55"}
Sep 18 19:43:51.096853 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:51.096811    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-587f57f874-krptn" event={"ID":"664889b8-50a0-4ef8-9b4e-b4ca6f67ba35","Type":"ContainerStarted","Data":"b8ce665bc61544e7f8af47f7d6ac9fc0d92e08c2b0b3388730f049d74d756b41"}
Sep 18 19:43:51.097341 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:51.096918    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-587f57f874-krptn" event={"ID":"664889b8-50a0-4ef8-9b4e-b4ca6f67ba35","Type":"ContainerStarted","Data":"8b0f85e64075bf9ceb594722acca56c263b7e166e03b2296dd65991023abc7f0"}
Sep 18 19:43:51.097341 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:51.097143    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-587f57f874-krptn"
Sep 18 19:43:51.131743 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:51.131690    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-587f57f874-krptn" podStartSLOduration=3.300172182 podStartE2EDuration="7.131672387s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.831500205s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:43:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:43:51.131297184 +0000 UTC m=+1791.014823645" watchObservedRunningTime="2026-09-18 19:43:51.131672387 +0000 UTC m=+1791.015198851"
Sep 18 19:43:52.079840 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:52.079811    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-5rffs"
Sep 18 19:43:52.080313 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:52.080289    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-5rffs" podUID="61e90e37-8b27-4d0b-b6c0-7272ae55e207" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 18 19:43:52.101967 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:52.101936    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-587f57f874-krptn"
Sep 18 19:43:52.102974 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:52.102948    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-587f57f874-krptn" podUID="664889b8-50a0-4ef8-9b4e-b4ca6f67ba35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 18 19:43:53.106951 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:53.106907    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-587f57f874-krptn" podUID="664889b8-50a0-4ef8-9b4e-b4ca6f67ba35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 18 19:43:58.111776 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:58.111745    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-587f57f874-krptn"
Sep 18 19:43:58.112436 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:43:58.112406    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-587f57f874-krptn" podUID="664889b8-50a0-4ef8-9b4e-b4ca6f67ba35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 18 19:44:02.080760 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:44:02.080710    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-5rffs" podUID="61e90e37-8b27-4d0b-b6c0-7272ae55e207" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 18 19:44:08.112551 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:44:08.112512    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-587f57f874-krptn" podUID="664889b8-50a0-4ef8-9b4e-b4ca6f67ba35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 18 19:44:12.081239 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:44:12.081195    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-5rffs" podUID="61e90e37-8b27-4d0b-b6c0-7272ae55e207" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 18 19:44:18.113158 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:44:18.113116    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-587f57f874-krptn" podUID="664889b8-50a0-4ef8-9b4e-b4ca6f67ba35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 18 19:44:22.081207 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:44:22.081162    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-5rffs" podUID="61e90e37-8b27-4d0b-b6c0-7272ae55e207" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 18 19:44:28.113165 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:44:28.113125    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-587f57f874-krptn" podUID="664889b8-50a0-4ef8-9b4e-b4ca6f67ba35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 18 19:44:32.080974 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:44:32.080929    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-5rffs" podUID="61e90e37-8b27-4d0b-b6c0-7272ae55e207" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 18 19:44:38.112714 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:44:38.112670    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-587f57f874-krptn" podUID="664889b8-50a0-4ef8-9b4e-b4ca6f67ba35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 18 19:44:42.080622 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:44:42.080581    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-5rffs" podUID="61e90e37-8b27-4d0b-b6c0-7272ae55e207" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 18 19:44:48.112866 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:44:48.112827    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-587f57f874-krptn" podUID="664889b8-50a0-4ef8-9b4e-b4ca6f67ba35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 18 19:44:52.081006 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:44:52.080962    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-5rffs" podUID="61e90e37-8b27-4d0b-b6c0-7272ae55e207" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 18 19:44:58.113242 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:44:58.113202    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-587f57f874-krptn" podUID="664889b8-50a0-4ef8-9b4e-b4ca6f67ba35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 18 19:45:01.716971 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:01.716931    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-5rffs" podUID="61e90e37-8b27-4d0b-b6c0-7272ae55e207" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 18 19:45:02.716267 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:02.716222    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-587f57f874-krptn" podUID="664889b8-50a0-4ef8-9b4e-b4ca6f67ba35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 18 19:45:07.935724 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:07.935688    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-5rffs"]
Sep 18 19:45:07.936270 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:07.935988    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-5rffs" podUID="61e90e37-8b27-4d0b-b6c0-7272ae55e207" containerName="kserve-container" containerID="cri-o://7ae046cf013af3289abb4d8c80d9b7c0562012bbceaa2dd93fab4b932f6f3292" gracePeriod=30
Sep 18 19:45:07.936270 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:07.936021    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-5rffs" podUID="61e90e37-8b27-4d0b-b6c0-7272ae55e207" containerName="kube-rbac-proxy" containerID="cri-o://d46a023c0aecb80426f15db107aef1504a63a4e428f11d1e6be954f4a25a9645" gracePeriod=30
Sep 18 19:45:07.937064 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:07.937042    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-5rffs"
Sep 18 19:45:08.466555 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:08.466520    2522 generic.go:356] "Generic (PLEG): container finished" podID="61e90e37-8b27-4d0b-b6c0-7272ae55e207" containerID="d46a023c0aecb80426f15db107aef1504a63a4e428f11d1e6be954f4a25a9645" exitCode=2
Sep 18 19:45:08.466773 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:08.466593    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-5rffs" event={"ID":"61e90e37-8b27-4d0b-b6c0-7272ae55e207","Type":"ContainerDied","Data":"d46a023c0aecb80426f15db107aef1504a63a4e428f11d1e6be954f4a25a9645"}
Sep 18 19:45:12.075048 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:12.075003    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-5rffs" podUID="61e90e37-8b27-4d0b-b6c0-7272ae55e207" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.48:8643/healthz\": dial tcp 10.132.0.48:8643: connect: connection refused"
Sep 18 19:45:12.721821 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:12.721791    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-587f57f874-krptn"
Sep 18 19:45:12.884556 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:12.884531    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-5rffs"
Sep 18 19:45:13.037287 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:13.037199    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z75jf\" (UniqueName: \"kubernetes.io/projected/61e90e37-8b27-4d0b-b6c0-7272ae55e207-kube-api-access-z75jf\") pod \"61e90e37-8b27-4d0b-b6c0-7272ae55e207\" (UID: \"61e90e37-8b27-4d0b-b6c0-7272ae55e207\") "
Sep 18 19:45:13.037287 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:13.037253    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61e90e37-8b27-4d0b-b6c0-7272ae55e207-kserve-provision-location\") pod \"61e90e37-8b27-4d0b-b6c0-7272ae55e207\" (UID: \"61e90e37-8b27-4d0b-b6c0-7272ae55e207\") "
Sep 18 19:45:13.037540 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:13.037298    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/61e90e37-8b27-4d0b-b6c0-7272ae55e207-kube-rbac-proxy-sar-config\") pod \"61e90e37-8b27-4d0b-b6c0-7272ae55e207\" (UID: \"61e90e37-8b27-4d0b-b6c0-7272ae55e207\") "
Sep 18 19:45:13.037540 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:13.037332    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61e90e37-8b27-4d0b-b6c0-7272ae55e207-proxy-tls\") pod \"61e90e37-8b27-4d0b-b6c0-7272ae55e207\" (UID: \"61e90e37-8b27-4d0b-b6c0-7272ae55e207\") "
Sep 18 19:45:13.037540 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:13.037346    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/61e90e37-8b27-4d0b-b6c0-7272ae55e207-cabundle-cert\") pod \"61e90e37-8b27-4d0b-b6c0-7272ae55e207\" (UID: \"61e90e37-8b27-4d0b-b6c0-7272ae55e207\") "
Sep 18 19:45:13.037741 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:13.037679    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61e90e37-8b27-4d0b-b6c0-7272ae55e207-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "61e90e37-8b27-4d0b-b6c0-7272ae55e207" (UID: "61e90e37-8b27-4d0b-b6c0-7272ae55e207"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:45:13.037741 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:13.037711    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61e90e37-8b27-4d0b-b6c0-7272ae55e207-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "61e90e37-8b27-4d0b-b6c0-7272ae55e207" (UID: "61e90e37-8b27-4d0b-b6c0-7272ae55e207"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:45:13.037816 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:13.037744    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61e90e37-8b27-4d0b-b6c0-7272ae55e207-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "61e90e37-8b27-4d0b-b6c0-7272ae55e207" (UID: "61e90e37-8b27-4d0b-b6c0-7272ae55e207"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:45:13.039418 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:13.039396    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61e90e37-8b27-4d0b-b6c0-7272ae55e207-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "61e90e37-8b27-4d0b-b6c0-7272ae55e207" (UID: "61e90e37-8b27-4d0b-b6c0-7272ae55e207"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:45:13.039505 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:13.039420    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61e90e37-8b27-4d0b-b6c0-7272ae55e207-kube-api-access-z75jf" (OuterVolumeSpecName: "kube-api-access-z75jf") pod "61e90e37-8b27-4d0b-b6c0-7272ae55e207" (UID: "61e90e37-8b27-4d0b-b6c0-7272ae55e207"). InnerVolumeSpecName "kube-api-access-z75jf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:45:13.138478 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:13.138446    2522 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61e90e37-8b27-4d0b-b6c0-7272ae55e207-proxy-tls\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:45:13.138478 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:13.138470    2522 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/61e90e37-8b27-4d0b-b6c0-7272ae55e207-cabundle-cert\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:45:13.138478 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:13.138479    2522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z75jf\" (UniqueName: \"kubernetes.io/projected/61e90e37-8b27-4d0b-b6c0-7272ae55e207-kube-api-access-z75jf\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:45:13.138478 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:13.138488    2522 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61e90e37-8b27-4d0b-b6c0-7272ae55e207-kserve-provision-location\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:45:13.138999 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:13.138497    2522 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/61e90e37-8b27-4d0b-b6c0-7272ae55e207-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:45:13.491630 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:13.491590    2522 generic.go:356] "Generic (PLEG): container finished" podID="61e90e37-8b27-4d0b-b6c0-7272ae55e207" containerID="7ae046cf013af3289abb4d8c80d9b7c0562012bbceaa2dd93fab4b932f6f3292" exitCode=0
Sep 18 19:45:13.491837 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:13.491657    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-5rffs"
Sep 18 19:45:13.491837 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:13.491655    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-5rffs" event={"ID":"61e90e37-8b27-4d0b-b6c0-7272ae55e207","Type":"ContainerDied","Data":"7ae046cf013af3289abb4d8c80d9b7c0562012bbceaa2dd93fab4b932f6f3292"}
Sep 18 19:45:13.491837 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:13.491783    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-5rffs" event={"ID":"61e90e37-8b27-4d0b-b6c0-7272ae55e207","Type":"ContainerDied","Data":"e00fe9ec4bfe845c7f59905468ff4424e69502c3b3dacfb1eb4763de603fc4d4"}
Sep 18 19:45:13.491837 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:13.491808    2522 scope.go:117] "RemoveContainer" containerID="d46a023c0aecb80426f15db107aef1504a63a4e428f11d1e6be954f4a25a9645"
Sep 18 19:45:13.500694 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:13.500678    2522 scope.go:117] "RemoveContainer" containerID="7ae046cf013af3289abb4d8c80d9b7c0562012bbceaa2dd93fab4b932f6f3292"
Sep 18 19:45:13.508596 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:13.508581    2522 scope.go:117] "RemoveContainer" containerID="82f8516454de95fd02c2bdc7e8db236ed43381a81b6cfaec6ebc5a26a41c50bb"
Sep 18 19:45:13.515765 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:13.515749    2522 scope.go:117] "RemoveContainer" containerID="d46a023c0aecb80426f15db107aef1504a63a4e428f11d1e6be954f4a25a9645"
Sep 18 19:45:13.516033 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:45:13.516012    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d46a023c0aecb80426f15db107aef1504a63a4e428f11d1e6be954f4a25a9645\": container with ID starting with d46a023c0aecb80426f15db107aef1504a63a4e428f11d1e6be954f4a25a9645 not found: ID does not exist" containerID="d46a023c0aecb80426f15db107aef1504a63a4e428f11d1e6be954f4a25a9645"
Sep 18 19:45:13.516079 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:13.516042    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d46a023c0aecb80426f15db107aef1504a63a4e428f11d1e6be954f4a25a9645"} err="failed to get container status \"d46a023c0aecb80426f15db107aef1504a63a4e428f11d1e6be954f4a25a9645\": rpc error: code = NotFound desc = could not find container \"d46a023c0aecb80426f15db107aef1504a63a4e428f11d1e6be954f4a25a9645\": container with ID starting with d46a023c0aecb80426f15db107aef1504a63a4e428f11d1e6be954f4a25a9645 not found: ID does not exist"
Sep 18 19:45:13.516079 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:13.516060    2522 scope.go:117] "RemoveContainer" containerID="7ae046cf013af3289abb4d8c80d9b7c0562012bbceaa2dd93fab4b932f6f3292"
Sep 18 19:45:13.516297 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:45:13.516277    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ae046cf013af3289abb4d8c80d9b7c0562012bbceaa2dd93fab4b932f6f3292\": container with ID starting with 7ae046cf013af3289abb4d8c80d9b7c0562012bbceaa2dd93fab4b932f6f3292 not found: ID does not exist" containerID="7ae046cf013af3289abb4d8c80d9b7c0562012bbceaa2dd93fab4b932f6f3292"
Sep 18 19:45:13.516363 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:13.516305    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ae046cf013af3289abb4d8c80d9b7c0562012bbceaa2dd93fab4b932f6f3292"} err="failed to get container status \"7ae046cf013af3289abb4d8c80d9b7c0562012bbceaa2dd93fab4b932f6f3292\": rpc error: code = NotFound desc = could not find container \"7ae046cf013af3289abb4d8c80d9b7c0562012bbceaa2dd93fab4b932f6f3292\": container with ID starting with 7ae046cf013af3289abb4d8c80d9b7c0562012bbceaa2dd93fab4b932f6f3292 not found: ID does not exist"
Sep 18 19:45:13.516363 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:13.516326    2522 scope.go:117] "RemoveContainer" containerID="82f8516454de95fd02c2bdc7e8db236ed43381a81b6cfaec6ebc5a26a41c50bb"
Sep 18 19:45:13.516556 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:45:13.516543    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82f8516454de95fd02c2bdc7e8db236ed43381a81b6cfaec6ebc5a26a41c50bb\": container with ID starting with 82f8516454de95fd02c2bdc7e8db236ed43381a81b6cfaec6ebc5a26a41c50bb not found: ID does not exist" containerID="82f8516454de95fd02c2bdc7e8db236ed43381a81b6cfaec6ebc5a26a41c50bb"
Sep 18 19:45:13.516623 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:13.516571    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82f8516454de95fd02c2bdc7e8db236ed43381a81b6cfaec6ebc5a26a41c50bb"} err="failed to get container status \"82f8516454de95fd02c2bdc7e8db236ed43381a81b6cfaec6ebc5a26a41c50bb\": rpc error: code = NotFound desc = could not find container \"82f8516454de95fd02c2bdc7e8db236ed43381a81b6cfaec6ebc5a26a41c50bb\": container with ID starting with 82f8516454de95fd02c2bdc7e8db236ed43381a81b6cfaec6ebc5a26a41c50bb not found: ID does not exist"
Sep 18 19:45:13.522213 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:13.522193    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-5rffs"]
Sep 18 19:45:13.529897 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:13.529876    2522 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-5rffs"]
Sep 18 19:45:14.727495 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:14.723604    2522 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61e90e37-8b27-4d0b-b6c0-7272ae55e207" path="/var/lib/kubelet/pods/61e90e37-8b27-4d0b-b6c0-7272ae55e207/volumes"
Sep 18 19:45:15.186178 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:15.186145    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-587f57f874-krptn"]
Sep 18 19:45:15.186415 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:15.186389    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-587f57f874-krptn" podUID="664889b8-50a0-4ef8-9b4e-b4ca6f67ba35" containerName="kserve-container" containerID="cri-o://8b0f85e64075bf9ceb594722acca56c263b7e166e03b2296dd65991023abc7f0" gracePeriod=30
Sep 18 19:45:15.186483 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:15.186399    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-587f57f874-krptn" podUID="664889b8-50a0-4ef8-9b4e-b4ca6f67ba35" containerName="kube-rbac-proxy" containerID="cri-o://b8ce665bc61544e7f8af47f7d6ac9fc0d92e08c2b0b3388730f049d74d756b41" gracePeriod=30
Sep 18 19:45:15.503581 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:15.503477    2522 generic.go:356] "Generic (PLEG): container finished" podID="664889b8-50a0-4ef8-9b4e-b4ca6f67ba35" containerID="b8ce665bc61544e7f8af47f7d6ac9fc0d92e08c2b0b3388730f049d74d756b41" exitCode=2
Sep 18 19:45:15.503581 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:15.503524    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-587f57f874-krptn" event={"ID":"664889b8-50a0-4ef8-9b4e-b4ca6f67ba35","Type":"ContainerDied","Data":"b8ce665bc61544e7f8af47f7d6ac9fc0d92e08c2b0b3388730f049d74d756b41"}
Sep 18 19:45:18.107493 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:18.107451    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-587f57f874-krptn" podUID="664889b8-50a0-4ef8-9b4e-b4ca6f67ba35" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.49:8643/healthz\": dial tcp 10.132.0.49:8643: connect: connection refused"
Sep 18 19:45:18.839330 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:18.839292    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk"]
Sep 18 19:45:18.839670 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:18.839657    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61e90e37-8b27-4d0b-b6c0-7272ae55e207" containerName="storage-initializer"
Sep 18 19:45:18.839730 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:18.839672    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="61e90e37-8b27-4d0b-b6c0-7272ae55e207" containerName="storage-initializer"
Sep 18 19:45:18.839730 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:18.839680    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61e90e37-8b27-4d0b-b6c0-7272ae55e207" containerName="kserve-container"
Sep 18 19:45:18.839730 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:18.839686    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="61e90e37-8b27-4d0b-b6c0-7272ae55e207" containerName="kserve-container"
Sep 18 19:45:18.839730 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:18.839708    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61e90e37-8b27-4d0b-b6c0-7272ae55e207" containerName="kube-rbac-proxy"
Sep 18 19:45:18.839730 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:18.839714    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="61e90e37-8b27-4d0b-b6c0-7272ae55e207" containerName="kube-rbac-proxy"
Sep 18 19:45:18.839912 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:18.839776    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="61e90e37-8b27-4d0b-b6c0-7272ae55e207" containerName="kserve-container"
Sep 18 19:45:18.839912 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:18.839785    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="61e90e37-8b27-4d0b-b6c0-7272ae55e207" containerName="kube-rbac-proxy"
Sep 18 19:45:18.843019 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:18.842990    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk"
Sep 18 19:45:18.846640 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:18.846622    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 18 19:45:18.847734 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:18.847717    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 18 19:45:18.855288 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:18.855265    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk"]
Sep 18 19:45:18.882489 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:18.882465    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db1cabe7-ba95-443e-b47f-6d5f67b028d4-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk\" (UID: \"db1cabe7-ba95-443e-b47f-6d5f67b028d4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk"
Sep 18 19:45:18.882650 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:18.882491    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-675d7\" (UniqueName: \"kubernetes.io/projected/db1cabe7-ba95-443e-b47f-6d5f67b028d4-kube-api-access-675d7\") pod \"isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk\" (UID: \"db1cabe7-ba95-443e-b47f-6d5f67b028d4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk"
Sep 18 19:45:18.882650 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:18.882528    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db1cabe7-ba95-443e-b47f-6d5f67b028d4-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk\" (UID: \"db1cabe7-ba95-443e-b47f-6d5f67b028d4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk"
Sep 18 19:45:18.882650 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:18.882616    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db1cabe7-ba95-443e-b47f-6d5f67b028d4-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk\" (UID: \"db1cabe7-ba95-443e-b47f-6d5f67b028d4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk"
Sep 18 19:45:18.882825 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:18.882669    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db1cabe7-ba95-443e-b47f-6d5f67b028d4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk\" (UID: \"db1cabe7-ba95-443e-b47f-6d5f67b028d4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk"
Sep 18 19:45:18.983145 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:18.983107    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db1cabe7-ba95-443e-b47f-6d5f67b028d4-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk\" (UID: \"db1cabe7-ba95-443e-b47f-6d5f67b028d4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk"
Sep 18 19:45:18.983145 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:18.983151    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db1cabe7-ba95-443e-b47f-6d5f67b028d4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk\" (UID: \"db1cabe7-ba95-443e-b47f-6d5f67b028d4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk"
Sep 18 19:45:18.983398 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:18.983192    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db1cabe7-ba95-443e-b47f-6d5f67b028d4-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk\" (UID: \"db1cabe7-ba95-443e-b47f-6d5f67b028d4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk"
Sep 18 19:45:18.983398 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:18.983210    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-675d7\" (UniqueName: \"kubernetes.io/projected/db1cabe7-ba95-443e-b47f-6d5f67b028d4-kube-api-access-675d7\") pod \"isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk\" (UID: \"db1cabe7-ba95-443e-b47f-6d5f67b028d4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk"
Sep 18 19:45:18.983398 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:18.983246    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db1cabe7-ba95-443e-b47f-6d5f67b028d4-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk\" (UID: \"db1cabe7-ba95-443e-b47f-6d5f67b028d4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk"
Sep 18 19:45:18.983714 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:18.983693    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db1cabe7-ba95-443e-b47f-6d5f67b028d4-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk\" (UID: \"db1cabe7-ba95-443e-b47f-6d5f67b028d4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk"
Sep 18 19:45:18.983973 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:18.983951    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db1cabe7-ba95-443e-b47f-6d5f67b028d4-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk\" (UID: \"db1cabe7-ba95-443e-b47f-6d5f67b028d4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk"
Sep 18 19:45:18.984072 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:18.984052    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db1cabe7-ba95-443e-b47f-6d5f67b028d4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk\" (UID: \"db1cabe7-ba95-443e-b47f-6d5f67b028d4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk"
Sep 18 19:45:18.985623 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:18.985602    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db1cabe7-ba95-443e-b47f-6d5f67b028d4-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk\" (UID: \"db1cabe7-ba95-443e-b47f-6d5f67b028d4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk"
Sep 18 19:45:18.991946 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:18.991902    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-675d7\" (UniqueName: \"kubernetes.io/projected/db1cabe7-ba95-443e-b47f-6d5f67b028d4-kube-api-access-675d7\") pod \"isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk\" (UID: \"db1cabe7-ba95-443e-b47f-6d5f67b028d4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk"
Sep 18 19:45:19.153330 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:19.153232    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk"
Sep 18 19:45:19.432071 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:19.432038    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk"]
Sep 18 19:45:19.432691 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:45:19.432664    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb1cabe7_ba95_443e_b47f_6d5f67b028d4.slice/crio-dbcee6bcb8b359cf8d6d42686ad7de7fffe2722f10d195faadc81953bb0826f6 WatchSource:0}: Error finding container dbcee6bcb8b359cf8d6d42686ad7de7fffe2722f10d195faadc81953bb0826f6: Status 404 returned error can't find the container with id dbcee6bcb8b359cf8d6d42686ad7de7fffe2722f10d195faadc81953bb0826f6
Sep 18 19:45:19.523574 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:19.523514    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk" event={"ID":"db1cabe7-ba95-443e-b47f-6d5f67b028d4","Type":"ContainerStarted","Data":"91f7c7b5045463ffaf0f37c413e9821de57dbd0ac1151e7f28685dff50893827"}
Sep 18 19:45:19.523725 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:19.523607    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk" event={"ID":"db1cabe7-ba95-443e-b47f-6d5f67b028d4","Type":"ContainerStarted","Data":"dbcee6bcb8b359cf8d6d42686ad7de7fffe2722f10d195faadc81953bb0826f6"}
Sep 18 19:45:20.433681 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:20.433654    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-587f57f874-krptn"
Sep 18 19:45:20.494919 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:20.494835    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/664889b8-50a0-4ef8-9b4e-b4ca6f67ba35-proxy-tls\") pod \"664889b8-50a0-4ef8-9b4e-b4ca6f67ba35\" (UID: \"664889b8-50a0-4ef8-9b4e-b4ca6f67ba35\") "
Sep 18 19:45:20.494919 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:20.494874    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/664889b8-50a0-4ef8-9b4e-b4ca6f67ba35-kserve-provision-location\") pod \"664889b8-50a0-4ef8-9b4e-b4ca6f67ba35\" (UID: \"664889b8-50a0-4ef8-9b4e-b4ca6f67ba35\") "
Sep 18 19:45:20.494919 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:20.494894    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zb96h\" (UniqueName: \"kubernetes.io/projected/664889b8-50a0-4ef8-9b4e-b4ca6f67ba35-kube-api-access-zb96h\") pod \"664889b8-50a0-4ef8-9b4e-b4ca6f67ba35\" (UID: \"664889b8-50a0-4ef8-9b4e-b4ca6f67ba35\") "
Sep 18 19:45:20.495208 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:20.494958    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/664889b8-50a0-4ef8-9b4e-b4ca6f67ba35-kube-rbac-proxy-sar-config\") pod \"664889b8-50a0-4ef8-9b4e-b4ca6f67ba35\" (UID: \"664889b8-50a0-4ef8-9b4e-b4ca6f67ba35\") "
Sep 18 19:45:20.495208 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:20.494988    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/664889b8-50a0-4ef8-9b4e-b4ca6f67ba35-cabundle-cert\") pod \"664889b8-50a0-4ef8-9b4e-b4ca6f67ba35\" (UID: \"664889b8-50a0-4ef8-9b4e-b4ca6f67ba35\") "
Sep 18 19:45:20.495317 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:20.495219    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/664889b8-50a0-4ef8-9b4e-b4ca6f67ba35-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "664889b8-50a0-4ef8-9b4e-b4ca6f67ba35" (UID: "664889b8-50a0-4ef8-9b4e-b4ca6f67ba35"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:45:20.495317 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:20.495283    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/664889b8-50a0-4ef8-9b4e-b4ca6f67ba35-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "664889b8-50a0-4ef8-9b4e-b4ca6f67ba35" (UID: "664889b8-50a0-4ef8-9b4e-b4ca6f67ba35"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:45:20.495455 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:20.495428    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/664889b8-50a0-4ef8-9b4e-b4ca6f67ba35-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "664889b8-50a0-4ef8-9b4e-b4ca6f67ba35" (UID: "664889b8-50a0-4ef8-9b4e-b4ca6f67ba35"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:45:20.497594 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:20.497549    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/664889b8-50a0-4ef8-9b4e-b4ca6f67ba35-kube-api-access-zb96h" (OuterVolumeSpecName: "kube-api-access-zb96h") pod "664889b8-50a0-4ef8-9b4e-b4ca6f67ba35" (UID: "664889b8-50a0-4ef8-9b4e-b4ca6f67ba35"). InnerVolumeSpecName "kube-api-access-zb96h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:45:20.497594 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:20.497551    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/664889b8-50a0-4ef8-9b4e-b4ca6f67ba35-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "664889b8-50a0-4ef8-9b4e-b4ca6f67ba35" (UID: "664889b8-50a0-4ef8-9b4e-b4ca6f67ba35"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:45:20.529633 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:20.529604    2522 generic.go:356] "Generic (PLEG): container finished" podID="664889b8-50a0-4ef8-9b4e-b4ca6f67ba35" containerID="8b0f85e64075bf9ceb594722acca56c263b7e166e03b2296dd65991023abc7f0" exitCode=0
Sep 18 19:45:20.529750 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:20.529654    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-587f57f874-krptn"
Sep 18 19:45:20.529750 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:20.529679    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-587f57f874-krptn" event={"ID":"664889b8-50a0-4ef8-9b4e-b4ca6f67ba35","Type":"ContainerDied","Data":"8b0f85e64075bf9ceb594722acca56c263b7e166e03b2296dd65991023abc7f0"}
Sep 18 19:45:20.529750 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:20.529734    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-587f57f874-krptn" event={"ID":"664889b8-50a0-4ef8-9b4e-b4ca6f67ba35","Type":"ContainerDied","Data":"2259e39cfbfbe36b0957447dcea2429ad1a84d6c3975da05ffb37fcdf204333f"}
Sep 18 19:45:20.529867 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:20.529759    2522 scope.go:117] "RemoveContainer" containerID="b8ce665bc61544e7f8af47f7d6ac9fc0d92e08c2b0b3388730f049d74d756b41"
Sep 18 19:45:20.538738 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:20.538715    2522 scope.go:117] "RemoveContainer" containerID="8b0f85e64075bf9ceb594722acca56c263b7e166e03b2296dd65991023abc7f0"
Sep 18 19:45:20.546377 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:20.546361    2522 scope.go:117] "RemoveContainer" containerID="aa05a28d436f5564f07fe6dc8d7c139d5cf6519838ea9235b65d9ff182b28f55"
Sep 18 19:45:20.553533 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:20.553515    2522 scope.go:117] "RemoveContainer" containerID="b8ce665bc61544e7f8af47f7d6ac9fc0d92e08c2b0b3388730f049d74d756b41"
Sep 18 19:45:20.553805 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:45:20.553787    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8ce665bc61544e7f8af47f7d6ac9fc0d92e08c2b0b3388730f049d74d756b41\": container with ID starting with b8ce665bc61544e7f8af47f7d6ac9fc0d92e08c2b0b3388730f049d74d756b41 not found: ID does not exist" containerID="b8ce665bc61544e7f8af47f7d6ac9fc0d92e08c2b0b3388730f049d74d756b41"
Sep 18 19:45:20.553877 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:20.553815    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8ce665bc61544e7f8af47f7d6ac9fc0d92e08c2b0b3388730f049d74d756b41"} err="failed to get container status \"b8ce665bc61544e7f8af47f7d6ac9fc0d92e08c2b0b3388730f049d74d756b41\": rpc error: code = NotFound desc = could not find container \"b8ce665bc61544e7f8af47f7d6ac9fc0d92e08c2b0b3388730f049d74d756b41\": container with ID starting with b8ce665bc61544e7f8af47f7d6ac9fc0d92e08c2b0b3388730f049d74d756b41 not found: ID does not exist"
Sep 18 19:45:20.553877 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:20.553837    2522 scope.go:117] "RemoveContainer" containerID="8b0f85e64075bf9ceb594722acca56c263b7e166e03b2296dd65991023abc7f0"
Sep 18 19:45:20.554061 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:45:20.554046    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b0f85e64075bf9ceb594722acca56c263b7e166e03b2296dd65991023abc7f0\": container with ID starting with 8b0f85e64075bf9ceb594722acca56c263b7e166e03b2296dd65991023abc7f0 not found: ID does not exist" containerID="8b0f85e64075bf9ceb594722acca56c263b7e166e03b2296dd65991023abc7f0"
Sep 18 19:45:20.554129 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:20.554070    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b0f85e64075bf9ceb594722acca56c263b7e166e03b2296dd65991023abc7f0"} err="failed to get container status \"8b0f85e64075bf9ceb594722acca56c263b7e166e03b2296dd65991023abc7f0\": rpc error: code = NotFound desc = could not find container \"8b0f85e64075bf9ceb594722acca56c263b7e166e03b2296dd65991023abc7f0\": container with ID starting with 8b0f85e64075bf9ceb594722acca56c263b7e166e03b2296dd65991023abc7f0 not found: ID does not exist"
Sep 18 19:45:20.554129 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:20.554091    2522 scope.go:117] "RemoveContainer" containerID="aa05a28d436f5564f07fe6dc8d7c139d5cf6519838ea9235b65d9ff182b28f55"
Sep 18 19:45:20.554307 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:45:20.554289    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa05a28d436f5564f07fe6dc8d7c139d5cf6519838ea9235b65d9ff182b28f55\": container with ID starting with aa05a28d436f5564f07fe6dc8d7c139d5cf6519838ea9235b65d9ff182b28f55 not found: ID does not exist" containerID="aa05a28d436f5564f07fe6dc8d7c139d5cf6519838ea9235b65d9ff182b28f55"
Sep 18 19:45:20.554350 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:20.554312    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa05a28d436f5564f07fe6dc8d7c139d5cf6519838ea9235b65d9ff182b28f55"} err="failed to get container status \"aa05a28d436f5564f07fe6dc8d7c139d5cf6519838ea9235b65d9ff182b28f55\": rpc error: code = NotFound desc = could not find container \"aa05a28d436f5564f07fe6dc8d7c139d5cf6519838ea9235b65d9ff182b28f55\": container with ID starting with aa05a28d436f5564f07fe6dc8d7c139d5cf6519838ea9235b65d9ff182b28f55 not found: ID does not exist"
Sep 18 19:45:20.559055 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:20.559034    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-587f57f874-krptn"]
Sep 18 19:45:20.570535 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:20.570513    2522 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-587f57f874-krptn"]
Sep 18 19:45:20.596011 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:20.595994    2522 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/664889b8-50a0-4ef8-9b4e-b4ca6f67ba35-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:45:20.596091 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:20.596013    2522 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/664889b8-50a0-4ef8-9b4e-b4ca6f67ba35-cabundle-cert\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:45:20.596091 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:20.596027    2522 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/664889b8-50a0-4ef8-9b4e-b4ca6f67ba35-proxy-tls\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:45:20.596091 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:20.596040    2522 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/664889b8-50a0-4ef8-9b4e-b4ca6f67ba35-kserve-provision-location\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:45:20.596091 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:20.596049    2522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb96h\" (UniqueName: \"kubernetes.io/projected/664889b8-50a0-4ef8-9b4e-b4ca6f67ba35-kube-api-access-zb96h\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:45:20.722189 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:20.722157    2522 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="664889b8-50a0-4ef8-9b4e-b4ca6f67ba35" path="/var/lib/kubelet/pods/664889b8-50a0-4ef8-9b4e-b4ca6f67ba35/volumes"
Sep 18 19:45:23.546974 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:23.546863    2522 generic.go:356] "Generic (PLEG): container finished" podID="db1cabe7-ba95-443e-b47f-6d5f67b028d4" containerID="91f7c7b5045463ffaf0f37c413e9821de57dbd0ac1151e7f28685dff50893827" exitCode=0
Sep 18 19:45:23.546974 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:23.546916    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk" event={"ID":"db1cabe7-ba95-443e-b47f-6d5f67b028d4","Type":"ContainerDied","Data":"91f7c7b5045463ffaf0f37c413e9821de57dbd0ac1151e7f28685dff50893827"}
Sep 18 19:45:24.553901 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:24.553867    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk" event={"ID":"db1cabe7-ba95-443e-b47f-6d5f67b028d4","Type":"ContainerStarted","Data":"edb9319e33bc0f027ae9a1722cb6a02f76a033627151a9ae54536319f956fc71"}
Sep 18 19:45:24.554334 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:24.553932    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk" event={"ID":"db1cabe7-ba95-443e-b47f-6d5f67b028d4","Type":"ContainerStarted","Data":"72091061c8efa280ae59dfecf132ab3c892230cd9e70d4e03babc27da2eacf52"}
Sep 18 19:45:24.554334 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:24.554143    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk"
Sep 18 19:45:24.584451 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:24.584390    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk" podStartSLOduration=2.774562613 podStartE2EDuration="6.584373182s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.809810569s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:45:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:45:24.580599512 +0000 UTC m=+1884.464125977" watchObservedRunningTime="2026-09-18 19:45:24.584373182 +0000 UTC m=+1884.467899644"
Sep 18 19:45:25.560197 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:25.560160    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk"
Sep 18 19:45:25.561540 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:25.561514    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk" podUID="db1cabe7-ba95-443e-b47f-6d5f67b028d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 18 19:45:26.565061 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:26.565015    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk" podUID="db1cabe7-ba95-443e-b47f-6d5f67b028d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 18 19:45:31.569451 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:31.569422    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk"
Sep 18 19:45:31.570004 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:31.569973    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk" podUID="db1cabe7-ba95-443e-b47f-6d5f67b028d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 18 19:45:41.570426 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:41.570383    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk" podUID="db1cabe7-ba95-443e-b47f-6d5f67b028d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 18 19:45:51.570631 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:45:51.570592    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk" podUID="db1cabe7-ba95-443e-b47f-6d5f67b028d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 18 19:46:01.570366 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:01.570326    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk" podUID="db1cabe7-ba95-443e-b47f-6d5f67b028d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 18 19:46:11.570892 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:11.570849    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk" podUID="db1cabe7-ba95-443e-b47f-6d5f67b028d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 18 19:46:21.569990 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:21.569948    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk" podUID="db1cabe7-ba95-443e-b47f-6d5f67b028d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 18 19:46:31.570686 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:31.570638    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk" podUID="db1cabe7-ba95-443e-b47f-6d5f67b028d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 18 19:46:41.571704 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:41.571673    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk"
Sep 18 19:46:48.921700 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:48.921665    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk"]
Sep 18 19:46:48.922173 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:48.922055    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk" podUID="db1cabe7-ba95-443e-b47f-6d5f67b028d4" containerName="kserve-container" containerID="cri-o://72091061c8efa280ae59dfecf132ab3c892230cd9e70d4e03babc27da2eacf52" gracePeriod=30
Sep 18 19:46:48.922255 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:48.922188    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk" podUID="db1cabe7-ba95-443e-b47f-6d5f67b028d4" containerName="kube-rbac-proxy" containerID="cri-o://edb9319e33bc0f027ae9a1722cb6a02f76a033627151a9ae54536319f956fc71" gracePeriod=30
Sep 18 19:46:49.964282 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:49.964243    2522 generic.go:356] "Generic (PLEG): container finished" podID="db1cabe7-ba95-443e-b47f-6d5f67b028d4" containerID="edb9319e33bc0f027ae9a1722cb6a02f76a033627151a9ae54536319f956fc71" exitCode=2
Sep 18 19:46:49.964692 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:49.964312    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk" event={"ID":"db1cabe7-ba95-443e-b47f-6d5f67b028d4","Type":"ContainerDied","Data":"edb9319e33bc0f027ae9a1722cb6a02f76a033627151a9ae54536319f956fc71"}
Sep 18 19:46:51.565739 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:51.565700    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk" podUID="db1cabe7-ba95-443e-b47f-6d5f67b028d4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.50:8643/healthz\": dial tcp 10.132.0.50:8643: connect: connection refused"
Sep 18 19:46:51.570034 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:51.570004    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk" podUID="db1cabe7-ba95-443e-b47f-6d5f67b028d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 18 19:46:54.186442 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.186372    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk"
Sep 18 19:46:54.218326 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.218296    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db1cabe7-ba95-443e-b47f-6d5f67b028d4-kube-rbac-proxy-sar-config\") pod \"db1cabe7-ba95-443e-b47f-6d5f67b028d4\" (UID: \"db1cabe7-ba95-443e-b47f-6d5f67b028d4\") "
Sep 18 19:46:54.218521 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.218345    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db1cabe7-ba95-443e-b47f-6d5f67b028d4-proxy-tls\") pod \"db1cabe7-ba95-443e-b47f-6d5f67b028d4\" (UID: \"db1cabe7-ba95-443e-b47f-6d5f67b028d4\") "
Sep 18 19:46:54.218521 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.218370    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db1cabe7-ba95-443e-b47f-6d5f67b028d4-cabundle-cert\") pod \"db1cabe7-ba95-443e-b47f-6d5f67b028d4\" (UID: \"db1cabe7-ba95-443e-b47f-6d5f67b028d4\") "
Sep 18 19:46:54.218521 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.218443    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-675d7\" (UniqueName: \"kubernetes.io/projected/db1cabe7-ba95-443e-b47f-6d5f67b028d4-kube-api-access-675d7\") pod \"db1cabe7-ba95-443e-b47f-6d5f67b028d4\" (UID: \"db1cabe7-ba95-443e-b47f-6d5f67b028d4\") "
Sep 18 19:46:54.218521 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.218471    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db1cabe7-ba95-443e-b47f-6d5f67b028d4-kserve-provision-location\") pod \"db1cabe7-ba95-443e-b47f-6d5f67b028d4\" (UID: \"db1cabe7-ba95-443e-b47f-6d5f67b028d4\") "
Sep 18 19:46:54.218781 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.218746    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db1cabe7-ba95-443e-b47f-6d5f67b028d4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "db1cabe7-ba95-443e-b47f-6d5f67b028d4" (UID: "db1cabe7-ba95-443e-b47f-6d5f67b028d4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:46:54.218984 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.218966    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db1cabe7-ba95-443e-b47f-6d5f67b028d4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "db1cabe7-ba95-443e-b47f-6d5f67b028d4" (UID: "db1cabe7-ba95-443e-b47f-6d5f67b028d4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:46:54.219056 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.219027    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db1cabe7-ba95-443e-b47f-6d5f67b028d4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "db1cabe7-ba95-443e-b47f-6d5f67b028d4" (UID: "db1cabe7-ba95-443e-b47f-6d5f67b028d4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:46:54.220651 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.220614    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db1cabe7-ba95-443e-b47f-6d5f67b028d4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "db1cabe7-ba95-443e-b47f-6d5f67b028d4" (UID: "db1cabe7-ba95-443e-b47f-6d5f67b028d4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:46:54.220757 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.220727    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db1cabe7-ba95-443e-b47f-6d5f67b028d4-kube-api-access-675d7" (OuterVolumeSpecName: "kube-api-access-675d7") pod "db1cabe7-ba95-443e-b47f-6d5f67b028d4" (UID: "db1cabe7-ba95-443e-b47f-6d5f67b028d4"). InnerVolumeSpecName "kube-api-access-675d7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:46:54.227671 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.227641    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-fxhm7"]
Sep 18 19:46:54.228148 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.228131    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db1cabe7-ba95-443e-b47f-6d5f67b028d4" containerName="kube-rbac-proxy"
Sep 18 19:46:54.228148 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.228149    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="db1cabe7-ba95-443e-b47f-6d5f67b028d4" containerName="kube-rbac-proxy"
Sep 18 19:46:54.228297 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.228190    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db1cabe7-ba95-443e-b47f-6d5f67b028d4" containerName="storage-initializer"
Sep 18 19:46:54.228297 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.228199    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="db1cabe7-ba95-443e-b47f-6d5f67b028d4" containerName="storage-initializer"
Sep 18 19:46:54.228297 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.228208    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="664889b8-50a0-4ef8-9b4e-b4ca6f67ba35" containerName="storage-initializer"
Sep 18 19:46:54.228297 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.228216    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="664889b8-50a0-4ef8-9b4e-b4ca6f67ba35" containerName="storage-initializer"
Sep 18 19:46:54.228297 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.228226    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db1cabe7-ba95-443e-b47f-6d5f67b028d4" containerName="kserve-container"
Sep 18 19:46:54.228297 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.228234    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="db1cabe7-ba95-443e-b47f-6d5f67b028d4" containerName="kserve-container"
Sep 18 19:46:54.228297 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.228246    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="664889b8-50a0-4ef8-9b4e-b4ca6f67ba35" containerName="kserve-container"
Sep 18 19:46:54.228297 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.228255    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="664889b8-50a0-4ef8-9b4e-b4ca6f67ba35" containerName="kserve-container"
Sep 18 19:46:54.228297 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.228268    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="664889b8-50a0-4ef8-9b4e-b4ca6f67ba35" containerName="kube-rbac-proxy"
Sep 18 19:46:54.228297 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.228276    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="664889b8-50a0-4ef8-9b4e-b4ca6f67ba35" containerName="kube-rbac-proxy"
Sep 18 19:46:54.228914 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.228367    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="664889b8-50a0-4ef8-9b4e-b4ca6f67ba35" containerName="kube-rbac-proxy"
Sep 18 19:46:54.228914 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.228382    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="db1cabe7-ba95-443e-b47f-6d5f67b028d4" containerName="kserve-container"
Sep 18 19:46:54.228914 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.228391    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="db1cabe7-ba95-443e-b47f-6d5f67b028d4" containerName="kube-rbac-proxy"
Sep 18 19:46:54.228914 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.228401    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="664889b8-50a0-4ef8-9b4e-b4ca6f67ba35" containerName="kserve-container"
Sep 18 19:46:54.232200 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.232178    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz"]
Sep 18 19:46:54.232313 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.232289    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-fxhm7"
Sep 18 19:46:54.235074 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.235053    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 18 19:46:54.235299 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.235283    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 18 19:46:54.235767 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.235748    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz"
Sep 18 19:46:54.239711 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.239690    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Sep 18 19:46:54.256930 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.256817    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-fxhm7"]
Sep 18 19:46:54.289581 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.289534    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz"]
Sep 18 19:46:54.320069 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.320032    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1dae9595-b8a2-40f6-9bf7-6d982d784649-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz\" (UID: \"1dae9595-b8a2-40f6-9bf7-6d982d784649\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz"
Sep 18 19:46:54.320069 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.320079    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5cda7e89-3d8f-454a-8ad8-d6fc55c42822-cabundle-cert\") pod \"isvc-canary-rollback-predictor-794fd55d6c-fxhm7\" (UID: \"5cda7e89-3d8f-454a-8ad8-d6fc55c42822\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-fxhm7"
Sep 18 19:46:54.320341 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.320104    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1dae9595-b8a2-40f6-9bf7-6d982d784649-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz\" (UID: \"1dae9595-b8a2-40f6-9bf7-6d982d784649\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz"
Sep 18 19:46:54.320341 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.320197    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cda7e89-3d8f-454a-8ad8-d6fc55c42822-proxy-tls\") pod \"isvc-canary-rollback-predictor-794fd55d6c-fxhm7\" (UID: \"5cda7e89-3d8f-454a-8ad8-d6fc55c42822\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-fxhm7"
Sep 18 19:46:54.320341 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.320243    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpl5m\" (UniqueName: \"kubernetes.io/projected/1dae9595-b8a2-40f6-9bf7-6d982d784649-kube-api-access-cpl5m\") pod \"isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz\" (UID: \"1dae9595-b8a2-40f6-9bf7-6d982d784649\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz"
Sep 18 19:46:54.320341 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.320299    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5cda7e89-3d8f-454a-8ad8-d6fc55c42822-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-794fd55d6c-fxhm7\" (UID: \"5cda7e89-3d8f-454a-8ad8-d6fc55c42822\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-fxhm7"
Sep 18 19:46:54.320341 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.320325    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1dae9595-b8a2-40f6-9bf7-6d982d784649-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz\" (UID: \"1dae9595-b8a2-40f6-9bf7-6d982d784649\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz"
Sep 18 19:46:54.320525 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.320349    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cda7e89-3d8f-454a-8ad8-d6fc55c42822-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-794fd55d6c-fxhm7\" (UID: \"5cda7e89-3d8f-454a-8ad8-d6fc55c42822\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-fxhm7"
Sep 18 19:46:54.320525 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.320373    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzr2c\" (UniqueName: \"kubernetes.io/projected/5cda7e89-3d8f-454a-8ad8-d6fc55c42822-kube-api-access-hzr2c\") pod \"isvc-canary-rollback-predictor-794fd55d6c-fxhm7\" (UID: \"5cda7e89-3d8f-454a-8ad8-d6fc55c42822\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-fxhm7"
Sep 18 19:46:54.320525 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.320388    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1dae9595-b8a2-40f6-9bf7-6d982d784649-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz\" (UID: \"1dae9595-b8a2-40f6-9bf7-6d982d784649\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz"
Sep 18 19:46:54.320525 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.320470    2522 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db1cabe7-ba95-443e-b47f-6d5f67b028d4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:46:54.320525 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.320489    2522 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db1cabe7-ba95-443e-b47f-6d5f67b028d4-proxy-tls\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:46:54.320525 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.320502    2522 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db1cabe7-ba95-443e-b47f-6d5f67b028d4-cabundle-cert\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:46:54.320525 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.320517    2522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-675d7\" (UniqueName: \"kubernetes.io/projected/db1cabe7-ba95-443e-b47f-6d5f67b028d4-kube-api-access-675d7\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:46:54.320778 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.320530    2522 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db1cabe7-ba95-443e-b47f-6d5f67b028d4-kserve-provision-location\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:46:54.420954 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.420918    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cda7e89-3d8f-454a-8ad8-d6fc55c42822-proxy-tls\") pod \"isvc-canary-rollback-predictor-794fd55d6c-fxhm7\" (UID: \"5cda7e89-3d8f-454a-8ad8-d6fc55c42822\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-fxhm7"
Sep 18 19:46:54.420954 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.420957    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpl5m\" (UniqueName: \"kubernetes.io/projected/1dae9595-b8a2-40f6-9bf7-6d982d784649-kube-api-access-cpl5m\") pod \"isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz\" (UID: \"1dae9595-b8a2-40f6-9bf7-6d982d784649\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz"
Sep 18 19:46:54.421235 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.420989    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5cda7e89-3d8f-454a-8ad8-d6fc55c42822-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-794fd55d6c-fxhm7\" (UID: \"5cda7e89-3d8f-454a-8ad8-d6fc55c42822\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-fxhm7"
Sep 18 19:46:54.421235 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.421013    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1dae9595-b8a2-40f6-9bf7-6d982d784649-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz\" (UID: \"1dae9595-b8a2-40f6-9bf7-6d982d784649\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz"
Sep 18 19:46:54.421235 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.421048    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cda7e89-3d8f-454a-8ad8-d6fc55c42822-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-794fd55d6c-fxhm7\" (UID: \"5cda7e89-3d8f-454a-8ad8-d6fc55c42822\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-fxhm7"
Sep 18 19:46:54.421235 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.421084    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzr2c\" (UniqueName: \"kubernetes.io/projected/5cda7e89-3d8f-454a-8ad8-d6fc55c42822-kube-api-access-hzr2c\") pod \"isvc-canary-rollback-predictor-794fd55d6c-fxhm7\" (UID: \"5cda7e89-3d8f-454a-8ad8-d6fc55c42822\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-fxhm7"
Sep 18 19:46:54.421235 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.421108    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1dae9595-b8a2-40f6-9bf7-6d982d784649-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz\" (UID: \"1dae9595-b8a2-40f6-9bf7-6d982d784649\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz"
Sep 18 19:46:54.421535 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.421258    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1dae9595-b8a2-40f6-9bf7-6d982d784649-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz\" (UID: \"1dae9595-b8a2-40f6-9bf7-6d982d784649\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz"
Sep 18 19:46:54.421535 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.421305    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5cda7e89-3d8f-454a-8ad8-d6fc55c42822-cabundle-cert\") pod \"isvc-canary-rollback-predictor-794fd55d6c-fxhm7\" (UID: \"5cda7e89-3d8f-454a-8ad8-d6fc55c42822\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-fxhm7"
Sep 18 19:46:54.421535 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.421333    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1dae9595-b8a2-40f6-9bf7-6d982d784649-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz\" (UID: \"1dae9595-b8a2-40f6-9bf7-6d982d784649\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz"
Sep 18 19:46:54.421535 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.421481    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5cda7e89-3d8f-454a-8ad8-d6fc55c42822-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-794fd55d6c-fxhm7\" (UID: \"5cda7e89-3d8f-454a-8ad8-d6fc55c42822\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-fxhm7"
Sep 18 19:46:54.421956 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.421925    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1dae9595-b8a2-40f6-9bf7-6d982d784649-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz\" (UID: \"1dae9595-b8a2-40f6-9bf7-6d982d784649\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz"
Sep 18 19:46:54.422084 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.421966    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cda7e89-3d8f-454a-8ad8-d6fc55c42822-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-794fd55d6c-fxhm7\" (UID: \"5cda7e89-3d8f-454a-8ad8-d6fc55c42822\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-fxhm7"
Sep 18 19:46:54.422084 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.421978    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5cda7e89-3d8f-454a-8ad8-d6fc55c42822-cabundle-cert\") pod \"isvc-canary-rollback-predictor-794fd55d6c-fxhm7\" (UID: \"5cda7e89-3d8f-454a-8ad8-d6fc55c42822\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-fxhm7"
Sep 18 19:46:54.422084 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.422025    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1dae9595-b8a2-40f6-9bf7-6d982d784649-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz\" (UID: \"1dae9595-b8a2-40f6-9bf7-6d982d784649\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz"
Sep 18 19:46:54.422208 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.422095    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1dae9595-b8a2-40f6-9bf7-6d982d784649-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz\" (UID: \"1dae9595-b8a2-40f6-9bf7-6d982d784649\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz"
Sep 18 19:46:54.423704 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.423683    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cda7e89-3d8f-454a-8ad8-d6fc55c42822-proxy-tls\") pod \"isvc-canary-rollback-predictor-794fd55d6c-fxhm7\" (UID: \"5cda7e89-3d8f-454a-8ad8-d6fc55c42822\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-fxhm7"
Sep 18 19:46:54.423890 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.423875    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1dae9595-b8a2-40f6-9bf7-6d982d784649-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz\" (UID: \"1dae9595-b8a2-40f6-9bf7-6d982d784649\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz"
Sep 18 19:46:54.430476 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.430454    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpl5m\" (UniqueName: \"kubernetes.io/projected/1dae9595-b8a2-40f6-9bf7-6d982d784649-kube-api-access-cpl5m\") pod \"isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz\" (UID: \"1dae9595-b8a2-40f6-9bf7-6d982d784649\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz"
Sep 18 19:46:54.432840 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.432823    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzr2c\" (UniqueName: \"kubernetes.io/projected/5cda7e89-3d8f-454a-8ad8-d6fc55c42822-kube-api-access-hzr2c\") pod \"isvc-canary-rollback-predictor-794fd55d6c-fxhm7\" (UID: \"5cda7e89-3d8f-454a-8ad8-d6fc55c42822\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-fxhm7"
Sep 18 19:46:54.553705 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.553617    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-fxhm7"
Sep 18 19:46:54.560922 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.560890    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz"
Sep 18 19:46:54.832272 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.832247    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-fxhm7"]
Sep 18 19:46:54.833726 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:46:54.833695    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cda7e89_3d8f_454a_8ad8_d6fc55c42822.slice/crio-cca4a0aed3ca5cd45edaf207dd033207277a04ef619b2eb2a7b5d5c3e27c8e19 WatchSource:0}: Error finding container cca4a0aed3ca5cd45edaf207dd033207277a04ef619b2eb2a7b5d5c3e27c8e19: Status 404 returned error can't find the container with id cca4a0aed3ca5cd45edaf207dd033207277a04ef619b2eb2a7b5d5c3e27c8e19
Sep 18 19:46:54.839451 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.839422    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz"]
Sep 18 19:46:54.842849 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:46:54.842822    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dae9595_b8a2_40f6_9bf7_6d982d784649.slice/crio-01cc5e6502aecdd308ea57b08921abb71f10ed6cb8843e2b064f478e76f0e353 WatchSource:0}: Error finding container 01cc5e6502aecdd308ea57b08921abb71f10ed6cb8843e2b064f478e76f0e353: Status 404 returned error can't find the container with id 01cc5e6502aecdd308ea57b08921abb71f10ed6cb8843e2b064f478e76f0e353
Sep 18 19:46:54.989898 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.989859    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz" event={"ID":"1dae9595-b8a2-40f6-9bf7-6d982d784649","Type":"ContainerStarted","Data":"9096aeb3f18484828608fa1cd4f02ede128b3375e0438e07822e1fceb6ce5735"}
Sep 18 19:46:54.990111 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.989941    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz" event={"ID":"1dae9595-b8a2-40f6-9bf7-6d982d784649","Type":"ContainerStarted","Data":"01cc5e6502aecdd308ea57b08921abb71f10ed6cb8843e2b064f478e76f0e353"}
Sep 18 19:46:54.991731 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.991692    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-fxhm7" event={"ID":"5cda7e89-3d8f-454a-8ad8-d6fc55c42822","Type":"ContainerStarted","Data":"89ff932c3a060aa17c89efd0b7895b47d54d4d652cca70f85a74f62582525c67"}
Sep 18 19:46:54.991843 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.991766    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-fxhm7" event={"ID":"5cda7e89-3d8f-454a-8ad8-d6fc55c42822","Type":"ContainerStarted","Data":"cca4a0aed3ca5cd45edaf207dd033207277a04ef619b2eb2a7b5d5c3e27c8e19"}
Sep 18 19:46:54.993946 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.993911    2522 generic.go:356] "Generic (PLEG): container finished" podID="db1cabe7-ba95-443e-b47f-6d5f67b028d4" containerID="72091061c8efa280ae59dfecf132ab3c892230cd9e70d4e03babc27da2eacf52" exitCode=0
Sep 18 19:46:54.994063 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.993951    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk" event={"ID":"db1cabe7-ba95-443e-b47f-6d5f67b028d4","Type":"ContainerDied","Data":"72091061c8efa280ae59dfecf132ab3c892230cd9e70d4e03babc27da2eacf52"}
Sep 18 19:46:54.994063 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.993992    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk" event={"ID":"db1cabe7-ba95-443e-b47f-6d5f67b028d4","Type":"ContainerDied","Data":"dbcee6bcb8b359cf8d6d42686ad7de7fffe2722f10d195faadc81953bb0826f6"}
Sep 18 19:46:54.994063 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.994002    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk"
Sep 18 19:46:54.994063 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:54.994020    2522 scope.go:117] "RemoveContainer" containerID="edb9319e33bc0f027ae9a1722cb6a02f76a033627151a9ae54536319f956fc71"
Sep 18 19:46:55.004840 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:55.004820    2522 scope.go:117] "RemoveContainer" containerID="72091061c8efa280ae59dfecf132ab3c892230cd9e70d4e03babc27da2eacf52"
Sep 18 19:46:55.014428 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:55.014392    2522 scope.go:117] "RemoveContainer" containerID="91f7c7b5045463ffaf0f37c413e9821de57dbd0ac1151e7f28685dff50893827"
Sep 18 19:46:55.023213 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:55.023189    2522 scope.go:117] "RemoveContainer" containerID="edb9319e33bc0f027ae9a1722cb6a02f76a033627151a9ae54536319f956fc71"
Sep 18 19:46:55.023606 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:46:55.023579    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edb9319e33bc0f027ae9a1722cb6a02f76a033627151a9ae54536319f956fc71\": container with ID starting with edb9319e33bc0f027ae9a1722cb6a02f76a033627151a9ae54536319f956fc71 not found: ID does not exist" containerID="edb9319e33bc0f027ae9a1722cb6a02f76a033627151a9ae54536319f956fc71"
Sep 18 19:46:55.023714 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:55.023611    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edb9319e33bc0f027ae9a1722cb6a02f76a033627151a9ae54536319f956fc71"} err="failed to get container status \"edb9319e33bc0f027ae9a1722cb6a02f76a033627151a9ae54536319f956fc71\": rpc error: code = NotFound desc = could not find container \"edb9319e33bc0f027ae9a1722cb6a02f76a033627151a9ae54536319f956fc71\": container with ID starting with edb9319e33bc0f027ae9a1722cb6a02f76a033627151a9ae54536319f956fc71 not found: ID does not exist"
Sep 18 19:46:55.023714 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:55.023631    2522 scope.go:117] "RemoveContainer" containerID="72091061c8efa280ae59dfecf132ab3c892230cd9e70d4e03babc27da2eacf52"
Sep 18 19:46:55.023916 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:46:55.023895    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72091061c8efa280ae59dfecf132ab3c892230cd9e70d4e03babc27da2eacf52\": container with ID starting with 72091061c8efa280ae59dfecf132ab3c892230cd9e70d4e03babc27da2eacf52 not found: ID does not exist" containerID="72091061c8efa280ae59dfecf132ab3c892230cd9e70d4e03babc27da2eacf52"
Sep 18 19:46:55.023985 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:55.023918    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72091061c8efa280ae59dfecf132ab3c892230cd9e70d4e03babc27da2eacf52"} err="failed to get container status \"72091061c8efa280ae59dfecf132ab3c892230cd9e70d4e03babc27da2eacf52\": rpc error: code = NotFound desc = could not find container \"72091061c8efa280ae59dfecf132ab3c892230cd9e70d4e03babc27da2eacf52\": container with ID starting with 72091061c8efa280ae59dfecf132ab3c892230cd9e70d4e03babc27da2eacf52 not found: ID does not exist"
Sep 18 19:46:55.023985 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:55.023932    2522 scope.go:117] "RemoveContainer" containerID="91f7c7b5045463ffaf0f37c413e9821de57dbd0ac1151e7f28685dff50893827"
Sep 18 19:46:55.024182 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:46:55.024165    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91f7c7b5045463ffaf0f37c413e9821de57dbd0ac1151e7f28685dff50893827\": container with ID starting with 91f7c7b5045463ffaf0f37c413e9821de57dbd0ac1151e7f28685dff50893827 not found: ID does not exist" containerID="91f7c7b5045463ffaf0f37c413e9821de57dbd0ac1151e7f28685dff50893827"
Sep 18 19:46:55.024251 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:55.024181    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91f7c7b5045463ffaf0f37c413e9821de57dbd0ac1151e7f28685dff50893827"} err="failed to get container status \"91f7c7b5045463ffaf0f37c413e9821de57dbd0ac1151e7f28685dff50893827\": rpc error: code = NotFound desc = could not find container \"91f7c7b5045463ffaf0f37c413e9821de57dbd0ac1151e7f28685dff50893827\": container with ID starting with 91f7c7b5045463ffaf0f37c413e9821de57dbd0ac1151e7f28685dff50893827 not found: ID does not exist"
Sep 18 19:46:55.033839 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:55.033809    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk"]
Sep 18 19:46:55.038043 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:55.038017    2522 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-f7npk"]
Sep 18 19:46:56.721595 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:56.721542    2522 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db1cabe7-ba95-443e-b47f-6d5f67b028d4" path="/var/lib/kubelet/pods/db1cabe7-ba95-443e-b47f-6d5f67b028d4/volumes"
Sep 18 19:46:59.018140 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:59.018053    2522 generic.go:356] "Generic (PLEG): container finished" podID="1dae9595-b8a2-40f6-9bf7-6d982d784649" containerID="9096aeb3f18484828608fa1cd4f02ede128b3375e0438e07822e1fceb6ce5735" exitCode=0
Sep 18 19:46:59.018140 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:59.018127    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz" event={"ID":"1dae9595-b8a2-40f6-9bf7-6d982d784649","Type":"ContainerDied","Data":"9096aeb3f18484828608fa1cd4f02ede128b3375e0438e07822e1fceb6ce5735"}
Sep 18 19:46:59.019794 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:59.019766    2522 generic.go:356] "Generic (PLEG): container finished" podID="5cda7e89-3d8f-454a-8ad8-d6fc55c42822" containerID="89ff932c3a060aa17c89efd0b7895b47d54d4d652cca70f85a74f62582525c67" exitCode=0
Sep 18 19:46:59.019883 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:46:59.019848    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-fxhm7" event={"ID":"5cda7e89-3d8f-454a-8ad8-d6fc55c42822","Type":"ContainerDied","Data":"89ff932c3a060aa17c89efd0b7895b47d54d4d652cca70f85a74f62582525c67"}
Sep 18 19:47:00.027845 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:47:00.027794    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz" event={"ID":"1dae9595-b8a2-40f6-9bf7-6d982d784649","Type":"ContainerStarted","Data":"71c3629db7e6caca1532d0c6dc4ca0f22b6957a4f41096f248149165d5ec541b"}
Sep 18 19:47:00.028356 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:47:00.027882    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz" event={"ID":"1dae9595-b8a2-40f6-9bf7-6d982d784649","Type":"ContainerStarted","Data":"98f392277b32b668fcdecd14ddd79e0fa597eec2f76f8ad9055161cdfe1014a4"}
Sep 18 19:47:00.028356 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:47:00.028123    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz"
Sep 18 19:47:00.030433 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:47:00.030402    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-fxhm7" event={"ID":"5cda7e89-3d8f-454a-8ad8-d6fc55c42822","Type":"ContainerStarted","Data":"213afa0299a9ff7460146cf43a1c538de2323873d78fa5b09e461dbb0f5cb2e2"}
Sep 18 19:47:00.030534 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:47:00.030475    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-fxhm7" event={"ID":"5cda7e89-3d8f-454a-8ad8-d6fc55c42822","Type":"ContainerStarted","Data":"88c851e12e533b2fa5c57fe61607554eb7202ad8b50b7bf6f1f2f3123ef30d35"}
Sep 18 19:47:00.030822 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:47:00.030804    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-fxhm7"
Sep 18 19:47:00.030869 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:47:00.030856    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-fxhm7"
Sep 18 19:47:00.032774 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:47:00.032742    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-fxhm7" podUID="5cda7e89-3d8f-454a-8ad8-d6fc55c42822" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 18 19:47:00.051286 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:47:00.051194    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz" podStartSLOduration=2.459677572 podStartE2EDuration="6.051178101s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.591500529s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:46:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:47:00.050363476 +0000 UTC m=+1979.933889949" watchObservedRunningTime="2026-09-18 19:47:00.051178101 +0000 UTC m=+1979.934704564"
Sep 18 19:47:00.069800 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:47:00.069746    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-fxhm7" podStartSLOduration=2.197353756 podStartE2EDuration="6.069730411s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.872376655s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:46:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:47:00.068817199 +0000 UTC m=+1979.952343662" watchObservedRunningTime="2026-09-18 19:47:00.069730411 +0000 UTC m=+1979.953256870"
Sep 18 19:47:01.036741 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:47:01.036689    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-fxhm7" podUID="5cda7e89-3d8f-454a-8ad8-d6fc55c42822" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 18 19:47:01.037187 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:47:01.036788    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz"
Sep 18 19:47:01.038324 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:47:01.038300    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz" podUID="1dae9595-b8a2-40f6-9bf7-6d982d784649" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 18 19:47:02.041628 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:47:02.041590    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz" podUID="1dae9595-b8a2-40f6-9bf7-6d982d784649" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 18 19:47:06.040776 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:47:06.040740    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-fxhm7"
Sep 18 19:47:06.041400 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:47:06.041372    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-fxhm7" podUID="5cda7e89-3d8f-454a-8ad8-d6fc55c42822" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 18 19:47:07.047176 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:47:07.047144    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz"
Sep 18 19:47:07.047799 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:47:07.047773    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz" podUID="1dae9595-b8a2-40f6-9bf7-6d982d784649" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 18 19:47:16.042190 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:47:16.042146    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-fxhm7" podUID="5cda7e89-3d8f-454a-8ad8-d6fc55c42822" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 18 19:47:17.048189 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:47:17.048146    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz" podUID="1dae9595-b8a2-40f6-9bf7-6d982d784649" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 18 19:47:26.041442 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:47:26.041396    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-fxhm7" podUID="5cda7e89-3d8f-454a-8ad8-d6fc55c42822" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 18 19:47:27.047946 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:47:27.047905    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz" podUID="1dae9595-b8a2-40f6-9bf7-6d982d784649" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 18 19:47:36.041292 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:47:36.041249    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-fxhm7" podUID="5cda7e89-3d8f-454a-8ad8-d6fc55c42822" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 18 19:47:37.048541 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:47:37.048497    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz" podUID="1dae9595-b8a2-40f6-9bf7-6d982d784649" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 18 19:47:42.850908 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:47:42.850870    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t"]
Sep 18 19:47:42.854717 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:47:42.854694    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t"
Sep 18 19:47:42.857674 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:47:42.857650    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\""
Sep 18 19:47:42.857829 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:47:42.857677    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-predictor-serving-cert\""
Sep 18 19:47:42.860717 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:47:42.860687    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t"]
Sep 18 19:47:42.935433 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:47:42.935394    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbe15859-6546-4c39-9afe-ec6a32ddb890-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t\" (UID: \"dbe15859-6546-4c39-9afe-ec6a32ddb890\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t"
Sep 18 19:47:42.935433 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:47:42.935437    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbe15859-6546-4c39-9afe-ec6a32ddb890-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t\" (UID: \"dbe15859-6546-4c39-9afe-ec6a32ddb890\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t"
Sep 18 19:47:42.935730 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:47:42.935523    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v8cz\" (UniqueName: \"kubernetes.io/projected/dbe15859-6546-4c39-9afe-ec6a32ddb890-kube-api-access-6v8cz\") pod \"isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t\" (UID: \"dbe15859-6546-4c39-9afe-ec6a32ddb890\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t"
Sep 18 19:47:42.935730 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:47:42.935585    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dbe15859-6546-4c39-9afe-ec6a32ddb890-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t\" (UID: \"dbe15859-6546-4c39-9afe-ec6a32ddb890\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t"
Sep 18 19:47:42.935730 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:47:42.935633    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbe15859-6546-4c39-9afe-ec6a32ddb890-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t\" (UID: \"dbe15859-6546-4c39-9afe-ec6a32ddb890\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t"
Sep 18 19:47:43.036231 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:47:43.036179    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v8cz\" (UniqueName: \"kubernetes.io/projected/dbe15859-6546-4c39-9afe-ec6a32ddb890-kube-api-access-6v8cz\") pod \"isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t\" (UID: \"dbe15859-6546-4c39-9afe-ec6a32ddb890\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t"
Sep 18 19:47:43.036231 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:47:43.036226    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dbe15859-6546-4c39-9afe-ec6a32ddb890-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t\" (UID: \"dbe15859-6546-4c39-9afe-ec6a32ddb890\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t"
Sep 18 19:47:43.036518 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:47:43.036273    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbe15859-6546-4c39-9afe-ec6a32ddb890-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t\" (UID: \"dbe15859-6546-4c39-9afe-ec6a32ddb890\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t"
Sep 18 19:47:43.036518 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:47:43.036346    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbe15859-6546-4c39-9afe-ec6a32ddb890-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t\" (UID: \"dbe15859-6546-4c39-9afe-ec6a32ddb890\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t"
Sep 18 19:47:43.036518 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:47:43.036375    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbe15859-6546-4c39-9afe-ec6a32ddb890-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t\" (UID: \"dbe15859-6546-4c39-9afe-ec6a32ddb890\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t"
Sep 18 19:47:43.036518 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:47:43.036485    2522 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-serving-cert: secret "isvc-tensorflow-runtime-predictor-serving-cert" not found
Sep 18 19:47:43.036769 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:47:43.036585    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbe15859-6546-4c39-9afe-ec6a32ddb890-proxy-tls podName:dbe15859-6546-4c39-9afe-ec6a32ddb890 nodeName:}" failed. No retries permitted until 2026-09-18 19:47:43.536538354 +0000 UTC m=+2023.420064808 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/dbe15859-6546-4c39-9afe-ec6a32ddb890-proxy-tls") pod "isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t" (UID: "dbe15859-6546-4c39-9afe-ec6a32ddb890") : secret "isvc-tensorflow-runtime-predictor-serving-cert" not found
Sep 18 19:47:43.036887 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:47:43.036864    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbe15859-6546-4c39-9afe-ec6a32ddb890-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t\" (UID: \"dbe15859-6546-4c39-9afe-ec6a32ddb890\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t"
Sep 18 19:47:43.037095 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:47:43.037072    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dbe15859-6546-4c39-9afe-ec6a32ddb890-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t\" (UID: \"dbe15859-6546-4c39-9afe-ec6a32ddb890\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t"
Sep 18 19:47:43.037095 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:47:43.037088    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbe15859-6546-4c39-9afe-ec6a32ddb890-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t\" (UID: \"dbe15859-6546-4c39-9afe-ec6a32ddb890\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t"
Sep 18 19:47:43.047141 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:47:43.047109    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v8cz\" (UniqueName: \"kubernetes.io/projected/dbe15859-6546-4c39-9afe-ec6a32ddb890-kube-api-access-6v8cz\") pod \"isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t\" (UID: \"dbe15859-6546-4c39-9afe-ec6a32ddb890\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t"
Sep 18 19:47:43.539605 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:47:43.539545    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbe15859-6546-4c39-9afe-ec6a32ddb890-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t\" (UID: \"dbe15859-6546-4c39-9afe-ec6a32ddb890\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t"
Sep 18 19:47:43.542032 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:47:43.542006    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbe15859-6546-4c39-9afe-ec6a32ddb890-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t\" (UID: \"dbe15859-6546-4c39-9afe-ec6a32ddb890\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t"
Sep 18 19:47:43.766539 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:47:43.766498    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t"
Sep 18 19:47:44.042591 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:47:44.042546    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t"]
Sep 18 19:47:44.043596 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:47:44.043551    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbe15859_6546_4c39_9afe_ec6a32ddb890.slice/crio-e2cd27dbd65ab4d35773ddb16f4380b5a53c0106269167c8407f576e15728c61 WatchSource:0}: Error finding container e2cd27dbd65ab4d35773ddb16f4380b5a53c0106269167c8407f576e15728c61: Status 404 returned error can't find the container with id e2cd27dbd65ab4d35773ddb16f4380b5a53c0106269167c8407f576e15728c61
Sep 18 19:47:44.242167 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:47:44.242131    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t" event={"ID":"dbe15859-6546-4c39-9afe-ec6a32ddb890","Type":"ContainerStarted","Data":"0a3c2db1f5dcd62b65ad9634b6336932fa99034d09f8beeddd92753f8c471b2e"}
Sep 18 19:47:44.242343 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:47:44.242192    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t" event={"ID":"dbe15859-6546-4c39-9afe-ec6a32ddb890","Type":"ContainerStarted","Data":"e2cd27dbd65ab4d35773ddb16f4380b5a53c0106269167c8407f576e15728c61"}
Sep 18 19:47:46.041550 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:47:46.041513    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-fxhm7" podUID="5cda7e89-3d8f-454a-8ad8-d6fc55c42822" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 18 19:47:47.048123 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:47:47.048085    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz" podUID="1dae9595-b8a2-40f6-9bf7-6d982d784649" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 18 19:47:49.268585 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:47:49.268527    2522 generic.go:356] "Generic (PLEG): container finished" podID="dbe15859-6546-4c39-9afe-ec6a32ddb890" containerID="0a3c2db1f5dcd62b65ad9634b6336932fa99034d09f8beeddd92753f8c471b2e" exitCode=0
Sep 18 19:47:49.269001 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:47:49.268610    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t" event={"ID":"dbe15859-6546-4c39-9afe-ec6a32ddb890","Type":"ContainerDied","Data":"0a3c2db1f5dcd62b65ad9634b6336932fa99034d09f8beeddd92753f8c471b2e"}
Sep 18 19:47:49.269866 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:47:49.269850    2522 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:47:53.294651 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:47:53.294612    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t" event={"ID":"dbe15859-6546-4c39-9afe-ec6a32ddb890","Type":"ContainerStarted","Data":"29829510436f4553a0178bed4150cce06d2b7243bb4f3bab2e13d99ec04573e3"}
Sep 18 19:47:53.295049 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:47:53.294685    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t" event={"ID":"dbe15859-6546-4c39-9afe-ec6a32ddb890","Type":"ContainerStarted","Data":"569126494d613d28180b42f4fc64a71e62be05a3890a17080ace06bea809cffa"}
Sep 18 19:47:53.295049 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:47:53.294888    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t"
Sep 18 19:47:53.321529 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:47:53.321439    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t" podStartSLOduration=3.059380374 podStartE2EDuration="11.321422931s" totalImagesPullingTime="3.800420466s" totalInitContainerRuntime="4.461622091s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:47:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:47:53.320063474 +0000 UTC m=+2033.203589937" watchObservedRunningTime="2026-09-18 19:47:53.321422931 +0000 UTC m=+2033.204949393"
Sep 18 19:47:54.301916 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:47:54.301882    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t"
Sep 18 19:47:54.303293 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:47:54.303262    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t" podUID="dbe15859-6546-4c39-9afe-ec6a32ddb890" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 18 19:47:55.307268 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:47:55.307223    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t" podUID="dbe15859-6546-4c39-9afe-ec6a32ddb890" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 18 19:47:56.041387 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:47:56.041344    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-fxhm7" podUID="5cda7e89-3d8f-454a-8ad8-d6fc55c42822" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 18 19:47:57.047718 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:47:57.047676    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz" podUID="1dae9595-b8a2-40f6-9bf7-6d982d784649" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 18 19:48:00.311761 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:00.311730    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t"
Sep 18 19:48:00.312444 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:00.312418    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t" podUID="dbe15859-6546-4c39-9afe-ec6a32ddb890" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 18 19:48:06.041283 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:06.041243    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-fxhm7" podUID="5cda7e89-3d8f-454a-8ad8-d6fc55c42822" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 18 19:48:07.047741 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:07.047702    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz" podUID="1dae9595-b8a2-40f6-9bf7-6d982d784649" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 18 19:48:10.313835 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:10.313806    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t"
Sep 18 19:48:16.041493 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:16.041450    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-fxhm7" podUID="5cda7e89-3d8f-454a-8ad8-d6fc55c42822" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 18 19:48:17.047685 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:17.047640    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz" podUID="1dae9595-b8a2-40f6-9bf7-6d982d784649" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 18 19:48:24.186852 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:24.186767    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t"]
Sep 18 19:48:24.187291 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:24.187026    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t" podUID="dbe15859-6546-4c39-9afe-ec6a32ddb890" containerName="kserve-container" containerID="cri-o://569126494d613d28180b42f4fc64a71e62be05a3890a17080ace06bea809cffa" gracePeriod=30
Sep 18 19:48:24.187291 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:24.187046    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t" podUID="dbe15859-6546-4c39-9afe-ec6a32ddb890" containerName="kube-rbac-proxy" containerID="cri-o://29829510436f4553a0178bed4150cce06d2b7243bb4f3bab2e13d99ec04573e3" gracePeriod=30
Sep 18 19:48:24.452437 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:24.452339    2522 generic.go:356] "Generic (PLEG): container finished" podID="dbe15859-6546-4c39-9afe-ec6a32ddb890" containerID="29829510436f4553a0178bed4150cce06d2b7243bb4f3bab2e13d99ec04573e3" exitCode=2
Sep 18 19:48:24.452437 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:24.452400    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t" event={"ID":"dbe15859-6546-4c39-9afe-ec6a32ddb890","Type":"ContainerDied","Data":"29829510436f4553a0178bed4150cce06d2b7243bb4f3bab2e13d99ec04573e3"}
Sep 18 19:48:25.307503 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:25.307455    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t" podUID="dbe15859-6546-4c39-9afe-ec6a32ddb890" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.53:8643/healthz\": dial tcp 10.132.0.53:8643: connect: connection refused"
Sep 18 19:48:26.042765 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:26.042727    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-fxhm7"
Sep 18 19:48:27.048548 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:27.048518    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz"
Sep 18 19:48:30.308330 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:30.308273    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t" podUID="dbe15859-6546-4c39-9afe-ec6a32ddb890" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.53:8643/healthz\": dial tcp 10.132.0.53:8643: connect: connection refused"
Sep 18 19:48:34.179489 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:34.179454    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz"]
Sep 18 19:48:34.179949 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:34.179703    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz" podUID="1dae9595-b8a2-40f6-9bf7-6d982d784649" containerName="kserve-container" containerID="cri-o://98f392277b32b668fcdecd14ddd79e0fa597eec2f76f8ad9055161cdfe1014a4" gracePeriod=30
Sep 18 19:48:34.179949 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:34.179752    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz" podUID="1dae9595-b8a2-40f6-9bf7-6d982d784649" containerName="kube-rbac-proxy" containerID="cri-o://71c3629db7e6caca1532d0c6dc4ca0f22b6957a4f41096f248149165d5ec541b" gracePeriod=30
Sep 18 19:48:34.514271 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:34.514185    2522 generic.go:356] "Generic (PLEG): container finished" podID="1dae9595-b8a2-40f6-9bf7-6d982d784649" containerID="71c3629db7e6caca1532d0c6dc4ca0f22b6957a4f41096f248149165d5ec541b" exitCode=2
Sep 18 19:48:34.514271 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:34.514257    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz" event={"ID":"1dae9595-b8a2-40f6-9bf7-6d982d784649","Type":"ContainerDied","Data":"71c3629db7e6caca1532d0c6dc4ca0f22b6957a4f41096f248149165d5ec541b"}
Sep 18 19:48:35.308220 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:35.308179    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t" podUID="dbe15859-6546-4c39-9afe-ec6a32ddb890" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.53:8643/healthz\": dial tcp 10.132.0.53:8643: connect: connection refused"
Sep 18 19:48:35.308649 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:35.308311    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t"
Sep 18 19:48:37.041919 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:37.041881    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz" podUID="1dae9595-b8a2-40f6-9bf7-6d982d784649" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.52:8643/healthz\": dial tcp 10.132.0.52:8643: connect: connection refused"
Sep 18 19:48:37.048337 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:37.048304    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz" podUID="1dae9595-b8a2-40f6-9bf7-6d982d784649" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 18 19:48:39.230910 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:39.230884    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz"
Sep 18 19:48:39.337277 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:39.337252    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1dae9595-b8a2-40f6-9bf7-6d982d784649-cabundle-cert\") pod \"1dae9595-b8a2-40f6-9bf7-6d982d784649\" (UID: \"1dae9595-b8a2-40f6-9bf7-6d982d784649\") "
Sep 18 19:48:39.337469 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:39.337303    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1dae9595-b8a2-40f6-9bf7-6d982d784649-kube-rbac-proxy-sar-config\") pod \"1dae9595-b8a2-40f6-9bf7-6d982d784649\" (UID: \"1dae9595-b8a2-40f6-9bf7-6d982d784649\") "
Sep 18 19:48:39.337469 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:39.337352    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1dae9595-b8a2-40f6-9bf7-6d982d784649-proxy-tls\") pod \"1dae9595-b8a2-40f6-9bf7-6d982d784649\" (UID: \"1dae9595-b8a2-40f6-9bf7-6d982d784649\") "
Sep 18 19:48:39.337469 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:39.337372    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1dae9595-b8a2-40f6-9bf7-6d982d784649-kserve-provision-location\") pod \"1dae9595-b8a2-40f6-9bf7-6d982d784649\" (UID: \"1dae9595-b8a2-40f6-9bf7-6d982d784649\") "
Sep 18 19:48:39.337469 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:39.337414    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cpl5m\" (UniqueName: \"kubernetes.io/projected/1dae9595-b8a2-40f6-9bf7-6d982d784649-kube-api-access-cpl5m\") pod \"1dae9595-b8a2-40f6-9bf7-6d982d784649\" (UID: \"1dae9595-b8a2-40f6-9bf7-6d982d784649\") "
Sep 18 19:48:39.337734 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:39.337653    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dae9595-b8a2-40f6-9bf7-6d982d784649-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1dae9595-b8a2-40f6-9bf7-6d982d784649" (UID: "1dae9595-b8a2-40f6-9bf7-6d982d784649"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:48:39.337794 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:39.337727    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dae9595-b8a2-40f6-9bf7-6d982d784649-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1dae9595-b8a2-40f6-9bf7-6d982d784649" (UID: "1dae9595-b8a2-40f6-9bf7-6d982d784649"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:48:39.337794 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:39.337770    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dae9595-b8a2-40f6-9bf7-6d982d784649-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1dae9595-b8a2-40f6-9bf7-6d982d784649" (UID: "1dae9595-b8a2-40f6-9bf7-6d982d784649"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:48:39.339557 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:39.339535    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dae9595-b8a2-40f6-9bf7-6d982d784649-kube-api-access-cpl5m" (OuterVolumeSpecName: "kube-api-access-cpl5m") pod "1dae9595-b8a2-40f6-9bf7-6d982d784649" (UID: "1dae9595-b8a2-40f6-9bf7-6d982d784649"). InnerVolumeSpecName "kube-api-access-cpl5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:48:39.339650 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:39.339606    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dae9595-b8a2-40f6-9bf7-6d982d784649-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1dae9595-b8a2-40f6-9bf7-6d982d784649" (UID: "1dae9595-b8a2-40f6-9bf7-6d982d784649"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:48:39.438638 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:39.438601    2522 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1dae9595-b8a2-40f6-9bf7-6d982d784649-proxy-tls\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:48:39.438638 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:39.438636    2522 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1dae9595-b8a2-40f6-9bf7-6d982d784649-kserve-provision-location\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:48:39.438638 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:39.438648    2522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpl5m\" (UniqueName: \"kubernetes.io/projected/1dae9595-b8a2-40f6-9bf7-6d982d784649-kube-api-access-cpl5m\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:48:39.438875 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:39.438657    2522 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1dae9595-b8a2-40f6-9bf7-6d982d784649-cabundle-cert\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:48:39.438875 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:39.438666    2522 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1dae9595-b8a2-40f6-9bf7-6d982d784649-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:48:39.541908 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:39.541874    2522 generic.go:356] "Generic (PLEG): container finished" podID="1dae9595-b8a2-40f6-9bf7-6d982d784649" containerID="98f392277b32b668fcdecd14ddd79e0fa597eec2f76f8ad9055161cdfe1014a4" exitCode=0
Sep 18 19:48:39.542087 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:39.541931    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz" event={"ID":"1dae9595-b8a2-40f6-9bf7-6d982d784649","Type":"ContainerDied","Data":"98f392277b32b668fcdecd14ddd79e0fa597eec2f76f8ad9055161cdfe1014a4"}
Sep 18 19:48:39.542087 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:39.541977    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz" event={"ID":"1dae9595-b8a2-40f6-9bf7-6d982d784649","Type":"ContainerDied","Data":"01cc5e6502aecdd308ea57b08921abb71f10ed6cb8843e2b064f478e76f0e353"}
Sep 18 19:48:39.542087 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:39.541933    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz"
Sep 18 19:48:39.542087 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:39.542004    2522 scope.go:117] "RemoveContainer" containerID="71c3629db7e6caca1532d0c6dc4ca0f22b6957a4f41096f248149165d5ec541b"
Sep 18 19:48:39.551048 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:39.551029    2522 scope.go:117] "RemoveContainer" containerID="98f392277b32b668fcdecd14ddd79e0fa597eec2f76f8ad9055161cdfe1014a4"
Sep 18 19:48:39.559710 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:39.559692    2522 scope.go:117] "RemoveContainer" containerID="9096aeb3f18484828608fa1cd4f02ede128b3375e0438e07822e1fceb6ce5735"
Sep 18 19:48:39.567505 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:39.567487    2522 scope.go:117] "RemoveContainer" containerID="71c3629db7e6caca1532d0c6dc4ca0f22b6957a4f41096f248149165d5ec541b"
Sep 18 19:48:39.567784 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:48:39.567761    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71c3629db7e6caca1532d0c6dc4ca0f22b6957a4f41096f248149165d5ec541b\": container with ID starting with 71c3629db7e6caca1532d0c6dc4ca0f22b6957a4f41096f248149165d5ec541b not found: ID does not exist" containerID="71c3629db7e6caca1532d0c6dc4ca0f22b6957a4f41096f248149165d5ec541b"
Sep 18 19:48:39.567842 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:39.567795    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71c3629db7e6caca1532d0c6dc4ca0f22b6957a4f41096f248149165d5ec541b"} err="failed to get container status \"71c3629db7e6caca1532d0c6dc4ca0f22b6957a4f41096f248149165d5ec541b\": rpc error: code = NotFound desc = could not find container \"71c3629db7e6caca1532d0c6dc4ca0f22b6957a4f41096f248149165d5ec541b\": container with ID starting with 71c3629db7e6caca1532d0c6dc4ca0f22b6957a4f41096f248149165d5ec541b not found: ID does not exist"
Sep 18 19:48:39.567842 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:39.567817    2522 scope.go:117] "RemoveContainer" containerID="98f392277b32b668fcdecd14ddd79e0fa597eec2f76f8ad9055161cdfe1014a4"
Sep 18 19:48:39.568114 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:48:39.568084    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98f392277b32b668fcdecd14ddd79e0fa597eec2f76f8ad9055161cdfe1014a4\": container with ID starting with 98f392277b32b668fcdecd14ddd79e0fa597eec2f76f8ad9055161cdfe1014a4 not found: ID does not exist" containerID="98f392277b32b668fcdecd14ddd79e0fa597eec2f76f8ad9055161cdfe1014a4"
Sep 18 19:48:39.568188 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:39.568112    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98f392277b32b668fcdecd14ddd79e0fa597eec2f76f8ad9055161cdfe1014a4"} err="failed to get container status \"98f392277b32b668fcdecd14ddd79e0fa597eec2f76f8ad9055161cdfe1014a4\": rpc error: code = NotFound desc = could not find container \"98f392277b32b668fcdecd14ddd79e0fa597eec2f76f8ad9055161cdfe1014a4\": container with ID starting with 98f392277b32b668fcdecd14ddd79e0fa597eec2f76f8ad9055161cdfe1014a4 not found: ID does not exist"
Sep 18 19:48:39.568188 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:39.568135    2522 scope.go:117] "RemoveContainer" containerID="9096aeb3f18484828608fa1cd4f02ede128b3375e0438e07822e1fceb6ce5735"
Sep 18 19:48:39.568367 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:48:39.568340    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9096aeb3f18484828608fa1cd4f02ede128b3375e0438e07822e1fceb6ce5735\": container with ID starting with 9096aeb3f18484828608fa1cd4f02ede128b3375e0438e07822e1fceb6ce5735 not found: ID does not exist" containerID="9096aeb3f18484828608fa1cd4f02ede128b3375e0438e07822e1fceb6ce5735"
Sep 18 19:48:39.568416 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:39.568363    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9096aeb3f18484828608fa1cd4f02ede128b3375e0438e07822e1fceb6ce5735"} err="failed to get container status \"9096aeb3f18484828608fa1cd4f02ede128b3375e0438e07822e1fceb6ce5735\": rpc error: code = NotFound desc = could not find container \"9096aeb3f18484828608fa1cd4f02ede128b3375e0438e07822e1fceb6ce5735\": container with ID starting with 9096aeb3f18484828608fa1cd4f02ede128b3375e0438e07822e1fceb6ce5735 not found: ID does not exist"
Sep 18 19:48:39.572967 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:39.572944    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz"]
Sep 18 19:48:39.583937 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:39.583904    2522 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6d75476b46-hd4hz"]
Sep 18 19:48:40.308496 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:40.308449    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t" podUID="dbe15859-6546-4c39-9afe-ec6a32ddb890" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.53:8643/healthz\": dial tcp 10.132.0.53:8643: connect: connection refused"
Sep 18 19:48:40.722387 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:40.722349    2522 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1dae9595-b8a2-40f6-9bf7-6d982d784649" path="/var/lib/kubelet/pods/1dae9595-b8a2-40f6-9bf7-6d982d784649/volumes"
Sep 18 19:48:45.307713 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:45.307674    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t" podUID="dbe15859-6546-4c39-9afe-ec6a32ddb890" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.53:8643/healthz\": dial tcp 10.132.0.53:8643: connect: connection refused"
Sep 18 19:48:50.308411 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:50.308359    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t" podUID="dbe15859-6546-4c39-9afe-ec6a32ddb890" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.53:8643/healthz\": dial tcp 10.132.0.53:8643: connect: connection refused"
Sep 18 19:48:54.615757 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:54.615724    2522 generic.go:356] "Generic (PLEG): container finished" podID="dbe15859-6546-4c39-9afe-ec6a32ddb890" containerID="569126494d613d28180b42f4fc64a71e62be05a3890a17080ace06bea809cffa" exitCode=137
Sep 18 19:48:54.616130 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:54.615803    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t" event={"ID":"dbe15859-6546-4c39-9afe-ec6a32ddb890","Type":"ContainerDied","Data":"569126494d613d28180b42f4fc64a71e62be05a3890a17080ace06bea809cffa"}
Sep 18 19:48:54.831500 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:54.831476    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t"
Sep 18 19:48:54.867246 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:54.867169    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbe15859-6546-4c39-9afe-ec6a32ddb890-proxy-tls\") pod \"dbe15859-6546-4c39-9afe-ec6a32ddb890\" (UID: \"dbe15859-6546-4c39-9afe-ec6a32ddb890\") "
Sep 18 19:48:54.867386 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:54.867249    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbe15859-6546-4c39-9afe-ec6a32ddb890-kube-rbac-proxy-sar-config\") pod \"dbe15859-6546-4c39-9afe-ec6a32ddb890\" (UID: \"dbe15859-6546-4c39-9afe-ec6a32ddb890\") "
Sep 18 19:48:54.867386 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:54.867312    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbe15859-6546-4c39-9afe-ec6a32ddb890-kserve-provision-location\") pod \"dbe15859-6546-4c39-9afe-ec6a32ddb890\" (UID: \"dbe15859-6546-4c39-9afe-ec6a32ddb890\") "
Sep 18 19:48:54.867386 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:54.867355    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6v8cz\" (UniqueName: \"kubernetes.io/projected/dbe15859-6546-4c39-9afe-ec6a32ddb890-kube-api-access-6v8cz\") pod \"dbe15859-6546-4c39-9afe-ec6a32ddb890\" (UID: \"dbe15859-6546-4c39-9afe-ec6a32ddb890\") "
Sep 18 19:48:54.867585 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:54.867411    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dbe15859-6546-4c39-9afe-ec6a32ddb890-cabundle-cert\") pod \"dbe15859-6546-4c39-9afe-ec6a32ddb890\" (UID: \"dbe15859-6546-4c39-9afe-ec6a32ddb890\") "
Sep 18 19:48:54.867648 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:54.867606    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbe15859-6546-4c39-9afe-ec6a32ddb890-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dbe15859-6546-4c39-9afe-ec6a32ddb890" (UID: "dbe15859-6546-4c39-9afe-ec6a32ddb890"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:48:54.867753 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:54.867730    2522 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbe15859-6546-4c39-9afe-ec6a32ddb890-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:48:54.867980 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:54.867942    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbe15859-6546-4c39-9afe-ec6a32ddb890-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dbe15859-6546-4c39-9afe-ec6a32ddb890" (UID: "dbe15859-6546-4c39-9afe-ec6a32ddb890"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:48:54.869497 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:54.869470    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbe15859-6546-4c39-9afe-ec6a32ddb890-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dbe15859-6546-4c39-9afe-ec6a32ddb890" (UID: "dbe15859-6546-4c39-9afe-ec6a32ddb890"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:48:54.869798 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:54.869778    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbe15859-6546-4c39-9afe-ec6a32ddb890-kube-api-access-6v8cz" (OuterVolumeSpecName: "kube-api-access-6v8cz") pod "dbe15859-6546-4c39-9afe-ec6a32ddb890" (UID: "dbe15859-6546-4c39-9afe-ec6a32ddb890"). InnerVolumeSpecName "kube-api-access-6v8cz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:48:54.876216 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:54.876195    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbe15859-6546-4c39-9afe-ec6a32ddb890-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dbe15859-6546-4c39-9afe-ec6a32ddb890" (UID: "dbe15859-6546-4c39-9afe-ec6a32ddb890"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:48:54.969130 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:54.969101    2522 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbe15859-6546-4c39-9afe-ec6a32ddb890-kserve-provision-location\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:48:54.969130 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:54.969127    2522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v8cz\" (UniqueName: \"kubernetes.io/projected/dbe15859-6546-4c39-9afe-ec6a32ddb890-kube-api-access-6v8cz\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:48:54.969130 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:54.969136    2522 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dbe15859-6546-4c39-9afe-ec6a32ddb890-cabundle-cert\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:48:54.969130 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:54.969147    2522 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbe15859-6546-4c39-9afe-ec6a32ddb890-proxy-tls\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:48:55.628036 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:55.628007    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t"
Sep 18 19:48:55.628438 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:55.628003    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t" event={"ID":"dbe15859-6546-4c39-9afe-ec6a32ddb890","Type":"ContainerDied","Data":"e2cd27dbd65ab4d35773ddb16f4380b5a53c0106269167c8407f576e15728c61"}
Sep 18 19:48:55.628438 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:55.628164    2522 scope.go:117] "RemoveContainer" containerID="29829510436f4553a0178bed4150cce06d2b7243bb4f3bab2e13d99ec04573e3"
Sep 18 19:48:55.637110 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:55.637089    2522 scope.go:117] "RemoveContainer" containerID="569126494d613d28180b42f4fc64a71e62be05a3890a17080ace06bea809cffa"
Sep 18 19:48:55.644499 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:55.644481    2522 scope.go:117] "RemoveContainer" containerID="0a3c2db1f5dcd62b65ad9634b6336932fa99034d09f8beeddd92753f8c471b2e"
Sep 18 19:48:55.652188 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:55.652164    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t"]
Sep 18 19:48:55.656623 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:55.656603    2522 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-588dc9cf7d-7pp2t"]
Sep 18 19:48:56.722581 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:48:56.722538    2522 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbe15859-6546-4c39-9afe-ec6a32ddb890" path="/var/lib/kubelet/pods/dbe15859-6546-4c39-9afe-ec6a32ddb890/volumes"
Sep 18 19:50:24.605282 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:24.605206    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-fxhm7"]
Sep 18 19:50:24.605877 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:24.605449    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-fxhm7" podUID="5cda7e89-3d8f-454a-8ad8-d6fc55c42822" containerName="kserve-container" containerID="cri-o://88c851e12e533b2fa5c57fe61607554eb7202ad8b50b7bf6f1f2f3123ef30d35" gracePeriod=30
Sep 18 19:50:24.605877 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:24.605493    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-fxhm7" podUID="5cda7e89-3d8f-454a-8ad8-d6fc55c42822" containerName="kube-rbac-proxy" containerID="cri-o://213afa0299a9ff7460146cf43a1c538de2323873d78fa5b09e461dbb0f5cb2e2" gracePeriod=30
Sep 18 19:50:25.047189 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:25.047094    2522 generic.go:356] "Generic (PLEG): container finished" podID="5cda7e89-3d8f-454a-8ad8-d6fc55c42822" containerID="213afa0299a9ff7460146cf43a1c538de2323873d78fa5b09e461dbb0f5cb2e2" exitCode=2
Sep 18 19:50:25.047189 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:25.047147    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-fxhm7" event={"ID":"5cda7e89-3d8f-454a-8ad8-d6fc55c42822","Type":"ContainerDied","Data":"213afa0299a9ff7460146cf43a1c538de2323873d78fa5b09e461dbb0f5cb2e2"}
Sep 18 19:50:26.037575 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:26.037524    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-fxhm7" podUID="5cda7e89-3d8f-454a-8ad8-d6fc55c42822" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.51:8643/healthz\": dial tcp 10.132.0.51:8643: connect: connection refused"
Sep 18 19:50:26.041959 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:26.041934    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-fxhm7" podUID="5cda7e89-3d8f-454a-8ad8-d6fc55c42822" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 18 19:50:29.678223 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.678193    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-fxhm7"
Sep 18 19:50:29.782173 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.782068    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cda7e89-3d8f-454a-8ad8-d6fc55c42822-kube-rbac-proxy-sar-config\") pod \"5cda7e89-3d8f-454a-8ad8-d6fc55c42822\" (UID: \"5cda7e89-3d8f-454a-8ad8-d6fc55c42822\") "
Sep 18 19:50:29.782173 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.782152    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cda7e89-3d8f-454a-8ad8-d6fc55c42822-proxy-tls\") pod \"5cda7e89-3d8f-454a-8ad8-d6fc55c42822\" (UID: \"5cda7e89-3d8f-454a-8ad8-d6fc55c42822\") "
Sep 18 19:50:29.782357 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.782183    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hzr2c\" (UniqueName: \"kubernetes.io/projected/5cda7e89-3d8f-454a-8ad8-d6fc55c42822-kube-api-access-hzr2c\") pod \"5cda7e89-3d8f-454a-8ad8-d6fc55c42822\" (UID: \"5cda7e89-3d8f-454a-8ad8-d6fc55c42822\") "
Sep 18 19:50:29.782357 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.782219    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5cda7e89-3d8f-454a-8ad8-d6fc55c42822-cabundle-cert\") pod \"5cda7e89-3d8f-454a-8ad8-d6fc55c42822\" (UID: \"5cda7e89-3d8f-454a-8ad8-d6fc55c42822\") "
Sep 18 19:50:29.782357 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.782272    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5cda7e89-3d8f-454a-8ad8-d6fc55c42822-kserve-provision-location\") pod \"5cda7e89-3d8f-454a-8ad8-d6fc55c42822\" (UID: \"5cda7e89-3d8f-454a-8ad8-d6fc55c42822\") "
Sep 18 19:50:29.782619 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.782543    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cda7e89-3d8f-454a-8ad8-d6fc55c42822-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5cda7e89-3d8f-454a-8ad8-d6fc55c42822" (UID: "5cda7e89-3d8f-454a-8ad8-d6fc55c42822"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:50:29.782619 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.782598    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cda7e89-3d8f-454a-8ad8-d6fc55c42822-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5cda7e89-3d8f-454a-8ad8-d6fc55c42822" (UID: "5cda7e89-3d8f-454a-8ad8-d6fc55c42822"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:50:29.782845 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.782705    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cda7e89-3d8f-454a-8ad8-d6fc55c42822-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5cda7e89-3d8f-454a-8ad8-d6fc55c42822" (UID: "5cda7e89-3d8f-454a-8ad8-d6fc55c42822"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:50:29.784328 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.784306    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cda7e89-3d8f-454a-8ad8-d6fc55c42822-kube-api-access-hzr2c" (OuterVolumeSpecName: "kube-api-access-hzr2c") pod "5cda7e89-3d8f-454a-8ad8-d6fc55c42822" (UID: "5cda7e89-3d8f-454a-8ad8-d6fc55c42822"). InnerVolumeSpecName "kube-api-access-hzr2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:50:29.784431 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.784325    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cda7e89-3d8f-454a-8ad8-d6fc55c42822-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5cda7e89-3d8f-454a-8ad8-d6fc55c42822" (UID: "5cda7e89-3d8f-454a-8ad8-d6fc55c42822"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:50:29.848622 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.848592    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn"]
Sep 18 19:50:29.848944 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.848931    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbe15859-6546-4c39-9afe-ec6a32ddb890" containerName="kserve-container"
Sep 18 19:50:29.848986 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.848947    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbe15859-6546-4c39-9afe-ec6a32ddb890" containerName="kserve-container"
Sep 18 19:50:29.848986 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.848959    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cda7e89-3d8f-454a-8ad8-d6fc55c42822" containerName="kserve-container"
Sep 18 19:50:29.848986 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.848965    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cda7e89-3d8f-454a-8ad8-d6fc55c42822" containerName="kserve-container"
Sep 18 19:50:29.848986 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.848973    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dae9595-b8a2-40f6-9bf7-6d982d784649" containerName="storage-initializer"
Sep 18 19:50:29.848986 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.848978    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dae9595-b8a2-40f6-9bf7-6d982d784649" containerName="storage-initializer"
Sep 18 19:50:29.848986 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.848985    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cda7e89-3d8f-454a-8ad8-d6fc55c42822" containerName="kube-rbac-proxy"
Sep 18 19:50:29.849202 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.848990    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cda7e89-3d8f-454a-8ad8-d6fc55c42822" containerName="kube-rbac-proxy"
Sep 18 19:50:29.849202 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.848999    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cda7e89-3d8f-454a-8ad8-d6fc55c42822" containerName="storage-initializer"
Sep 18 19:50:29.849202 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.849005    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cda7e89-3d8f-454a-8ad8-d6fc55c42822" containerName="storage-initializer"
Sep 18 19:50:29.849202 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.849013    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbe15859-6546-4c39-9afe-ec6a32ddb890" containerName="storage-initializer"
Sep 18 19:50:29.849202 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.849018    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbe15859-6546-4c39-9afe-ec6a32ddb890" containerName="storage-initializer"
Sep 18 19:50:29.849202 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.849028    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dae9595-b8a2-40f6-9bf7-6d982d784649" containerName="kserve-container"
Sep 18 19:50:29.849202 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.849034    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dae9595-b8a2-40f6-9bf7-6d982d784649" containerName="kserve-container"
Sep 18 19:50:29.849202 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.849040    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dae9595-b8a2-40f6-9bf7-6d982d784649" containerName="kube-rbac-proxy"
Sep 18 19:50:29.849202 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.849045    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dae9595-b8a2-40f6-9bf7-6d982d784649" containerName="kube-rbac-proxy"
Sep 18 19:50:29.849202 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.849051    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbe15859-6546-4c39-9afe-ec6a32ddb890" containerName="kube-rbac-proxy"
Sep 18 19:50:29.849202 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.849055    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbe15859-6546-4c39-9afe-ec6a32ddb890" containerName="kube-rbac-proxy"
Sep 18 19:50:29.849202 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.849105    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dae9595-b8a2-40f6-9bf7-6d982d784649" containerName="kube-rbac-proxy"
Sep 18 19:50:29.849202 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.849112    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cda7e89-3d8f-454a-8ad8-d6fc55c42822" containerName="kube-rbac-proxy"
Sep 18 19:50:29.849202 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.849117    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbe15859-6546-4c39-9afe-ec6a32ddb890" containerName="kserve-container"
Sep 18 19:50:29.849202 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.849124    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cda7e89-3d8f-454a-8ad8-d6fc55c42822" containerName="kserve-container"
Sep 18 19:50:29.849202 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.849129    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbe15859-6546-4c39-9afe-ec6a32ddb890" containerName="kube-rbac-proxy"
Sep 18 19:50:29.849202 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.849137    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dae9595-b8a2-40f6-9bf7-6d982d784649" containerName="kserve-container"
Sep 18 19:50:29.852443 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.852425    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn"
Sep 18 19:50:29.855057 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.855037    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Sep 18 19:50:29.855792 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.855777    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 18 19:50:29.879300 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.879275    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8"]
Sep 18 19:50:29.882734 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.882714    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8"
Sep 18 19:50:29.882954 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.882932    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1597b7b3-9d5e-47be-bf09-82f72fcad09e-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn\" (UID: \"1597b7b3-9d5e-47be-bf09-82f72fcad09e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn"
Sep 18 19:50:29.883000 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.882976    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1597b7b3-9d5e-47be-bf09-82f72fcad09e-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn\" (UID: \"1597b7b3-9d5e-47be-bf09-82f72fcad09e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn"
Sep 18 19:50:29.883053 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.883001    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1597b7b3-9d5e-47be-bf09-82f72fcad09e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn\" (UID: \"1597b7b3-9d5e-47be-bf09-82f72fcad09e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn"
Sep 18 19:50:29.883053 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.883046    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1597b7b3-9d5e-47be-bf09-82f72fcad09e-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn\" (UID: \"1597b7b3-9d5e-47be-bf09-82f72fcad09e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn"
Sep 18 19:50:29.883153 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.883063    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlggt\" (UniqueName: \"kubernetes.io/projected/1597b7b3-9d5e-47be-bf09-82f72fcad09e-kube-api-access-qlggt\") pod \"isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn\" (UID: \"1597b7b3-9d5e-47be-bf09-82f72fcad09e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn"
Sep 18 19:50:29.883194 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.883174    2522 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cda7e89-3d8f-454a-8ad8-d6fc55c42822-proxy-tls\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:50:29.883229 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.883199    2522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzr2c\" (UniqueName: \"kubernetes.io/projected/5cda7e89-3d8f-454a-8ad8-d6fc55c42822-kube-api-access-hzr2c\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:50:29.883229 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.883214    2522 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5cda7e89-3d8f-454a-8ad8-d6fc55c42822-cabundle-cert\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:50:29.883229 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.883227    2522 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5cda7e89-3d8f-454a-8ad8-d6fc55c42822-kserve-provision-location\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:50:29.883322 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.883236    2522 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cda7e89-3d8f-454a-8ad8-d6fc55c42822-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:50:29.889959 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.889938    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 18 19:50:29.906913 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.906885    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn"]
Sep 18 19:50:29.919242 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.919216    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8"]
Sep 18 19:50:29.983753 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.983724    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1597b7b3-9d5e-47be-bf09-82f72fcad09e-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn\" (UID: \"1597b7b3-9d5e-47be-bf09-82f72fcad09e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn"
Sep 18 19:50:29.983753 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.983757    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b82a9095-65cd-4b23-8e76-3d8b894b915d-cabundle-cert\") pod \"isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8\" (UID: \"b82a9095-65cd-4b23-8e76-3d8b894b915d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8"
Sep 18 19:50:29.984023 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.983785    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b82a9095-65cd-4b23-8e76-3d8b894b915d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8\" (UID: \"b82a9095-65cd-4b23-8e76-3d8b894b915d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8"
Sep 18 19:50:29.984023 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.983831    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1597b7b3-9d5e-47be-bf09-82f72fcad09e-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn\" (UID: \"1597b7b3-9d5e-47be-bf09-82f72fcad09e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn"
Sep 18 19:50:29.984023 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.983850    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1597b7b3-9d5e-47be-bf09-82f72fcad09e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn\" (UID: \"1597b7b3-9d5e-47be-bf09-82f72fcad09e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn"
Sep 18 19:50:29.984023 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.983888    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1597b7b3-9d5e-47be-bf09-82f72fcad09e-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn\" (UID: \"1597b7b3-9d5e-47be-bf09-82f72fcad09e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn"
Sep 18 19:50:29.984023 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.983914    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlggt\" (UniqueName: \"kubernetes.io/projected/1597b7b3-9d5e-47be-bf09-82f72fcad09e-kube-api-access-qlggt\") pod \"isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn\" (UID: \"1597b7b3-9d5e-47be-bf09-82f72fcad09e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn"
Sep 18 19:50:29.984023 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.983963    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b82a9095-65cd-4b23-8e76-3d8b894b915d-proxy-tls\") pod \"isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8\" (UID: \"b82a9095-65cd-4b23-8e76-3d8b894b915d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8"
Sep 18 19:50:29.984023 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:50:29.983968    2522 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-serving-cert: secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Sep 18 19:50:29.984023 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.984005    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b82a9095-65cd-4b23-8e76-3d8b894b915d-kserve-provision-location\") pod \"isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8\" (UID: \"b82a9095-65cd-4b23-8e76-3d8b894b915d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8"
Sep 18 19:50:29.984428 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:50:29.984046    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1597b7b3-9d5e-47be-bf09-82f72fcad09e-proxy-tls podName:1597b7b3-9d5e-47be-bf09-82f72fcad09e nodeName:}" failed. No retries permitted until 2026-09-18 19:50:30.484026632 +0000 UTC m=+2190.367553077 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1597b7b3-9d5e-47be-bf09-82f72fcad09e-proxy-tls") pod "isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn" (UID: "1597b7b3-9d5e-47be-bf09-82f72fcad09e") : secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Sep 18 19:50:29.984428 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.984128    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7nfr\" (UniqueName: \"kubernetes.io/projected/b82a9095-65cd-4b23-8e76-3d8b894b915d-kube-api-access-t7nfr\") pod \"isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8\" (UID: \"b82a9095-65cd-4b23-8e76-3d8b894b915d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8"
Sep 18 19:50:29.984428 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.984191    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1597b7b3-9d5e-47be-bf09-82f72fcad09e-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn\" (UID: \"1597b7b3-9d5e-47be-bf09-82f72fcad09e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn"
Sep 18 19:50:29.984547 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.984495    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1597b7b3-9d5e-47be-bf09-82f72fcad09e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn\" (UID: \"1597b7b3-9d5e-47be-bf09-82f72fcad09e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn"
Sep 18 19:50:29.984547 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:29.984506    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1597b7b3-9d5e-47be-bf09-82f72fcad09e-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn\" (UID: \"1597b7b3-9d5e-47be-bf09-82f72fcad09e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn"
Sep 18 19:50:30.017467 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:30.017441    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlggt\" (UniqueName: \"kubernetes.io/projected/1597b7b3-9d5e-47be-bf09-82f72fcad09e-kube-api-access-qlggt\") pod \"isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn\" (UID: \"1597b7b3-9d5e-47be-bf09-82f72fcad09e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn"
Sep 18 19:50:30.070712 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:30.070660    2522 generic.go:356] "Generic (PLEG): container finished" podID="5cda7e89-3d8f-454a-8ad8-d6fc55c42822" containerID="88c851e12e533b2fa5c57fe61607554eb7202ad8b50b7bf6f1f2f3123ef30d35" exitCode=0
Sep 18 19:50:30.070849 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:30.070744    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-fxhm7"
Sep 18 19:50:30.070849 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:30.070760    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-fxhm7" event={"ID":"5cda7e89-3d8f-454a-8ad8-d6fc55c42822","Type":"ContainerDied","Data":"88c851e12e533b2fa5c57fe61607554eb7202ad8b50b7bf6f1f2f3123ef30d35"}
Sep 18 19:50:30.070849 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:30.070823    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-fxhm7" event={"ID":"5cda7e89-3d8f-454a-8ad8-d6fc55c42822","Type":"ContainerDied","Data":"cca4a0aed3ca5cd45edaf207dd033207277a04ef619b2eb2a7b5d5c3e27c8e19"}
Sep 18 19:50:30.070979 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:30.070852    2522 scope.go:117] "RemoveContainer" containerID="213afa0299a9ff7460146cf43a1c538de2323873d78fa5b09e461dbb0f5cb2e2"
Sep 18 19:50:30.085992 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:30.085963    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b82a9095-65cd-4b23-8e76-3d8b894b915d-cabundle-cert\") pod \"isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8\" (UID: \"b82a9095-65cd-4b23-8e76-3d8b894b915d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8"
Sep 18 19:50:30.086320 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:30.086302    2522 scope.go:117] "RemoveContainer" containerID="88c851e12e533b2fa5c57fe61607554eb7202ad8b50b7bf6f1f2f3123ef30d35"
Sep 18 19:50:30.087243 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:30.087210    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b82a9095-65cd-4b23-8e76-3d8b894b915d-cabundle-cert\") pod \"isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8\" (UID: \"b82a9095-65cd-4b23-8e76-3d8b894b915d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8"
Sep 18 19:50:30.088151 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:30.088068    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b82a9095-65cd-4b23-8e76-3d8b894b915d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8\" (UID: \"b82a9095-65cd-4b23-8e76-3d8b894b915d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8"
Sep 18 19:50:30.088677 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:30.088646    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b82a9095-65cd-4b23-8e76-3d8b894b915d-proxy-tls\") pod \"isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8\" (UID: \"b82a9095-65cd-4b23-8e76-3d8b894b915d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8"
Sep 18 19:50:30.089190 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:30.089165    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b82a9095-65cd-4b23-8e76-3d8b894b915d-kserve-provision-location\") pod \"isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8\" (UID: \"b82a9095-65cd-4b23-8e76-3d8b894b915d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8"
Sep 18 19:50:30.089301 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:30.089220    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7nfr\" (UniqueName: \"kubernetes.io/projected/b82a9095-65cd-4b23-8e76-3d8b894b915d-kube-api-access-t7nfr\") pod \"isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8\" (UID: \"b82a9095-65cd-4b23-8e76-3d8b894b915d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8"
Sep 18 19:50:30.089603 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:30.089552    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b82a9095-65cd-4b23-8e76-3d8b894b915d-kserve-provision-location\") pod \"isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8\" (UID: \"b82a9095-65cd-4b23-8e76-3d8b894b915d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8"
Sep 18 19:50:30.089744 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:30.089662    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b82a9095-65cd-4b23-8e76-3d8b894b915d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8\" (UID: \"b82a9095-65cd-4b23-8e76-3d8b894b915d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8"
Sep 18 19:50:30.091509 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:30.091488    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b82a9095-65cd-4b23-8e76-3d8b894b915d-proxy-tls\") pod \"isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8\" (UID: \"b82a9095-65cd-4b23-8e76-3d8b894b915d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8"
Sep 18 19:50:30.096957 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:30.096939    2522 scope.go:117] "RemoveContainer" containerID="89ff932c3a060aa17c89efd0b7895b47d54d4d652cca70f85a74f62582525c67"
Sep 18 19:50:30.099351 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:30.099323    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7nfr\" (UniqueName: \"kubernetes.io/projected/b82a9095-65cd-4b23-8e76-3d8b894b915d-kube-api-access-t7nfr\") pod \"isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8\" (UID: \"b82a9095-65cd-4b23-8e76-3d8b894b915d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8"
Sep 18 19:50:30.104670 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:30.104649    2522 scope.go:117] "RemoveContainer" containerID="213afa0299a9ff7460146cf43a1c538de2323873d78fa5b09e461dbb0f5cb2e2"
Sep 18 19:50:30.104947 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:50:30.104927    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"213afa0299a9ff7460146cf43a1c538de2323873d78fa5b09e461dbb0f5cb2e2\": container with ID starting with 213afa0299a9ff7460146cf43a1c538de2323873d78fa5b09e461dbb0f5cb2e2 not found: ID does not exist" containerID="213afa0299a9ff7460146cf43a1c538de2323873d78fa5b09e461dbb0f5cb2e2"
Sep 18 19:50:30.105006 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:30.104954    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"213afa0299a9ff7460146cf43a1c538de2323873d78fa5b09e461dbb0f5cb2e2"} err="failed to get container status \"213afa0299a9ff7460146cf43a1c538de2323873d78fa5b09e461dbb0f5cb2e2\": rpc error: code = NotFound desc = could not find container \"213afa0299a9ff7460146cf43a1c538de2323873d78fa5b09e461dbb0f5cb2e2\": container with ID starting with 213afa0299a9ff7460146cf43a1c538de2323873d78fa5b09e461dbb0f5cb2e2 not found: ID does not exist"
Sep 18 19:50:30.105006 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:30.104971    2522 scope.go:117] "RemoveContainer" containerID="88c851e12e533b2fa5c57fe61607554eb7202ad8b50b7bf6f1f2f3123ef30d35"
Sep 18 19:50:30.105232 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:50:30.105212    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88c851e12e533b2fa5c57fe61607554eb7202ad8b50b7bf6f1f2f3123ef30d35\": container with ID starting with 88c851e12e533b2fa5c57fe61607554eb7202ad8b50b7bf6f1f2f3123ef30d35 not found: ID does not exist" containerID="88c851e12e533b2fa5c57fe61607554eb7202ad8b50b7bf6f1f2f3123ef30d35"
Sep 18 19:50:30.105273 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:30.105239    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88c851e12e533b2fa5c57fe61607554eb7202ad8b50b7bf6f1f2f3123ef30d35"} err="failed to get container status \"88c851e12e533b2fa5c57fe61607554eb7202ad8b50b7bf6f1f2f3123ef30d35\": rpc error: code = NotFound desc = could not find container \"88c851e12e533b2fa5c57fe61607554eb7202ad8b50b7bf6f1f2f3123ef30d35\": container with ID starting with 88c851e12e533b2fa5c57fe61607554eb7202ad8b50b7bf6f1f2f3123ef30d35 not found: ID does not exist"
Sep 18 19:50:30.105273 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:30.105255    2522 scope.go:117] "RemoveContainer" containerID="89ff932c3a060aa17c89efd0b7895b47d54d4d652cca70f85a74f62582525c67"
Sep 18 19:50:30.105473 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:50:30.105457    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89ff932c3a060aa17c89efd0b7895b47d54d4d652cca70f85a74f62582525c67\": container with ID starting with 89ff932c3a060aa17c89efd0b7895b47d54d4d652cca70f85a74f62582525c67 not found: ID does not exist" containerID="89ff932c3a060aa17c89efd0b7895b47d54d4d652cca70f85a74f62582525c67"
Sep 18 19:50:30.105535 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:30.105477    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89ff932c3a060aa17c89efd0b7895b47d54d4d652cca70f85a74f62582525c67"} err="failed to get container status \"89ff932c3a060aa17c89efd0b7895b47d54d4d652cca70f85a74f62582525c67\": rpc error: code = NotFound desc = could not find container \"89ff932c3a060aa17c89efd0b7895b47d54d4d652cca70f85a74f62582525c67\": container with ID starting with 89ff932c3a060aa17c89efd0b7895b47d54d4d652cca70f85a74f62582525c67 not found: ID does not exist"
Sep 18 19:50:30.120526 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:30.120501    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-fxhm7"]
Sep 18 19:50:30.136419 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:30.136397    2522 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-fxhm7"]
Sep 18 19:50:30.192205 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:30.192170    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8"
Sep 18 19:50:30.480200 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:30.480120    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8"]
Sep 18 19:50:30.483991 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:50:30.483964    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb82a9095_65cd_4b23_8e76_3d8b894b915d.slice/crio-33992750a359f5a9ed60d7b300ab72a6697c64a269b4e536521c50287eeb2ad2 WatchSource:0}: Error finding container 33992750a359f5a9ed60d7b300ab72a6697c64a269b4e536521c50287eeb2ad2: Status 404 returned error can't find the container with id 33992750a359f5a9ed60d7b300ab72a6697c64a269b4e536521c50287eeb2ad2
Sep 18 19:50:30.493482 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:30.493460    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1597b7b3-9d5e-47be-bf09-82f72fcad09e-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn\" (UID: \"1597b7b3-9d5e-47be-bf09-82f72fcad09e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn"
Sep 18 19:50:30.495897 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:30.495874    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1597b7b3-9d5e-47be-bf09-82f72fcad09e-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn\" (UID: \"1597b7b3-9d5e-47be-bf09-82f72fcad09e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn"
Sep 18 19:50:30.723993 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:30.723951    2522 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cda7e89-3d8f-454a-8ad8-d6fc55c42822" path="/var/lib/kubelet/pods/5cda7e89-3d8f-454a-8ad8-d6fc55c42822/volumes"
Sep 18 19:50:30.763526 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:30.763447    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn"
Sep 18 19:50:31.051493 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:31.051411    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn"]
Sep 18 19:50:31.055784 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:50:31.055754    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1597b7b3_9d5e_47be_bf09_82f72fcad09e.slice/crio-f0e3603bb694ba381abb06cbd21698286c0a9966b5f9d30135cc8dfeee04bb98 WatchSource:0}: Error finding container f0e3603bb694ba381abb06cbd21698286c0a9966b5f9d30135cc8dfeee04bb98: Status 404 returned error can't find the container with id f0e3603bb694ba381abb06cbd21698286c0a9966b5f9d30135cc8dfeee04bb98
Sep 18 19:50:31.079772 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:31.079742    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8" event={"ID":"b82a9095-65cd-4b23-8e76-3d8b894b915d","Type":"ContainerStarted","Data":"e0a9573f001e97e5ff3df860a8abcb76ebd975242a252713b8c3c430a821226b"}
Sep 18 19:50:31.079888 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:31.079811    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8" event={"ID":"b82a9095-65cd-4b23-8e76-3d8b894b915d","Type":"ContainerStarted","Data":"33992750a359f5a9ed60d7b300ab72a6697c64a269b4e536521c50287eeb2ad2"}
Sep 18 19:50:31.080894 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:31.080875    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn" event={"ID":"1597b7b3-9d5e-47be-bf09-82f72fcad09e","Type":"ContainerStarted","Data":"f0e3603bb694ba381abb06cbd21698286c0a9966b5f9d30135cc8dfeee04bb98"}
Sep 18 19:50:32.086742 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:32.086705    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn" event={"ID":"1597b7b3-9d5e-47be-bf09-82f72fcad09e","Type":"ContainerStarted","Data":"1f5c729f5643d5579c3f18209fff44c7cf1e95856cc7e7a3256a7747dc165e01"}
Sep 18 19:50:35.102257 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:35.102155    2522 generic.go:356] "Generic (PLEG): container finished" podID="b82a9095-65cd-4b23-8e76-3d8b894b915d" containerID="e0a9573f001e97e5ff3df860a8abcb76ebd975242a252713b8c3c430a821226b" exitCode=0
Sep 18 19:50:35.102257 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:35.102235    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8" event={"ID":"b82a9095-65cd-4b23-8e76-3d8b894b915d","Type":"ContainerDied","Data":"e0a9573f001e97e5ff3df860a8abcb76ebd975242a252713b8c3c430a821226b"}
Sep 18 19:50:35.103973 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:35.103948    2522 generic.go:356] "Generic (PLEG): container finished" podID="1597b7b3-9d5e-47be-bf09-82f72fcad09e" containerID="1f5c729f5643d5579c3f18209fff44c7cf1e95856cc7e7a3256a7747dc165e01" exitCode=0
Sep 18 19:50:35.104075 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:35.104014    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn" event={"ID":"1597b7b3-9d5e-47be-bf09-82f72fcad09e","Type":"ContainerDied","Data":"1f5c729f5643d5579c3f18209fff44c7cf1e95856cc7e7a3256a7747dc165e01"}
Sep 18 19:50:36.110506 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:36.110464    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8" event={"ID":"b82a9095-65cd-4b23-8e76-3d8b894b915d","Type":"ContainerStarted","Data":"6194fe3f2218dcc5f6bddb49cd442ecc1d9c7e9ddcc24701532e31e64e6ab117"}
Sep 18 19:50:36.111018 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:36.110542    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8" event={"ID":"b82a9095-65cd-4b23-8e76-3d8b894b915d","Type":"ContainerStarted","Data":"9dcc54b4f373ed643c4fb269547501a617d2b4b998938c5073f9db15cfe57ee1"}
Sep 18 19:50:36.111018 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:36.110783    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8"
Sep 18 19:50:36.112739 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:36.112718    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn" event={"ID":"1597b7b3-9d5e-47be-bf09-82f72fcad09e","Type":"ContainerStarted","Data":"76c53f3c16a7d6415108695ae27ab639e02d4239d0e59066817acfd0f6ef70b2"}
Sep 18 19:50:36.112846 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:36.112760    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn" event={"ID":"1597b7b3-9d5e-47be-bf09-82f72fcad09e","Type":"ContainerStarted","Data":"8dbda5a82633890a359a1bc7d0fd1f6aa999e3c95398fe8b710e92fada3e93ae"}
Sep 18 19:50:36.112940 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:36.112924    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn"
Sep 18 19:50:36.112985 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:36.112970    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn"
Sep 18 19:50:36.114416 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:36.114393    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn" podUID="1597b7b3-9d5e-47be-bf09-82f72fcad09e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 18 19:50:36.137829 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:36.137781    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8" podStartSLOduration=3.312157229 podStartE2EDuration="7.137767906s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.825610677s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:50:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:50:36.134951031 +0000 UTC m=+2196.018477517" watchObservedRunningTime="2026-09-18 19:50:36.137767906 +0000 UTC m=+2196.021294367"
Sep 18 19:50:36.168041 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:36.167986    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn" podStartSLOduration=3.440013358 podStartE2EDuration="7.167973878s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.72796052s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:50:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:50:36.167245705 +0000 UTC m=+2196.050772167" watchObservedRunningTime="2026-09-18 19:50:36.167973878 +0000 UTC m=+2196.051500405"
Sep 18 19:50:37.117549 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:37.117504    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn" podUID="1597b7b3-9d5e-47be-bf09-82f72fcad09e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 18 19:50:37.118009 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:37.117726    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8"
Sep 18 19:50:37.119167 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:37.119137    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8" podUID="b82a9095-65cd-4b23-8e76-3d8b894b915d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 18 19:50:38.123029 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:38.122990    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8" podUID="b82a9095-65cd-4b23-8e76-3d8b894b915d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 18 19:50:42.122457 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:42.122425    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn"
Sep 18 19:50:42.123099 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:42.123068    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn" podUID="1597b7b3-9d5e-47be-bf09-82f72fcad09e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 18 19:50:43.128767 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:43.128734    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8"
Sep 18 19:50:43.129519 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:43.129490    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8" podUID="b82a9095-65cd-4b23-8e76-3d8b894b915d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 18 19:50:52.123251 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:52.123210    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn" podUID="1597b7b3-9d5e-47be-bf09-82f72fcad09e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 18 19:50:53.130243 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:50:53.130191    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8" podUID="b82a9095-65cd-4b23-8e76-3d8b894b915d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 18 19:51:02.123850 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:51:02.123811    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn" podUID="1597b7b3-9d5e-47be-bf09-82f72fcad09e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 18 19:51:03.129732 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:51:03.129693    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8" podUID="b82a9095-65cd-4b23-8e76-3d8b894b915d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 18 19:51:12.123372 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:51:12.123330    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn" podUID="1597b7b3-9d5e-47be-bf09-82f72fcad09e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 18 19:51:13.130090 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:51:13.130047    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8" podUID="b82a9095-65cd-4b23-8e76-3d8b894b915d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 18 19:51:22.123381 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:51:22.123341    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn" podUID="1597b7b3-9d5e-47be-bf09-82f72fcad09e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 18 19:51:23.130178 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:51:23.130129    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8" podUID="b82a9095-65cd-4b23-8e76-3d8b894b915d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 18 19:51:32.123348 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:51:32.123303    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn" podUID="1597b7b3-9d5e-47be-bf09-82f72fcad09e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 18 19:51:33.130104 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:51:33.130049    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8" podUID="b82a9095-65cd-4b23-8e76-3d8b894b915d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 18 19:51:42.123485 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:51:42.123442    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn" podUID="1597b7b3-9d5e-47be-bf09-82f72fcad09e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 18 19:51:43.130054 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:51:43.130011    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8" podUID="b82a9095-65cd-4b23-8e76-3d8b894b915d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 18 19:51:44.716290 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:51:44.716253    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn" podUID="1597b7b3-9d5e-47be-bf09-82f72fcad09e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 18 19:51:45.716721 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:51:45.716681    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8" podUID="b82a9095-65cd-4b23-8e76-3d8b894b915d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 18 19:51:54.721889 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:51:54.721857    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn"
Sep 18 19:51:54.856687 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:51:54.856649    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8"]
Sep 18 19:51:54.856976 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:51:54.856941    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8" podUID="b82a9095-65cd-4b23-8e76-3d8b894b915d" containerName="kserve-container" containerID="cri-o://9dcc54b4f373ed643c4fb269547501a617d2b4b998938c5073f9db15cfe57ee1" gracePeriod=30
Sep 18 19:51:54.857139 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:51:54.856943    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8" podUID="b82a9095-65cd-4b23-8e76-3d8b894b915d" containerName="kube-rbac-proxy" containerID="cri-o://6194fe3f2218dcc5f6bddb49cd442ecc1d9c7e9ddcc24701532e31e64e6ab117" gracePeriod=30
Sep 18 19:51:54.857913 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:51:54.857696    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8"
Sep 18 19:51:55.496103 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:51:55.496069    2522 generic.go:356] "Generic (PLEG): container finished" podID="b82a9095-65cd-4b23-8e76-3d8b894b915d" containerID="6194fe3f2218dcc5f6bddb49cd442ecc1d9c7e9ddcc24701532e31e64e6ab117" exitCode=2
Sep 18 19:51:55.496301 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:51:55.496133    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8" event={"ID":"b82a9095-65cd-4b23-8e76-3d8b894b915d","Type":"ContainerDied","Data":"6194fe3f2218dcc5f6bddb49cd442ecc1d9c7e9ddcc24701532e31e64e6ab117"}
Sep 18 19:51:58.123926 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:51:58.123871    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8" podUID="b82a9095-65cd-4b23-8e76-3d8b894b915d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.55:8643/healthz\": dial tcp 10.132.0.55:8643: connect: connection refused"
Sep 18 19:51:59.901885 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:51:59.901856    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8"
Sep 18 19:51:59.980085 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:51:59.980051    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b82a9095-65cd-4b23-8e76-3d8b894b915d-cabundle-cert\") pod \"b82a9095-65cd-4b23-8e76-3d8b894b915d\" (UID: \"b82a9095-65cd-4b23-8e76-3d8b894b915d\") "
Sep 18 19:51:59.980268 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:51:59.980098    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b82a9095-65cd-4b23-8e76-3d8b894b915d-kserve-provision-location\") pod \"b82a9095-65cd-4b23-8e76-3d8b894b915d\" (UID: \"b82a9095-65cd-4b23-8e76-3d8b894b915d\") "
Sep 18 19:51:59.980268 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:51:59.980117    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t7nfr\" (UniqueName: \"kubernetes.io/projected/b82a9095-65cd-4b23-8e76-3d8b894b915d-kube-api-access-t7nfr\") pod \"b82a9095-65cd-4b23-8e76-3d8b894b915d\" (UID: \"b82a9095-65cd-4b23-8e76-3d8b894b915d\") "
Sep 18 19:51:59.980268 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:51:59.980172    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b82a9095-65cd-4b23-8e76-3d8b894b915d-kube-rbac-proxy-sar-config\") pod \"b82a9095-65cd-4b23-8e76-3d8b894b915d\" (UID: \"b82a9095-65cd-4b23-8e76-3d8b894b915d\") "
Sep 18 19:51:59.980268 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:51:59.980234    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b82a9095-65cd-4b23-8e76-3d8b894b915d-proxy-tls\") pod \"b82a9095-65cd-4b23-8e76-3d8b894b915d\" (UID: \"b82a9095-65cd-4b23-8e76-3d8b894b915d\") "
Sep 18 19:51:59.980548 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:51:59.980520    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b82a9095-65cd-4b23-8e76-3d8b894b915d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b82a9095-65cd-4b23-8e76-3d8b894b915d" (UID: "b82a9095-65cd-4b23-8e76-3d8b894b915d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:51:59.980648 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:51:59.980588    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b82a9095-65cd-4b23-8e76-3d8b894b915d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b82a9095-65cd-4b23-8e76-3d8b894b915d" (UID: "b82a9095-65cd-4b23-8e76-3d8b894b915d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:51:59.980701 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:51:59.980651    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b82a9095-65cd-4b23-8e76-3d8b894b915d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b82a9095-65cd-4b23-8e76-3d8b894b915d" (UID: "b82a9095-65cd-4b23-8e76-3d8b894b915d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:51:59.982378 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:51:59.982352    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b82a9095-65cd-4b23-8e76-3d8b894b915d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b82a9095-65cd-4b23-8e76-3d8b894b915d" (UID: "b82a9095-65cd-4b23-8e76-3d8b894b915d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:51:59.982378 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:51:59.982365    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b82a9095-65cd-4b23-8e76-3d8b894b915d-kube-api-access-t7nfr" (OuterVolumeSpecName: "kube-api-access-t7nfr") pod "b82a9095-65cd-4b23-8e76-3d8b894b915d" (UID: "b82a9095-65cd-4b23-8e76-3d8b894b915d"). InnerVolumeSpecName "kube-api-access-t7nfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:52:00.081736 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:00.081694    2522 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b82a9095-65cd-4b23-8e76-3d8b894b915d-cabundle-cert\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:52:00.081736 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:00.081730    2522 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b82a9095-65cd-4b23-8e76-3d8b894b915d-kserve-provision-location\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:52:00.081736 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:00.081745    2522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7nfr\" (UniqueName: \"kubernetes.io/projected/b82a9095-65cd-4b23-8e76-3d8b894b915d-kube-api-access-t7nfr\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:52:00.081989 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:00.081758    2522 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b82a9095-65cd-4b23-8e76-3d8b894b915d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:52:00.081989 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:00.081771    2522 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b82a9095-65cd-4b23-8e76-3d8b894b915d-proxy-tls\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:52:00.522786 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:00.522748    2522 generic.go:356] "Generic (PLEG): container finished" podID="b82a9095-65cd-4b23-8e76-3d8b894b915d" containerID="9dcc54b4f373ed643c4fb269547501a617d2b4b998938c5073f9db15cfe57ee1" exitCode=0
Sep 18 19:52:00.522999 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:00.522826    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8"
Sep 18 19:52:00.522999 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:00.522832    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8" event={"ID":"b82a9095-65cd-4b23-8e76-3d8b894b915d","Type":"ContainerDied","Data":"9dcc54b4f373ed643c4fb269547501a617d2b4b998938c5073f9db15cfe57ee1"}
Sep 18 19:52:00.522999 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:00.522946    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8" event={"ID":"b82a9095-65cd-4b23-8e76-3d8b894b915d","Type":"ContainerDied","Data":"33992750a359f5a9ed60d7b300ab72a6697c64a269b4e536521c50287eeb2ad2"}
Sep 18 19:52:00.522999 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:00.522982    2522 scope.go:117] "RemoveContainer" containerID="6194fe3f2218dcc5f6bddb49cd442ecc1d9c7e9ddcc24701532e31e64e6ab117"
Sep 18 19:52:00.532286 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:00.532271    2522 scope.go:117] "RemoveContainer" containerID="9dcc54b4f373ed643c4fb269547501a617d2b4b998938c5073f9db15cfe57ee1"
Sep 18 19:52:00.540203 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:00.540187    2522 scope.go:117] "RemoveContainer" containerID="e0a9573f001e97e5ff3df860a8abcb76ebd975242a252713b8c3c430a821226b"
Sep 18 19:52:00.547942 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:00.547919    2522 scope.go:117] "RemoveContainer" containerID="6194fe3f2218dcc5f6bddb49cd442ecc1d9c7e9ddcc24701532e31e64e6ab117"
Sep 18 19:52:00.548222 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:52:00.548203    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6194fe3f2218dcc5f6bddb49cd442ecc1d9c7e9ddcc24701532e31e64e6ab117\": container with ID starting with 6194fe3f2218dcc5f6bddb49cd442ecc1d9c7e9ddcc24701532e31e64e6ab117 not found: ID does not exist" containerID="6194fe3f2218dcc5f6bddb49cd442ecc1d9c7e9ddcc24701532e31e64e6ab117"
Sep 18 19:52:00.548285 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:00.548228    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6194fe3f2218dcc5f6bddb49cd442ecc1d9c7e9ddcc24701532e31e64e6ab117"} err="failed to get container status \"6194fe3f2218dcc5f6bddb49cd442ecc1d9c7e9ddcc24701532e31e64e6ab117\": rpc error: code = NotFound desc = could not find container \"6194fe3f2218dcc5f6bddb49cd442ecc1d9c7e9ddcc24701532e31e64e6ab117\": container with ID starting with 6194fe3f2218dcc5f6bddb49cd442ecc1d9c7e9ddcc24701532e31e64e6ab117 not found: ID does not exist"
Sep 18 19:52:00.548285 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:00.548246    2522 scope.go:117] "RemoveContainer" containerID="9dcc54b4f373ed643c4fb269547501a617d2b4b998938c5073f9db15cfe57ee1"
Sep 18 19:52:00.548468 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:52:00.548455    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dcc54b4f373ed643c4fb269547501a617d2b4b998938c5073f9db15cfe57ee1\": container with ID starting with 9dcc54b4f373ed643c4fb269547501a617d2b4b998938c5073f9db15cfe57ee1 not found: ID does not exist" containerID="9dcc54b4f373ed643c4fb269547501a617d2b4b998938c5073f9db15cfe57ee1"
Sep 18 19:52:00.548512 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:00.548472    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dcc54b4f373ed643c4fb269547501a617d2b4b998938c5073f9db15cfe57ee1"} err="failed to get container status \"9dcc54b4f373ed643c4fb269547501a617d2b4b998938c5073f9db15cfe57ee1\": rpc error: code = NotFound desc = could not find container \"9dcc54b4f373ed643c4fb269547501a617d2b4b998938c5073f9db15cfe57ee1\": container with ID starting with 9dcc54b4f373ed643c4fb269547501a617d2b4b998938c5073f9db15cfe57ee1 not found: ID does not exist"
Sep 18 19:52:00.548512 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:00.548484    2522 scope.go:117] "RemoveContainer" containerID="e0a9573f001e97e5ff3df860a8abcb76ebd975242a252713b8c3c430a821226b"
Sep 18 19:52:00.548748 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:52:00.548725    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0a9573f001e97e5ff3df860a8abcb76ebd975242a252713b8c3c430a821226b\": container with ID starting with e0a9573f001e97e5ff3df860a8abcb76ebd975242a252713b8c3c430a821226b not found: ID does not exist" containerID="e0a9573f001e97e5ff3df860a8abcb76ebd975242a252713b8c3c430a821226b"
Sep 18 19:52:00.548828 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:00.548758    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0a9573f001e97e5ff3df860a8abcb76ebd975242a252713b8c3c430a821226b"} err="failed to get container status \"e0a9573f001e97e5ff3df860a8abcb76ebd975242a252713b8c3c430a821226b\": rpc error: code = NotFound desc = could not find container \"e0a9573f001e97e5ff3df860a8abcb76ebd975242a252713b8c3c430a821226b\": container with ID starting with e0a9573f001e97e5ff3df860a8abcb76ebd975242a252713b8c3c430a821226b not found: ID does not exist"
Sep 18 19:52:00.553932 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:00.553913    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8"]
Sep 18 19:52:00.558895 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:00.558878    2522 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-tgcd8"]
Sep 18 19:52:00.723109 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:00.723075    2522 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b82a9095-65cd-4b23-8e76-3d8b894b915d" path="/var/lib/kubelet/pods/b82a9095-65cd-4b23-8e76-3d8b894b915d/volumes"
Sep 18 19:52:09.607055 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:09.607020    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn"]
Sep 18 19:52:09.607596 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:09.607305    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn" podUID="1597b7b3-9d5e-47be-bf09-82f72fcad09e" containerName="kserve-container" containerID="cri-o://8dbda5a82633890a359a1bc7d0fd1f6aa999e3c95398fe8b710e92fada3e93ae" gracePeriod=30
Sep 18 19:52:09.607596 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:09.607344    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn" podUID="1597b7b3-9d5e-47be-bf09-82f72fcad09e" containerName="kube-rbac-proxy" containerID="cri-o://76c53f3c16a7d6415108695ae27ab639e02d4239d0e59066817acfd0f6ef70b2" gracePeriod=30
Sep 18 19:52:10.573292 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:10.573260    2522 generic.go:356] "Generic (PLEG): container finished" podID="1597b7b3-9d5e-47be-bf09-82f72fcad09e" containerID="76c53f3c16a7d6415108695ae27ab639e02d4239d0e59066817acfd0f6ef70b2" exitCode=2
Sep 18 19:52:10.573469 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:10.573301    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn" event={"ID":"1597b7b3-9d5e-47be-bf09-82f72fcad09e","Type":"ContainerDied","Data":"76c53f3c16a7d6415108695ae27ab639e02d4239d0e59066817acfd0f6ef70b2"}
Sep 18 19:52:12.118270 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:12.118223    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn" podUID="1597b7b3-9d5e-47be-bf09-82f72fcad09e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.54:8643/healthz\": dial tcp 10.132.0.54:8643: connect: connection refused"
Sep 18 19:52:14.596825 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:14.596788    2522 generic.go:356] "Generic (PLEG): container finished" podID="1597b7b3-9d5e-47be-bf09-82f72fcad09e" containerID="8dbda5a82633890a359a1bc7d0fd1f6aa999e3c95398fe8b710e92fada3e93ae" exitCode=0
Sep 18 19:52:14.597216 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:14.596822    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn" event={"ID":"1597b7b3-9d5e-47be-bf09-82f72fcad09e","Type":"ContainerDied","Data":"8dbda5a82633890a359a1bc7d0fd1f6aa999e3c95398fe8b710e92fada3e93ae"}
Sep 18 19:52:14.653661 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:14.653634    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn"
Sep 18 19:52:14.709667 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:14.709583    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1597b7b3-9d5e-47be-bf09-82f72fcad09e-kserve-provision-location\") pod \"1597b7b3-9d5e-47be-bf09-82f72fcad09e\" (UID: \"1597b7b3-9d5e-47be-bf09-82f72fcad09e\") "
Sep 18 19:52:14.709667 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:14.709639    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1597b7b3-9d5e-47be-bf09-82f72fcad09e-cabundle-cert\") pod \"1597b7b3-9d5e-47be-bf09-82f72fcad09e\" (UID: \"1597b7b3-9d5e-47be-bf09-82f72fcad09e\") "
Sep 18 19:52:14.709887 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:14.709713    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1597b7b3-9d5e-47be-bf09-82f72fcad09e-proxy-tls\") pod \"1597b7b3-9d5e-47be-bf09-82f72fcad09e\" (UID: \"1597b7b3-9d5e-47be-bf09-82f72fcad09e\") "
Sep 18 19:52:14.709887 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:14.709736    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1597b7b3-9d5e-47be-bf09-82f72fcad09e-kube-rbac-proxy-sar-config\") pod \"1597b7b3-9d5e-47be-bf09-82f72fcad09e\" (UID: \"1597b7b3-9d5e-47be-bf09-82f72fcad09e\") "
Sep 18 19:52:14.709887 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:14.709763    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qlggt\" (UniqueName: \"kubernetes.io/projected/1597b7b3-9d5e-47be-bf09-82f72fcad09e-kube-api-access-qlggt\") pod \"1597b7b3-9d5e-47be-bf09-82f72fcad09e\" (UID: \"1597b7b3-9d5e-47be-bf09-82f72fcad09e\") "
Sep 18 19:52:14.710058 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:14.709920    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1597b7b3-9d5e-47be-bf09-82f72fcad09e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1597b7b3-9d5e-47be-bf09-82f72fcad09e" (UID: "1597b7b3-9d5e-47be-bf09-82f72fcad09e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:52:14.710058 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:14.710028    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1597b7b3-9d5e-47be-bf09-82f72fcad09e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1597b7b3-9d5e-47be-bf09-82f72fcad09e" (UID: "1597b7b3-9d5e-47be-bf09-82f72fcad09e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:52:14.710170 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:14.710062    2522 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1597b7b3-9d5e-47be-bf09-82f72fcad09e-kserve-provision-location\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:52:14.710170 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:14.710124    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1597b7b3-9d5e-47be-bf09-82f72fcad09e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1597b7b3-9d5e-47be-bf09-82f72fcad09e" (UID: "1597b7b3-9d5e-47be-bf09-82f72fcad09e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:52:14.711890 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:14.711865    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1597b7b3-9d5e-47be-bf09-82f72fcad09e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1597b7b3-9d5e-47be-bf09-82f72fcad09e" (UID: "1597b7b3-9d5e-47be-bf09-82f72fcad09e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:52:14.712040 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:14.712019    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1597b7b3-9d5e-47be-bf09-82f72fcad09e-kube-api-access-qlggt" (OuterVolumeSpecName: "kube-api-access-qlggt") pod "1597b7b3-9d5e-47be-bf09-82f72fcad09e" (UID: "1597b7b3-9d5e-47be-bf09-82f72fcad09e"). InnerVolumeSpecName "kube-api-access-qlggt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:52:14.811068 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:14.811029    2522 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1597b7b3-9d5e-47be-bf09-82f72fcad09e-proxy-tls\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:52:14.811068 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:14.811062    2522 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1597b7b3-9d5e-47be-bf09-82f72fcad09e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:52:14.811068 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:14.811074    2522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlggt\" (UniqueName: \"kubernetes.io/projected/1597b7b3-9d5e-47be-bf09-82f72fcad09e-kube-api-access-qlggt\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:52:14.811317 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:14.811084    2522 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1597b7b3-9d5e-47be-bf09-82f72fcad09e-cabundle-cert\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:52:15.603255 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:15.603229    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn"
Sep 18 19:52:15.603684 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:15.603223    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn" event={"ID":"1597b7b3-9d5e-47be-bf09-82f72fcad09e","Type":"ContainerDied","Data":"f0e3603bb694ba381abb06cbd21698286c0a9966b5f9d30135cc8dfeee04bb98"}
Sep 18 19:52:15.603684 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:15.603364    2522 scope.go:117] "RemoveContainer" containerID="76c53f3c16a7d6415108695ae27ab639e02d4239d0e59066817acfd0f6ef70b2"
Sep 18 19:52:15.611879 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:15.611862    2522 scope.go:117] "RemoveContainer" containerID="8dbda5a82633890a359a1bc7d0fd1f6aa999e3c95398fe8b710e92fada3e93ae"
Sep 18 19:52:15.619389 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:15.619369    2522 scope.go:117] "RemoveContainer" containerID="1f5c729f5643d5579c3f18209fff44c7cf1e95856cc7e7a3256a7747dc165e01"
Sep 18 19:52:15.632143 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:15.632123    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn"]
Sep 18 19:52:15.638714 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:15.638695    2522 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-dc6dbfb86-jcbsn"]
Sep 18 19:52:16.721304 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:16.721271    2522 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1597b7b3-9d5e-47be-bf09-82f72fcad09e" path="/var/lib/kubelet/pods/1597b7b3-9d5e-47be-bf09-82f72fcad09e/volumes"
Sep 18 19:52:19.924184 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:19.924146    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-59b9d969c9-cv995"]
Sep 18 19:52:19.924621 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:19.924483    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1597b7b3-9d5e-47be-bf09-82f72fcad09e" containerName="kserve-container"
Sep 18 19:52:19.924621 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:19.924494    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="1597b7b3-9d5e-47be-bf09-82f72fcad09e" containerName="kserve-container"
Sep 18 19:52:19.924621 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:19.924501    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b82a9095-65cd-4b23-8e76-3d8b894b915d" containerName="kube-rbac-proxy"
Sep 18 19:52:19.924621 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:19.924507    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="b82a9095-65cd-4b23-8e76-3d8b894b915d" containerName="kube-rbac-proxy"
Sep 18 19:52:19.924621 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:19.924525    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1597b7b3-9d5e-47be-bf09-82f72fcad09e" containerName="kube-rbac-proxy"
Sep 18 19:52:19.924621 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:19.924532    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="1597b7b3-9d5e-47be-bf09-82f72fcad09e" containerName="kube-rbac-proxy"
Sep 18 19:52:19.924621 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:19.924544    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b82a9095-65cd-4b23-8e76-3d8b894b915d" containerName="storage-initializer"
Sep 18 19:52:19.924621 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:19.924550    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="b82a9095-65cd-4b23-8e76-3d8b894b915d" containerName="storage-initializer"
Sep 18 19:52:19.924621 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:19.924556    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b82a9095-65cd-4b23-8e76-3d8b894b915d" containerName="kserve-container"
Sep 18 19:52:19.924621 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:19.924573    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="b82a9095-65cd-4b23-8e76-3d8b894b915d" containerName="kserve-container"
Sep 18 19:52:19.924621 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:19.924579    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1597b7b3-9d5e-47be-bf09-82f72fcad09e" containerName="storage-initializer"
Sep 18 19:52:19.924621 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:19.924584    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="1597b7b3-9d5e-47be-bf09-82f72fcad09e" containerName="storage-initializer"
Sep 18 19:52:19.925044 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:19.924634    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="1597b7b3-9d5e-47be-bf09-82f72fcad09e" containerName="kserve-container"
Sep 18 19:52:19.925044 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:19.924641    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="1597b7b3-9d5e-47be-bf09-82f72fcad09e" containerName="kube-rbac-proxy"
Sep 18 19:52:19.925044 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:19.924647    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="b82a9095-65cd-4b23-8e76-3d8b894b915d" containerName="kube-rbac-proxy"
Sep 18 19:52:19.925044 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:19.924654    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="b82a9095-65cd-4b23-8e76-3d8b894b915d" containerName="kserve-container"
Sep 18 19:52:19.929213 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:19.929195    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-59b9d969c9-cv995"
Sep 18 19:52:19.932617 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:19.932590    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\""
Sep 18 19:52:19.932765 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:19.932635    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 19:52:19.932765 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:19.932733    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-predictor-serving-cert\""
Sep 18 19:52:19.932895 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:19.932770    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 19:52:19.932895 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:19.932849    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 19:52:19.933006 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:19.932904    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 19:52:19.933324 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:19.933306    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5b562\""
Sep 18 19:52:19.941118 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:19.941098    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-59b9d969c9-cv995"]
Sep 18 19:52:20.054379 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:20.054334    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff83910c-18d1-4b06-8978-4fe64bc307ed-kserve-provision-location\") pod \"isvc-lightgbm-predictor-59b9d969c9-cv995\" (UID: \"ff83910c-18d1-4b06-8978-4fe64bc307ed\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-59b9d969c9-cv995"
Sep 18 19:52:20.054596 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:20.054403    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff83910c-18d1-4b06-8978-4fe64bc307ed-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-59b9d969c9-cv995\" (UID: \"ff83910c-18d1-4b06-8978-4fe64bc307ed\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-59b9d969c9-cv995"
Sep 18 19:52:20.054596 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:20.054463    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff83910c-18d1-4b06-8978-4fe64bc307ed-cabundle-cert\") pod \"isvc-lightgbm-predictor-59b9d969c9-cv995\" (UID: \"ff83910c-18d1-4b06-8978-4fe64bc307ed\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-59b9d969c9-cv995"
Sep 18 19:52:20.054596 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:20.054583    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff83910c-18d1-4b06-8978-4fe64bc307ed-proxy-tls\") pod \"isvc-lightgbm-predictor-59b9d969c9-cv995\" (UID: \"ff83910c-18d1-4b06-8978-4fe64bc307ed\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-59b9d969c9-cv995"
Sep 18 19:52:20.054720 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:20.054616    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kr87\" (UniqueName: \"kubernetes.io/projected/ff83910c-18d1-4b06-8978-4fe64bc307ed-kube-api-access-4kr87\") pod \"isvc-lightgbm-predictor-59b9d969c9-cv995\" (UID: \"ff83910c-18d1-4b06-8978-4fe64bc307ed\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-59b9d969c9-cv995"
Sep 18 19:52:20.155275 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:20.155239    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff83910c-18d1-4b06-8978-4fe64bc307ed-kserve-provision-location\") pod \"isvc-lightgbm-predictor-59b9d969c9-cv995\" (UID: \"ff83910c-18d1-4b06-8978-4fe64bc307ed\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-59b9d969c9-cv995"
Sep 18 19:52:20.155275 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:20.155286    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff83910c-18d1-4b06-8978-4fe64bc307ed-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-59b9d969c9-cv995\" (UID: \"ff83910c-18d1-4b06-8978-4fe64bc307ed\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-59b9d969c9-cv995"
Sep 18 19:52:20.155552 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:20.155465    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff83910c-18d1-4b06-8978-4fe64bc307ed-cabundle-cert\") pod \"isvc-lightgbm-predictor-59b9d969c9-cv995\" (UID: \"ff83910c-18d1-4b06-8978-4fe64bc307ed\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-59b9d969c9-cv995"
Sep 18 19:52:20.155552 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:20.155539    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff83910c-18d1-4b06-8978-4fe64bc307ed-proxy-tls\") pod \"isvc-lightgbm-predictor-59b9d969c9-cv995\" (UID: \"ff83910c-18d1-4b06-8978-4fe64bc307ed\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-59b9d969c9-cv995"
Sep 18 19:52:20.155706 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:20.155592    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kr87\" (UniqueName: \"kubernetes.io/projected/ff83910c-18d1-4b06-8978-4fe64bc307ed-kube-api-access-4kr87\") pod \"isvc-lightgbm-predictor-59b9d969c9-cv995\" (UID: \"ff83910c-18d1-4b06-8978-4fe64bc307ed\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-59b9d969c9-cv995"
Sep 18 19:52:20.155765 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:52:20.155696    2522 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-predictor-serving-cert: secret "isvc-lightgbm-predictor-serving-cert" not found
Sep 18 19:52:20.155818 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:20.155765    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff83910c-18d1-4b06-8978-4fe64bc307ed-kserve-provision-location\") pod \"isvc-lightgbm-predictor-59b9d969c9-cv995\" (UID: \"ff83910c-18d1-4b06-8978-4fe64bc307ed\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-59b9d969c9-cv995"
Sep 18 19:52:20.155818 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:52:20.155780    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff83910c-18d1-4b06-8978-4fe64bc307ed-proxy-tls podName:ff83910c-18d1-4b06-8978-4fe64bc307ed nodeName:}" failed. No retries permitted until 2026-09-18 19:52:20.655759794 +0000 UTC m=+2300.539286249 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ff83910c-18d1-4b06-8978-4fe64bc307ed-proxy-tls") pod "isvc-lightgbm-predictor-59b9d969c9-cv995" (UID: "ff83910c-18d1-4b06-8978-4fe64bc307ed") : secret "isvc-lightgbm-predictor-serving-cert" not found
Sep 18 19:52:20.156150 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:20.156128    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff83910c-18d1-4b06-8978-4fe64bc307ed-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-59b9d969c9-cv995\" (UID: \"ff83910c-18d1-4b06-8978-4fe64bc307ed\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-59b9d969c9-cv995"
Sep 18 19:52:20.156188 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:20.156149    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff83910c-18d1-4b06-8978-4fe64bc307ed-cabundle-cert\") pod \"isvc-lightgbm-predictor-59b9d969c9-cv995\" (UID: \"ff83910c-18d1-4b06-8978-4fe64bc307ed\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-59b9d969c9-cv995"
Sep 18 19:52:20.168295 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:20.168274    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kr87\" (UniqueName: \"kubernetes.io/projected/ff83910c-18d1-4b06-8978-4fe64bc307ed-kube-api-access-4kr87\") pod \"isvc-lightgbm-predictor-59b9d969c9-cv995\" (UID: \"ff83910c-18d1-4b06-8978-4fe64bc307ed\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-59b9d969c9-cv995"
Sep 18 19:52:20.659314 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:20.659278    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff83910c-18d1-4b06-8978-4fe64bc307ed-proxy-tls\") pod \"isvc-lightgbm-predictor-59b9d969c9-cv995\" (UID: \"ff83910c-18d1-4b06-8978-4fe64bc307ed\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-59b9d969c9-cv995"
Sep 18 19:52:20.661719 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:20.661692    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff83910c-18d1-4b06-8978-4fe64bc307ed-proxy-tls\") pod \"isvc-lightgbm-predictor-59b9d969c9-cv995\" (UID: \"ff83910c-18d1-4b06-8978-4fe64bc307ed\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-59b9d969c9-cv995"
Sep 18 19:52:20.840175 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:20.840133    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-59b9d969c9-cv995"
Sep 18 19:52:21.112266 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:21.112241    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-59b9d969c9-cv995"]
Sep 18 19:52:21.113741 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:52:21.113712    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff83910c_18d1_4b06_8978_4fe64bc307ed.slice/crio-f6ad2ed882738b34cbfe4dc0d8fefabbcca32178dc97be7cdcb7ba01c77164f9 WatchSource:0}: Error finding container f6ad2ed882738b34cbfe4dc0d8fefabbcca32178dc97be7cdcb7ba01c77164f9: Status 404 returned error can't find the container with id f6ad2ed882738b34cbfe4dc0d8fefabbcca32178dc97be7cdcb7ba01c77164f9
Sep 18 19:52:21.632426 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:21.632387    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-59b9d969c9-cv995" event={"ID":"ff83910c-18d1-4b06-8978-4fe64bc307ed","Type":"ContainerStarted","Data":"9dc9351f6cfe84b8b67c7523b1ffa3a53d9d840a3c760d07d1f9de638b3d518d"}
Sep 18 19:52:21.632600 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:21.632464    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-59b9d969c9-cv995" event={"ID":"ff83910c-18d1-4b06-8978-4fe64bc307ed","Type":"ContainerStarted","Data":"f6ad2ed882738b34cbfe4dc0d8fefabbcca32178dc97be7cdcb7ba01c77164f9"}
Sep 18 19:52:25.652377 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:25.652335    2522 generic.go:356] "Generic (PLEG): container finished" podID="ff83910c-18d1-4b06-8978-4fe64bc307ed" containerID="9dc9351f6cfe84b8b67c7523b1ffa3a53d9d840a3c760d07d1f9de638b3d518d" exitCode=0
Sep 18 19:52:25.652809 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:25.652407    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-59b9d969c9-cv995" event={"ID":"ff83910c-18d1-4b06-8978-4fe64bc307ed","Type":"ContainerDied","Data":"9dc9351f6cfe84b8b67c7523b1ffa3a53d9d840a3c760d07d1f9de638b3d518d"}
Sep 18 19:52:32.693829 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:32.693789    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-59b9d969c9-cv995" event={"ID":"ff83910c-18d1-4b06-8978-4fe64bc307ed","Type":"ContainerStarted","Data":"0934c9b9e8ac88f20b1b54e22908cbb242193f5b4752d19ea504bc03ef1c715b"}
Sep 18 19:52:32.694402 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:32.693871    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-59b9d969c9-cv995" event={"ID":"ff83910c-18d1-4b06-8978-4fe64bc307ed","Type":"ContainerStarted","Data":"d8cbb70ce96e90d8c4488ee3662be065b9fb44f2e878865803a2e073b0681645"}
Sep 18 19:52:32.694402 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:32.694091    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-59b9d969c9-cv995"
Sep 18 19:52:32.716805 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:32.716755    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-59b9d969c9-cv995" podStartSLOduration=3.535705483 podStartE2EDuration="13.716739974s" totalImagesPullingTime="6.444713561s" totalInitContainerRuntime="3.73632093s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:52:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:52:32.716419533 +0000 UTC m=+2312.599945995" watchObservedRunningTime="2026-09-18 19:52:32.716739974 +0000 UTC m=+2312.600266436"
Sep 18 19:52:33.698553 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:33.698516    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-59b9d969c9-cv995"
Sep 18 19:52:33.700024 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:33.699995    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-59b9d969c9-cv995" podUID="ff83910c-18d1-4b06-8978-4fe64bc307ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 18 19:52:34.704118 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:34.704081    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-59b9d969c9-cv995" podUID="ff83910c-18d1-4b06-8978-4fe64bc307ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 18 19:52:39.708543 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:39.708510    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-59b9d969c9-cv995"
Sep 18 19:52:39.709141 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:39.709116    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-59b9d969c9-cv995" podUID="ff83910c-18d1-4b06-8978-4fe64bc307ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 18 19:52:49.709303 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:49.709254    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-59b9d969c9-cv995" podUID="ff83910c-18d1-4b06-8978-4fe64bc307ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 18 19:52:59.709219 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:52:59.709124    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-59b9d969c9-cv995" podUID="ff83910c-18d1-4b06-8978-4fe64bc307ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 18 19:53:09.710041 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:53:09.709999    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-59b9d969c9-cv995" podUID="ff83910c-18d1-4b06-8978-4fe64bc307ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 18 19:53:19.709830 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:53:19.709791    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-59b9d969c9-cv995" podUID="ff83910c-18d1-4b06-8978-4fe64bc307ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 18 19:53:29.709534 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:53:29.709490    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-59b9d969c9-cv995" podUID="ff83910c-18d1-4b06-8978-4fe64bc307ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 18 19:53:39.709572 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:53:39.709522    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-59b9d969c9-cv995" podUID="ff83910c-18d1-4b06-8978-4fe64bc307ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 18 19:53:49.709386 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:53:49.709342    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-59b9d969c9-cv995" podUID="ff83910c-18d1-4b06-8978-4fe64bc307ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 18 19:53:56.716969 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:53:56.716903    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-59b9d969c9-cv995" podUID="ff83910c-18d1-4b06-8978-4fe64bc307ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 18 19:54:06.721925 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:54:06.721896    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-59b9d969c9-cv995"
Sep 18 19:54:10.909698 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:54:10.909666    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-59b9d969c9-cv995"]
Sep 18 19:54:10.910134 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:54:10.909913    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-59b9d969c9-cv995" podUID="ff83910c-18d1-4b06-8978-4fe64bc307ed" containerName="kserve-container" containerID="cri-o://d8cbb70ce96e90d8c4488ee3662be065b9fb44f2e878865803a2e073b0681645" gracePeriod=30
Sep 18 19:54:10.910134 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:54:10.909931    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-59b9d969c9-cv995" podUID="ff83910c-18d1-4b06-8978-4fe64bc307ed" containerName="kube-rbac-proxy" containerID="cri-o://0934c9b9e8ac88f20b1b54e22908cbb242193f5b4752d19ea504bc03ef1c715b" gracePeriod=30
Sep 18 19:54:11.155539 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:54:11.155508    2522 generic.go:356] "Generic (PLEG): container finished" podID="ff83910c-18d1-4b06-8978-4fe64bc307ed" containerID="0934c9b9e8ac88f20b1b54e22908cbb242193f5b4752d19ea504bc03ef1c715b" exitCode=2
Sep 18 19:54:11.155701 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:54:11.155595    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-59b9d969c9-cv995" event={"ID":"ff83910c-18d1-4b06-8978-4fe64bc307ed","Type":"ContainerDied","Data":"0934c9b9e8ac88f20b1b54e22908cbb242193f5b4752d19ea504bc03ef1c715b"}
Sep 18 19:54:14.704733 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:54:14.704692    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-59b9d969c9-cv995" podUID="ff83910c-18d1-4b06-8978-4fe64bc307ed" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.56:8643/healthz\": dial tcp 10.132.0.56:8643: connect: connection refused"
Sep 18 19:54:15.846630 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:54:15.846601    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-59b9d969c9-cv995"
Sep 18 19:54:15.954478 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:54:15.954406    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff83910c-18d1-4b06-8978-4fe64bc307ed-proxy-tls\") pod \"ff83910c-18d1-4b06-8978-4fe64bc307ed\" (UID: \"ff83910c-18d1-4b06-8978-4fe64bc307ed\") "
Sep 18 19:54:15.954478 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:54:15.954467    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff83910c-18d1-4b06-8978-4fe64bc307ed-kserve-provision-location\") pod \"ff83910c-18d1-4b06-8978-4fe64bc307ed\" (UID: \"ff83910c-18d1-4b06-8978-4fe64bc307ed\") "
Sep 18 19:54:15.954742 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:54:15.954513    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff83910c-18d1-4b06-8978-4fe64bc307ed-cabundle-cert\") pod \"ff83910c-18d1-4b06-8978-4fe64bc307ed\" (UID: \"ff83910c-18d1-4b06-8978-4fe64bc307ed\") "
Sep 18 19:54:15.954742 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:54:15.954532    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff83910c-18d1-4b06-8978-4fe64bc307ed-kube-rbac-proxy-sar-config\") pod \"ff83910c-18d1-4b06-8978-4fe64bc307ed\" (UID: \"ff83910c-18d1-4b06-8978-4fe64bc307ed\") "
Sep 18 19:54:15.954742 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:54:15.954594    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4kr87\" (UniqueName: \"kubernetes.io/projected/ff83910c-18d1-4b06-8978-4fe64bc307ed-kube-api-access-4kr87\") pod \"ff83910c-18d1-4b06-8978-4fe64bc307ed\" (UID: \"ff83910c-18d1-4b06-8978-4fe64bc307ed\") "
Sep 18 19:54:15.954916 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:54:15.954858    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff83910c-18d1-4b06-8978-4fe64bc307ed-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ff83910c-18d1-4b06-8978-4fe64bc307ed" (UID: "ff83910c-18d1-4b06-8978-4fe64bc307ed"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:54:15.954965 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:54:15.954907    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff83910c-18d1-4b06-8978-4fe64bc307ed-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ff83910c-18d1-4b06-8978-4fe64bc307ed" (UID: "ff83910c-18d1-4b06-8978-4fe64bc307ed"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:54:15.955011 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:54:15.954959    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff83910c-18d1-4b06-8978-4fe64bc307ed-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ff83910c-18d1-4b06-8978-4fe64bc307ed" (UID: "ff83910c-18d1-4b06-8978-4fe64bc307ed"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:54:15.956556 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:54:15.956536    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff83910c-18d1-4b06-8978-4fe64bc307ed-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ff83910c-18d1-4b06-8978-4fe64bc307ed" (UID: "ff83910c-18d1-4b06-8978-4fe64bc307ed"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:54:15.956690 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:54:15.956672    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff83910c-18d1-4b06-8978-4fe64bc307ed-kube-api-access-4kr87" (OuterVolumeSpecName: "kube-api-access-4kr87") pod "ff83910c-18d1-4b06-8978-4fe64bc307ed" (UID: "ff83910c-18d1-4b06-8978-4fe64bc307ed"). InnerVolumeSpecName "kube-api-access-4kr87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:54:16.055772 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:54:16.055743    2522 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff83910c-18d1-4b06-8978-4fe64bc307ed-kserve-provision-location\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:54:16.055772 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:54:16.055773    2522 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff83910c-18d1-4b06-8978-4fe64bc307ed-cabundle-cert\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:54:16.055962 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:54:16.055789    2522 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff83910c-18d1-4b06-8978-4fe64bc307ed-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:54:16.055962 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:54:16.055803    2522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kr87\" (UniqueName: \"kubernetes.io/projected/ff83910c-18d1-4b06-8978-4fe64bc307ed-kube-api-access-4kr87\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:54:16.055962 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:54:16.055816    2522 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff83910c-18d1-4b06-8978-4fe64bc307ed-proxy-tls\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 19:54:16.181181 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:54:16.181147    2522 generic.go:356] "Generic (PLEG): container finished" podID="ff83910c-18d1-4b06-8978-4fe64bc307ed" containerID="d8cbb70ce96e90d8c4488ee3662be065b9fb44f2e878865803a2e073b0681645" exitCode=0
Sep 18 19:54:16.181349 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:54:16.181194    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-59b9d969c9-cv995"
Sep 18 19:54:16.181349 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:54:16.181228    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-59b9d969c9-cv995" event={"ID":"ff83910c-18d1-4b06-8978-4fe64bc307ed","Type":"ContainerDied","Data":"d8cbb70ce96e90d8c4488ee3662be065b9fb44f2e878865803a2e073b0681645"}
Sep 18 19:54:16.181349 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:54:16.181304    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-59b9d969c9-cv995" event={"ID":"ff83910c-18d1-4b06-8978-4fe64bc307ed","Type":"ContainerDied","Data":"f6ad2ed882738b34cbfe4dc0d8fefabbcca32178dc97be7cdcb7ba01c77164f9"}
Sep 18 19:54:16.181349 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:54:16.181319    2522 scope.go:117] "RemoveContainer" containerID="0934c9b9e8ac88f20b1b54e22908cbb242193f5b4752d19ea504bc03ef1c715b"
Sep 18 19:54:16.192638 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:54:16.192541    2522 scope.go:117] "RemoveContainer" containerID="d8cbb70ce96e90d8c4488ee3662be065b9fb44f2e878865803a2e073b0681645"
Sep 18 19:54:16.200909 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:54:16.200891    2522 scope.go:117] "RemoveContainer" containerID="9dc9351f6cfe84b8b67c7523b1ffa3a53d9d840a3c760d07d1f9de638b3d518d"
Sep 18 19:54:16.208887 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:54:16.208869    2522 scope.go:117] "RemoveContainer" containerID="0934c9b9e8ac88f20b1b54e22908cbb242193f5b4752d19ea504bc03ef1c715b"
Sep 18 19:54:16.209315 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:54:16.209294    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0934c9b9e8ac88f20b1b54e22908cbb242193f5b4752d19ea504bc03ef1c715b\": container with ID starting with 0934c9b9e8ac88f20b1b54e22908cbb242193f5b4752d19ea504bc03ef1c715b not found: ID does not exist" containerID="0934c9b9e8ac88f20b1b54e22908cbb242193f5b4752d19ea504bc03ef1c715b"
Sep 18 19:54:16.209406 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:54:16.209325    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0934c9b9e8ac88f20b1b54e22908cbb242193f5b4752d19ea504bc03ef1c715b"} err="failed to get container status \"0934c9b9e8ac88f20b1b54e22908cbb242193f5b4752d19ea504bc03ef1c715b\": rpc error: code = NotFound desc = could not find container \"0934c9b9e8ac88f20b1b54e22908cbb242193f5b4752d19ea504bc03ef1c715b\": container with ID starting with 0934c9b9e8ac88f20b1b54e22908cbb242193f5b4752d19ea504bc03ef1c715b not found: ID does not exist"
Sep 18 19:54:16.209406 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:54:16.209342    2522 scope.go:117] "RemoveContainer" containerID="d8cbb70ce96e90d8c4488ee3662be065b9fb44f2e878865803a2e073b0681645"
Sep 18 19:54:16.209617 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:54:16.209598    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8cbb70ce96e90d8c4488ee3662be065b9fb44f2e878865803a2e073b0681645\": container with ID starting with d8cbb70ce96e90d8c4488ee3662be065b9fb44f2e878865803a2e073b0681645 not found: ID does not exist" containerID="d8cbb70ce96e90d8c4488ee3662be065b9fb44f2e878865803a2e073b0681645"
Sep 18 19:54:16.209680 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:54:16.209621    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8cbb70ce96e90d8c4488ee3662be065b9fb44f2e878865803a2e073b0681645"} err="failed to get container status \"d8cbb70ce96e90d8c4488ee3662be065b9fb44f2e878865803a2e073b0681645\": rpc error: code = NotFound desc = could not find container \"d8cbb70ce96e90d8c4488ee3662be065b9fb44f2e878865803a2e073b0681645\": container with ID starting with d8cbb70ce96e90d8c4488ee3662be065b9fb44f2e878865803a2e073b0681645 not found: ID does not exist"
Sep 18 19:54:16.209680 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:54:16.209637    2522 scope.go:117] "RemoveContainer" containerID="9dc9351f6cfe84b8b67c7523b1ffa3a53d9d840a3c760d07d1f9de638b3d518d"
Sep 18 19:54:16.209916 ip-10-0-131-221 kubenswrapper[2522]: E0918 19:54:16.209896    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dc9351f6cfe84b8b67c7523b1ffa3a53d9d840a3c760d07d1f9de638b3d518d\": container with ID starting with 9dc9351f6cfe84b8b67c7523b1ffa3a53d9d840a3c760d07d1f9de638b3d518d not found: ID does not exist" containerID="9dc9351f6cfe84b8b67c7523b1ffa3a53d9d840a3c760d07d1f9de638b3d518d"
Sep 18 19:54:16.210010 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:54:16.209921    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dc9351f6cfe84b8b67c7523b1ffa3a53d9d840a3c760d07d1f9de638b3d518d"} err="failed to get container status \"9dc9351f6cfe84b8b67c7523b1ffa3a53d9d840a3c760d07d1f9de638b3d518d\": rpc error: code = NotFound desc = could not find container \"9dc9351f6cfe84b8b67c7523b1ffa3a53d9d840a3c760d07d1f9de638b3d518d\": container with ID starting with 9dc9351f6cfe84b8b67c7523b1ffa3a53d9d840a3c760d07d1f9de638b3d518d not found: ID does not exist"
Sep 18 19:54:16.210547 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:54:16.210529    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-59b9d969c9-cv995"]
Sep 18 19:54:16.222577 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:54:16.222531    2522 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-59b9d969c9-cv995"]
Sep 18 19:54:16.722508 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:54:16.722475    2522 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff83910c-18d1-4b06-8978-4fe64bc307ed" path="/var/lib/kubelet/pods/ff83910c-18d1-4b06-8978-4fe64bc307ed/volumes"
Sep 18 19:58:48.541329 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:58:48.541296    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc"]
Sep 18 19:58:48.541819 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:58:48.541660    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff83910c-18d1-4b06-8978-4fe64bc307ed" containerName="kserve-container"
Sep 18 19:58:48.541819 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:58:48.541674    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff83910c-18d1-4b06-8978-4fe64bc307ed" containerName="kserve-container"
Sep 18 19:58:48.541819 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:58:48.541685    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff83910c-18d1-4b06-8978-4fe64bc307ed" containerName="storage-initializer"
Sep 18 19:58:48.541819 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:58:48.541691    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff83910c-18d1-4b06-8978-4fe64bc307ed" containerName="storage-initializer"
Sep 18 19:58:48.541819 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:58:48.541698    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff83910c-18d1-4b06-8978-4fe64bc307ed" containerName="kube-rbac-proxy"
Sep 18 19:58:48.541819 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:58:48.541703    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff83910c-18d1-4b06-8978-4fe64bc307ed" containerName="kube-rbac-proxy"
Sep 18 19:58:48.541819 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:58:48.541756    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff83910c-18d1-4b06-8978-4fe64bc307ed" containerName="kube-rbac-proxy"
Sep 18 19:58:48.541819 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:58:48.541766    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff83910c-18d1-4b06-8978-4fe64bc307ed" containerName="kserve-container"
Sep 18 19:58:48.549815 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:58:48.549798    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc"
Sep 18 19:58:48.553136 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:58:48.553114    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 19:58:48.553272 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:58:48.553124    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-predictor-serving-cert\""
Sep 18 19:58:48.553471 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:58:48.553454    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 19:58:48.553532 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:58:48.553509    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 19:58:48.553734 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:58:48.553720    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\""
Sep 18 19:58:48.554073 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:58:48.554058    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5b562\""
Sep 18 19:58:48.554145 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:58:48.554131    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 19:58:48.560791 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:58:48.560770    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc"]
Sep 18 19:58:48.696583 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:58:48.696533    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/343a5d81-88f0-4b68-8b4f-1d8f673ceacd-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc\" (UID: \"343a5d81-88f0-4b68-8b4f-1d8f673ceacd\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc"
Sep 18 19:58:48.696747 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:58:48.696595    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpxm6\" (UniqueName: \"kubernetes.io/projected/343a5d81-88f0-4b68-8b4f-1d8f673ceacd-kube-api-access-lpxm6\") pod \"isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc\" (UID: \"343a5d81-88f0-4b68-8b4f-1d8f673ceacd\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc"
Sep 18 19:58:48.696747 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:58:48.696632    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/343a5d81-88f0-4b68-8b4f-1d8f673ceacd-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc\" (UID: \"343a5d81-88f0-4b68-8b4f-1d8f673ceacd\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc"
Sep 18 19:58:48.696747 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:58:48.696732    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/343a5d81-88f0-4b68-8b4f-1d8f673ceacd-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc\" (UID: \"343a5d81-88f0-4b68-8b4f-1d8f673ceacd\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc"
Sep 18 19:58:48.696856 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:58:48.696771    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/343a5d81-88f0-4b68-8b4f-1d8f673ceacd-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc\" (UID: \"343a5d81-88f0-4b68-8b4f-1d8f673ceacd\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc"
Sep 18 19:58:48.797697 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:58:48.797612    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/343a5d81-88f0-4b68-8b4f-1d8f673ceacd-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc\" (UID: \"343a5d81-88f0-4b68-8b4f-1d8f673ceacd\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc"
Sep 18 19:58:48.797697 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:58:48.797685    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/343a5d81-88f0-4b68-8b4f-1d8f673ceacd-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc\" (UID: \"343a5d81-88f0-4b68-8b4f-1d8f673ceacd\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc"
Sep 18 19:58:48.797917 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:58:48.797722    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/343a5d81-88f0-4b68-8b4f-1d8f673ceacd-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc\" (UID: \"343a5d81-88f0-4b68-8b4f-1d8f673ceacd\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc"
Sep 18 19:58:48.797917 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:58:48.797764    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpxm6\" (UniqueName: \"kubernetes.io/projected/343a5d81-88f0-4b68-8b4f-1d8f673ceacd-kube-api-access-lpxm6\") pod \"isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc\" (UID: \"343a5d81-88f0-4b68-8b4f-1d8f673ceacd\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc"
Sep 18 19:58:48.797917 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:58:48.797801    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/343a5d81-88f0-4b68-8b4f-1d8f673ceacd-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc\" (UID: \"343a5d81-88f0-4b68-8b4f-1d8f673ceacd\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc"
Sep 18 19:58:48.798122 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:58:48.798097    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/343a5d81-88f0-4b68-8b4f-1d8f673ceacd-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc\" (UID: \"343a5d81-88f0-4b68-8b4f-1d8f673ceacd\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc"
Sep 18 19:58:48.798351 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:58:48.798333    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/343a5d81-88f0-4b68-8b4f-1d8f673ceacd-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc\" (UID: \"343a5d81-88f0-4b68-8b4f-1d8f673ceacd\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc"
Sep 18 19:58:48.798409 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:58:48.798393    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/343a5d81-88f0-4b68-8b4f-1d8f673ceacd-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc\" (UID: \"343a5d81-88f0-4b68-8b4f-1d8f673ceacd\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc"
Sep 18 19:58:48.800393 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:58:48.800370    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/343a5d81-88f0-4b68-8b4f-1d8f673ceacd-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc\" (UID: \"343a5d81-88f0-4b68-8b4f-1d8f673ceacd\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc"
Sep 18 19:58:48.807286 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:58:48.807263    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpxm6\" (UniqueName: \"kubernetes.io/projected/343a5d81-88f0-4b68-8b4f-1d8f673ceacd-kube-api-access-lpxm6\") pod \"isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc\" (UID: \"343a5d81-88f0-4b68-8b4f-1d8f673ceacd\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc"
Sep 18 19:58:48.862227 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:58:48.862195    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc"
Sep 18 19:58:49.138130 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:58:49.138107    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc"]
Sep 18 19:58:49.140218 ip-10-0-131-221 kubenswrapper[2522]: W0918 19:58:49.140192    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod343a5d81_88f0_4b68_8b4f_1d8f673ceacd.slice/crio-9c42d1ebdb72ea8e223eea9bc2a9caf63631611bad7cdf8111ec7ebbfe28ad5b WatchSource:0}: Error finding container 9c42d1ebdb72ea8e223eea9bc2a9caf63631611bad7cdf8111ec7ebbfe28ad5b: Status 404 returned error can't find the container with id 9c42d1ebdb72ea8e223eea9bc2a9caf63631611bad7cdf8111ec7ebbfe28ad5b
Sep 18 19:58:49.416545 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:58:49.416450    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc" event={"ID":"343a5d81-88f0-4b68-8b4f-1d8f673ceacd","Type":"ContainerStarted","Data":"5ac5d5314a7cc67aa6cedab072c7f4750940db624ff933d220f3f8ef1fcc45d5"}
Sep 18 19:58:49.416545 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:58:49.416535    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc" event={"ID":"343a5d81-88f0-4b68-8b4f-1d8f673ceacd","Type":"ContainerStarted","Data":"9c42d1ebdb72ea8e223eea9bc2a9caf63631611bad7cdf8111ec7ebbfe28ad5b"}
Sep 18 19:58:53.435000 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:58:53.434915    2522 generic.go:356] "Generic (PLEG): container finished" podID="343a5d81-88f0-4b68-8b4f-1d8f673ceacd" containerID="5ac5d5314a7cc67aa6cedab072c7f4750940db624ff933d220f3f8ef1fcc45d5" exitCode=0
Sep 18 19:58:53.435000 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:58:53.434985    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc" event={"ID":"343a5d81-88f0-4b68-8b4f-1d8f673ceacd","Type":"ContainerDied","Data":"5ac5d5314a7cc67aa6cedab072c7f4750940db624ff933d220f3f8ef1fcc45d5"}
Sep 18 19:58:53.498913 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:58:53.498895    2522 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:58:54.444024 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:58:54.443923    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc" event={"ID":"343a5d81-88f0-4b68-8b4f-1d8f673ceacd","Type":"ContainerStarted","Data":"7427d62eb8e9336f2ca1f752c1734a6a512daed5e0ba6e818c874fc859fa893a"}
Sep 18 19:58:56.457620 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:58:56.457496    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc" event={"ID":"343a5d81-88f0-4b68-8b4f-1d8f673ceacd","Type":"ContainerStarted","Data":"d80a26a5f4049fb37a3b9fc41f9afabd98f9d5123ae75c57f2d948f762c003f2"}
Sep 18 19:58:56.457985 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:58:56.457760    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc"
Sep 18 19:58:56.457985 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:58:56.457814    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc"
Sep 18 19:58:56.457985 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:58:56.457849    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc" event={"ID":"343a5d81-88f0-4b68-8b4f-1d8f673ceacd","Type":"ContainerStarted","Data":"7b1f2cf402a0d745684ebd45bb23319a1a1e5892ebc34ee3d6c69089e2cd7961"}
Sep 18 19:58:56.482255 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:58:56.482193    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc" podStartSLOduration=2.16914077 podStartE2EDuration="8.48217514s" totalImagesPullingTime="2.643294172s" totalInitContainerRuntime="3.669740198s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:58:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:58:56.481706267 +0000 UTC m=+2696.365232728" watchObservedRunningTime="2026-09-18 19:58:56.48217514 +0000 UTC m=+2696.365701602"
Sep 18 19:58:57.462620 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:58:57.462585    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc"
Sep 18 19:59:03.473333 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:59:03.473304    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc"
Sep 18 19:59:23.475018 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:59:23.474972    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc" podUID="343a5d81-88f0-4b68-8b4f-1d8f673ceacd" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.57:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 18 19:59:28.862947 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:59:28.862896    2522 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc" podUID="343a5d81-88f0-4b68-8b4f-1d8f673ceacd" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.57:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 18 19:59:33.638544 ip-10-0-131-221 kubenswrapper[2522]: I0918 19:59:33.638512    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc"
Sep 18 20:00:03.639838 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:00:03.639808    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc"
Sep 18 20:00:08.550016 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:00:08.549978    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc"]
Sep 18 20:00:08.550486 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:00:08.550299    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc" podUID="343a5d81-88f0-4b68-8b4f-1d8f673ceacd" containerName="kserve-container" containerID="cri-o://7427d62eb8e9336f2ca1f752c1734a6a512daed5e0ba6e818c874fc859fa893a" gracePeriod=30
Sep 18 20:00:08.550486 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:00:08.550370    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc" podUID="343a5d81-88f0-4b68-8b4f-1d8f673ceacd" containerName="kube-rbac-proxy" containerID="cri-o://d80a26a5f4049fb37a3b9fc41f9afabd98f9d5123ae75c57f2d948f762c003f2" gracePeriod=30
Sep 18 20:00:08.550486 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:00:08.550394    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc" podUID="343a5d81-88f0-4b68-8b4f-1d8f673ceacd" containerName="kserve-agent" containerID="cri-o://7b1f2cf402a0d745684ebd45bb23319a1a1e5892ebc34ee3d6c69089e2cd7961" gracePeriod=30
Sep 18 20:00:08.792083 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:00:08.792052    2522 generic.go:356] "Generic (PLEG): container finished" podID="343a5d81-88f0-4b68-8b4f-1d8f673ceacd" containerID="d80a26a5f4049fb37a3b9fc41f9afabd98f9d5123ae75c57f2d948f762c003f2" exitCode=2
Sep 18 20:00:08.792251 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:00:08.792126    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc" event={"ID":"343a5d81-88f0-4b68-8b4f-1d8f673ceacd","Type":"ContainerDied","Data":"d80a26a5f4049fb37a3b9fc41f9afabd98f9d5123ae75c57f2d948f762c003f2"}
Sep 18 20:00:10.810631 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:00:10.810527    2522 generic.go:356] "Generic (PLEG): container finished" podID="343a5d81-88f0-4b68-8b4f-1d8f673ceacd" containerID="7427d62eb8e9336f2ca1f752c1734a6a512daed5e0ba6e818c874fc859fa893a" exitCode=0
Sep 18 20:00:10.810631 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:00:10.810596    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc" event={"ID":"343a5d81-88f0-4b68-8b4f-1d8f673ceacd","Type":"ContainerDied","Data":"7427d62eb8e9336f2ca1f752c1734a6a512daed5e0ba6e818c874fc859fa893a"}
Sep 18 20:00:13.467938 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:00:13.467888    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc" podUID="343a5d81-88f0-4b68-8b4f-1d8f673ceacd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.57:8643/healthz\": dial tcp 10.132.0.57:8643: connect: connection refused"
Sep 18 20:00:13.474364 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:00:13.474337    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc" podUID="343a5d81-88f0-4b68-8b4f-1d8f673ceacd" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.57:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 18 20:00:18.467970 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:00:18.467926    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc" podUID="343a5d81-88f0-4b68-8b4f-1d8f673ceacd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.57:8643/healthz\": dial tcp 10.132.0.57:8643: connect: connection refused"
Sep 18 20:00:23.467452 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:00:23.467356    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc" podUID="343a5d81-88f0-4b68-8b4f-1d8f673ceacd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.57:8643/healthz\": dial tcp 10.132.0.57:8643: connect: connection refused"
Sep 18 20:00:23.467844 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:00:23.467493    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc"
Sep 18 20:00:23.474691 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:00:23.474654    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc" podUID="343a5d81-88f0-4b68-8b4f-1d8f673ceacd" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.57:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 18 20:00:28.467576 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:00:28.467531    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc" podUID="343a5d81-88f0-4b68-8b4f-1d8f673ceacd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.57:8643/healthz\": dial tcp 10.132.0.57:8643: connect: connection refused"
Sep 18 20:00:33.467686 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:00:33.467647    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc" podUID="343a5d81-88f0-4b68-8b4f-1d8f673ceacd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.57:8643/healthz\": dial tcp 10.132.0.57:8643: connect: connection refused"
Sep 18 20:00:33.473979 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:00:33.473950    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc" podUID="343a5d81-88f0-4b68-8b4f-1d8f673ceacd" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.57:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 18 20:00:33.474072 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:00:33.474059    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc"
Sep 18 20:00:38.467956 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:00:38.467912    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc" podUID="343a5d81-88f0-4b68-8b4f-1d8f673ceacd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.57:8643/healthz\": dial tcp 10.132.0.57:8643: connect: connection refused"
Sep 18 20:00:38.697767 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:00:38.697742    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc"
Sep 18 20:00:38.763747 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:00:38.763681    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lpxm6\" (UniqueName: \"kubernetes.io/projected/343a5d81-88f0-4b68-8b4f-1d8f673ceacd-kube-api-access-lpxm6\") pod \"343a5d81-88f0-4b68-8b4f-1d8f673ceacd\" (UID: \"343a5d81-88f0-4b68-8b4f-1d8f673ceacd\") "
Sep 18 20:00:38.763747 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:00:38.763732    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/343a5d81-88f0-4b68-8b4f-1d8f673ceacd-cabundle-cert\") pod \"343a5d81-88f0-4b68-8b4f-1d8f673ceacd\" (UID: \"343a5d81-88f0-4b68-8b4f-1d8f673ceacd\") "
Sep 18 20:00:38.763958 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:00:38.763755    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/343a5d81-88f0-4b68-8b4f-1d8f673ceacd-proxy-tls\") pod \"343a5d81-88f0-4b68-8b4f-1d8f673ceacd\" (UID: \"343a5d81-88f0-4b68-8b4f-1d8f673ceacd\") "
Sep 18 20:00:38.763958 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:00:38.763827    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/343a5d81-88f0-4b68-8b4f-1d8f673ceacd-kserve-provision-location\") pod \"343a5d81-88f0-4b68-8b4f-1d8f673ceacd\" (UID: \"343a5d81-88f0-4b68-8b4f-1d8f673ceacd\") "
Sep 18 20:00:38.763958 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:00:38.763851    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/343a5d81-88f0-4b68-8b4f-1d8f673ceacd-kube-rbac-proxy-sar-config\") pod \"343a5d81-88f0-4b68-8b4f-1d8f673ceacd\" (UID: \"343a5d81-88f0-4b68-8b4f-1d8f673ceacd\") "
Sep 18 20:00:38.764141 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:00:38.764110    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/343a5d81-88f0-4b68-8b4f-1d8f673ceacd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "343a5d81-88f0-4b68-8b4f-1d8f673ceacd" (UID: "343a5d81-88f0-4b68-8b4f-1d8f673ceacd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:00:38.764203 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:00:38.764171    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/343a5d81-88f0-4b68-8b4f-1d8f673ceacd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "343a5d81-88f0-4b68-8b4f-1d8f673ceacd" (UID: "343a5d81-88f0-4b68-8b4f-1d8f673ceacd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:00:38.764246 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:00:38.764223    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/343a5d81-88f0-4b68-8b4f-1d8f673ceacd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "343a5d81-88f0-4b68-8b4f-1d8f673ceacd" (UID: "343a5d81-88f0-4b68-8b4f-1d8f673ceacd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:00:38.765876 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:00:38.765854    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/343a5d81-88f0-4b68-8b4f-1d8f673ceacd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "343a5d81-88f0-4b68-8b4f-1d8f673ceacd" (UID: "343a5d81-88f0-4b68-8b4f-1d8f673ceacd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:00:38.765979 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:00:38.765892    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/343a5d81-88f0-4b68-8b4f-1d8f673ceacd-kube-api-access-lpxm6" (OuterVolumeSpecName: "kube-api-access-lpxm6") pod "343a5d81-88f0-4b68-8b4f-1d8f673ceacd" (UID: "343a5d81-88f0-4b68-8b4f-1d8f673ceacd"). InnerVolumeSpecName "kube-api-access-lpxm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:00:38.865365 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:00:38.865339    2522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpxm6\" (UniqueName: \"kubernetes.io/projected/343a5d81-88f0-4b68-8b4f-1d8f673ceacd-kube-api-access-lpxm6\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 20:00:38.865365 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:00:38.865364    2522 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/343a5d81-88f0-4b68-8b4f-1d8f673ceacd-cabundle-cert\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 20:00:38.865537 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:00:38.865376    2522 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/343a5d81-88f0-4b68-8b4f-1d8f673ceacd-proxy-tls\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 20:00:38.865537 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:00:38.865402    2522 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/343a5d81-88f0-4b68-8b4f-1d8f673ceacd-kserve-provision-location\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 20:00:38.865537 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:00:38.865412    2522 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/343a5d81-88f0-4b68-8b4f-1d8f673ceacd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 20:00:38.948809 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:00:38.948777    2522 generic.go:356] "Generic (PLEG): container finished" podID="343a5d81-88f0-4b68-8b4f-1d8f673ceacd" containerID="7b1f2cf402a0d745684ebd45bb23319a1a1e5892ebc34ee3d6c69089e2cd7961" exitCode=0
Sep 18 20:00:38.948959 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:00:38.948838    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc" event={"ID":"343a5d81-88f0-4b68-8b4f-1d8f673ceacd","Type":"ContainerDied","Data":"7b1f2cf402a0d745684ebd45bb23319a1a1e5892ebc34ee3d6c69089e2cd7961"}
Sep 18 20:00:38.948959 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:00:38.948848    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc"
Sep 18 20:00:38.948959 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:00:38.948933    2522 scope.go:117] "RemoveContainer" containerID="d80a26a5f4049fb37a3b9fc41f9afabd98f9d5123ae75c57f2d948f762c003f2"
Sep 18 20:00:38.949084 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:00:38.948923    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc" event={"ID":"343a5d81-88f0-4b68-8b4f-1d8f673ceacd","Type":"ContainerDied","Data":"9c42d1ebdb72ea8e223eea9bc2a9caf63631611bad7cdf8111ec7ebbfe28ad5b"}
Sep 18 20:00:38.958433 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:00:38.958415    2522 scope.go:117] "RemoveContainer" containerID="7b1f2cf402a0d745684ebd45bb23319a1a1e5892ebc34ee3d6c69089e2cd7961"
Sep 18 20:00:38.965721 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:00:38.965702    2522 scope.go:117] "RemoveContainer" containerID="7427d62eb8e9336f2ca1f752c1734a6a512daed5e0ba6e818c874fc859fa893a"
Sep 18 20:00:38.973056 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:00:38.973039    2522 scope.go:117] "RemoveContainer" containerID="5ac5d5314a7cc67aa6cedab072c7f4750940db624ff933d220f3f8ef1fcc45d5"
Sep 18 20:00:38.973321 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:00:38.973303    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc"]
Sep 18 20:00:38.976803 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:00:38.976782    2522 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7f8b8b68f-z6lrc"]
Sep 18 20:00:38.980610 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:00:38.980588    2522 scope.go:117] "RemoveContainer" containerID="d80a26a5f4049fb37a3b9fc41f9afabd98f9d5123ae75c57f2d948f762c003f2"
Sep 18 20:00:38.980886 ip-10-0-131-221 kubenswrapper[2522]: E0918 20:00:38.980868    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d80a26a5f4049fb37a3b9fc41f9afabd98f9d5123ae75c57f2d948f762c003f2\": container with ID starting with d80a26a5f4049fb37a3b9fc41f9afabd98f9d5123ae75c57f2d948f762c003f2 not found: ID does not exist" containerID="d80a26a5f4049fb37a3b9fc41f9afabd98f9d5123ae75c57f2d948f762c003f2"
Sep 18 20:00:38.980951 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:00:38.980893    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d80a26a5f4049fb37a3b9fc41f9afabd98f9d5123ae75c57f2d948f762c003f2"} err="failed to get container status \"d80a26a5f4049fb37a3b9fc41f9afabd98f9d5123ae75c57f2d948f762c003f2\": rpc error: code = NotFound desc = could not find container \"d80a26a5f4049fb37a3b9fc41f9afabd98f9d5123ae75c57f2d948f762c003f2\": container with ID starting with d80a26a5f4049fb37a3b9fc41f9afabd98f9d5123ae75c57f2d948f762c003f2 not found: ID does not exist"
Sep 18 20:00:38.980951 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:00:38.980911    2522 scope.go:117] "RemoveContainer" containerID="7b1f2cf402a0d745684ebd45bb23319a1a1e5892ebc34ee3d6c69089e2cd7961"
Sep 18 20:00:38.981167 ip-10-0-131-221 kubenswrapper[2522]: E0918 20:00:38.981149    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b1f2cf402a0d745684ebd45bb23319a1a1e5892ebc34ee3d6c69089e2cd7961\": container with ID starting with 7b1f2cf402a0d745684ebd45bb23319a1a1e5892ebc34ee3d6c69089e2cd7961 not found: ID does not exist" containerID="7b1f2cf402a0d745684ebd45bb23319a1a1e5892ebc34ee3d6c69089e2cd7961"
Sep 18 20:00:38.981211 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:00:38.981172    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b1f2cf402a0d745684ebd45bb23319a1a1e5892ebc34ee3d6c69089e2cd7961"} err="failed to get container status \"7b1f2cf402a0d745684ebd45bb23319a1a1e5892ebc34ee3d6c69089e2cd7961\": rpc error: code = NotFound desc = could not find container \"7b1f2cf402a0d745684ebd45bb23319a1a1e5892ebc34ee3d6c69089e2cd7961\": container with ID starting with 7b1f2cf402a0d745684ebd45bb23319a1a1e5892ebc34ee3d6c69089e2cd7961 not found: ID does not exist"
Sep 18 20:00:38.981211 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:00:38.981188    2522 scope.go:117] "RemoveContainer" containerID="7427d62eb8e9336f2ca1f752c1734a6a512daed5e0ba6e818c874fc859fa893a"
Sep 18 20:00:38.981438 ip-10-0-131-221 kubenswrapper[2522]: E0918 20:00:38.981422    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7427d62eb8e9336f2ca1f752c1734a6a512daed5e0ba6e818c874fc859fa893a\": container with ID starting with 7427d62eb8e9336f2ca1f752c1734a6a512daed5e0ba6e818c874fc859fa893a not found: ID does not exist" containerID="7427d62eb8e9336f2ca1f752c1734a6a512daed5e0ba6e818c874fc859fa893a"
Sep 18 20:00:38.981476 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:00:38.981441    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7427d62eb8e9336f2ca1f752c1734a6a512daed5e0ba6e818c874fc859fa893a"} err="failed to get container status \"7427d62eb8e9336f2ca1f752c1734a6a512daed5e0ba6e818c874fc859fa893a\": rpc error: code = NotFound desc = could not find container \"7427d62eb8e9336f2ca1f752c1734a6a512daed5e0ba6e818c874fc859fa893a\": container with ID starting with 7427d62eb8e9336f2ca1f752c1734a6a512daed5e0ba6e818c874fc859fa893a not found: ID does not exist"
Sep 18 20:00:38.981476 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:00:38.981454    2522 scope.go:117] "RemoveContainer" containerID="5ac5d5314a7cc67aa6cedab072c7f4750940db624ff933d220f3f8ef1fcc45d5"
Sep 18 20:00:38.981700 ip-10-0-131-221 kubenswrapper[2522]: E0918 20:00:38.981679    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ac5d5314a7cc67aa6cedab072c7f4750940db624ff933d220f3f8ef1fcc45d5\": container with ID starting with 5ac5d5314a7cc67aa6cedab072c7f4750940db624ff933d220f3f8ef1fcc45d5 not found: ID does not exist" containerID="5ac5d5314a7cc67aa6cedab072c7f4750940db624ff933d220f3f8ef1fcc45d5"
Sep 18 20:00:38.981825 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:00:38.981704    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ac5d5314a7cc67aa6cedab072c7f4750940db624ff933d220f3f8ef1fcc45d5"} err="failed to get container status \"5ac5d5314a7cc67aa6cedab072c7f4750940db624ff933d220f3f8ef1fcc45d5\": rpc error: code = NotFound desc = could not find container \"5ac5d5314a7cc67aa6cedab072c7f4750940db624ff933d220f3f8ef1fcc45d5\": container with ID starting with 5ac5d5314a7cc67aa6cedab072c7f4750940db624ff933d220f3f8ef1fcc45d5 not found: ID does not exist"
Sep 18 20:00:40.721658 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:00:40.721628    2522 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="343a5d81-88f0-4b68-8b4f-1d8f673ceacd" path="/var/lib/kubelet/pods/343a5d81-88f0-4b68-8b4f-1d8f673ceacd/volumes"
Sep 18 20:02:02.504624 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:02.504515    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-55f6d867dc-hjf4q"]
Sep 18 20:02:02.505192 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:02.505031    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="343a5d81-88f0-4b68-8b4f-1d8f673ceacd" containerName="kserve-agent"
Sep 18 20:02:02.505192 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:02.505051    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="343a5d81-88f0-4b68-8b4f-1d8f673ceacd" containerName="kserve-agent"
Sep 18 20:02:02.505192 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:02.505082    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="343a5d81-88f0-4b68-8b4f-1d8f673ceacd" containerName="kube-rbac-proxy"
Sep 18 20:02:02.505192 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:02.505091    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="343a5d81-88f0-4b68-8b4f-1d8f673ceacd" containerName="kube-rbac-proxy"
Sep 18 20:02:02.505192 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:02.505107    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="343a5d81-88f0-4b68-8b4f-1d8f673ceacd" containerName="storage-initializer"
Sep 18 20:02:02.505192 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:02.505116    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="343a5d81-88f0-4b68-8b4f-1d8f673ceacd" containerName="storage-initializer"
Sep 18 20:02:02.505192 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:02.505138    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="343a5d81-88f0-4b68-8b4f-1d8f673ceacd" containerName="kserve-container"
Sep 18 20:02:02.505192 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:02.505146    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="343a5d81-88f0-4b68-8b4f-1d8f673ceacd" containerName="kserve-container"
Sep 18 20:02:02.505645 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:02.505227    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="343a5d81-88f0-4b68-8b4f-1d8f673ceacd" containerName="kube-rbac-proxy"
Sep 18 20:02:02.505645 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:02.505244    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="343a5d81-88f0-4b68-8b4f-1d8f673ceacd" containerName="kserve-container"
Sep 18 20:02:02.505645 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:02.505253    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="343a5d81-88f0-4b68-8b4f-1d8f673ceacd" containerName="kserve-agent"
Sep 18 20:02:02.508584 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:02.508550    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-55f6d867dc-hjf4q"
Sep 18 20:02:02.511260 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:02.511226    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\""
Sep 18 20:02:02.511371 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:02.511249    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\""
Sep 18 20:02:02.511371 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:02.511353    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 18 20:02:02.511498 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:02.511314    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 20:02:02.511595 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:02.511579    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 20:02:02.512825 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:02.512803    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5b562\""
Sep 18 20:02:02.524369 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:02.524345    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-55f6d867dc-hjf4q"]
Sep 18 20:02:02.639622 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:02.639582    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz2zw\" (UniqueName: \"kubernetes.io/projected/ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807-kube-api-access-cz2zw\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-55f6d867dc-hjf4q\" (UID: \"ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-55f6d867dc-hjf4q"
Sep 18 20:02:02.639791 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:02.639688    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-55f6d867dc-hjf4q\" (UID: \"ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-55f6d867dc-hjf4q"
Sep 18 20:02:02.639791 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:02.639722    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-55f6d867dc-hjf4q\" (UID: \"ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-55f6d867dc-hjf4q"
Sep 18 20:02:02.639791 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:02.639757    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-55f6d867dc-hjf4q\" (UID: \"ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-55f6d867dc-hjf4q"
Sep 18 20:02:02.741004 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:02.740974    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-55f6d867dc-hjf4q\" (UID: \"ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-55f6d867dc-hjf4q"
Sep 18 20:02:02.741172 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:02.741019    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-55f6d867dc-hjf4q\" (UID: \"ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-55f6d867dc-hjf4q"
Sep 18 20:02:02.741172 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:02.741041    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz2zw\" (UniqueName: \"kubernetes.io/projected/ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807-kube-api-access-cz2zw\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-55f6d867dc-hjf4q\" (UID: \"ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-55f6d867dc-hjf4q"
Sep 18 20:02:02.741172 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:02.741083    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-55f6d867dc-hjf4q\" (UID: \"ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-55f6d867dc-hjf4q"
Sep 18 20:02:02.741172 ip-10-0-131-221 kubenswrapper[2522]: E0918 20:02:02.741100    2522 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-global-fail-predictor-serving-cert" not found
Sep 18 20:02:02.741405 ip-10-0-131-221 kubenswrapper[2522]: E0918 20:02:02.741180    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807-proxy-tls podName:ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807 nodeName:}" failed. No retries permitted until 2026-09-18 20:02:03.241157133 +0000 UTC m=+2883.124683579 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807-proxy-tls") pod "isvc-sklearn-s3-tls-global-fail-predictor-55f6d867dc-hjf4q" (UID: "ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807") : secret "isvc-sklearn-s3-tls-global-fail-predictor-serving-cert" not found
Sep 18 20:02:02.741471 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:02.741445    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-55f6d867dc-hjf4q\" (UID: \"ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-55f6d867dc-hjf4q"
Sep 18 20:02:02.741816 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:02.741799    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-55f6d867dc-hjf4q\" (UID: \"ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-55f6d867dc-hjf4q"
Sep 18 20:02:02.750911 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:02.750884    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz2zw\" (UniqueName: \"kubernetes.io/projected/ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807-kube-api-access-cz2zw\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-55f6d867dc-hjf4q\" (UID: \"ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-55f6d867dc-hjf4q"
Sep 18 20:02:03.245708 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:03.245674    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-55f6d867dc-hjf4q\" (UID: \"ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-55f6d867dc-hjf4q"
Sep 18 20:02:03.248030 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:03.247998    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-55f6d867dc-hjf4q\" (UID: \"ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-55f6d867dc-hjf4q"
Sep 18 20:02:03.418435 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:03.418403    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-55f6d867dc-hjf4q"
Sep 18 20:02:03.692853 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:03.692828    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-55f6d867dc-hjf4q"]
Sep 18 20:02:03.695188 ip-10-0-131-221 kubenswrapper[2522]: W0918 20:02:03.695158    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebb5c3fd_491c_4ee4_ba76_b7d3e97c1807.slice/crio-f6bba6cb3f5a86d05b9dcbf06c96b2810ac0c809071f17ee7ee35824e030925b WatchSource:0}: Error finding container f6bba6cb3f5a86d05b9dcbf06c96b2810ac0c809071f17ee7ee35824e030925b: Status 404 returned error can't find the container with id f6bba6cb3f5a86d05b9dcbf06c96b2810ac0c809071f17ee7ee35824e030925b
Sep 18 20:02:04.337154 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:04.337116    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-55f6d867dc-hjf4q" event={"ID":"ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807","Type":"ContainerStarted","Data":"bbe001dcc4f755530e7f4b41427a16dec753adb0c358a873de09dcd5df4e5049"}
Sep 18 20:02:04.337331 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:04.337208    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-55f6d867dc-hjf4q" event={"ID":"ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807","Type":"ContainerStarted","Data":"f6bba6cb3f5a86d05b9dcbf06c96b2810ac0c809071f17ee7ee35824e030925b"}
Sep 18 20:02:09.361248 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:09.361222    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-55f6d867dc-hjf4q_ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807/storage-initializer/0.log"
Sep 18 20:02:09.361650 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:09.361263    2522 generic.go:356] "Generic (PLEG): container finished" podID="ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807" containerID="bbe001dcc4f755530e7f4b41427a16dec753adb0c358a873de09dcd5df4e5049" exitCode=1
Sep 18 20:02:09.361650 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:09.361307    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-55f6d867dc-hjf4q" event={"ID":"ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807","Type":"ContainerDied","Data":"bbe001dcc4f755530e7f4b41427a16dec753adb0c358a873de09dcd5df4e5049"}
Sep 18 20:02:10.373737 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:10.373702    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-55f6d867dc-hjf4q_ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807/storage-initializer/0.log"
Sep 18 20:02:10.374126 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:10.373907    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-55f6d867dc-hjf4q" event={"ID":"ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807","Type":"ContainerStarted","Data":"17fd27a06d1526a4276ae15dc177538095411f6c4cb379ff7e611e7c9c37e374"}
Sep 18 20:02:12.660884 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:12.660847    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-55f6d867dc-hjf4q"]
Sep 18 20:02:12.661286 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:12.661077    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-55f6d867dc-hjf4q" podUID="ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807" containerName="storage-initializer" containerID="cri-o://17fd27a06d1526a4276ae15dc177538095411f6c4cb379ff7e611e7c9c37e374" gracePeriod=30
Sep 18 20:02:14.598856 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:14.598832    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-55f6d867dc-hjf4q_ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807/storage-initializer/1.log"
Sep 18 20:02:14.599296 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:14.599281    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-55f6d867dc-hjf4q_ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807/storage-initializer/0.log"
Sep 18 20:02:14.601598 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:14.601584    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-55f6d867dc-hjf4q"
Sep 18 20:02:14.746403 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:14.746333    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807-proxy-tls\") pod \"ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807\" (UID: \"ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807\") "
Sep 18 20:02:14.746403 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:14.746393    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807-kserve-provision-location\") pod \"ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807\" (UID: \"ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807\") "
Sep 18 20:02:14.746610 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:14.746422    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cz2zw\" (UniqueName: \"kubernetes.io/projected/ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807-kube-api-access-cz2zw\") pod \"ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807\" (UID: \"ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807\") "
Sep 18 20:02:14.746610 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:14.746446    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807-kube-rbac-proxy-sar-config\") pod \"ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807\" (UID: \"ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807\") "
Sep 18 20:02:14.746793 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:14.746739    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807" (UID: "ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:02:14.746904 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:14.746881    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807" (UID: "ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:02:14.748443 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:14.748418    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807" (UID: "ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:02:14.748582 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:14.748540    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807-kube-api-access-cz2zw" (OuterVolumeSpecName: "kube-api-access-cz2zw") pod "ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807" (UID: "ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807"). InnerVolumeSpecName "kube-api-access-cz2zw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:02:14.847794 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:14.847766    2522 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807-proxy-tls\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 20:02:14.847794 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:14.847791    2522 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807-kserve-provision-location\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 20:02:14.847794 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:14.847804    2522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cz2zw\" (UniqueName: \"kubernetes.io/projected/ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807-kube-api-access-cz2zw\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 20:02:14.848025 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:14.847813    2522 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 20:02:15.397490 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:15.397460    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-55f6d867dc-hjf4q_ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807/storage-initializer/1.log"
Sep 18 20:02:15.397862 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:15.397846    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-55f6d867dc-hjf4q_ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807/storage-initializer/0.log"
Sep 18 20:02:15.397933 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:15.397886    2522 generic.go:356] "Generic (PLEG): container finished" podID="ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807" containerID="17fd27a06d1526a4276ae15dc177538095411f6c4cb379ff7e611e7c9c37e374" exitCode=1
Sep 18 20:02:15.398015 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:15.397923    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-55f6d867dc-hjf4q" event={"ID":"ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807","Type":"ContainerDied","Data":"17fd27a06d1526a4276ae15dc177538095411f6c4cb379ff7e611e7c9c37e374"}
Sep 18 20:02:15.398015 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:15.397951    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-55f6d867dc-hjf4q"
Sep 18 20:02:15.398015 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:15.398004    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-55f6d867dc-hjf4q" event={"ID":"ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807","Type":"ContainerDied","Data":"f6bba6cb3f5a86d05b9dcbf06c96b2810ac0c809071f17ee7ee35824e030925b"}
Sep 18 20:02:15.398175 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:15.398046    2522 scope.go:117] "RemoveContainer" containerID="17fd27a06d1526a4276ae15dc177538095411f6c4cb379ff7e611e7c9c37e374"
Sep 18 20:02:15.407383 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:15.407364    2522 scope.go:117] "RemoveContainer" containerID="bbe001dcc4f755530e7f4b41427a16dec753adb0c358a873de09dcd5df4e5049"
Sep 18 20:02:15.414645 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:15.414626    2522 scope.go:117] "RemoveContainer" containerID="17fd27a06d1526a4276ae15dc177538095411f6c4cb379ff7e611e7c9c37e374"
Sep 18 20:02:15.414897 ip-10-0-131-221 kubenswrapper[2522]: E0918 20:02:15.414878    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17fd27a06d1526a4276ae15dc177538095411f6c4cb379ff7e611e7c9c37e374\": container with ID starting with 17fd27a06d1526a4276ae15dc177538095411f6c4cb379ff7e611e7c9c37e374 not found: ID does not exist" containerID="17fd27a06d1526a4276ae15dc177538095411f6c4cb379ff7e611e7c9c37e374"
Sep 18 20:02:15.414964 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:15.414905    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17fd27a06d1526a4276ae15dc177538095411f6c4cb379ff7e611e7c9c37e374"} err="failed to get container status \"17fd27a06d1526a4276ae15dc177538095411f6c4cb379ff7e611e7c9c37e374\": rpc error: code = NotFound desc = could not find container \"17fd27a06d1526a4276ae15dc177538095411f6c4cb379ff7e611e7c9c37e374\": container with ID starting with 17fd27a06d1526a4276ae15dc177538095411f6c4cb379ff7e611e7c9c37e374 not found: ID does not exist"
Sep 18 20:02:15.414964 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:15.414921    2522 scope.go:117] "RemoveContainer" containerID="bbe001dcc4f755530e7f4b41427a16dec753adb0c358a873de09dcd5df4e5049"
Sep 18 20:02:15.415152 ip-10-0-131-221 kubenswrapper[2522]: E0918 20:02:15.415135    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbe001dcc4f755530e7f4b41427a16dec753adb0c358a873de09dcd5df4e5049\": container with ID starting with bbe001dcc4f755530e7f4b41427a16dec753adb0c358a873de09dcd5df4e5049 not found: ID does not exist" containerID="bbe001dcc4f755530e7f4b41427a16dec753adb0c358a873de09dcd5df4e5049"
Sep 18 20:02:15.415193 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:15.415155    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbe001dcc4f755530e7f4b41427a16dec753adb0c358a873de09dcd5df4e5049"} err="failed to get container status \"bbe001dcc4f755530e7f4b41427a16dec753adb0c358a873de09dcd5df4e5049\": rpc error: code = NotFound desc = could not find container \"bbe001dcc4f755530e7f4b41427a16dec753adb0c358a873de09dcd5df4e5049\": container with ID starting with bbe001dcc4f755530e7f4b41427a16dec753adb0c358a873de09dcd5df4e5049 not found: ID does not exist"
Sep 18 20:02:15.439908 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:15.439886    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-55f6d867dc-hjf4q"]
Sep 18 20:02:15.445998 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:15.445979    2522 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-55f6d867dc-hjf4q"]
Sep 18 20:02:16.726940 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:02:16.726907    2522 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807" path="/var/lib/kubelet/pods/ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807/volumes"
Sep 18 20:03:54.838462 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:03:54.838427    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-74d5d6bcf-f6pb7"]
Sep 18 20:03:54.839045 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:03:54.838805    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807" containerName="storage-initializer"
Sep 18 20:03:54.839045 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:03:54.838816    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807" containerName="storage-initializer"
Sep 18 20:03:54.839045 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:03:54.838828    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807" containerName="storage-initializer"
Sep 18 20:03:54.839045 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:03:54.838833    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807" containerName="storage-initializer"
Sep 18 20:03:54.839045 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:03:54.838892    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807" containerName="storage-initializer"
Sep 18 20:03:54.839045 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:03:54.838988    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebb5c3fd-491c-4ee4-ba76-b7d3e97c1807" containerName="storage-initializer"
Sep 18 20:03:54.847182 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:03:54.847152    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-74d5d6bcf-f6pb7"
Sep 18 20:03:54.849993 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:03:54.849972    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5b562\""
Sep 18 20:03:54.850113 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:03:54.850074    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\""
Sep 18 20:03:54.850327 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:03:54.850308    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 20:03:54.851051 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:03:54.851032    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\""
Sep 18 20:03:54.851159 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:03:54.851035    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 18 20:03:54.851159 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:03:54.851064    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 20:03:54.858321 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:03:54.858299    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-74d5d6bcf-f6pb7"]
Sep 18 20:03:54.917968 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:03:54.917942    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9j7h\" (UniqueName: \"kubernetes.io/projected/7b1fab5e-c65f-4f66-9aa5-b5a2c6217420-kube-api-access-h9j7h\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-74d5d6bcf-f6pb7\" (UID: \"7b1fab5e-c65f-4f66-9aa5-b5a2c6217420\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-74d5d6bcf-f6pb7"
Sep 18 20:03:54.918096 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:03:54.917990    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b1fab5e-c65f-4f66-9aa5-b5a2c6217420-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-74d5d6bcf-f6pb7\" (UID: \"7b1fab5e-c65f-4f66-9aa5-b5a2c6217420\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-74d5d6bcf-f6pb7"
Sep 18 20:03:54.918096 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:03:54.918023    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b1fab5e-c65f-4f66-9aa5-b5a2c6217420-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-74d5d6bcf-f6pb7\" (UID: \"7b1fab5e-c65f-4f66-9aa5-b5a2c6217420\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-74d5d6bcf-f6pb7"
Sep 18 20:03:54.918171 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:03:54.918088    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b1fab5e-c65f-4f66-9aa5-b5a2c6217420-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-74d5d6bcf-f6pb7\" (UID: \"7b1fab5e-c65f-4f66-9aa5-b5a2c6217420\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-74d5d6bcf-f6pb7"
Sep 18 20:03:55.018457 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:03:55.018425    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b1fab5e-c65f-4f66-9aa5-b5a2c6217420-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-74d5d6bcf-f6pb7\" (UID: \"7b1fab5e-c65f-4f66-9aa5-b5a2c6217420\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-74d5d6bcf-f6pb7"
Sep 18 20:03:55.018611 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:03:55.018462    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b1fab5e-c65f-4f66-9aa5-b5a2c6217420-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-74d5d6bcf-f6pb7\" (UID: \"7b1fab5e-c65f-4f66-9aa5-b5a2c6217420\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-74d5d6bcf-f6pb7"
Sep 18 20:03:55.018611 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:03:55.018489    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b1fab5e-c65f-4f66-9aa5-b5a2c6217420-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-74d5d6bcf-f6pb7\" (UID: \"7b1fab5e-c65f-4f66-9aa5-b5a2c6217420\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-74d5d6bcf-f6pb7"
Sep 18 20:03:55.018611 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:03:55.018534    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9j7h\" (UniqueName: \"kubernetes.io/projected/7b1fab5e-c65f-4f66-9aa5-b5a2c6217420-kube-api-access-h9j7h\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-74d5d6bcf-f6pb7\" (UID: \"7b1fab5e-c65f-4f66-9aa5-b5a2c6217420\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-74d5d6bcf-f6pb7"
Sep 18 20:03:55.018869 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:03:55.018852    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b1fab5e-c65f-4f66-9aa5-b5a2c6217420-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-74d5d6bcf-f6pb7\" (UID: \"7b1fab5e-c65f-4f66-9aa5-b5a2c6217420\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-74d5d6bcf-f6pb7"
Sep 18 20:03:55.019144 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:03:55.019118    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b1fab5e-c65f-4f66-9aa5-b5a2c6217420-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-74d5d6bcf-f6pb7\" (UID: \"7b1fab5e-c65f-4f66-9aa5-b5a2c6217420\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-74d5d6bcf-f6pb7"
Sep 18 20:03:55.020931 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:03:55.020906    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b1fab5e-c65f-4f66-9aa5-b5a2c6217420-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-74d5d6bcf-f6pb7\" (UID: \"7b1fab5e-c65f-4f66-9aa5-b5a2c6217420\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-74d5d6bcf-f6pb7"
Sep 18 20:03:55.027045 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:03:55.027013    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9j7h\" (UniqueName: \"kubernetes.io/projected/7b1fab5e-c65f-4f66-9aa5-b5a2c6217420-kube-api-access-h9j7h\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-74d5d6bcf-f6pb7\" (UID: \"7b1fab5e-c65f-4f66-9aa5-b5a2c6217420\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-74d5d6bcf-f6pb7"
Sep 18 20:03:55.160416 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:03:55.160330    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-74d5d6bcf-f6pb7"
Sep 18 20:03:55.436709 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:03:55.436681    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-74d5d6bcf-f6pb7"]
Sep 18 20:03:55.437968 ip-10-0-131-221 kubenswrapper[2522]: W0918 20:03:55.437938    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b1fab5e_c65f_4f66_9aa5_b5a2c6217420.slice/crio-34b29eca93f71a0b9d3c6b285f4b64440833bc1ee2cf2ae678d55f408c26c035 WatchSource:0}: Error finding container 34b29eca93f71a0b9d3c6b285f4b64440833bc1ee2cf2ae678d55f408c26c035: Status 404 returned error can't find the container with id 34b29eca93f71a0b9d3c6b285f4b64440833bc1ee2cf2ae678d55f408c26c035
Sep 18 20:03:55.842013 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:03:55.841973    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-74d5d6bcf-f6pb7" event={"ID":"7b1fab5e-c65f-4f66-9aa5-b5a2c6217420","Type":"ContainerStarted","Data":"becfd3cf73a6c202e60c6b79580c1cd78808b56216a8a6d111527d1dbf7fe9b8"}
Sep 18 20:03:55.842392 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:03:55.842049    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-74d5d6bcf-f6pb7" event={"ID":"7b1fab5e-c65f-4f66-9aa5-b5a2c6217420","Type":"ContainerStarted","Data":"34b29eca93f71a0b9d3c6b285f4b64440833bc1ee2cf2ae678d55f408c26c035"}
Sep 18 20:03:59.864895 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:03:59.864867    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-74d5d6bcf-f6pb7_7b1fab5e-c65f-4f66-9aa5-b5a2c6217420/storage-initializer/0.log"
Sep 18 20:03:59.865356 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:03:59.864900    2522 generic.go:356] "Generic (PLEG): container finished" podID="7b1fab5e-c65f-4f66-9aa5-b5a2c6217420" containerID="becfd3cf73a6c202e60c6b79580c1cd78808b56216a8a6d111527d1dbf7fe9b8" exitCode=1
Sep 18 20:03:59.865356 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:03:59.864982    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-74d5d6bcf-f6pb7" event={"ID":"7b1fab5e-c65f-4f66-9aa5-b5a2c6217420","Type":"ContainerDied","Data":"becfd3cf73a6c202e60c6b79580c1cd78808b56216a8a6d111527d1dbf7fe9b8"}
Sep 18 20:04:00.782665 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:04:00.782637    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-74d5d6bcf-f6pb7_7b1fab5e-c65f-4f66-9aa5-b5a2c6217420/storage-initializer/0.log"
Sep 18 20:04:00.790149 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:04:00.790128    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-74d5d6bcf-f6pb7_7b1fab5e-c65f-4f66-9aa5-b5a2c6217420/storage-initializer/0.log"
Sep 18 20:04:00.872188 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:04:00.872159    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-74d5d6bcf-f6pb7_7b1fab5e-c65f-4f66-9aa5-b5a2c6217420/storage-initializer/0.log"
Sep 18 20:04:00.872684 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:04:00.872250    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-74d5d6bcf-f6pb7" event={"ID":"7b1fab5e-c65f-4f66-9aa5-b5a2c6217420","Type":"ContainerStarted","Data":"b7e3c812a95eb9ace88353a7f7f6258fb8c43f5c5aea5461703bbc86663f1350"}
Sep 18 20:04:03.888156 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:04:03.888126    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-74d5d6bcf-f6pb7_7b1fab5e-c65f-4f66-9aa5-b5a2c6217420/storage-initializer/1.log"
Sep 18 20:04:03.888644 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:04:03.888588    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-74d5d6bcf-f6pb7_7b1fab5e-c65f-4f66-9aa5-b5a2c6217420/storage-initializer/0.log"
Sep 18 20:04:03.888644 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:04:03.888628    2522 generic.go:356] "Generic (PLEG): container finished" podID="7b1fab5e-c65f-4f66-9aa5-b5a2c6217420" containerID="b7e3c812a95eb9ace88353a7f7f6258fb8c43f5c5aea5461703bbc86663f1350" exitCode=1
Sep 18 20:04:03.888763 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:04:03.888698    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-74d5d6bcf-f6pb7" event={"ID":"7b1fab5e-c65f-4f66-9aa5-b5a2c6217420","Type":"ContainerDied","Data":"b7e3c812a95eb9ace88353a7f7f6258fb8c43f5c5aea5461703bbc86663f1350"}
Sep 18 20:04:03.888763 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:04:03.888760    2522 scope.go:117] "RemoveContainer" containerID="becfd3cf73a6c202e60c6b79580c1cd78808b56216a8a6d111527d1dbf7fe9b8"
Sep 18 20:04:03.889156 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:04:03.889137    2522 scope.go:117] "RemoveContainer" containerID="becfd3cf73a6c202e60c6b79580c1cd78808b56216a8a6d111527d1dbf7fe9b8"
Sep 18 20:04:03.899449 ip-10-0-131-221 kubenswrapper[2522]: E0918 20:04:03.899428    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-custom-fail-predictor-74d5d6bcf-f6pb7_kserve-ci-e2e-test(7b1fab5e-c65f-4f66-9aa5-b5a2c6217420)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-74d5d6bcf-f6pb7" podUID="7b1fab5e-c65f-4f66-9aa5-b5a2c6217420"
Sep 18 20:04:03.899594 ip-10-0-131-221 kubenswrapper[2522]: E0918 20:04:03.899544    2522 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-custom-fail-predictor-74d5d6bcf-f6pb7_kserve-ci-e2e-test_7b1fab5e-c65f-4f66-9aa5-b5a2c6217420_0 in pod sandbox 34b29eca93f71a0b9d3c6b285f4b64440833bc1ee2cf2ae678d55f408c26c035 from index: no such id: 'becfd3cf73a6c202e60c6b79580c1cd78808b56216a8a6d111527d1dbf7fe9b8'" containerID="becfd3cf73a6c202e60c6b79580c1cd78808b56216a8a6d111527d1dbf7fe9b8"
Sep 18 20:04:03.899702 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:04:03.899606    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"becfd3cf73a6c202e60c6b79580c1cd78808b56216a8a6d111527d1dbf7fe9b8"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-custom-fail-predictor-74d5d6bcf-f6pb7_kserve-ci-e2e-test_7b1fab5e-c65f-4f66-9aa5-b5a2c6217420_0 in pod sandbox 34b29eca93f71a0b9d3c6b285f4b64440833bc1ee2cf2ae678d55f408c26c035 from index: no such id: 'becfd3cf73a6c202e60c6b79580c1cd78808b56216a8a6d111527d1dbf7fe9b8'"
Sep 18 20:04:04.868778 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:04:04.868748    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-74d5d6bcf-f6pb7"]
Sep 18 20:04:04.894911 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:04:04.894884    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-74d5d6bcf-f6pb7_7b1fab5e-c65f-4f66-9aa5-b5a2c6217420/storage-initializer/1.log"
Sep 18 20:04:05.027795 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:04:05.027770    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-74d5d6bcf-f6pb7_7b1fab5e-c65f-4f66-9aa5-b5a2c6217420/storage-initializer/1.log"
Sep 18 20:04:05.031188 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:04:05.031171    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-74d5d6bcf-f6pb7"
Sep 18 20:04:05.106193 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:04:05.106165    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b1fab5e-c65f-4f66-9aa5-b5a2c6217420-kserve-provision-location\") pod \"7b1fab5e-c65f-4f66-9aa5-b5a2c6217420\" (UID: \"7b1fab5e-c65f-4f66-9aa5-b5a2c6217420\") "
Sep 18 20:04:05.106330 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:04:05.106219    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b1fab5e-c65f-4f66-9aa5-b5a2c6217420-kube-rbac-proxy-sar-config\") pod \"7b1fab5e-c65f-4f66-9aa5-b5a2c6217420\" (UID: \"7b1fab5e-c65f-4f66-9aa5-b5a2c6217420\") "
Sep 18 20:04:05.106330 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:04:05.106260    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9j7h\" (UniqueName: \"kubernetes.io/projected/7b1fab5e-c65f-4f66-9aa5-b5a2c6217420-kube-api-access-h9j7h\") pod \"7b1fab5e-c65f-4f66-9aa5-b5a2c6217420\" (UID: \"7b1fab5e-c65f-4f66-9aa5-b5a2c6217420\") "
Sep 18 20:04:05.106330 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:04:05.106299    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b1fab5e-c65f-4f66-9aa5-b5a2c6217420-proxy-tls\") pod \"7b1fab5e-c65f-4f66-9aa5-b5a2c6217420\" (UID: \"7b1fab5e-c65f-4f66-9aa5-b5a2c6217420\") "
Sep 18 20:04:05.106487 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:04:05.106461    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b1fab5e-c65f-4f66-9aa5-b5a2c6217420-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7b1fab5e-c65f-4f66-9aa5-b5a2c6217420" (UID: "7b1fab5e-c65f-4f66-9aa5-b5a2c6217420"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:04:05.106601 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:04:05.106581    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b1fab5e-c65f-4f66-9aa5-b5a2c6217420-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7b1fab5e-c65f-4f66-9aa5-b5a2c6217420" (UID: "7b1fab5e-c65f-4f66-9aa5-b5a2c6217420"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:04:05.108381 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:04:05.108359    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b1fab5e-c65f-4f66-9aa5-b5a2c6217420-kube-api-access-h9j7h" (OuterVolumeSpecName: "kube-api-access-h9j7h") pod "7b1fab5e-c65f-4f66-9aa5-b5a2c6217420" (UID: "7b1fab5e-c65f-4f66-9aa5-b5a2c6217420"). InnerVolumeSpecName "kube-api-access-h9j7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:04:05.108381 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:04:05.108368    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b1fab5e-c65f-4f66-9aa5-b5a2c6217420-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7b1fab5e-c65f-4f66-9aa5-b5a2c6217420" (UID: "7b1fab5e-c65f-4f66-9aa5-b5a2c6217420"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:04:05.207091 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:04:05.207006    2522 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b1fab5e-c65f-4f66-9aa5-b5a2c6217420-proxy-tls\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 20:04:05.207091 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:04:05.207039    2522 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b1fab5e-c65f-4f66-9aa5-b5a2c6217420-kserve-provision-location\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 20:04:05.207091 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:04:05.207052    2522 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b1fab5e-c65f-4f66-9aa5-b5a2c6217420-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 20:04:05.207091 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:04:05.207065    2522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9j7h\" (UniqueName: \"kubernetes.io/projected/7b1fab5e-c65f-4f66-9aa5-b5a2c6217420-kube-api-access-h9j7h\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 20:04:05.900912 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:04:05.900881    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-74d5d6bcf-f6pb7_7b1fab5e-c65f-4f66-9aa5-b5a2c6217420/storage-initializer/1.log"
Sep 18 20:04:05.901398 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:04:05.900990    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-74d5d6bcf-f6pb7"
Sep 18 20:04:05.901398 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:04:05.901014    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-74d5d6bcf-f6pb7" event={"ID":"7b1fab5e-c65f-4f66-9aa5-b5a2c6217420","Type":"ContainerDied","Data":"34b29eca93f71a0b9d3c6b285f4b64440833bc1ee2cf2ae678d55f408c26c035"}
Sep 18 20:04:05.901398 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:04:05.901086    2522 scope.go:117] "RemoveContainer" containerID="b7e3c812a95eb9ace88353a7f7f6258fb8c43f5c5aea5461703bbc86663f1350"
Sep 18 20:04:05.948857 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:04:05.948828    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-74d5d6bcf-f6pb7"]
Sep 18 20:04:05.952308 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:04:05.952281    2522 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-74d5d6bcf-f6pb7"]
Sep 18 20:04:06.722112 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:04:06.722075    2522 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b1fab5e-c65f-4f66-9aa5-b5a2c6217420" path="/var/lib/kubelet/pods/7b1fab5e-c65f-4f66-9aa5-b5a2c6217420/volumes"
Sep 18 20:05:37.067070 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:37.067038    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bfddbbb78-tnh5d"]
Sep 18 20:05:37.067534 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:37.067368    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b1fab5e-c65f-4f66-9aa5-b5a2c6217420" containerName="storage-initializer"
Sep 18 20:05:37.067534 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:37.067379    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b1fab5e-c65f-4f66-9aa5-b5a2c6217420" containerName="storage-initializer"
Sep 18 20:05:37.067534 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:37.067397    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b1fab5e-c65f-4f66-9aa5-b5a2c6217420" containerName="storage-initializer"
Sep 18 20:05:37.067534 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:37.067402    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b1fab5e-c65f-4f66-9aa5-b5a2c6217420" containerName="storage-initializer"
Sep 18 20:05:37.067534 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:37.067462    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b1fab5e-c65f-4f66-9aa5-b5a2c6217420" containerName="storage-initializer"
Sep 18 20:05:37.067730 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:37.067592    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b1fab5e-c65f-4f66-9aa5-b5a2c6217420" containerName="storage-initializer"
Sep 18 20:05:37.070799 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:37.070779    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bfddbbb78-tnh5d"
Sep 18 20:05:37.074611 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:37.074591    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\""
Sep 18 20:05:37.074761 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:37.074610    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 20:05:37.074761 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:37.074620    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5b562\""
Sep 18 20:05:37.074761 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:37.074608    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert\""
Sep 18 20:05:37.074983 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:37.074951    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 18 20:05:37.075101 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:37.075085    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 20:05:37.082489 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:37.082467    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bfddbbb78-tnh5d"]
Sep 18 20:05:37.165266 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:37.165237    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1fb55ce-a411-4157-8c7f-b4ab4358b2bb-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-bfddbbb78-tnh5d\" (UID: \"c1fb55ce-a411-4157-8c7f-b4ab4358b2bb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bfddbbb78-tnh5d"
Sep 18 20:05:37.165415 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:37.165297    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx98q\" (UniqueName: \"kubernetes.io/projected/c1fb55ce-a411-4157-8c7f-b4ab4358b2bb-kube-api-access-lx98q\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-bfddbbb78-tnh5d\" (UID: \"c1fb55ce-a411-4157-8c7f-b4ab4358b2bb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bfddbbb78-tnh5d"
Sep 18 20:05:37.165415 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:37.165364    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1fb55ce-a411-4157-8c7f-b4ab4358b2bb-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-bfddbbb78-tnh5d\" (UID: \"c1fb55ce-a411-4157-8c7f-b4ab4358b2bb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bfddbbb78-tnh5d"
Sep 18 20:05:37.165415 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:37.165402    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1fb55ce-a411-4157-8c7f-b4ab4358b2bb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-bfddbbb78-tnh5d\" (UID: \"c1fb55ce-a411-4157-8c7f-b4ab4358b2bb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bfddbbb78-tnh5d"
Sep 18 20:05:37.265942 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:37.265909    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1fb55ce-a411-4157-8c7f-b4ab4358b2bb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-bfddbbb78-tnh5d\" (UID: \"c1fb55ce-a411-4157-8c7f-b4ab4358b2bb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bfddbbb78-tnh5d"
Sep 18 20:05:37.266112 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:37.265973    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1fb55ce-a411-4157-8c7f-b4ab4358b2bb-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-bfddbbb78-tnh5d\" (UID: \"c1fb55ce-a411-4157-8c7f-b4ab4358b2bb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bfddbbb78-tnh5d"
Sep 18 20:05:37.266112 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:37.266012    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx98q\" (UniqueName: \"kubernetes.io/projected/c1fb55ce-a411-4157-8c7f-b4ab4358b2bb-kube-api-access-lx98q\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-bfddbbb78-tnh5d\" (UID: \"c1fb55ce-a411-4157-8c7f-b4ab4358b2bb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bfddbbb78-tnh5d"
Sep 18 20:05:37.266112 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:37.266038    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1fb55ce-a411-4157-8c7f-b4ab4358b2bb-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-bfddbbb78-tnh5d\" (UID: \"c1fb55ce-a411-4157-8c7f-b4ab4358b2bb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bfddbbb78-tnh5d"
Sep 18 20:05:37.266447 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:37.266420    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1fb55ce-a411-4157-8c7f-b4ab4358b2bb-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-bfddbbb78-tnh5d\" (UID: \"c1fb55ce-a411-4157-8c7f-b4ab4358b2bb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bfddbbb78-tnh5d"
Sep 18 20:05:37.266625 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:37.266608    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1fb55ce-a411-4157-8c7f-b4ab4358b2bb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-bfddbbb78-tnh5d\" (UID: \"c1fb55ce-a411-4157-8c7f-b4ab4358b2bb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bfddbbb78-tnh5d"
Sep 18 20:05:37.268445 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:37.268429    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1fb55ce-a411-4157-8c7f-b4ab4358b2bb-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-bfddbbb78-tnh5d\" (UID: \"c1fb55ce-a411-4157-8c7f-b4ab4358b2bb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bfddbbb78-tnh5d"
Sep 18 20:05:37.275222 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:37.275202    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx98q\" (UniqueName: \"kubernetes.io/projected/c1fb55ce-a411-4157-8c7f-b4ab4358b2bb-kube-api-access-lx98q\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-bfddbbb78-tnh5d\" (UID: \"c1fb55ce-a411-4157-8c7f-b4ab4358b2bb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bfddbbb78-tnh5d"
Sep 18 20:05:37.381644 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:37.381578    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bfddbbb78-tnh5d"
Sep 18 20:05:37.658958 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:37.658875    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bfddbbb78-tnh5d"]
Sep 18 20:05:37.662415 ip-10-0-131-221 kubenswrapper[2522]: W0918 20:05:37.662386    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1fb55ce_a411_4157_8c7f_b4ab4358b2bb.slice/crio-b25778efe0089662d87a73b2aec1b33d797797583e43fb7c663b400d14292387 WatchSource:0}: Error finding container b25778efe0089662d87a73b2aec1b33d797797583e43fb7c663b400d14292387: Status 404 returned error can't find the container with id b25778efe0089662d87a73b2aec1b33d797797583e43fb7c663b400d14292387
Sep 18 20:05:38.328580 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:38.328535    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bfddbbb78-tnh5d" event={"ID":"c1fb55ce-a411-4157-8c7f-b4ab4358b2bb","Type":"ContainerStarted","Data":"bb29031a1bc24345cee8cdd0682000063aa6b92d47f49bc46b23c777ed4645c2"}
Sep 18 20:05:38.329021 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:38.328615    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bfddbbb78-tnh5d" event={"ID":"c1fb55ce-a411-4157-8c7f-b4ab4358b2bb","Type":"ContainerStarted","Data":"b25778efe0089662d87a73b2aec1b33d797797583e43fb7c663b400d14292387"}
Sep 18 20:05:41.347941 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:41.347909    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-bfddbbb78-tnh5d_c1fb55ce-a411-4157-8c7f-b4ab4358b2bb/storage-initializer/0.log"
Sep 18 20:05:41.348344 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:41.347949    2522 generic.go:356] "Generic (PLEG): container finished" podID="c1fb55ce-a411-4157-8c7f-b4ab4358b2bb" containerID="bb29031a1bc24345cee8cdd0682000063aa6b92d47f49bc46b23c777ed4645c2" exitCode=1
Sep 18 20:05:41.348344 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:41.348029    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bfddbbb78-tnh5d" event={"ID":"c1fb55ce-a411-4157-8c7f-b4ab4358b2bb","Type":"ContainerDied","Data":"bb29031a1bc24345cee8cdd0682000063aa6b92d47f49bc46b23c777ed4645c2"}
Sep 18 20:05:42.354673 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:42.354644    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-bfddbbb78-tnh5d_c1fb55ce-a411-4157-8c7f-b4ab4358b2bb/storage-initializer/0.log"
Sep 18 20:05:42.355072 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:42.354700    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bfddbbb78-tnh5d" event={"ID":"c1fb55ce-a411-4157-8c7f-b4ab4358b2bb","Type":"ContainerStarted","Data":"d83fe62cf44af03f36b03bf782bdb1d7d5335efe27d90d40ba9e6222321fa284"}
Sep 18 20:05:45.370104 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:45.370074    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-bfddbbb78-tnh5d_c1fb55ce-a411-4157-8c7f-b4ab4358b2bb/storage-initializer/1.log"
Sep 18 20:05:45.370524 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:45.370509    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-bfddbbb78-tnh5d_c1fb55ce-a411-4157-8c7f-b4ab4358b2bb/storage-initializer/0.log"
Sep 18 20:05:45.370597 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:45.370545    2522 generic.go:356] "Generic (PLEG): container finished" podID="c1fb55ce-a411-4157-8c7f-b4ab4358b2bb" containerID="d83fe62cf44af03f36b03bf782bdb1d7d5335efe27d90d40ba9e6222321fa284" exitCode=1
Sep 18 20:05:45.370654 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:45.370592    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bfddbbb78-tnh5d" event={"ID":"c1fb55ce-a411-4157-8c7f-b4ab4358b2bb","Type":"ContainerDied","Data":"d83fe62cf44af03f36b03bf782bdb1d7d5335efe27d90d40ba9e6222321fa284"}
Sep 18 20:05:45.370712 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:45.370660    2522 scope.go:117] "RemoveContainer" containerID="bb29031a1bc24345cee8cdd0682000063aa6b92d47f49bc46b23c777ed4645c2"
Sep 18 20:05:45.371022 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:45.370997    2522 scope.go:117] "RemoveContainer" containerID="bb29031a1bc24345cee8cdd0682000063aa6b92d47f49bc46b23c777ed4645c2"
Sep 18 20:05:45.381662 ip-10-0-131-221 kubenswrapper[2522]: E0918 20:05:45.381636    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-serving-fail-predictor-bfddbbb78-tnh5d_kserve-ci-e2e-test(c1fb55ce-a411-4157-8c7f-b4ab4358b2bb)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bfddbbb78-tnh5d" podUID="c1fb55ce-a411-4157-8c7f-b4ab4358b2bb"
Sep 18 20:05:45.381790 ip-10-0-131-221 kubenswrapper[2522]: E0918 20:05:45.381763    2522 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-serving-fail-predictor-bfddbbb78-tnh5d_kserve-ci-e2e-test_c1fb55ce-a411-4157-8c7f-b4ab4358b2bb_0 in pod sandbox b25778efe0089662d87a73b2aec1b33d797797583e43fb7c663b400d14292387 from index: no such id: 'bb29031a1bc24345cee8cdd0682000063aa6b92d47f49bc46b23c777ed4645c2'" containerID="bb29031a1bc24345cee8cdd0682000063aa6b92d47f49bc46b23c777ed4645c2"
Sep 18 20:05:45.381855 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:45.381809    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb29031a1bc24345cee8cdd0682000063aa6b92d47f49bc46b23c777ed4645c2"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-serving-fail-predictor-bfddbbb78-tnh5d_kserve-ci-e2e-test_c1fb55ce-a411-4157-8c7f-b4ab4358b2bb_0 in pod sandbox b25778efe0089662d87a73b2aec1b33d797797583e43fb7c663b400d14292387 from index: no such id: 'bb29031a1bc24345cee8cdd0682000063aa6b92d47f49bc46b23c777ed4645c2'"
Sep 18 20:05:46.377081 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:46.377053    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-bfddbbb78-tnh5d_c1fb55ce-a411-4157-8c7f-b4ab4358b2bb/storage-initializer/1.log"
Sep 18 20:05:47.050061 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:47.050030    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bfddbbb78-tnh5d"]
Sep 18 20:05:47.180708 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:47.180687    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-bfddbbb78-tnh5d_c1fb55ce-a411-4157-8c7f-b4ab4358b2bb/storage-initializer/1.log"
Sep 18 20:05:47.184094 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:47.184078    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bfddbbb78-tnh5d"
Sep 18 20:05:47.234411 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:47.234381    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lx98q\" (UniqueName: \"kubernetes.io/projected/c1fb55ce-a411-4157-8c7f-b4ab4358b2bb-kube-api-access-lx98q\") pod \"c1fb55ce-a411-4157-8c7f-b4ab4358b2bb\" (UID: \"c1fb55ce-a411-4157-8c7f-b4ab4358b2bb\") "
Sep 18 20:05:47.234633 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:47.234426    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1fb55ce-a411-4157-8c7f-b4ab4358b2bb-kserve-provision-location\") pod \"c1fb55ce-a411-4157-8c7f-b4ab4358b2bb\" (UID: \"c1fb55ce-a411-4157-8c7f-b4ab4358b2bb\") "
Sep 18 20:05:47.234633 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:47.234467    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1fb55ce-a411-4157-8c7f-b4ab4358b2bb-proxy-tls\") pod \"c1fb55ce-a411-4157-8c7f-b4ab4358b2bb\" (UID: \"c1fb55ce-a411-4157-8c7f-b4ab4358b2bb\") "
Sep 18 20:05:47.234633 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:47.234496    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1fb55ce-a411-4157-8c7f-b4ab4358b2bb-kube-rbac-proxy-sar-config\") pod \"c1fb55ce-a411-4157-8c7f-b4ab4358b2bb\" (UID: \"c1fb55ce-a411-4157-8c7f-b4ab4358b2bb\") "
Sep 18 20:05:47.234831 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:47.234773    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1fb55ce-a411-4157-8c7f-b4ab4358b2bb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c1fb55ce-a411-4157-8c7f-b4ab4358b2bb" (UID: "c1fb55ce-a411-4157-8c7f-b4ab4358b2bb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:05:47.234944 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:47.234920    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1fb55ce-a411-4157-8c7f-b4ab4358b2bb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c1fb55ce-a411-4157-8c7f-b4ab4358b2bb" (UID: "c1fb55ce-a411-4157-8c7f-b4ab4358b2bb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:05:47.236594 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:47.236572    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1fb55ce-a411-4157-8c7f-b4ab4358b2bb-kube-api-access-lx98q" (OuterVolumeSpecName: "kube-api-access-lx98q") pod "c1fb55ce-a411-4157-8c7f-b4ab4358b2bb" (UID: "c1fb55ce-a411-4157-8c7f-b4ab4358b2bb"). InnerVolumeSpecName "kube-api-access-lx98q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:05:47.236695 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:47.236678    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1fb55ce-a411-4157-8c7f-b4ab4358b2bb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c1fb55ce-a411-4157-8c7f-b4ab4358b2bb" (UID: "c1fb55ce-a411-4157-8c7f-b4ab4358b2bb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:05:47.335960 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:47.335932    2522 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1fb55ce-a411-4157-8c7f-b4ab4358b2bb-proxy-tls\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 20:05:47.335960 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:47.335961    2522 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1fb55ce-a411-4157-8c7f-b4ab4358b2bb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 20:05:47.336142 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:47.335972    2522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx98q\" (UniqueName: \"kubernetes.io/projected/c1fb55ce-a411-4157-8c7f-b4ab4358b2bb-kube-api-access-lx98q\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 20:05:47.336142 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:47.335982    2522 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1fb55ce-a411-4157-8c7f-b4ab4358b2bb-kserve-provision-location\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 20:05:47.382576 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:47.382547    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-bfddbbb78-tnh5d_c1fb55ce-a411-4157-8c7f-b4ab4358b2bb/storage-initializer/1.log"
Sep 18 20:05:47.383001 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:47.382627    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bfddbbb78-tnh5d" event={"ID":"c1fb55ce-a411-4157-8c7f-b4ab4358b2bb","Type":"ContainerDied","Data":"b25778efe0089662d87a73b2aec1b33d797797583e43fb7c663b400d14292387"}
Sep 18 20:05:47.383001 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:47.382670    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bfddbbb78-tnh5d"
Sep 18 20:05:47.383001 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:47.382709    2522 scope.go:117] "RemoveContainer" containerID="d83fe62cf44af03f36b03bf782bdb1d7d5335efe27d90d40ba9e6222321fa284"
Sep 18 20:05:47.428633 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:47.428606    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bfddbbb78-tnh5d"]
Sep 18 20:05:47.439071 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:47.439036    2522 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bfddbbb78-tnh5d"]
Sep 18 20:05:48.207389 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:48.207352    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-86649cc556-bd42t"]
Sep 18 20:05:48.207745 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:48.207731    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1fb55ce-a411-4157-8c7f-b4ab4358b2bb" containerName="storage-initializer"
Sep 18 20:05:48.207811 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:48.207746    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1fb55ce-a411-4157-8c7f-b4ab4358b2bb" containerName="storage-initializer"
Sep 18 20:05:48.207811 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:48.207757    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1fb55ce-a411-4157-8c7f-b4ab4358b2bb" containerName="storage-initializer"
Sep 18 20:05:48.207811 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:48.207762    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1fb55ce-a411-4157-8c7f-b4ab4358b2bb" containerName="storage-initializer"
Sep 18 20:05:48.207982 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:48.207825    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1fb55ce-a411-4157-8c7f-b4ab4358b2bb" containerName="storage-initializer"
Sep 18 20:05:48.207982 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:48.207835    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1fb55ce-a411-4157-8c7f-b4ab4358b2bb" containerName="storage-initializer"
Sep 18 20:05:48.212252 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:48.212233    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-86649cc556-bd42t"
Sep 18 20:05:48.214897 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:48.214858    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-predictor-serving-cert\""
Sep 18 20:05:48.215073 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:48.214893    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:05:48.215073 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:48.214920    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 20:05:48.215215 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:48.215156    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 18 20:05:48.215275 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:48.215212    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5b562\""
Sep 18 20:05:48.215914 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:48.215899    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 20:05:48.216009 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:48.215989    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 20:05:48.224242 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:48.224222    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-86649cc556-bd42t"]
Sep 18 20:05:48.244552 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:48.244525    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-86649cc556-bd42t\" (UID: \"6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-86649cc556-bd42t"
Sep 18 20:05:48.244651 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:48.244576    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvcrd\" (UniqueName: \"kubernetes.io/projected/6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b-kube-api-access-lvcrd\") pod \"isvc-pmml-v2-kserve-predictor-86649cc556-bd42t\" (UID: \"6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-86649cc556-bd42t"
Sep 18 20:05:48.244703 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:48.244652    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-86649cc556-bd42t\" (UID: \"6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-86649cc556-bd42t"
Sep 18 20:05:48.244703 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:48.244673    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-86649cc556-bd42t\" (UID: \"6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-86649cc556-bd42t"
Sep 18 20:05:48.244777 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:48.244702    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-86649cc556-bd42t\" (UID: \"6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-86649cc556-bd42t"
Sep 18 20:05:48.345503 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:48.345466    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-86649cc556-bd42t\" (UID: \"6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-86649cc556-bd42t"
Sep 18 20:05:48.345503 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:48.345510    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvcrd\" (UniqueName: \"kubernetes.io/projected/6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b-kube-api-access-lvcrd\") pod \"isvc-pmml-v2-kserve-predictor-86649cc556-bd42t\" (UID: \"6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-86649cc556-bd42t"
Sep 18 20:05:48.345769 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:48.345639    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-86649cc556-bd42t\" (UID: \"6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-86649cc556-bd42t"
Sep 18 20:05:48.345769 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:48.345689    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-86649cc556-bd42t\" (UID: \"6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-86649cc556-bd42t"
Sep 18 20:05:48.345769 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:48.345716    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-86649cc556-bd42t\" (UID: \"6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-86649cc556-bd42t"
Sep 18 20:05:48.346035 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:48.346009    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-86649cc556-bd42t\" (UID: \"6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-86649cc556-bd42t"
Sep 18 20:05:48.346249 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:48.346232    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-86649cc556-bd42t\" (UID: \"6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-86649cc556-bd42t"
Sep 18 20:05:48.346323 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:48.346307    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-86649cc556-bd42t\" (UID: \"6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-86649cc556-bd42t"
Sep 18 20:05:48.348090 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:48.348068    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-86649cc556-bd42t\" (UID: \"6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-86649cc556-bd42t"
Sep 18 20:05:48.355348 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:48.355328    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvcrd\" (UniqueName: \"kubernetes.io/projected/6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b-kube-api-access-lvcrd\") pod \"isvc-pmml-v2-kserve-predictor-86649cc556-bd42t\" (UID: \"6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-86649cc556-bd42t"
Sep 18 20:05:48.523050 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:48.522970    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-86649cc556-bd42t"
Sep 18 20:05:48.721987 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:48.721948    2522 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1fb55ce-a411-4157-8c7f-b4ab4358b2bb" path="/var/lib/kubelet/pods/c1fb55ce-a411-4157-8c7f-b4ab4358b2bb/volumes"
Sep 18 20:05:48.796959 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:48.796817    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-86649cc556-bd42t"]
Sep 18 20:05:48.799265 ip-10-0-131-221 kubenswrapper[2522]: W0918 20:05:48.799236    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6135d4f8_8b8f_494b_a007_fe6a3c0d7f4b.slice/crio-01eef767de8dd4c72541a5893c2badbf82264ab9e350c8a8cbe2142f8d0475f1 WatchSource:0}: Error finding container 01eef767de8dd4c72541a5893c2badbf82264ab9e350c8a8cbe2142f8d0475f1: Status 404 returned error can't find the container with id 01eef767de8dd4c72541a5893c2badbf82264ab9e350c8a8cbe2142f8d0475f1
Sep 18 20:05:49.397535 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:49.397490    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-86649cc556-bd42t" event={"ID":"6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b","Type":"ContainerStarted","Data":"8a967c65e73db8c43beb29de27f9d6a92c89b0d3bff18ec96f668d592ca7aacb"}
Sep 18 20:05:49.397732 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:49.397579    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-86649cc556-bd42t" event={"ID":"6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b","Type":"ContainerStarted","Data":"01eef767de8dd4c72541a5893c2badbf82264ab9e350c8a8cbe2142f8d0475f1"}
Sep 18 20:05:53.419017 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:53.418977    2522 generic.go:356] "Generic (PLEG): container finished" podID="6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b" containerID="8a967c65e73db8c43beb29de27f9d6a92c89b0d3bff18ec96f668d592ca7aacb" exitCode=0
Sep 18 20:05:53.419424 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:53.419053    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-86649cc556-bd42t" event={"ID":"6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b","Type":"ContainerDied","Data":"8a967c65e73db8c43beb29de27f9d6a92c89b0d3bff18ec96f668d592ca7aacb"}
Sep 18 20:05:53.420252 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:05:53.420235    2522 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 20:06:01.468433 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:06:01.468393    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-86649cc556-bd42t" event={"ID":"6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b","Type":"ContainerStarted","Data":"209f5d14287bb0dfe477720cc24784c50d38170ed868a86de05bc627014d2a7c"}
Sep 18 20:06:01.468928 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:06:01.468480    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-86649cc556-bd42t" event={"ID":"6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b","Type":"ContainerStarted","Data":"13fb33a60e6de332b8994767ad135d38cd10322b192ba13cd92b6f6720853181"}
Sep 18 20:06:01.468928 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:06:01.468738    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-86649cc556-bd42t"
Sep 18 20:06:01.468928 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:06:01.468790    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-86649cc556-bd42t"
Sep 18 20:06:01.470361 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:06:01.470332    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-86649cc556-bd42t" podUID="6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Sep 18 20:06:01.496222 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:06:01.496175    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-86649cc556-bd42t" podStartSLOduration=2.182048823 podStartE2EDuration="13.496160903s" totalImagesPullingTime="7.311764168s" totalInitContainerRuntime="4.002347912s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:05:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:06:01.49350365 +0000 UTC m=+3121.377030111" watchObservedRunningTime="2026-09-18 20:06:01.496160903 +0000 UTC m=+3121.379687365"
Sep 18 20:06:02.473282 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:06:02.473245    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-86649cc556-bd42t" podUID="6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Sep 18 20:06:07.477473 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:06:07.477438    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-86649cc556-bd42t"
Sep 18 20:06:07.478114 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:06:07.478081    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-86649cc556-bd42t" podUID="6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Sep 18 20:06:17.479041 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:06:17.479000    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-86649cc556-bd42t" podUID="6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Sep 18 20:06:27.479064 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:06:27.478980    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-86649cc556-bd42t" podUID="6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Sep 18 20:06:37.479052 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:06:37.479006    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-86649cc556-bd42t" podUID="6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Sep 18 20:06:47.478784 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:06:47.478743    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-86649cc556-bd42t" podUID="6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Sep 18 20:06:57.479051 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:06:57.479007    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-86649cc556-bd42t" podUID="6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Sep 18 20:07:07.479025 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:07:07.478982    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-86649cc556-bd42t" podUID="6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Sep 18 20:07:17.478894 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:07:17.478844    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-86649cc556-bd42t" podUID="6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Sep 18 20:07:27.479164 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:07:27.479132    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-86649cc556-bd42t"
Sep 18 20:07:30.010297 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:07:30.010259    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-86649cc556-bd42t"]
Sep 18 20:07:30.010796 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:07:30.010547    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-86649cc556-bd42t" podUID="6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b" containerName="kserve-container" containerID="cri-o://13fb33a60e6de332b8994767ad135d38cd10322b192ba13cd92b6f6720853181" gracePeriod=30
Sep 18 20:07:30.010796 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:07:30.010572    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-86649cc556-bd42t" podUID="6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b" containerName="kube-rbac-proxy" containerID="cri-o://209f5d14287bb0dfe477720cc24784c50d38170ed868a86de05bc627014d2a7c" gracePeriod=30
Sep 18 20:07:30.876298 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:07:30.876261    2522 generic.go:356] "Generic (PLEG): container finished" podID="6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b" containerID="209f5d14287bb0dfe477720cc24784c50d38170ed868a86de05bc627014d2a7c" exitCode=2
Sep 18 20:07:30.876478 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:07:30.876324    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-86649cc556-bd42t" event={"ID":"6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b","Type":"ContainerDied","Data":"209f5d14287bb0dfe477720cc24784c50d38170ed868a86de05bc627014d2a7c"}
Sep 18 20:07:32.474409 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:07:32.474363    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-86649cc556-bd42t" podUID="6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.61:8643/healthz\": dial tcp 10.132.0.61:8643: connect: connection refused"
Sep 18 20:07:34.153801 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:07:34.153773    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-86649cc556-bd42t"
Sep 18 20:07:34.247385 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:07:34.247291    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b-cabundle-cert\") pod \"6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b\" (UID: \"6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b\") "
Sep 18 20:07:34.247385 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:07:34.247348    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lvcrd\" (UniqueName: \"kubernetes.io/projected/6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b-kube-api-access-lvcrd\") pod \"6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b\" (UID: \"6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b\") "
Sep 18 20:07:34.247385 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:07:34.247377    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b-proxy-tls\") pod \"6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b\" (UID: \"6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b\") "
Sep 18 20:07:34.247738 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:07:34.247410    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b-kserve-provision-location\") pod \"6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b\" (UID: \"6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b\") "
Sep 18 20:07:34.247738 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:07:34.247439    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b-kube-rbac-proxy-sar-config\") pod \"6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b\" (UID: \"6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b\") "
Sep 18 20:07:34.247847 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:07:34.247789    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b" (UID: "6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:07:34.247847 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:07:34.247809    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b" (UID: "6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:07:34.247948 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:07:34.247924    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b" (UID: "6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:07:34.249487 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:07:34.249465    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b" (UID: "6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:07:34.249540 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:07:34.249520    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b-kube-api-access-lvcrd" (OuterVolumeSpecName: "kube-api-access-lvcrd") pod "6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b" (UID: "6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b"). InnerVolumeSpecName "kube-api-access-lvcrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:07:34.348024 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:07:34.347971    2522 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b-cabundle-cert\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 20:07:34.348024 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:07:34.348017    2522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvcrd\" (UniqueName: \"kubernetes.io/projected/6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b-kube-api-access-lvcrd\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 20:07:34.348024 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:07:34.348032    2522 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b-proxy-tls\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 20:07:34.348024 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:07:34.348040    2522 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b-kserve-provision-location\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 20:07:34.348301 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:07:34.348050    2522 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 20:07:34.895643 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:07:34.895608    2522 generic.go:356] "Generic (PLEG): container finished" podID="6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b" containerID="13fb33a60e6de332b8994767ad135d38cd10322b192ba13cd92b6f6720853181" exitCode=0
Sep 18 20:07:34.895834 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:07:34.895660    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-86649cc556-bd42t"
Sep 18 20:07:34.895834 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:07:34.895672    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-86649cc556-bd42t" event={"ID":"6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b","Type":"ContainerDied","Data":"13fb33a60e6de332b8994767ad135d38cd10322b192ba13cd92b6f6720853181"}
Sep 18 20:07:34.895834 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:07:34.895722    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-86649cc556-bd42t" event={"ID":"6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b","Type":"ContainerDied","Data":"01eef767de8dd4c72541a5893c2badbf82264ab9e350c8a8cbe2142f8d0475f1"}
Sep 18 20:07:34.895834 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:07:34.895747    2522 scope.go:117] "RemoveContainer" containerID="209f5d14287bb0dfe477720cc24784c50d38170ed868a86de05bc627014d2a7c"
Sep 18 20:07:34.903908 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:07:34.903886    2522 scope.go:117] "RemoveContainer" containerID="13fb33a60e6de332b8994767ad135d38cd10322b192ba13cd92b6f6720853181"
Sep 18 20:07:34.911417 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:07:34.911397    2522 scope.go:117] "RemoveContainer" containerID="8a967c65e73db8c43beb29de27f9d6a92c89b0d3bff18ec96f668d592ca7aacb"
Sep 18 20:07:34.919192 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:07:34.919175    2522 scope.go:117] "RemoveContainer" containerID="209f5d14287bb0dfe477720cc24784c50d38170ed868a86de05bc627014d2a7c"
Sep 18 20:07:34.919491 ip-10-0-131-221 kubenswrapper[2522]: E0918 20:07:34.919472    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"209f5d14287bb0dfe477720cc24784c50d38170ed868a86de05bc627014d2a7c\": container with ID starting with 209f5d14287bb0dfe477720cc24784c50d38170ed868a86de05bc627014d2a7c not found: ID does not exist" containerID="209f5d14287bb0dfe477720cc24784c50d38170ed868a86de05bc627014d2a7c"
Sep 18 20:07:34.919609 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:07:34.919496    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"209f5d14287bb0dfe477720cc24784c50d38170ed868a86de05bc627014d2a7c"} err="failed to get container status \"209f5d14287bb0dfe477720cc24784c50d38170ed868a86de05bc627014d2a7c\": rpc error: code = NotFound desc = could not find container \"209f5d14287bb0dfe477720cc24784c50d38170ed868a86de05bc627014d2a7c\": container with ID starting with 209f5d14287bb0dfe477720cc24784c50d38170ed868a86de05bc627014d2a7c not found: ID does not exist"
Sep 18 20:07:34.919609 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:07:34.919513    2522 scope.go:117] "RemoveContainer" containerID="13fb33a60e6de332b8994767ad135d38cd10322b192ba13cd92b6f6720853181"
Sep 18 20:07:34.919806 ip-10-0-131-221 kubenswrapper[2522]: E0918 20:07:34.919787    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13fb33a60e6de332b8994767ad135d38cd10322b192ba13cd92b6f6720853181\": container with ID starting with 13fb33a60e6de332b8994767ad135d38cd10322b192ba13cd92b6f6720853181 not found: ID does not exist" containerID="13fb33a60e6de332b8994767ad135d38cd10322b192ba13cd92b6f6720853181"
Sep 18 20:07:34.919854 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:07:34.919811    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13fb33a60e6de332b8994767ad135d38cd10322b192ba13cd92b6f6720853181"} err="failed to get container status \"13fb33a60e6de332b8994767ad135d38cd10322b192ba13cd92b6f6720853181\": rpc error: code = NotFound desc = could not find container \"13fb33a60e6de332b8994767ad135d38cd10322b192ba13cd92b6f6720853181\": container with ID starting with 13fb33a60e6de332b8994767ad135d38cd10322b192ba13cd92b6f6720853181 not found: ID does not exist"
Sep 18 20:07:34.919854 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:07:34.919827    2522 scope.go:117] "RemoveContainer" containerID="8a967c65e73db8c43beb29de27f9d6a92c89b0d3bff18ec96f668d592ca7aacb"
Sep 18 20:07:34.919973 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:07:34.919953    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-86649cc556-bd42t"]
Sep 18 20:07:34.920082 ip-10-0-131-221 kubenswrapper[2522]: E0918 20:07:34.920063    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a967c65e73db8c43beb29de27f9d6a92c89b0d3bff18ec96f668d592ca7aacb\": container with ID starting with 8a967c65e73db8c43beb29de27f9d6a92c89b0d3bff18ec96f668d592ca7aacb not found: ID does not exist" containerID="8a967c65e73db8c43beb29de27f9d6a92c89b0d3bff18ec96f668d592ca7aacb"
Sep 18 20:07:34.920122 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:07:34.920089    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a967c65e73db8c43beb29de27f9d6a92c89b0d3bff18ec96f668d592ca7aacb"} err="failed to get container status \"8a967c65e73db8c43beb29de27f9d6a92c89b0d3bff18ec96f668d592ca7aacb\": rpc error: code = NotFound desc = could not find container \"8a967c65e73db8c43beb29de27f9d6a92c89b0d3bff18ec96f668d592ca7aacb\": container with ID starting with 8a967c65e73db8c43beb29de27f9d6a92c89b0d3bff18ec96f668d592ca7aacb not found: ID does not exist"
Sep 18 20:07:34.922715 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:07:34.922694    2522 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-86649cc556-bd42t"]
Sep 18 20:07:36.722220 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:07:36.722186    2522 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b" path="/var/lib/kubelet/pods/6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b/volumes"
Sep 18 20:10:50.539668 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:10:50.539636    2522 ???:1] "http: TLS handshake error from 10.0.135.205:39506: EOF"
Sep 18 20:10:50.543903 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:10:50.543884    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-g679p_24ec319c-fec4-4b8b-bcb6-6331ec8574fa/global-pull-secret-syncer/0.log"
Sep 18 20:10:50.807174 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:10:50.807093    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-s7ckz_e31f28a0-45fc-45ae-a307-578e281565a9/konnectivity-agent/0.log"
Sep 18 20:10:50.914312 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:10:50.914283    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-221.ec2.internal_523614729ae9b688582b641ea2fafde6/haproxy/0.log"
Sep 18 20:10:54.703829 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:10:54.703725    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d0c8d507-b773-4b4e-b84c-3d387bbd2223/alertmanager/0.log"
Sep 18 20:10:54.733751 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:10:54.733724    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d0c8d507-b773-4b4e-b84c-3d387bbd2223/config-reloader/0.log"
Sep 18 20:10:54.774148 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:10:54.774118    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d0c8d507-b773-4b4e-b84c-3d387bbd2223/kube-rbac-proxy-web/0.log"
Sep 18 20:10:54.823947 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:10:54.823923    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d0c8d507-b773-4b4e-b84c-3d387bbd2223/kube-rbac-proxy/0.log"
Sep 18 20:10:54.897877 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:10:54.897846    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d0c8d507-b773-4b4e-b84c-3d387bbd2223/kube-rbac-proxy-metric/0.log"
Sep 18 20:10:54.995725 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:10:54.995654    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d0c8d507-b773-4b4e-b84c-3d387bbd2223/prom-label-proxy/0.log"
Sep 18 20:10:55.078668 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:10:55.078620    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d0c8d507-b773-4b4e-b84c-3d387bbd2223/init-config-reloader/0.log"
Sep 18 20:10:55.260736 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:10:55.260662    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-58tlh_2317359e-59e6-4734-8ac4-f01b0bb27d57/kube-state-metrics/0.log"
Sep 18 20:10:55.301690 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:10:55.301654    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-58tlh_2317359e-59e6-4734-8ac4-f01b0bb27d57/kube-rbac-proxy-main/0.log"
Sep 18 20:10:55.344724 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:10:55.344700    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-58tlh_2317359e-59e6-4734-8ac4-f01b0bb27d57/kube-rbac-proxy-self/0.log"
Sep 18 20:10:55.388313 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:10:55.388289    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-fdd59846c-7qgqt_ae2fbef4-e449-47e1-9017-199dec3639b2/metrics-server/0.log"
Sep 18 20:10:55.618030 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:10:55.617992    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cfpmj_896188ce-60fe-4f42-bced-50927342f530/node-exporter/0.log"
Sep 18 20:10:55.655266 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:10:55.655243    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cfpmj_896188ce-60fe-4f42-bced-50927342f530/kube-rbac-proxy/0.log"
Sep 18 20:10:55.693754 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:10:55.693730    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cfpmj_896188ce-60fe-4f42-bced-50927342f530/init-textfile/0.log"
Sep 18 20:10:55.891145 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:10:55.891060    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-q9q7d_1d034d6d-be14-44c8-a72b-b7351a3f4a40/kube-rbac-proxy-main/0.log"
Sep 18 20:10:55.932732 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:10:55.932701    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-q9q7d_1d034d6d-be14-44c8-a72b-b7351a3f4a40/kube-rbac-proxy-self/0.log"
Sep 18 20:10:55.985277 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:10:55.985252    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-q9q7d_1d034d6d-be14-44c8-a72b-b7351a3f4a40/openshift-state-metrics/0.log"
Sep 18 20:10:56.834913 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:10:56.834885    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5c6747dc7b-zm6d2_533ce2b0-2a99-4011-a5d9-c85e3cd87170/prometheus-operator-admission-webhook/0.log"
Sep 18 20:10:58.190901 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:10:58.190866    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-92dww/perf-node-gather-daemonset-mxgpx"]
Sep 18 20:10:58.191360 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:10:58.191268    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b" containerName="kube-rbac-proxy"
Sep 18 20:10:58.191360 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:10:58.191284    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b" containerName="kube-rbac-proxy"
Sep 18 20:10:58.191360 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:10:58.191299    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b" containerName="storage-initializer"
Sep 18 20:10:58.191360 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:10:58.191305    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b" containerName="storage-initializer"
Sep 18 20:10:58.191360 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:10:58.191311    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b" containerName="kserve-container"
Sep 18 20:10:58.191360 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:10:58.191316    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b" containerName="kserve-container"
Sep 18 20:10:58.191606 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:10:58.191367    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b" containerName="kserve-container"
Sep 18 20:10:58.191606 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:10:58.191378    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="6135d4f8-8b8f-494b-a007-fe6a3c0d7f4b" containerName="kube-rbac-proxy"
Sep 18 20:10:58.194481 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:10:58.194463    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-92dww/perf-node-gather-daemonset-mxgpx"
Sep 18 20:10:58.197262 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:10:58.197243    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-92dww\"/\"default-dockercfg-2fh4v\""
Sep 18 20:10:58.197540 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:10:58.197521    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-92dww\"/\"kube-root-ca.crt\""
Sep 18 20:10:58.198181 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:10:58.198167    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-92dww\"/\"openshift-service-ca.crt\""
Sep 18 20:10:58.205279 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:10:58.205258    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-92dww/perf-node-gather-daemonset-mxgpx"]
Sep 18 20:10:58.348845 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:10:58.348809    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/87fa4a79-0275-4fa3-8ace-d9a36a99bc77-podres\") pod \"perf-node-gather-daemonset-mxgpx\" (UID: \"87fa4a79-0275-4fa3-8ace-d9a36a99bc77\") " pod="openshift-must-gather-92dww/perf-node-gather-daemonset-mxgpx"
Sep 18 20:10:58.349062 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:10:58.348873    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/87fa4a79-0275-4fa3-8ace-d9a36a99bc77-lib-modules\") pod \"perf-node-gather-daemonset-mxgpx\" (UID: \"87fa4a79-0275-4fa3-8ace-d9a36a99bc77\") " pod="openshift-must-gather-92dww/perf-node-gather-daemonset-mxgpx"
Sep 18 20:10:58.349062 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:10:58.348893    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf8rm\" (UniqueName: \"kubernetes.io/projected/87fa4a79-0275-4fa3-8ace-d9a36a99bc77-kube-api-access-nf8rm\") pod \"perf-node-gather-daemonset-mxgpx\" (UID: \"87fa4a79-0275-4fa3-8ace-d9a36a99bc77\") " pod="openshift-must-gather-92dww/perf-node-gather-daemonset-mxgpx"
Sep 18 20:10:58.349062 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:10:58.348946    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/87fa4a79-0275-4fa3-8ace-d9a36a99bc77-sys\") pod \"perf-node-gather-daemonset-mxgpx\" (UID: \"87fa4a79-0275-4fa3-8ace-d9a36a99bc77\") " pod="openshift-must-gather-92dww/perf-node-gather-daemonset-mxgpx"
Sep 18 20:10:58.349062 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:10:58.349013    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/87fa4a79-0275-4fa3-8ace-d9a36a99bc77-proc\") pod \"perf-node-gather-daemonset-mxgpx\" (UID: \"87fa4a79-0275-4fa3-8ace-d9a36a99bc77\") " pod="openshift-must-gather-92dww/perf-node-gather-daemonset-mxgpx"
Sep 18 20:10:58.450041 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:10:58.449955    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/87fa4a79-0275-4fa3-8ace-d9a36a99bc77-lib-modules\") pod \"perf-node-gather-daemonset-mxgpx\" (UID: \"87fa4a79-0275-4fa3-8ace-d9a36a99bc77\") " pod="openshift-must-gather-92dww/perf-node-gather-daemonset-mxgpx"
Sep 18 20:10:58.450041 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:10:58.449992    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf8rm\" (UniqueName: \"kubernetes.io/projected/87fa4a79-0275-4fa3-8ace-d9a36a99bc77-kube-api-access-nf8rm\") pod \"perf-node-gather-daemonset-mxgpx\" (UID: \"87fa4a79-0275-4fa3-8ace-d9a36a99bc77\") " pod="openshift-must-gather-92dww/perf-node-gather-daemonset-mxgpx"
Sep 18 20:10:58.450041 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:10:58.450023    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/87fa4a79-0275-4fa3-8ace-d9a36a99bc77-sys\") pod \"perf-node-gather-daemonset-mxgpx\" (UID: \"87fa4a79-0275-4fa3-8ace-d9a36a99bc77\") " pod="openshift-must-gather-92dww/perf-node-gather-daemonset-mxgpx"
Sep 18 20:10:58.450300 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:10:58.450073    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/87fa4a79-0275-4fa3-8ace-d9a36a99bc77-proc\") pod \"perf-node-gather-daemonset-mxgpx\" (UID: \"87fa4a79-0275-4fa3-8ace-d9a36a99bc77\") " pod="openshift-must-gather-92dww/perf-node-gather-daemonset-mxgpx"
Sep 18 20:10:58.450300 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:10:58.450112    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/87fa4a79-0275-4fa3-8ace-d9a36a99bc77-podres\") pod \"perf-node-gather-daemonset-mxgpx\" (UID: \"87fa4a79-0275-4fa3-8ace-d9a36a99bc77\") " pod="openshift-must-gather-92dww/perf-node-gather-daemonset-mxgpx"
Sep 18 20:10:58.450300 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:10:58.450165    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/87fa4a79-0275-4fa3-8ace-d9a36a99bc77-lib-modules\") pod \"perf-node-gather-daemonset-mxgpx\" (UID: \"87fa4a79-0275-4fa3-8ace-d9a36a99bc77\") " pod="openshift-must-gather-92dww/perf-node-gather-daemonset-mxgpx"
Sep 18 20:10:58.450300 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:10:58.450174    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/87fa4a79-0275-4fa3-8ace-d9a36a99bc77-sys\") pod \"perf-node-gather-daemonset-mxgpx\" (UID: \"87fa4a79-0275-4fa3-8ace-d9a36a99bc77\") " pod="openshift-must-gather-92dww/perf-node-gather-daemonset-mxgpx"
Sep 18 20:10:58.450300 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:10:58.450186    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/87fa4a79-0275-4fa3-8ace-d9a36a99bc77-proc\") pod \"perf-node-gather-daemonset-mxgpx\" (UID: \"87fa4a79-0275-4fa3-8ace-d9a36a99bc77\") " pod="openshift-must-gather-92dww/perf-node-gather-daemonset-mxgpx"
Sep 18 20:10:58.450300 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:10:58.450212    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/87fa4a79-0275-4fa3-8ace-d9a36a99bc77-podres\") pod \"perf-node-gather-daemonset-mxgpx\" (UID: \"87fa4a79-0275-4fa3-8ace-d9a36a99bc77\") " pod="openshift-must-gather-92dww/perf-node-gather-daemonset-mxgpx"
Sep 18 20:10:58.458953 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:10:58.458930    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf8rm\" (UniqueName: \"kubernetes.io/projected/87fa4a79-0275-4fa3-8ace-d9a36a99bc77-kube-api-access-nf8rm\") pod \"perf-node-gather-daemonset-mxgpx\" (UID: \"87fa4a79-0275-4fa3-8ace-d9a36a99bc77\") " pod="openshift-must-gather-92dww/perf-node-gather-daemonset-mxgpx"
Sep 18 20:10:58.505184 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:10:58.505154    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-92dww/perf-node-gather-daemonset-mxgpx"
Sep 18 20:10:58.510260 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:10:58.510240    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6446c8d75d-sszkh_00d2da2b-3370-4f0a-8104-9bc429576c69/networking-console-plugin/0.log"
Sep 18 20:10:58.780076 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:10:58.780043    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-92dww/perf-node-gather-daemonset-mxgpx"]
Sep 18 20:10:58.782878 ip-10-0-131-221 kubenswrapper[2522]: W0918 20:10:58.782850    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod87fa4a79_0275_4fa3_8ace_d9a36a99bc77.slice/crio-7596967c5ee401685d77d6402ffb82c58baba999e8503ed8e679db200d7f5082 WatchSource:0}: Error finding container 7596967c5ee401685d77d6402ffb82c58baba999e8503ed8e679db200d7f5082: Status 404 returned error can't find the container with id 7596967c5ee401685d77d6402ffb82c58baba999e8503ed8e679db200d7f5082
Sep 18 20:10:58.834179 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:10:58.834151    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-92dww/perf-node-gather-daemonset-mxgpx" event={"ID":"87fa4a79-0275-4fa3-8ace-d9a36a99bc77","Type":"ContainerStarted","Data":"7596967c5ee401685d77d6402ffb82c58baba999e8503ed8e679db200d7f5082"}
Sep 18 20:10:59.644399 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:10:59.644364    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6594f4c75-f57kg_25102f99-24b3-4c22-81ac-ee32e1fceabf/download-server/0.log"
Sep 18 20:10:59.840308 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:10:59.840277    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-92dww/perf-node-gather-daemonset-mxgpx" event={"ID":"87fa4a79-0275-4fa3-8ace-d9a36a99bc77","Type":"ContainerStarted","Data":"4ebc32c76bfb4dd52fdfe0b165ff33bc6c0a0d59f41347c628e06c075cfa0190"}
Sep 18 20:10:59.840492 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:10:59.840377    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-92dww/perf-node-gather-daemonset-mxgpx"
Sep 18 20:10:59.881528 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:10:59.881474    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-92dww/perf-node-gather-daemonset-mxgpx" podStartSLOduration=1.881458616 podStartE2EDuration="1.881458616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:10:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:10:59.878165964 +0000 UTC m=+3419.761692425" watchObservedRunningTime="2026-09-18 20:10:59.881458616 +0000 UTC m=+3419.764985077"
Sep 18 20:11:00.292884 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:11:00.292851    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-s69xh_879188dd-2181-4f6c-9337-1356aa59fdc0/volume-data-source-validator/0.log"
Sep 18 20:11:01.503464 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:11:01.503432    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mrd75_72d866b0-9634-442e-bd20-d90321a00452/dns/0.log"
Sep 18 20:11:01.541981 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:11:01.541957    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mrd75_72d866b0-9634-442e-bd20-d90321a00452/kube-rbac-proxy/0.log"
Sep 18 20:11:01.638471 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:11:01.638433    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-w4tjm_5b796f3a-2d43-47a8-8d73-a34a2af2d62b/dns-node-resolver/0.log"
Sep 18 20:11:02.217281 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:11:02.217254    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hmfqw_438a5295-6f51-49ed-99d1-96a8fe037805/node-ca/0.log"
Sep 18 20:11:03.616914 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:11:03.616882    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pmrmr_47bbbb88-fe11-4227-bddc-4e124994d36a/serve-healthcheck-canary/0.log"
Sep 18 20:11:04.284462 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:11:04.284438    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8bdr7_399aa0ce-d6ee-4fd0-a027-edf66721c7f2/kube-rbac-proxy/0.log"
Sep 18 20:11:04.324013 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:11:04.323985    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8bdr7_399aa0ce-d6ee-4fd0-a027-edf66721c7f2/exporter/0.log"
Sep 18 20:11:04.356931 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:11:04.356906    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8bdr7_399aa0ce-d6ee-4fd0-a027-edf66721c7f2/extractor/0.log"
Sep 18 20:11:05.858576 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:11:05.858535    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-92dww/perf-node-gather-daemonset-mxgpx"
Sep 18 20:11:07.567467 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:11:07.567439    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-645f9c8fd6-m8mlb_69e3cc75-b8c2-4777-974f-b017538db378/manager/0.log"
Sep 18 20:11:08.543078 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:11:08.543049    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-qc86j_c05217f4-19ec-45ed-ac0c-9898689234c2/seaweedfs/0.log"
Sep 18 20:11:08.604360 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:11:08.604341    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-custom-8b44d767d-l987k_02dd1836-2988-4996-9a32-07600c3ee746/seaweedfs-tls-custom/0.log"
Sep 18 20:11:08.646142 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:11:08.646112    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-serving-67b7477c6d-b5mp8_c83c09a0-1343-4c29-8a40-b5a508d54f60/seaweedfs-tls-serving/0.log"
Sep 18 20:11:13.983297 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:11:13.983262    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-qkrw2_09a74ca7-303a-49c9-9967-98074bbfbb17/kube-storage-version-migrator-operator/1.log"
Sep 18 20:11:13.984332 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:11:13.984311    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-qkrw2_09a74ca7-303a-49c9-9967-98074bbfbb17/kube-storage-version-migrator-operator/0.log"
Sep 18 20:11:15.344604 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:11:15.344549    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jrg4l_6c2a6c5f-990b-4ceb-9308-f7a6e29e22a2/kube-multus-additional-cni-plugins/0.log"
Sep 18 20:11:15.380878 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:11:15.380852    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jrg4l_6c2a6c5f-990b-4ceb-9308-f7a6e29e22a2/egress-router-binary-copy/0.log"
Sep 18 20:11:15.435271 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:11:15.435243    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jrg4l_6c2a6c5f-990b-4ceb-9308-f7a6e29e22a2/cni-plugins/0.log"
Sep 18 20:11:15.484327 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:11:15.484294    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jrg4l_6c2a6c5f-990b-4ceb-9308-f7a6e29e22a2/bond-cni-plugin/0.log"
Sep 18 20:11:15.534706 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:11:15.534682    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jrg4l_6c2a6c5f-990b-4ceb-9308-f7a6e29e22a2/routeoverride-cni/0.log"
Sep 18 20:11:15.569537 ip-10-0-131-221 kubenswrapper[2522]: I0918 20:11:15.569507    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jrg4l_6c2a6c5f-990b-4ceb-9308-f7a6e29e22a2/whereabouts-cni-bincopy/0.log"