Jul 30 17:00:36.807570 ip-10-0-136-68 systemd[1]: Starting Kubernetes Kubelet...
Jul 30 17:00:37.187464 ip-10-0-136-68 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 30 17:00:37.187464 ip-10-0-136-68 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 30 17:00:37.187464 ip-10-0-136-68 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 30 17:00:37.187464 ip-10-0-136-68 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 30 17:00:37.187464 ip-10-0-136-68 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 30 17:00:37.189474 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.189354    2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 30 17:00:37.195421 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195390    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 30 17:00:37.195421 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195417    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 30 17:00:37.195421 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195422    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 30 17:00:37.195421 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195426    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 30 17:00:37.195421 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195432    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 30 17:00:37.195752 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195436    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 30 17:00:37.195752 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195440    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 30 17:00:37.195752 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195445    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 30 17:00:37.195752 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195449    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 30 17:00:37.195752 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195453    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 30 17:00:37.195752 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195456    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 30 17:00:37.195752 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195460    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 30 17:00:37.195752 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195464    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 30 17:00:37.195752 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195468    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 30 17:00:37.195752 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195471    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 30 17:00:37.195752 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195475    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 30 17:00:37.195752 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195479    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 30 17:00:37.195752 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195483    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 30 17:00:37.195752 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195487    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 30 17:00:37.195752 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195496    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 30 17:00:37.195752 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195501    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 30 17:00:37.195752 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195504    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 30 17:00:37.195752 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195508    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 30 17:00:37.195752 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195512    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 30 17:00:37.195752 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195516    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 30 17:00:37.196426 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195523    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 30 17:00:37.196426 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195529    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 30 17:00:37.196426 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195533    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 30 17:00:37.196426 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195538    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 30 17:00:37.196426 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195541    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 30 17:00:37.196426 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195546    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 30 17:00:37.196426 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195550    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 30 17:00:37.196426 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195554    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 30 17:00:37.196426 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195558    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 30 17:00:37.196426 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195562    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 30 17:00:37.196426 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195565    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 30 17:00:37.196426 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195569    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 30 17:00:37.196426 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195574    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 30 17:00:37.196426 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195579    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 30 17:00:37.196426 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195587    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 30 17:00:37.196426 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195594    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 30 17:00:37.196426 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195599    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 30 17:00:37.196426 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195604    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 30 17:00:37.196426 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195608    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 30 17:00:37.196902 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195613    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 30 17:00:37.196902 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195617    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 30 17:00:37.196902 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195622    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 30 17:00:37.196902 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195626    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 30 17:00:37.196902 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195630    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 30 17:00:37.196902 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195634    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 30 17:00:37.196902 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195640    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 30 17:00:37.196902 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195647    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 30 17:00:37.196902 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195651    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 30 17:00:37.196902 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195656    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 30 17:00:37.196902 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195660    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 30 17:00:37.196902 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195665    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 30 17:00:37.196902 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195669    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 30 17:00:37.196902 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195673    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 30 17:00:37.196902 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195678    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 30 17:00:37.196902 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195682    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 30 17:00:37.196902 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195686    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 30 17:00:37.196902 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195691    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 30 17:00:37.196902 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195697    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 30 17:00:37.196902 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195702    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 30 17:00:37.197443 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195706    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 30 17:00:37.197443 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195710    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 30 17:00:37.197443 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195715    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 30 17:00:37.197443 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195719    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 30 17:00:37.197443 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195724    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 30 17:00:37.197443 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195729    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 30 17:00:37.197443 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195733    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 30 17:00:37.197443 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195739    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 30 17:00:37.197443 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195743    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 30 17:00:37.197443 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195747    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 30 17:00:37.197443 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195751    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 30 17:00:37.197443 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195755    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 30 17:00:37.197443 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195760    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 30 17:00:37.197443 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195764    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 30 17:00:37.197443 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195768    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 30 17:00:37.197443 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195772    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 30 17:00:37.197443 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195776    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 30 17:00:37.197443 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195781    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 30 17:00:37.197443 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195785    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 30 17:00:37.197443 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195790    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 30 17:00:37.198085 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195794    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 30 17:00:37.198085 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195798    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 30 17:00:37.198085 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195802    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 30 17:00:37.198085 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195837    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 30 17:00:37.198085 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195844    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 30 17:00:37.198085 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195849    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 30 17:00:37.198085 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195854    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 30 17:00:37.198085 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195859    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 30 17:00:37.198085 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195864    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 30 17:00:37.198085 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195869    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 30 17:00:37.198085 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195873    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 30 17:00:37.198085 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195878    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 30 17:00:37.198085 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195885    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 30 17:00:37.198085 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195892    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 30 17:00:37.198085 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.195897    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 30 17:00:37.198085 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.196878    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 30 17:00:37.198085 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.196892    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 30 17:00:37.198085 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.196896    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 30 17:00:37.198085 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.196900    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 30 17:00:37.198586 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.196904    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 30 17:00:37.198586 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.196907    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 30 17:00:37.198586 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.196910    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 30 17:00:37.198586 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.196913    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 30 17:00:37.198586 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.196917    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 30 17:00:37.198586 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.196919    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 30 17:00:37.198586 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.196922    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 30 17:00:37.198586 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.196925    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 30 17:00:37.198586 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.196928    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 30 17:00:37.198586 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.196931    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 30 17:00:37.198586 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.196934    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 30 17:00:37.198586 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.196938    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 30 17:00:37.198586 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.196940    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 30 17:00:37.198586 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.196943    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 30 17:00:37.198586 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.196946    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 30 17:00:37.198586 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.196948    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 30 17:00:37.198586 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.196951    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 30 17:00:37.198586 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.196954    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 30 17:00:37.198586 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.196956    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 30 17:00:37.198586 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.196959    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 30 17:00:37.199091 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.196961    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 30 17:00:37.199091 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.196964    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 30 17:00:37.199091 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.196967    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 30 17:00:37.199091 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.196970    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 30 17:00:37.199091 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.196973    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 30 17:00:37.199091 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.196975    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 30 17:00:37.199091 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.196978    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 30 17:00:37.199091 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.196980    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 30 17:00:37.199091 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.196983    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 30 17:00:37.199091 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.196987    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 30 17:00:37.199091 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.196991    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 30 17:00:37.199091 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.196994    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 30 17:00:37.199091 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.196996    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 30 17:00:37.199091 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.196999    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 30 17:00:37.199091 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197002    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 30 17:00:37.199091 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197005    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 30 17:00:37.199091 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197007    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 30 17:00:37.199091 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197009    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 30 17:00:37.199091 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197012    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 30 17:00:37.199091 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197016    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 30 17:00:37.199605 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197020    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 30 17:00:37.199605 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197023    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 30 17:00:37.199605 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197026    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 30 17:00:37.199605 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197028    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 30 17:00:37.199605 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197031    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 30 17:00:37.199605 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197035    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 30 17:00:37.199605 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197037    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 30 17:00:37.199605 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197040    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 30 17:00:37.199605 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197044    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 30 17:00:37.199605 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197048    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 30 17:00:37.199605 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197050    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 30 17:00:37.199605 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197053    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 30 17:00:37.199605 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197057    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 30 17:00:37.199605 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197060    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 30 17:00:37.199605 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197063    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 30 17:00:37.199605 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197066    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 30 17:00:37.199605 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197068    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 30 17:00:37.199605 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197071    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 30 17:00:37.199605 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197074    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 30 17:00:37.200112 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197077    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 30 17:00:37.200112 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197080    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 30 17:00:37.200112 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197083    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 30 17:00:37.200112 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197086    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 30 17:00:37.200112 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197089    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 30 17:00:37.200112 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197092    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 30 17:00:37.200112 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197095    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 30 17:00:37.200112 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197098    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 30 17:00:37.200112 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197100    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 30 17:00:37.200112 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197103    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 30 17:00:37.200112 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197105    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 30 17:00:37.200112 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197108    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 30 17:00:37.200112 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197110    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 30 17:00:37.200112 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197113    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 30 17:00:37.200112 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197116    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 30 17:00:37.200112 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197118    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 30 17:00:37.200112 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197120    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 30 17:00:37.200112 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197124    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 30 17:00:37.200112 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197128    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 30 17:00:37.200610 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197132    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 30 17:00:37.200610 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197135    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 30 17:00:37.200610 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197138    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 30 17:00:37.200610 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197140    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 30 17:00:37.200610 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197143    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 30 17:00:37.200610 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197145    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 30 17:00:37.200610 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197148    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 30 17:00:37.200610 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197151    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 30 17:00:37.200610 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197154    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 30 17:00:37.200610 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197157    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 30 17:00:37.200610 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197159    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 30 17:00:37.200610 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197162    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 30 17:00:37.200610 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197164    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 30 17:00:37.200610 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197184    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 30 17:00:37.200610 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197187    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 30 17:00:37.200610 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197190    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 30 17:00:37.200610 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197192    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 30 17:00:37.200610 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197682    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 30 17:00:37.200610 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197690    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 30 17:00:37.200610 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197694    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 30 17:00:37.201103 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197697    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 30 17:00:37.201103 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197700    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 30 17:00:37.201103 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197703    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 30 17:00:37.201103 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197706    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 30 17:00:37.201103 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197708    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 30 17:00:37.201103 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197711    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 30 17:00:37.201103 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197714    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 30 17:00:37.201103 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197717    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 30 17:00:37.201103 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197719    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 30 17:00:37.201103 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197722    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 30 17:00:37.201103 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197724    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 30 17:00:37.201103 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197727    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 30 17:00:37.201103 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197729    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 30 17:00:37.201103 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197732    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 30 17:00:37.201103 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197735    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 30 17:00:37.201103 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197737    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 30 17:00:37.201103 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197740    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 30 17:00:37.201103 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197742    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 30 17:00:37.201103 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197745    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 30 17:00:37.201103 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197748    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 30 17:00:37.201103 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197751    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 30 17:00:37.201685 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197753    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 30 17:00:37.201685 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197760    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 30 17:00:37.201685 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197762    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 30 17:00:37.201685 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197765    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 30 17:00:37.201685 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197768    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 30 17:00:37.201685 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197771    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 30 17:00:37.201685 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197773    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 30 17:00:37.201685 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197776    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 30 17:00:37.201685 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197780    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 30 17:00:37.201685 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197782    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 30 17:00:37.201685 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197785    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 30 17:00:37.201685 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197787    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 30 17:00:37.201685 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197790    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 30 17:00:37.201685 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197793    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 30 17:00:37.201685 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197797    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 30 17:00:37.201685 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197801    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 30 17:00:37.201685 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197820    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 30 17:00:37.201685 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197825    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 30 17:00:37.201685 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197829    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 30 17:00:37.202160 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197832    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 30 17:00:37.202160 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197835    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 30 17:00:37.202160 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197838    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 30 17:00:37.202160 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197842    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 30 17:00:37.202160 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197844    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 30 17:00:37.202160 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197847    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 30 17:00:37.202160 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197850    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 30 17:00:37.202160 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197852    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 30 17:00:37.202160 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197855    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 30 17:00:37.202160 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197858    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 30 17:00:37.202160 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197860    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 30 17:00:37.202160 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197863    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 30 17:00:37.202160 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197865    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 30 17:00:37.202160 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197868    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 30 17:00:37.202160 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197873    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 30 17:00:37.202160 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197876    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 30 17:00:37.202160 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197879    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 30 17:00:37.202160 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197882    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 30 17:00:37.202160 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197885    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 30 17:00:37.202643 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197888    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 30 17:00:37.202643 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197890    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 30 17:00:37.202643 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197893    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 30 17:00:37.202643 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197897    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 30 17:00:37.202643 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197900    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 30 17:00:37.202643 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197903    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 30 17:00:37.202643 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197905    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 30 17:00:37.202643 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197909    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 30 17:00:37.202643 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197911    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 30 17:00:37.202643 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197914    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 30 17:00:37.202643 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197917    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 30 17:00:37.202643 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197919    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 30 17:00:37.202643 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197922    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 30 17:00:37.202643 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197924    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 30 17:00:37.202643 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197928    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 30 17:00:37.202643 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197931    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 30 17:00:37.202643 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197933    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 30 17:00:37.202643 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197937    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 30 17:00:37.202643 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197940    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 30 17:00:37.203208 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197942    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 30 17:00:37.203208 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197945    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 30 17:00:37.203208 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197948    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 30 17:00:37.203208 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197951    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 30 17:00:37.203208 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197954    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 30 17:00:37.203208 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197957    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 30 17:00:37.203208 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197959    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 30 17:00:37.203208 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197961    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 30 17:00:37.203208 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197966    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 30 17:00:37.203208 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197968    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 30 17:00:37.203208 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197972    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 30 17:00:37.203208 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197974    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 30 17:00:37.203208 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197977    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 30 17:00:37.203208 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197981    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 30 17:00:37.203208 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197985    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 30 17:00:37.203208 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197988    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 30 17:00:37.203208 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197991    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 30 17:00:37.203208 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.197994    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 30 17:00:37.203208 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198075    2580 flags.go:64] FLAG: --address="0.0.0.0"
Jul 30 17:00:37.203208 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198084    2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 30 17:00:37.203699 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198092    2580 flags.go:64] FLAG: --anonymous-auth="true"
Jul 30 17:00:37.203699 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198097    2580 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 30 17:00:37.203699 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198101    2580 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 30 17:00:37.203699 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198105    2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 30 17:00:37.203699 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198109    2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 30 17:00:37.203699 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198114    2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 30 17:00:37.203699 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198118    2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 30 17:00:37.203699 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198121    2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 30 17:00:37.203699 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198125    2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 30 17:00:37.203699 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198129    2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 30 17:00:37.203699 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198132    2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 30 17:00:37.203699 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198135    2580 flags.go:64] FLAG: --cgroup-root=""
Jul 30 17:00:37.203699 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198139    2580 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 30 17:00:37.203699 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198142    2580 flags.go:64] FLAG: --client-ca-file=""
Jul 30 17:00:37.203699 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198145    2580 flags.go:64] FLAG: --cloud-provider="external"
Jul 30 17:00:37.203699 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198148    2580 flags.go:64] FLAG: --cluster-dns="[]"
Jul 30 17:00:37.203699 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198153    2580 flags.go:64] FLAG: --cluster-domain=""
Jul 30 17:00:37.203699 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198156    2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 30 17:00:37.203699 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198159    2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 30 17:00:37.203699 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198162    2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 30 17:00:37.203699 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198183    2580 flags.go:64] FLAG: --container-log-max-files="5"
Jul 30 17:00:37.203699 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198190    2580 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 30 17:00:37.203699 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198195    2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 30 17:00:37.204275 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198200    2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 30 17:00:37.204275 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198204    2580 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 30 17:00:37.204275 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198208    2580 flags.go:64] FLAG: --contention-profiling="false"
Jul 30 17:00:37.204275 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198211    2580 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 30 17:00:37.204275 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198214    2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 30 17:00:37.204275 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198217    2580 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 30 17:00:37.204275 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198220    2580 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 30 17:00:37.204275 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198226    2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 30 17:00:37.204275 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198229    2580 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 30 17:00:37.204275 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198232    2580 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 30 17:00:37.204275 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198235    2580 flags.go:64] FLAG: --enable-load-reader="false"
Jul 30 17:00:37.204275 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198238    2580 flags.go:64] FLAG: --enable-server="true"
Jul 30 17:00:37.204275 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198241    2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 30 17:00:37.204275 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198247    2580 flags.go:64] FLAG: --event-burst="100"
Jul 30 17:00:37.204275 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198250    2580 flags.go:64] FLAG: --event-qps="50"
Jul 30 17:00:37.204275 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198253    2580 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 30 17:00:37.204275 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198256    2580 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 30 17:00:37.204275 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198259    2580 flags.go:64] FLAG: --eviction-hard=""
Jul 30 17:00:37.204275 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198263    2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 30 17:00:37.204275 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198266    2580 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 30 17:00:37.204275 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198269    2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 30 17:00:37.204275 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198272    2580 flags.go:64] FLAG: --eviction-soft=""
Jul 30 17:00:37.204275 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198274    2580 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 30 17:00:37.204275 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198279    2580 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 30 17:00:37.204275 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198283    2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 30 17:00:37.204903 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198286    2580 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 30 17:00:37.204903 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198289    2580 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 30 17:00:37.204903 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198292    2580 flags.go:64] FLAG: --fail-swap-on="true"
Jul 30 17:00:37.204903 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198295    2580 flags.go:64] FLAG: --feature-gates=""
Jul 30 17:00:37.204903 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198299    2580 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 30 17:00:37.204903 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198302    2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 30 17:00:37.204903 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198306    2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 30 17:00:37.204903 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198309    2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 30 17:00:37.204903 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198312    2580 flags.go:64] FLAG: --healthz-port="10248"
Jul 30 17:00:37.204903 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198315    2580 flags.go:64] FLAG: --help="false"
Jul 30 17:00:37.204903 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198319    2580 flags.go:64] FLAG: --hostname-override="ip-10-0-136-68.ec2.internal"
Jul 30 17:00:37.204903 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198322    2580 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 30 17:00:37.204903 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198325    2580 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 30 17:00:37.204903 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198328    2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 30 17:00:37.204903 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198332    2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 30 17:00:37.204903 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198336    2580 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 30 17:00:37.204903 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198339    2580 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 30 17:00:37.204903 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198342    2580 flags.go:64] FLAG: --image-service-endpoint=""
Jul 30 17:00:37.204903 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198345    2580 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 30 17:00:37.204903 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198348    2580 flags.go:64] FLAG: --kube-api-burst="100"
Jul 30 17:00:37.204903 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198351    2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 30 17:00:37.204903 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198354    2580 flags.go:64] FLAG: --kube-api-qps="50"
Jul 30 17:00:37.204903 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198357    2580 flags.go:64] FLAG: --kube-reserved=""
Jul 30 17:00:37.204903 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198361    2580 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 30 17:00:37.205527 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198364    2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 30 17:00:37.205527 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198367    2580 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 30 17:00:37.205527 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198370    2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 30 17:00:37.205527 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198373    2580 flags.go:64] FLAG: --lock-file=""
Jul 30 17:00:37.205527 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198376    2580 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 30 17:00:37.205527 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198379    2580 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 30 17:00:37.205527 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198384    2580 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 30 17:00:37.205527 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198389    2580 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 30 17:00:37.205527 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198393    2580 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 30 17:00:37.205527 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198396    2580 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 30 17:00:37.205527 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198399    2580 flags.go:64] FLAG: --logging-format="text"
Jul 30 17:00:37.205527 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198402    2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 30 17:00:37.205527 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198406    2580 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 30 17:00:37.205527 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198409    2580 flags.go:64] FLAG: --manifest-url=""
Jul 30 17:00:37.205527 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198412    2580 flags.go:64] FLAG: --manifest-url-header=""
Jul 30 17:00:37.205527 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198417    2580 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 30 17:00:37.205527 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198420    2580 flags.go:64] FLAG: --max-open-files="1000000"
Jul 30 17:00:37.205527 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198430    2580 flags.go:64] FLAG: --max-pods="110"
Jul 30 17:00:37.205527 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198434    2580 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 30 17:00:37.205527 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198437    2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 30 17:00:37.205527 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198440    2580 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 30 17:00:37.205527 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198443    2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 30 17:00:37.205527 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198446    2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 30 17:00:37.205527 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198450    2580 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 30 17:00:37.205527 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198460    2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 30 17:00:37.206147 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198469    2580 flags.go:64] FLAG: --node-status-max-images="50"
Jul 30 17:00:37.206147 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198472    2580 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 30 17:00:37.206147 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198476    2580 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 30 17:00:37.206147 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198479    2580 flags.go:64] FLAG: --pod-cidr=""
Jul 30 17:00:37.206147 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198482    2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 30 17:00:37.206147 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198488    2580 flags.go:64] FLAG: --pod-manifest-path=""
Jul 30 17:00:37.206147 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198490    2580 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 30 17:00:37.206147 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198494    2580 flags.go:64] FLAG: --pods-per-core="0"
Jul 30 17:00:37.206147 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198497    2580 flags.go:64] FLAG: --port="10250"
Jul 30 17:00:37.206147 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198500    2580 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 30 17:00:37.206147 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198503    2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00a9557abe07ffd2e"
Jul 30 17:00:37.206147 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198506    2580 flags.go:64] FLAG: --qos-reserved=""
Jul 30 17:00:37.206147 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198509    2580 flags.go:64] FLAG: --read-only-port="10255"
Jul 30 17:00:37.206147 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198515    2580 flags.go:64] FLAG: --register-node="true"
Jul 30 17:00:37.206147 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198518    2580 flags.go:64] FLAG: --register-with-taints=""
Jul 30 17:00:37.206147 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198522    2580 flags.go:64] FLAG: --registry-burst="10"
Jul 30 17:00:37.206147 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198525    2580 flags.go:64] FLAG: --registry-qps="5"
Jul 30 17:00:37.206147 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198528    2580 flags.go:64] FLAG: --reserved-cpus=""
Jul 30 17:00:37.206147 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198531    2580 flags.go:64] FLAG: --reserved-memory=""
Jul 30 17:00:37.206147 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198535    2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 30 17:00:37.206147 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198538    2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 30 17:00:37.206147 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198542    2580 flags.go:64] FLAG: --rotate-certificates="false"
Jul 30 17:00:37.206147 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198545    2580 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 30 17:00:37.206147 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198548    2580 flags.go:64] FLAG: --runonce="false"
Jul 30 17:00:37.206147 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198550    2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 30 17:00:37.206841 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198554    2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 30 17:00:37.206841 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198556    2580 flags.go:64] FLAG: --seccomp-default="false"
Jul 30 17:00:37.206841 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198559    2580 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 30 17:00:37.206841 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198562    2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 30 17:00:37.206841 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198565    2580 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 30 17:00:37.206841 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198569    2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 30 17:00:37.206841 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198572    2580 flags.go:64] FLAG: --storage-driver-password="root"
Jul 30 17:00:37.206841 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198576    2580 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 30 17:00:37.206841 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198579    2580 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 30 17:00:37.206841 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198581    2580 flags.go:64] FLAG: --storage-driver-user="root"
Jul 30 17:00:37.206841 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198584    2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 30 17:00:37.206841 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198588    2580 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 30 17:00:37.206841 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198591    2580 flags.go:64] FLAG: --system-cgroups=""
Jul 30 17:00:37.206841 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198594    2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 30 17:00:37.206841 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198599    2580 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 30 17:00:37.206841 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198603    2580 flags.go:64] FLAG: --tls-cert-file=""
Jul 30 17:00:37.206841 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198606    2580 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 30 17:00:37.206841 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198610    2580 flags.go:64] FLAG: --tls-min-version=""
Jul 30 17:00:37.206841 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198613    2580 flags.go:64] FLAG: --tls-private-key-file=""
Jul 30 17:00:37.206841 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198616    2580 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 30 17:00:37.206841 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198621    2580 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 30 17:00:37.206841 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198625    2580 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 30 17:00:37.206841 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198628    2580 flags.go:64] FLAG: --v="2"
Jul 30 17:00:37.206841 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198633    2580 flags.go:64] FLAG: --version="false"
Jul 30 17:00:37.206841 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198637    2580 flags.go:64] FLAG: --vmodule=""
Jul 30 17:00:37.207842 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198641    2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 30 17:00:37.207842 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.198644    2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 30 17:00:37.207842 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198806    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 30 17:00:37.207842 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198812    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 30 17:00:37.207842 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198815    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 30 17:00:37.207842 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198818    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 30 17:00:37.207842 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198821    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 30 17:00:37.207842 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198824    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 30 17:00:37.207842 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198827    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 30 17:00:37.207842 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198830    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 30 17:00:37.207842 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198833    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 30 17:00:37.207842 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198836    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 30 17:00:37.207842 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198839    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 30 17:00:37.207842 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198842    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 30 17:00:37.207842 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198845    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 30 17:00:37.207842 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198848    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 30 17:00:37.207842 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198851    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 30 17:00:37.207842 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198854    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 30 17:00:37.207842 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198857    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 30 17:00:37.208383 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198859    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 30 17:00:37.208383 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198862    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 30 17:00:37.208383 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198864    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 30 17:00:37.208383 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198867    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 30 17:00:37.208383 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198869    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 30 17:00:37.208383 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198872    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 30 17:00:37.208383 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198875    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 30 17:00:37.208383 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198877    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 30 17:00:37.208383 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198881    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 30 17:00:37.208383 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198884    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 30 17:00:37.208383 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198886    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 30 17:00:37.208383 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198889    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 30 17:00:37.208383 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198891    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 30 17:00:37.208383 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198896    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 30 17:00:37.208383 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198899    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 30 17:00:37.208383 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198903    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 30 17:00:37.208383 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198907    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 30 17:00:37.208383 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198910    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 30 17:00:37.208383 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198913    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 30 17:00:37.208864 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198915    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 30 17:00:37.208864 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198918    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 30 17:00:37.208864 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198921    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 30 17:00:37.208864 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198923    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 30 17:00:37.208864 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198926    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 30 17:00:37.208864 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198930    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 30 17:00:37.208864 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198932    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 30 17:00:37.208864 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198935    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 30 17:00:37.208864 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198937    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 30 17:00:37.208864 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198940    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 30 17:00:37.208864 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198943    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 30 17:00:37.208864 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198946    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 30 17:00:37.208864 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198948    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 30 17:00:37.208864 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198951    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 30 17:00:37.208864 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198953    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 30 17:00:37.208864 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198956    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 30 17:00:37.208864 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198958    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 30 17:00:37.208864 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198961    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 30 17:00:37.208864 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198963    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 30 17:00:37.208864 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198966    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 30 17:00:37.209411 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198968    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 30 17:00:37.209411 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198972    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 30 17:00:37.209411 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198975    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 30 17:00:37.209411 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198977    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 30 17:00:37.209411 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198980    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 30 17:00:37.209411 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198983    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 30 17:00:37.209411 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198985    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 30 17:00:37.209411 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198988    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 30 17:00:37.209411 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198992    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 30 17:00:37.209411 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198994    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 30 17:00:37.209411 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.198997    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 30 17:00:37.209411 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.199000    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 30 17:00:37.209411 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.199002    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 30 17:00:37.209411 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.199005    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 30 17:00:37.209411 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.199007    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 30 17:00:37.209411 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.199010    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 30 17:00:37.209411 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.199012    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 30 17:00:37.209411 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.199015    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 30 17:00:37.209411 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.199017    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 30 17:00:37.209411 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.199020    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 30 17:00:37.209897 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.199023    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 30 17:00:37.209897 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.199027    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 30 17:00:37.209897 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.199030    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 30 17:00:37.209897 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.199033    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 30 17:00:37.209897 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.199037    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 30 17:00:37.209897 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.199040    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 30 17:00:37.209897 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.199042    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 30 17:00:37.209897 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.199045    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 30 17:00:37.209897 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.199048    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 30 17:00:37.209897 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.199050    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 30 17:00:37.209897 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.199053    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 30 17:00:37.209897 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.199055    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 30 17:00:37.209897 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.199059    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 30 17:00:37.209897 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.199064    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 30 17:00:37.209897 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.199067    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 30 17:00:37.209897 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.199070    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 30 17:00:37.209897 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.199072    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 30 17:00:37.209897 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.199075    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 30 17:00:37.209897 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.199078    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 30 17:00:37.210381 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.199080    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 30 17:00:37.210381 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.199084    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 30 17:00:37.210381 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.199087    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 30 17:00:37.210381 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.199090    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 30 17:00:37.210381 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.199689    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 30 17:00:37.210381 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.208216    2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 30 17:00:37.210381 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.208240    2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 30 17:00:37.210381 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.208263    2580 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 30 17:00:37.210381 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.208270    2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 30 17:00:37.210381 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208335    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 30 17:00:37.210381 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208340    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 30 17:00:37.210381 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208344    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 30 17:00:37.210381 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208349    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 30 17:00:37.210381 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208353    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 30 17:00:37.210745 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208357    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 30 17:00:37.210745 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208360    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 30 17:00:37.210745 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208363    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 30 17:00:37.210745 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208366    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 30 17:00:37.210745 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208370    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 30 17:00:37.210745 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208373    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 30 17:00:37.210745 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208375    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 30 17:00:37.210745 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208378    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 30 17:00:37.210745 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208381    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 30 17:00:37.210745 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208383    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 30 17:00:37.210745 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208386    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 30 17:00:37.210745 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208390    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 30 17:00:37.210745 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208393    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 30 17:00:37.210745 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208396    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 30 17:00:37.210745 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208398    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 30 17:00:37.210745 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208401    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 30 17:00:37.210745 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208404    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 30 17:00:37.210745 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208407    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 30 17:00:37.210745 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208410    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 30 17:00:37.211229 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208413    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 30 17:00:37.211229 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208415    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 30 17:00:37.211229 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208418    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 30 17:00:37.211229 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208422    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 30 17:00:37.211229 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208425    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 30 17:00:37.211229 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208428    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 30 17:00:37.211229 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208431    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 30 17:00:37.211229 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208434    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 30 17:00:37.211229 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208437    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 30 17:00:37.211229 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208440    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 30 17:00:37.211229 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208442    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 30 17:00:37.211229 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208445    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 30 17:00:37.211229 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208447    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 30 17:00:37.211229 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208450    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 30 17:00:37.211229 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208453    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 30 17:00:37.211229 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208455    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 30 17:00:37.211229 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208458    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 30 17:00:37.211229 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208460    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 30 17:00:37.211229 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208463    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 30 17:00:37.211229 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208466    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 30 17:00:37.211720 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208468    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 30 17:00:37.211720 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208471    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 30 17:00:37.211720 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208473    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 30 17:00:37.211720 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208476    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 30 17:00:37.211720 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208478    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 30 17:00:37.211720 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208481    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 30 17:00:37.211720 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208485    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 30 17:00:37.211720 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208489    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 30 17:00:37.211720 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208492    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 30 17:00:37.211720 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208495    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 30 17:00:37.211720 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208497    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 30 17:00:37.211720 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208500    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 30 17:00:37.211720 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208502    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 30 17:00:37.211720 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208505    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 30 17:00:37.211720 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208507    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 30 17:00:37.211720 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208511    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 30 17:00:37.211720 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208514    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 30 17:00:37.211720 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208517    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 30 17:00:37.211720 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208520    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 30 17:00:37.211720 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208523    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 30 17:00:37.212269 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208526    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 30 17:00:37.212269 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208528    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 30 17:00:37.212269 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208531    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 30 17:00:37.212269 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208534    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 30 17:00:37.212269 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208536    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 30 17:00:37.212269 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208539    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 30 17:00:37.212269 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208541    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 30 17:00:37.212269 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208544    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 30 17:00:37.212269 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208546    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 30 17:00:37.212269 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208548    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 30 17:00:37.212269 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208551    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 30 17:00:37.212269 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208554    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 30 17:00:37.212269 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208557    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 30 17:00:37.212269 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208561    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 30 17:00:37.212269 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208564    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 30 17:00:37.212269 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208569    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 30 17:00:37.212269 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208572    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 30 17:00:37.212269 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208575    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 30 17:00:37.212269 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208578    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 30 17:00:37.212771 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208581    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 30 17:00:37.212771 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208584    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 30 17:00:37.212771 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208587    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 30 17:00:37.212771 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208590    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 30 17:00:37.212771 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208592    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 30 17:00:37.212771 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208595    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 30 17:00:37.212771 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208598    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 30 17:00:37.212771 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208600    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 30 17:00:37.212771 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208603    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 30 17:00:37.212771 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208606    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 30 17:00:37.212771 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208609    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 30 17:00:37.212771 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208612    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 30 17:00:37.212771 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208615    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 30 17:00:37.212771 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208618    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 30 17:00:37.212771 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208621    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 30 17:00:37.212771 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208623    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 30 17:00:37.213187 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.208629    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 30 17:00:37.213187 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208745    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 30 17:00:37.213187 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208749    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 30 17:00:37.213187 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208752    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 30 17:00:37.213187 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208755    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 30 17:00:37.213187 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208757    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 30 17:00:37.213187 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208760    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 30 17:00:37.213187 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208762    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 30 17:00:37.213187 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208765    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 30 17:00:37.213187 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208767    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 30 17:00:37.213187 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208770    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 30 17:00:37.213187 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208773    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 30 17:00:37.213187 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208775    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 30 17:00:37.213187 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208778    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 30 17:00:37.213187 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208781    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 30 17:00:37.213573 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208783    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 30 17:00:37.213573 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208786    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 30 17:00:37.213573 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208788    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 30 17:00:37.213573 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208790    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 30 17:00:37.213573 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208793    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 30 17:00:37.213573 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208795    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 30 17:00:37.213573 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208798    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 30 17:00:37.213573 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208801    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 30 17:00:37.213573 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208803    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 30 17:00:37.213573 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208806    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 30 17:00:37.213573 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208809    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 30 17:00:37.213573 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208812    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 30 17:00:37.213573 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208815    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 30 17:00:37.213573 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208818    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 30 17:00:37.213573 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208820    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 30 17:00:37.213573 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208823    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 30 17:00:37.213573 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208825    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 30 17:00:37.213573 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208828    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 30 17:00:37.213573 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208830    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 30 17:00:37.213573 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208833    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 30 17:00:37.214051 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208836    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 30 17:00:37.214051 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208838    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 30 17:00:37.214051 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208841    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 30 17:00:37.214051 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208844    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 30 17:00:37.214051 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208846    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 30 17:00:37.214051 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208848    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 30 17:00:37.214051 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208851    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 30 17:00:37.214051 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208854    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 30 17:00:37.214051 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208856    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 30 17:00:37.214051 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208859    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 30 17:00:37.214051 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208861    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 30 17:00:37.214051 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208864    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 30 17:00:37.214051 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208867    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 30 17:00:37.214051 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208870    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 30 17:00:37.214051 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208873    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 30 17:00:37.214051 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208877    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 30 17:00:37.214051 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208879    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 30 17:00:37.214051 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208884    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 30 17:00:37.214051 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208887    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 30 17:00:37.214533 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208890    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 30 17:00:37.214533 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208893    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 30 17:00:37.214533 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208896    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 30 17:00:37.214533 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208899    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 30 17:00:37.214533 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208903    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 30 17:00:37.214533 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208907    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 30 17:00:37.214533 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208910    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 30 17:00:37.214533 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208912    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 30 17:00:37.214533 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208915    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 30 17:00:37.214533 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208918    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 30 17:00:37.214533 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208920    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 30 17:00:37.214533 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208923    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 30 17:00:37.214533 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208925    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 30 17:00:37.214533 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208928    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 30 17:00:37.214533 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208930    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 30 17:00:37.214533 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208933    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 30 17:00:37.214533 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208935    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 30 17:00:37.214533 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208938    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 30 17:00:37.214533 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208940    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 30 17:00:37.214997 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208943    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 30 17:00:37.214997 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208945    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 30 17:00:37.214997 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208948    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 30 17:00:37.214997 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208950    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 30 17:00:37.214997 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208953    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 30 17:00:37.214997 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208955    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 30 17:00:37.214997 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208959    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 30 17:00:37.214997 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208961    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 30 17:00:37.214997 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208964    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 30 17:00:37.214997 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208967    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 30 17:00:37.214997 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208969    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 30 17:00:37.214997 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208972    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 30 17:00:37.214997 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208974    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 30 17:00:37.214997 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208977    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 30 17:00:37.214997 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208980    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 30 17:00:37.214997 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208983    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 30 17:00:37.214997 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208985    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 30 17:00:37.214997 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208988    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 30 17:00:37.214997 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208990    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 30 17:00:37.214997 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208993    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 30 17:00:37.215509 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208995    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 30 17:00:37.215509 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.208998    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 30 17:00:37.215509 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.209001    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 30 17:00:37.215509 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.209003    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 30 17:00:37.215509 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.209006    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 30 17:00:37.215509 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.209009    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 30 17:00:37.215509 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:37.209011    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 30 17:00:37.215509 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.209016    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 30 17:00:37.215509 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.209148    2580 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 30 17:00:37.215509 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.211538    2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 30 17:00:37.215509 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.212936    2580 server.go:1019] "Starting client certificate rotation"
Jul 30 17:00:37.215509 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.213037    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 30 17:00:37.215509 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.213072    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 30 17:00:37.215900 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.215787    2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 30 17:00:37.220262 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.220198    2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 30 17:00:37.230668 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.230640    2580 log.go:25] "Validated CRI v1 runtime API"
Jul 30 17:00:37.233337 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.233311    2580 log.go:25] "Validated CRI v1 image API"
Jul 30 17:00:37.234321 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.234304    2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 30 17:00:37.237248 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.237220    2580 fs.go:135] Filesystem UUIDs: map[2c04e78a-1bdf-4d96-90aa-b35042ccbe43:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 9621d5da-add3-4dce-b689-77d3ecf2dd15:/dev/nvme0n1p4]
Jul 30 17:00:37.237383 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.237244    2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 30 17:00:37.244140 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.243970    2580 manager.go:216] Machine: {Timestamp:2026-07-30 17:00:37.242576119 +0000 UTC m=+0.317876000 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098338 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23d8f8899fb315c92a493ff62da2b6 SystemUUID:ec23d8f8-899f-b315-c92a-493ff62da2b6 BootID:23acffb1-6c23-45b4-bb9a-a93d6a6da70b Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:fd:fa:a4:e5:19 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:fd:fa:a4:e5:19 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ce:e4:1e:bf:cb:29 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 30 17:00:37.244140 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.244115    2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 30 17:00:37.244372 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.244276    2580 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 30 17:00:37.247326 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.247284    2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 30 17:00:37.247486 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.247328    2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-68.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 30 17:00:37.248104 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.248091    2580 topology_manager.go:138] "Creating topology manager with none policy"
Jul 30 17:00:37.248150 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.248107    2580 container_manager_linux.go:309] "Creating device plugin manager"
Jul 30 17:00:37.248150 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.248116    2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 30 17:00:37.248686 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.248674    2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 30 17:00:37.250508 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.250493    2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 30 17:00:37.252397 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.252374    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 30 17:00:37.254160 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.254146    2580 state_mem.go:36] "Initialized new in-memory state store"
Jul 30 17:00:37.254400 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.254388    2580 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 30 17:00:37.254443 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.254431    2580 kubelet.go:486] "Attempting to sync node with API server"
Jul 30 17:00:37.254472 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.254456    2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 30 17:00:37.254472 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.254468    2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 30 17:00:37.254544 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.254478    2580 kubelet.go:398] "Adding apiserver pod source"
Jul 30 17:00:37.254544 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.254489    2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 30 17:00:37.255121 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.255104    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 30 17:00:37.255202 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.255137    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 30 17:00:37.258618 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.258592    2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 30 17:00:37.260352 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.260335    2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 30 17:00:37.260462 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.260366    2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 30 17:00:37.261136 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.261124    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 30 17:00:37.261194 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.261140    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 30 17:00:37.261194 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.261146    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 30 17:00:37.261194 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.261152    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 30 17:00:37.261194 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.261161    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 30 17:00:37.261194 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.261183    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 30 17:00:37.261194 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.261192    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 30 17:00:37.261372 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.261198    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 30 17:00:37.261372 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.261205    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 30 17:00:37.261372 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.261211    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 30 17:00:37.261372 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.261221    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 30 17:00:37.261372 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.261226    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 30 17:00:37.261982 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.261972    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 30 17:00:37.261982 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.261982    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 30 17:00:37.265655 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.265635    2580 server.go:1268] "Started kubelet"
Jul 30 17:00:37.265770 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.265717    2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 30 17:00:37.265944 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.265925    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-r66d6"
Jul 30 17:00:37.266144 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.266015    2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 30 17:00:37.266144 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.266129    2580 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 30 17:00:37.266287 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:37.266212    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 30 17:00:37.266338 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.266294    2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-68.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 30 17:00:37.266502 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:37.266482    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-68.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 30 17:00:37.266829 ip-10-0-136-68 systemd[1]: Started Kubernetes Kubelet.
Jul 30 17:00:37.268279 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.268258    2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 30 17:00:37.268542 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.268527    2580 server.go:310] "Adding debug handlers to kubelet server"
Jul 30 17:00:37.271525 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:37.271501    2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 30 17:00:37.272380 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:37.271523    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-68.ec2.internal.18c71fe30a3e9901  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-68.ec2.internal,UID:ip-10-0-136-68.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-68.ec2.internal,},FirstTimestamp:2026-07-30 17:00:37.265602817 +0000 UTC m=+0.340902699,LastTimestamp:2026-07-30 17:00:37.265602817 +0000 UTC m=+0.340902699,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-68.ec2.internal,}"
Jul 30 17:00:37.273849 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.273831    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-r66d6"
Jul 30 17:00:37.278930 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.278909    2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 30 17:00:37.278930 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.278930    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 30 17:00:37.279132 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:37.279110    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-68.ec2.internal\" not found"
Jul 30 17:00:37.279748 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.279725    2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 30 17:00:37.279748 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.279741    2580 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 30 17:00:37.279748 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.279754    2580 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 30 17:00:37.279915 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.279892    2580 reconstruct.go:97] "Volume reconstruction finished"
Jul 30 17:00:37.279915 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.279901    2580 reconciler.go:29] "Reconciler: start to sync state"
Jul 30 17:00:37.280109 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.280092    2580 factory.go:55] Registering systemd factory
Jul 30 17:00:37.280109 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.280111    2580 factory.go:223] Registration of the systemd container factory successfully
Jul 30 17:00:37.280370 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.280356    2580 factory.go:153] Registering CRI-O factory
Jul 30 17:00:37.280431 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.280384    2580 factory.go:223] Registration of the crio container factory successfully
Jul 30 17:00:37.280485 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.280446    2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 30 17:00:37.280485 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.280473    2580 factory.go:103] Registering Raw factory
Jul 30 17:00:37.280571 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.280489    2580 manager.go:1196] Started watching for new ooms in manager
Jul 30 17:00:37.281062 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.281046    2580 manager.go:349] Starting recovery of all containers
Jul 30 17:00:37.281518 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:37.281468    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 30 17:00:37.282032 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:37.282009    2580 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-68.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 30 17:00:37.295697 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.295486    2580 manager.go:354] Recovery completed
Jul 30 17:00:37.301424 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.301404    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 30 17:00:37.303434 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.303405    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-68.ec2.internal" event="NodeHasSufficientMemory"
Jul 30 17:00:37.303558 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.303443    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-68.ec2.internal" event="NodeHasNoDiskPressure"
Jul 30 17:00:37.303558 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.303457    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-68.ec2.internal" event="NodeHasSufficientPID"
Jul 30 17:00:37.305608 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.305583    2580 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 30 17:00:37.305608 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.305602    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 30 17:00:37.305760 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.305605    2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 30 17:00:37.305760 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.305670    2580 state_mem.go:36] "Initialized new in-memory state store"
Jul 30 17:00:37.306437 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.306417    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-68.ec2.internal" event="NodeHasSufficientMemory"
Jul 30 17:00:37.306532 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.306454    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-68.ec2.internal" event="NodeHasNoDiskPressure"
Jul 30 17:00:37.306532 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.306468    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-68.ec2.internal" event="NodeHasSufficientPID"
Jul 30 17:00:37.306987 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.306971    2580 policy_none.go:49] "None policy: Start"
Jul 30 17:00:37.307047 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.306992    2580 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 30 17:00:37.307047 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.307004    2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 30 17:00:37.308100 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.308089    2580 policy_none.go:47] "Start"
Jul 30 17:00:37.347909 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.347886    2580 manager.go:337] "Starting Device Plugin manager"
Jul 30 17:00:37.348050 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:37.347927    2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 30 17:00:37.348050 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.347937    2580 server.go:87] "Starting device plugin registration server"
Jul 30 17:00:37.348301 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.348287    2580 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 30 17:00:37.348340 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.348307    2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 30 17:00:37.348402 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.348382    2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 30 17:00:37.348484 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.348473    2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 30 17:00:37.348484 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.348484    2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 30 17:00:37.349085 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:37.349063    2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 30 17:00:37.349215 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:37.349120    2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-68.ec2.internal\" not found"
Jul 30 17:00:37.449473 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.449389    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 30 17:00:37.450631 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.450424    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-68.ec2.internal" event="NodeHasSufficientMemory"
Jul 30 17:00:37.450631 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.450457    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-68.ec2.internal" event="NodeHasNoDiskPressure"
Jul 30 17:00:37.450631 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.450468    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-68.ec2.internal" event="NodeHasSufficientPID"
Jul 30 17:00:37.450631 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.450502    2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-68.ec2.internal"
Jul 30 17:00:37.459264 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.459238    2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-68.ec2.internal"
Jul 30 17:00:37.459264 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:37.459270    2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-68.ec2.internal\": node \"ip-10-0-136-68.ec2.internal\" not found"
Jul 30 17:00:37.472664 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:37.472634    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-68.ec2.internal\" not found"
Jul 30 17:00:37.475198 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.475162    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 30 17:00:37.476443 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.476417    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 30 17:00:37.476555 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.476484    2580 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 30 17:00:37.476555 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.476515    2580 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 30 17:00:37.476663 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:37.476559    2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 30 17:00:37.478522 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.478496    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 30 17:00:37.573742 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:37.573690    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-68.ec2.internal\" not found"
Jul 30 17:00:37.576852 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.576827    2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-68.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-68.ec2.internal"]
Jul 30 17:00:37.576929 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.576919    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 30 17:00:37.580931 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.580908    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-68.ec2.internal" event="NodeHasSufficientMemory"
Jul 30 17:00:37.581040 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.580951    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-68.ec2.internal" event="NodeHasNoDiskPressure"
Jul 30 17:00:37.581040 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.580966    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-68.ec2.internal" event="NodeHasSufficientPID"
Jul 30 17:00:37.582041 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.582017    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c5609348d151739648e886175062b0fc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-68.ec2.internal\" (UID: \"c5609348d151739648e886175062b0fc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-68.ec2.internal"
Jul 30 17:00:37.582129 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.582052    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5609348d151739648e886175062b0fc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-68.ec2.internal\" (UID: \"c5609348d151739648e886175062b0fc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-68.ec2.internal"
Jul 30 17:00:37.582622 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.582605    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 30 17:00:37.582725 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.582709    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-68.ec2.internal"
Jul 30 17:00:37.582779 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.582748    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 30 17:00:37.583554 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.583538    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-68.ec2.internal" event="NodeHasSufficientMemory"
Jul 30 17:00:37.583653 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.583568    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-68.ec2.internal" event="NodeHasNoDiskPressure"
Jul 30 17:00:37.583653 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.583538    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-68.ec2.internal" event="NodeHasSufficientMemory"
Jul 30 17:00:37.583653 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.583595    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-68.ec2.internal" event="NodeHasNoDiskPressure"
Jul 30 17:00:37.583653 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.583608    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-68.ec2.internal" event="NodeHasSufficientPID"
Jul 30 17:00:37.583653 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.583579    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-68.ec2.internal" event="NodeHasSufficientPID"
Jul 30 17:00:37.584952 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.584938    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-68.ec2.internal"
Jul 30 17:00:37.585021 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.584966    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 30 17:00:37.585908 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.585875    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-68.ec2.internal" event="NodeHasSufficientMemory"
Jul 30 17:00:37.585908 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.585909    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-68.ec2.internal" event="NodeHasNoDiskPressure"
Jul 30 17:00:37.586034 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.585932    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-68.ec2.internal" event="NodeHasSufficientPID"
Jul 30 17:00:37.609187 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:37.609144    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-68.ec2.internal\" not found" node="ip-10-0-136-68.ec2.internal"
Jul 30 17:00:37.614360 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:37.614338    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-68.ec2.internal\" not found" node="ip-10-0-136-68.ec2.internal"
Jul 30 17:00:37.674084 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:37.674048    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-68.ec2.internal\" not found"
Jul 30 17:00:37.682635 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.682602    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c5609348d151739648e886175062b0fc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-68.ec2.internal\" (UID: \"c5609348d151739648e886175062b0fc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-68.ec2.internal"
Jul 30 17:00:37.682779 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.682638    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5609348d151739648e886175062b0fc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-68.ec2.internal\" (UID: \"c5609348d151739648e886175062b0fc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-68.ec2.internal"
Jul 30 17:00:37.682779 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.682669    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/74460565773fc50cdf3aee810c639568-config\") pod \"kube-apiserver-proxy-ip-10-0-136-68.ec2.internal\" (UID: \"74460565773fc50cdf3aee810c639568\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-68.ec2.internal"
Jul 30 17:00:37.682779 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.682721    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c5609348d151739648e886175062b0fc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-68.ec2.internal\" (UID: \"c5609348d151739648e886175062b0fc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-68.ec2.internal"
Jul 30 17:00:37.682779 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.682725    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5609348d151739648e886175062b0fc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-68.ec2.internal\" (UID: \"c5609348d151739648e886175062b0fc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-68.ec2.internal"
Jul 30 17:00:37.774683 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:37.774570    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-68.ec2.internal\" not found"
Jul 30 17:00:37.783022 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.782991    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/74460565773fc50cdf3aee810c639568-config\") pod \"kube-apiserver-proxy-ip-10-0-136-68.ec2.internal\" (UID: \"74460565773fc50cdf3aee810c639568\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-68.ec2.internal"
Jul 30 17:00:37.783095 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.783039    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/74460565773fc50cdf3aee810c639568-config\") pod \"kube-apiserver-proxy-ip-10-0-136-68.ec2.internal\" (UID: \"74460565773fc50cdf3aee810c639568\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-68.ec2.internal"
Jul 30 17:00:37.875234 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:37.875163    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-68.ec2.internal\" not found"
Jul 30 17:00:37.912675 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.912648    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-68.ec2.internal"
Jul 30 17:00:37.918710 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:37.918688    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-68.ec2.internal"
Jul 30 17:00:37.975948 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:37.975901    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-68.ec2.internal\" not found"
Jul 30 17:00:38.076573 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:38.076469    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-68.ec2.internal\" not found"
Jul 30 17:00:38.122990 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.122960    2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 30 17:00:38.172899 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.172866    2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 30 17:00:38.179873 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.179846    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-68.ec2.internal"
Jul 30 17:00:38.191931 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.191908    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 30 17:00:38.192746 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.192730    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-68.ec2.internal"
Jul 30 17:00:38.199913 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.199892    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 30 17:00:38.212072 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.212047    2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 30 17:00:38.212246 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.212228    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 30 17:00:38.212312 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.212244    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 30 17:00:38.212312 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.212246    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 30 17:00:38.256413 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.256380    2580 apiserver.go:52] "Watching apiserver"
Jul 30 17:00:38.276080 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.276046    2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 30 17:00:38.276479 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.276455    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-k7lnc","kube-system/konnectivity-agent-d8nwh","kube-system/kube-apiserver-proxy-ip-10-0-136-68.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbxnv","openshift-cluster-node-tuning-operator/tuned-j4f5j","openshift-dns/node-resolver-c77l8","openshift-image-registry/node-ca-tdkrg","openshift-multus/multus-2f2qc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-68.ec2.internal","openshift-multus/multus-additional-cni-plugins-mz6bl","openshift-multus/network-metrics-daemon-kbsf2","openshift-network-diagnostics/network-check-target-ltj2t","openshift-network-operator/iptables-alerter-46vf4","openshift-ovn-kubernetes/ovnkube-node-tblxl"]
Jul 30 17:00:38.278079 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.278058    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-j4f5j"
Jul 30 17:00:38.279033 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.279014    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 30 17:00:38.279213 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.279160    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-d8nwh"
Jul 30 17:00:38.279472 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.279427    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-29 16:55:37 +0000 UTC" deadline="2028-02-12 06:08:23.313642388 +0000 UTC"
Jul 30 17:00:38.279547 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.279472    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13477h7m45.03417437s"
Jul 30 17:00:38.280415 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.280392    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kbsf2"
Jul 30 17:00:38.280509 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:38.280467    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kbsf2" podUID="4f8b90e9-e705-4b9c-89f7-047151a2b098"
Jul 30 17:00:38.280578 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.280550    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 30 17:00:38.280578 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.280569    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-b4w4b\""
Jul 30 17:00:38.280677 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.280650    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 30 17:00:38.281490 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.281475    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 30 17:00:38.281536 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.281488    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-ndzx5\""
Jul 30 17:00:38.281584 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.281479    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 30 17:00:38.281627 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.281607    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbxnv"
Jul 30 17:00:38.282870 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.282854    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tdkrg"
Jul 30 17:00:38.283698 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.283681    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 30 17:00:38.283784 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.283762    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 30 17:00:38.283927 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.283798    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 30 17:00:38.283927 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.283854    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-4mspj\""
Jul 30 17:00:38.284033 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.284018    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-c77l8"
Jul 30 17:00:38.284951 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.284937    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 30 17:00:38.285242 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.285230    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k7lnc"
Jul 30 17:00:38.285326 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:38.285293    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k7lnc" podUID="ad2d9b30-7d70-4236-9398-93ff51843c30"
Jul 30 17:00:38.285483 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.285469    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-z94lz\""
Jul 30 17:00:38.285610 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.285596    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 30 17:00:38.285670 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.285644    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 30 17:00:38.285879 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.285862    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/408b59ca-0265-474f-994c-799e6e74dc90-etc-modprobe-d\") pod \"tuned-j4f5j\" (UID: \"408b59ca-0265-474f-994c-799e6e74dc90\") " pod="openshift-cluster-node-tuning-operator/tuned-j4f5j"
Jul 30 17:00:38.285948 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.285892    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/440bb960-b449-499c-b4b8-dabc4cc9d3cf-agent-certs\") pod \"konnectivity-agent-d8nwh\" (UID: \"440bb960-b449-499c-b4b8-dabc4cc9d3cf\") " pod="kube-system/konnectivity-agent-d8nwh"
Jul 30 17:00:38.285948 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.285919    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/440bb960-b449-499c-b4b8-dabc4cc9d3cf-konnectivity-ca\") pod \"konnectivity-agent-d8nwh\" (UID: \"440bb960-b449-499c-b4b8-dabc4cc9d3cf\") " pod="kube-system/konnectivity-agent-d8nwh"
Jul 30 17:00:38.286045 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.285946    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92pcn\" (UniqueName: \"kubernetes.io/projected/4f8b90e9-e705-4b9c-89f7-047151a2b098-kube-api-access-92pcn\") pod \"network-metrics-daemon-kbsf2\" (UID: \"4f8b90e9-e705-4b9c-89f7-047151a2b098\") " pod="openshift-multus/network-metrics-daemon-kbsf2"
Jul 30 17:00:38.286045 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.285976    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/408b59ca-0265-474f-994c-799e6e74dc90-etc-sysconfig\") pod \"tuned-j4f5j\" (UID: \"408b59ca-0265-474f-994c-799e6e74dc90\") " pod="openshift-cluster-node-tuning-operator/tuned-j4f5j"
Jul 30 17:00:38.286045 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.286000    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/408b59ca-0265-474f-994c-799e6e74dc90-etc-sysctl-conf\") pod \"tuned-j4f5j\" (UID: \"408b59ca-0265-474f-994c-799e6e74dc90\") " pod="openshift-cluster-node-tuning-operator/tuned-j4f5j"
Jul 30 17:00:38.286045 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.286022    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/408b59ca-0265-474f-994c-799e6e74dc90-etc-tuned\") pod \"tuned-j4f5j\" (UID: \"408b59ca-0265-474f-994c-799e6e74dc90\") " pod="openshift-cluster-node-tuning-operator/tuned-j4f5j"
Jul 30 17:00:38.286226 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.286049    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l6nm\" (UniqueName: \"kubernetes.io/projected/408b59ca-0265-474f-994c-799e6e74dc90-kube-api-access-7l6nm\") pod \"tuned-j4f5j\" (UID: \"408b59ca-0265-474f-994c-799e6e74dc90\") " pod="openshift-cluster-node-tuning-operator/tuned-j4f5j"
Jul 30 17:00:38.286226 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.286075    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/408b59ca-0265-474f-994c-799e6e74dc90-etc-systemd\") pod \"tuned-j4f5j\" (UID: \"408b59ca-0265-474f-994c-799e6e74dc90\") " pod="openshift-cluster-node-tuning-operator/tuned-j4f5j"
Jul 30 17:00:38.286226 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.286110    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/408b59ca-0265-474f-994c-799e6e74dc90-tmp\") pod \"tuned-j4f5j\" (UID: \"408b59ca-0265-474f-994c-799e6e74dc90\") " pod="openshift-cluster-node-tuning-operator/tuned-j4f5j"
Jul 30 17:00:38.286226 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.286149    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f8b90e9-e705-4b9c-89f7-047151a2b098-metrics-certs\") pod \"network-metrics-daemon-kbsf2\" (UID: \"4f8b90e9-e705-4b9c-89f7-047151a2b098\") " pod="openshift-multus/network-metrics-daemon-kbsf2"
Jul 30 17:00:38.286226 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.286193    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/408b59ca-0265-474f-994c-799e6e74dc90-lib-modules\") pod \"tuned-j4f5j\" (UID: \"408b59ca-0265-474f-994c-799e6e74dc90\") " pod="openshift-cluster-node-tuning-operator/tuned-j4f5j"
Jul 30 17:00:38.286226 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.286216    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/408b59ca-0265-474f-994c-799e6e74dc90-var-lib-kubelet\") pod \"tuned-j4f5j\" (UID: \"408b59ca-0265-474f-994c-799e6e74dc90\") " pod="openshift-cluster-node-tuning-operator/tuned-j4f5j"
Jul 30 17:00:38.286438 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.286226    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 30 17:00:38.286438 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.286239    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/408b59ca-0265-474f-994c-799e6e74dc90-host\") pod \"tuned-j4f5j\" (UID: \"408b59ca-0265-474f-994c-799e6e74dc90\") " pod="openshift-cluster-node-tuning-operator/tuned-j4f5j"
Jul 30 17:00:38.286438 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.286267    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/408b59ca-0265-474f-994c-799e6e74dc90-etc-kubernetes\") pod \"tuned-j4f5j\" (UID: \"408b59ca-0265-474f-994c-799e6e74dc90\") " pod="openshift-cluster-node-tuning-operator/tuned-j4f5j"
Jul 30 17:00:38.286438 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.286290    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/408b59ca-0265-474f-994c-799e6e74dc90-etc-sysctl-d\") pod \"tuned-j4f5j\" (UID: \"408b59ca-0265-474f-994c-799e6e74dc90\") " pod="openshift-cluster-node-tuning-operator/tuned-j4f5j"
Jul 30 17:00:38.286438 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.286311    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/408b59ca-0265-474f-994c-799e6e74dc90-run\") pod \"tuned-j4f5j\" (UID: \"408b59ca-0265-474f-994c-799e6e74dc90\") " pod="openshift-cluster-node-tuning-operator/tuned-j4f5j"
Jul 30 17:00:38.286438 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.286331    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/408b59ca-0265-474f-994c-799e6e74dc90-sys\") pod \"tuned-j4f5j\" (UID: \"408b59ca-0265-474f-994c-799e6e74dc90\") " pod="openshift-cluster-node-tuning-operator/tuned-j4f5j"
Jul 30 17:00:38.286637 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.286617    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6n7g5\""
Jul 30 17:00:38.286748 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.286733    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2f2qc"
Jul 30 17:00:38.287202 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.287182    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 30 17:00:38.288075 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.288061    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mz6bl"
Jul 30 17:00:38.289442 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.289279    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 30 17:00:38.289442 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.289337    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 30 17:00:38.289442 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.289349    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 30 17:00:38.289442 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.289360    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-728cp\""
Jul 30 17:00:38.289689 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.289471    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 30 17:00:38.289689 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.289577    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ltj2t"
Jul 30 17:00:38.289689 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:38.289629    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ltj2t" podUID="33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e"
Jul 30 17:00:38.290106 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.290092    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 30 17:00:38.290341 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.290329    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 30 17:00:38.290982 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.290964    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-46vf4"
Jul 30 17:00:38.291499 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.291480    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-fmppd\""
Jul 30 17:00:38.292542 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.292522    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:00:38.293137 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.293121    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 30 17:00:38.293469 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.293453    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-k7r7w\""
Jul 30 17:00:38.293573 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.293552    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 30 17:00:38.293684 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.293670    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 30 17:00:38.294560 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.294545    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 30 17:00:38.294637 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.294612    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 30 17:00:38.295118 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.295104    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 30 17:00:38.295210 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.295104    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 30 17:00:38.295646 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.295628    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 30 17:00:38.295724 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.295663    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 30 17:00:38.295724 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.295716    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-nkpv9\""
Jul 30 17:00:38.295862 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.295842    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 30 17:00:38.313074 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.313045    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-sn7jk"
Jul 30 17:00:38.322020 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.322000    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-sn7jk"
Jul 30 17:00:38.380758 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.380734    2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 30 17:00:38.386772 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.386743    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/38224201-2e3f-4863-b749-06ec97bf2c1d-registration-dir\") pod \"aws-ebs-csi-driver-node-fbxnv\" (UID: \"38224201-2e3f-4863-b749-06ec97bf2c1d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbxnv"
Jul 30 17:00:38.386863 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.386779    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/408b59ca-0265-474f-994c-799e6e74dc90-host\") pod \"tuned-j4f5j\" (UID: \"408b59ca-0265-474f-994c-799e6e74dc90\") " pod="openshift-cluster-node-tuning-operator/tuned-j4f5j"
Jul 30 17:00:38.386863 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.386800    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1-host-var-lib-cni-bin\") pod \"multus-2f2qc\" (UID: \"752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1\") " pod="openshift-multus/multus-2f2qc"
Jul 30 17:00:38.386863 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.386819    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/408b59ca-0265-474f-994c-799e6e74dc90-etc-modprobe-d\") pod \"tuned-j4f5j\" (UID: \"408b59ca-0265-474f-994c-799e6e74dc90\") " pod="openshift-cluster-node-tuning-operator/tuned-j4f5j"
Jul 30 17:00:38.386967 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.386864    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/440bb960-b449-499c-b4b8-dabc4cc9d3cf-konnectivity-ca\") pod \"konnectivity-agent-d8nwh\" (UID: \"440bb960-b449-499c-b4b8-dabc4cc9d3cf\") " pod="kube-system/konnectivity-agent-d8nwh"
Jul 30 17:00:38.386967 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.386916    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f366886f-2162-4153-a077-55dcd0fef1e8-system-cni-dir\") pod \"multus-additional-cni-plugins-mz6bl\" (UID: \"f366886f-2162-4153-a077-55dcd0fef1e8\") " pod="openshift-multus/multus-additional-cni-plugins-mz6bl"
Jul 30 17:00:38.387035 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.386971    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/408b59ca-0265-474f-994c-799e6e74dc90-host\") pod \"tuned-j4f5j\" (UID: \"408b59ca-0265-474f-994c-799e6e74dc90\") " pod="openshift-cluster-node-tuning-operator/tuned-j4f5j"
Jul 30 17:00:38.387035 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.386995    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/408b59ca-0265-474f-994c-799e6e74dc90-etc-modprobe-d\") pod \"tuned-j4f5j\" (UID: \"408b59ca-0265-474f-994c-799e6e74dc90\") " pod="openshift-cluster-node-tuning-operator/tuned-j4f5j"
Jul 30 17:00:38.387035 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.387014    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1-etc-kubernetes\") pod \"multus-2f2qc\" (UID: \"752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1\") " pod="openshift-multus/multus-2f2qc"
Jul 30 17:00:38.387158 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.387035    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zd7m\" (UniqueName: \"kubernetes.io/projected/752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1-kube-api-access-5zd7m\") pod \"multus-2f2qc\" (UID: \"752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1\") " pod="openshift-multus/multus-2f2qc"
Jul 30 17:00:38.387158 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.387051    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/19e7774c-aa26-4558-a50c-0c5ff05d9767-run-ovn\") pod \"ovnkube-node-tblxl\" (UID: \"19e7774c-aa26-4558-a50c-0c5ff05d9767\") " pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:00:38.387158 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.387071    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f366886f-2162-4153-a077-55dcd0fef1e8-cni-binary-copy\") pod \"multus-additional-cni-plugins-mz6bl\" (UID: \"f366886f-2162-4153-a077-55dcd0fef1e8\") " pod="openshift-multus/multus-additional-cni-plugins-mz6bl"
Jul 30 17:00:38.387158 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.387097    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/408b59ca-0265-474f-994c-799e6e74dc90-etc-tuned\") pod \"tuned-j4f5j\" (UID: \"408b59ca-0265-474f-994c-799e6e74dc90\") " pod="openshift-cluster-node-tuning-operator/tuned-j4f5j"
Jul 30 17:00:38.387158 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.387115    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f366886f-2162-4153-a077-55dcd0fef1e8-cnibin\") pod \"multus-additional-cni-plugins-mz6bl\" (UID: \"f366886f-2162-4153-a077-55dcd0fef1e8\") " pod="openshift-multus/multus-additional-cni-plugins-mz6bl"
Jul 30 17:00:38.387158 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.387144    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1-host-var-lib-cni-multus\") pod \"multus-2f2qc\" (UID: \"752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1\") " pod="openshift-multus/multus-2f2qc"
Jul 30 17:00:38.387374 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.387183    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/19e7774c-aa26-4558-a50c-0c5ff05d9767-systemd-units\") pod \"ovnkube-node-tblxl\" (UID: \"19e7774c-aa26-4558-a50c-0c5ff05d9767\") " pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:00:38.387374 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.387200    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4c8x\" (UniqueName: \"kubernetes.io/projected/19e7774c-aa26-4558-a50c-0c5ff05d9767-kube-api-access-z4c8x\") pod \"ovnkube-node-tblxl\" (UID: \"19e7774c-aa26-4558-a50c-0c5ff05d9767\") " pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:00:38.387374 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.387214    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgq7j\" (UniqueName: \"kubernetes.io/projected/38224201-2e3f-4863-b749-06ec97bf2c1d-kube-api-access-hgq7j\") pod \"aws-ebs-csi-driver-node-fbxnv\" (UID: \"38224201-2e3f-4863-b749-06ec97bf2c1d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbxnv"
Jul 30 17:00:38.387374 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.387234    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ad2d9b30-7d70-4236-9398-93ff51843c30-kubelet-config\") pod \"global-pull-secret-syncer-k7lnc\" (UID: \"ad2d9b30-7d70-4236-9398-93ff51843c30\") " pod="kube-system/global-pull-secret-syncer-k7lnc"
Jul 30 17:00:38.387374 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.387256    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1-multus-daemon-config\") pod \"multus-2f2qc\" (UID: \"752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1\") " pod="openshift-multus/multus-2f2qc"
Jul 30 17:00:38.387374 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.387280    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a47afe28-69d9-4b1b-8e40-b28645745766-iptables-alerter-script\") pod \"iptables-alerter-46vf4\" (UID: \"a47afe28-69d9-4b1b-8e40-b28645745766\") " pod="openshift-network-operator/iptables-alerter-46vf4"
Jul 30 17:00:38.387374 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.387306    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1-host-run-k8s-cni-cncf-io\") pod \"multus-2f2qc\" (UID: \"752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1\") " pod="openshift-multus/multus-2f2qc"
Jul 30 17:00:38.387374 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.387325    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/19e7774c-aa26-4558-a50c-0c5ff05d9767-node-log\") pod \"ovnkube-node-tblxl\" (UID: \"19e7774c-aa26-4558-a50c-0c5ff05d9767\") " pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:00:38.387374 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.387340    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/38224201-2e3f-4863-b749-06ec97bf2c1d-etc-selinux\") pod \"aws-ebs-csi-driver-node-fbxnv\" (UID: \"38224201-2e3f-4863-b749-06ec97bf2c1d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbxnv"
Jul 30 17:00:38.387374 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.387359    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ad2d9b30-7d70-4236-9398-93ff51843c30-original-pull-secret\") pod \"global-pull-secret-syncer-k7lnc\" (UID: \"ad2d9b30-7d70-4236-9398-93ff51843c30\") " pod="kube-system/global-pull-secret-syncer-k7lnc"
Jul 30 17:00:38.387374 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.387374    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1-system-cni-dir\") pod \"multus-2f2qc\" (UID: \"752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1\") " pod="openshift-multus/multus-2f2qc"
Jul 30 17:00:38.387712 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.387388    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1-cni-binary-copy\") pod \"multus-2f2qc\" (UID: \"752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1\") " pod="openshift-multus/multus-2f2qc"
Jul 30 17:00:38.387712 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.387408    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1-hostroot\") pod \"multus-2f2qc\" (UID: \"752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1\") " pod="openshift-multus/multus-2f2qc"
Jul 30 17:00:38.387712 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.387432    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1-host-run-multus-certs\") pod \"multus-2f2qc\" (UID: \"752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1\") " pod="openshift-multus/multus-2f2qc"
Jul 30 17:00:38.387712 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.387448    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svhd5\" (UniqueName: \"kubernetes.io/projected/33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e-kube-api-access-svhd5\") pod \"network-check-target-ltj2t\" (UID: \"33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e\") " pod="openshift-network-diagnostics/network-check-target-ltj2t"
Jul 30 17:00:38.387712 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.387466    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19e7774c-aa26-4558-a50c-0c5ff05d9767-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tblxl\" (UID: \"19e7774c-aa26-4558-a50c-0c5ff05d9767\") " pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:00:38.387712 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.387504    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ad2d9b30-7d70-4236-9398-93ff51843c30-dbus\") pod \"global-pull-secret-syncer-k7lnc\" (UID: \"ad2d9b30-7d70-4236-9398-93ff51843c30\") " pod="kube-system/global-pull-secret-syncer-k7lnc"
Jul 30 17:00:38.387712 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.387539    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1-host-run-netns\") pod \"multus-2f2qc\" (UID: \"752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1\") " pod="openshift-multus/multus-2f2qc"
Jul 30 17:00:38.387712 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.387563    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/19e7774c-aa26-4558-a50c-0c5ff05d9767-host-run-netns\") pod \"ovnkube-node-tblxl\" (UID: \"19e7774c-aa26-4558-a50c-0c5ff05d9767\") " pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:00:38.387712 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.387581    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/19e7774c-aa26-4558-a50c-0c5ff05d9767-log-socket\") pod \"ovnkube-node-tblxl\" (UID: \"19e7774c-aa26-4558-a50c-0c5ff05d9767\") " pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:00:38.387712 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.387595    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/19e7774c-aa26-4558-a50c-0c5ff05d9767-host-cni-netd\") pod \"ovnkube-node-tblxl\" (UID: \"19e7774c-aa26-4558-a50c-0c5ff05d9767\") " pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:00:38.387712 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.387653    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f366886f-2162-4153-a077-55dcd0fef1e8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mz6bl\" (UID: \"f366886f-2162-4153-a077-55dcd0fef1e8\") " pod="openshift-multus/multus-additional-cni-plugins-mz6bl"
Jul 30 17:00:38.387712 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.387675    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7sm5\" (UniqueName: \"kubernetes.io/projected/e4e1630f-f153-4be6-883a-1fbeca875a69-kube-api-access-c7sm5\") pod \"node-resolver-c77l8\" (UID: \"e4e1630f-f153-4be6-883a-1fbeca875a69\") " pod="openshift-dns/node-resolver-c77l8"
Jul 30 17:00:38.387712 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.387690    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/adf2f8d3-a0f4-4514-b6e3-b1507fa0c0bb-host\") pod \"node-ca-tdkrg\" (UID: \"adf2f8d3-a0f4-4514-b6e3-b1507fa0c0bb\") " pod="openshift-image-registry/node-ca-tdkrg"
Jul 30 17:00:38.387712 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.387709    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/408b59ca-0265-474f-994c-799e6e74dc90-lib-modules\") pod \"tuned-j4f5j\" (UID: \"408b59ca-0265-474f-994c-799e6e74dc90\") " pod="openshift-cluster-node-tuning-operator/tuned-j4f5j"
Jul 30 17:00:38.388271 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.387729    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/408b59ca-0265-474f-994c-799e6e74dc90-etc-sysctl-d\") pod \"tuned-j4f5j\" (UID: \"408b59ca-0265-474f-994c-799e6e74dc90\") " pod="openshift-cluster-node-tuning-operator/tuned-j4f5j"
Jul 30 17:00:38.388271 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.387763    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/408b59ca-0265-474f-994c-799e6e74dc90-run\") pod \"tuned-j4f5j\" (UID: \"408b59ca-0265-474f-994c-799e6e74dc90\") " pod="openshift-cluster-node-tuning-operator/tuned-j4f5j"
Jul 30 17:00:38.388271 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.387806    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/408b59ca-0265-474f-994c-799e6e74dc90-lib-modules\") pod \"tuned-j4f5j\" (UID: \"408b59ca-0265-474f-994c-799e6e74dc90\") " pod="openshift-cluster-node-tuning-operator/tuned-j4f5j"
Jul 30 17:00:38.388271 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.387805    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/408b59ca-0265-474f-994c-799e6e74dc90-sys\") pod \"tuned-j4f5j\" (UID: \"408b59ca-0265-474f-994c-799e6e74dc90\") " pod="openshift-cluster-node-tuning-operator/tuned-j4f5j"
Jul 30 17:00:38.388271 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.387835    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j89dv\" (UniqueName: \"kubernetes.io/projected/f366886f-2162-4153-a077-55dcd0fef1e8-kube-api-access-j89dv\") pod \"multus-additional-cni-plugins-mz6bl\" (UID: \"f366886f-2162-4153-a077-55dcd0fef1e8\") " pod="openshift-multus/multus-additional-cni-plugins-mz6bl"
Jul 30 17:00:38.388271 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.387848    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/408b59ca-0265-474f-994c-799e6e74dc90-sys\") pod \"tuned-j4f5j\" (UID: \"408b59ca-0265-474f-994c-799e6e74dc90\") " pod="openshift-cluster-node-tuning-operator/tuned-j4f5j"
Jul 30 17:00:38.388271 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.387866    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/408b59ca-0265-474f-994c-799e6e74dc90-run\") pod \"tuned-j4f5j\" (UID: \"408b59ca-0265-474f-994c-799e6e74dc90\") " pod="openshift-cluster-node-tuning-operator/tuned-j4f5j"
Jul 30 17:00:38.388271 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.387870    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/408b59ca-0265-474f-994c-799e6e74dc90-etc-sysctl-d\") pod \"tuned-j4f5j\" (UID: \"408b59ca-0265-474f-994c-799e6e74dc90\") " pod="openshift-cluster-node-tuning-operator/tuned-j4f5j"
Jul 30 17:00:38.388271 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.387892    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4e1630f-f153-4be6-883a-1fbeca875a69-tmp-dir\") pod \"node-resolver-c77l8\" (UID: \"e4e1630f-f153-4be6-883a-1fbeca875a69\") " pod="openshift-dns/node-resolver-c77l8"
Jul 30 17:00:38.388271 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.387931    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1-multus-cni-dir\") pod \"multus-2f2qc\" (UID: \"752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1\") " pod="openshift-multus/multus-2f2qc"
Jul 30 17:00:38.388271 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.387958    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/19e7774c-aa26-4558-a50c-0c5ff05d9767-env-overrides\") pod \"ovnkube-node-tblxl\" (UID: \"19e7774c-aa26-4558-a50c-0c5ff05d9767\") " pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:00:38.388271 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.387986    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/440bb960-b449-499c-b4b8-dabc4cc9d3cf-agent-certs\") pod \"konnectivity-agent-d8nwh\" (UID: \"440bb960-b449-499c-b4b8-dabc4cc9d3cf\") " pod="kube-system/konnectivity-agent-d8nwh"
Jul 30 17:00:38.388271 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.388011    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19e7774c-aa26-4558-a50c-0c5ff05d9767-etc-openvswitch\") pod \"ovnkube-node-tblxl\" (UID: \"19e7774c-aa26-4558-a50c-0c5ff05d9767\") " pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:00:38.388271 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.388039    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/19e7774c-aa26-4558-a50c-0c5ff05d9767-ovn-node-metrics-cert\") pod \"ovnkube-node-tblxl\" (UID: \"19e7774c-aa26-4558-a50c-0c5ff05d9767\") " pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:00:38.388271 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.388064    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/38224201-2e3f-4863-b749-06ec97bf2c1d-sys-fs\") pod \"aws-ebs-csi-driver-node-fbxnv\" (UID: \"38224201-2e3f-4863-b749-06ec97bf2c1d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbxnv"
Jul 30 17:00:38.388271 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.388096    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/adf2f8d3-a0f4-4514-b6e3-b1507fa0c0bb-serviceca\") pod \"node-ca-tdkrg\" (UID: \"adf2f8d3-a0f4-4514-b6e3-b1507fa0c0bb\") " pod="openshift-image-registry/node-ca-tdkrg"
Jul 30 17:00:38.388271 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.388119    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28dfd\" (UniqueName: \"kubernetes.io/projected/adf2f8d3-a0f4-4514-b6e3-b1507fa0c0bb-kube-api-access-28dfd\") pod \"node-ca-tdkrg\" (UID: \"adf2f8d3-a0f4-4514-b6e3-b1507fa0c0bb\") " pod="openshift-image-registry/node-ca-tdkrg"
Jul 30 17:00:38.389051 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.388136    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/408b59ca-0265-474f-994c-799e6e74dc90-etc-sysconfig\") pod \"tuned-j4f5j\" (UID: \"408b59ca-0265-474f-994c-799e6e74dc90\") " pod="openshift-cluster-node-tuning-operator/tuned-j4f5j"
Jul 30 17:00:38.389051 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.388152    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/408b59ca-0265-474f-994c-799e6e74dc90-etc-sysctl-conf\") pod \"tuned-j4f5j\" (UID: \"408b59ca-0265-474f-994c-799e6e74dc90\") " pod="openshift-cluster-node-tuning-operator/tuned-j4f5j"
Jul 30 17:00:38.389051 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.388194    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l6nm\" (UniqueName: \"kubernetes.io/projected/408b59ca-0265-474f-994c-799e6e74dc90-kube-api-access-7l6nm\") pod \"tuned-j4f5j\" (UID: \"408b59ca-0265-474f-994c-799e6e74dc90\") " pod="openshift-cluster-node-tuning-operator/tuned-j4f5j"
Jul 30 17:00:38.389051 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.388209    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f366886f-2162-4153-a077-55dcd0fef1e8-os-release\") pod \"multus-additional-cni-plugins-mz6bl\" (UID: \"f366886f-2162-4153-a077-55dcd0fef1e8\") " pod="openshift-multus/multus-additional-cni-plugins-mz6bl"
Jul 30 17:00:38.389051 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.388222    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/408b59ca-0265-474f-994c-799e6e74dc90-etc-sysconfig\") pod \"tuned-j4f5j\" (UID: \"408b59ca-0265-474f-994c-799e6e74dc90\") " pod="openshift-cluster-node-tuning-operator/tuned-j4f5j"
Jul 30 17:00:38.389051 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.388234    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f366886f-2162-4153-a077-55dcd0fef1e8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mz6bl\" (UID: \"f366886f-2162-4153-a077-55dcd0fef1e8\") " pod="openshift-multus/multus-additional-cni-plugins-mz6bl"
Jul 30 17:00:38.389051 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.388256    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1-cnibin\") pod \"multus-2f2qc\" (UID: \"752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1\") " pod="openshift-multus/multus-2f2qc"
Jul 30 17:00:38.389051 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.388335    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/408b59ca-0265-474f-994c-799e6e74dc90-etc-sysctl-conf\") pod \"tuned-j4f5j\" (UID: \"408b59ca-0265-474f-994c-799e6e74dc90\") " pod="openshift-cluster-node-tuning-operator/tuned-j4f5j"
Jul 30 17:00:38.389051 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.388349    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a47afe28-69d9-4b1b-8e40-b28645745766-host-slash\") pod \"iptables-alerter-46vf4\" (UID: \"a47afe28-69d9-4b1b-8e40-b28645745766\") " pod="openshift-network-operator/iptables-alerter-46vf4"
Jul 30 17:00:38.389051 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.388377    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19e7774c-aa26-4558-a50c-0c5ff05d9767-var-lib-openvswitch\") pod \"ovnkube-node-tblxl\" (UID: \"19e7774c-aa26-4558-a50c-0c5ff05d9767\") " pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:00:38.389051 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.388417    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/408b59ca-0265-474f-994c-799e6e74dc90-etc-systemd\") pod \"tuned-j4f5j\" (UID: \"408b59ca-0265-474f-994c-799e6e74dc90\") " pod="openshift-cluster-node-tuning-operator/tuned-j4f5j"
Jul 30 17:00:38.389051 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.388457    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f366886f-2162-4153-a077-55dcd0fef1e8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mz6bl\" (UID: \"f366886f-2162-4153-a077-55dcd0fef1e8\") " pod="openshift-multus/multus-additional-cni-plugins-mz6bl"
Jul 30 17:00:38.389051 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.388474    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/408b59ca-0265-474f-994c-799e6e74dc90-etc-systemd\") pod \"tuned-j4f5j\" (UID: \"408b59ca-0265-474f-994c-799e6e74dc90\") " pod="openshift-cluster-node-tuning-operator/tuned-j4f5j"
Jul 30 17:00:38.389051 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.388479    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e4e1630f-f153-4be6-883a-1fbeca875a69-hosts-file\") pod \"node-resolver-c77l8\" (UID: \"e4e1630f-f153-4be6-883a-1fbeca875a69\") " pod="openshift-dns/node-resolver-c77l8"
Jul 30 17:00:38.389051 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.388510    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19e7774c-aa26-4558-a50c-0c5ff05d9767-host-run-ovn-kubernetes\") pod \"ovnkube-node-tblxl\" (UID: \"19e7774c-aa26-4558-a50c-0c5ff05d9767\") " pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:00:38.389051 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.388531    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/19e7774c-aa26-4558-a50c-0c5ff05d9767-ovnkube-config\") pod \"ovnkube-node-tblxl\" (UID: \"19e7774c-aa26-4558-a50c-0c5ff05d9767\") " pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:00:38.389051 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.388547    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/19e7774c-aa26-4558-a50c-0c5ff05d9767-ovnkube-script-lib\") pod \"ovnkube-node-tblxl\" (UID: \"19e7774c-aa26-4558-a50c-0c5ff05d9767\") " pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:00:38.389689 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.388545    2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 30 17:00:38.389689 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.388561    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/38224201-2e3f-4863-b749-06ec97bf2c1d-device-dir\") pod \"aws-ebs-csi-driver-node-fbxnv\" (UID: \"38224201-2e3f-4863-b749-06ec97bf2c1d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbxnv"
Jul 30 17:00:38.389689 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.388585    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/408b59ca-0265-474f-994c-799e6e74dc90-var-lib-kubelet\") pod \"tuned-j4f5j\" (UID: \"408b59ca-0265-474f-994c-799e6e74dc90\") " pod="openshift-cluster-node-tuning-operator/tuned-j4f5j"
Jul 30 17:00:38.389689 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.388618    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/408b59ca-0265-474f-994c-799e6e74dc90-var-lib-kubelet\") pod \"tuned-j4f5j\" (UID: \"408b59ca-0265-474f-994c-799e6e74dc90\") " pod="openshift-cluster-node-tuning-operator/tuned-j4f5j"
Jul 30 17:00:38.389689 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.388641    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/408b59ca-0265-474f-994c-799e6e74dc90-etc-kubernetes\") pod \"tuned-j4f5j\" (UID: \"408b59ca-0265-474f-994c-799e6e74dc90\") " pod="openshift-cluster-node-tuning-operator/tuned-j4f5j"
Jul 30 17:00:38.389689 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.388670    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/408b59ca-0265-474f-994c-799e6e74dc90-etc-kubernetes\") pod \"tuned-j4f5j\" (UID: \"408b59ca-0265-474f-994c-799e6e74dc90\") " pod="openshift-cluster-node-tuning-operator/tuned-j4f5j"
Jul 30 17:00:38.389689 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.388673    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19e7774c-aa26-4558-a50c-0c5ff05d9767-host-slash\") pod \"ovnkube-node-tblxl\" (UID: \"19e7774c-aa26-4558-a50c-0c5ff05d9767\") " pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:00:38.389689 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.388699    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/38224201-2e3f-4863-b749-06ec97bf2c1d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fbxnv\" (UID: \"38224201-2e3f-4863-b749-06ec97bf2c1d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbxnv"
Jul 30 17:00:38.389689 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.388725    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/38224201-2e3f-4863-b749-06ec97bf2c1d-socket-dir\") pod \"aws-ebs-csi-driver-node-fbxnv\" (UID: \"38224201-2e3f-4863-b749-06ec97bf2c1d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbxnv"
Jul 30 17:00:38.389689 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.388753    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92pcn\" (UniqueName: \"kubernetes.io/projected/4f8b90e9-e705-4b9c-89f7-047151a2b098-kube-api-access-92pcn\") pod \"network-metrics-daemon-kbsf2\" (UID: \"4f8b90e9-e705-4b9c-89f7-047151a2b098\") " pod="openshift-multus/network-metrics-daemon-kbsf2"
Jul 30 17:00:38.389689 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.388780    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxfsr\" (UniqueName: \"kubernetes.io/projected/a47afe28-69d9-4b1b-8e40-b28645745766-kube-api-access-bxfsr\") pod \"iptables-alerter-46vf4\" (UID: \"a47afe28-69d9-4b1b-8e40-b28645745766\") " pod="openshift-network-operator/iptables-alerter-46vf4"
Jul 30 17:00:38.389689 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.388805    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/19e7774c-aa26-4558-a50c-0c5ff05d9767-host-kubelet\") pod \"ovnkube-node-tblxl\" (UID: \"19e7774c-aa26-4558-a50c-0c5ff05d9767\") " pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:00:38.389689 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.388828    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1-os-release\") pod \"multus-2f2qc\" (UID: \"752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1\") " pod="openshift-multus/multus-2f2qc"
Jul 30 17:00:38.389689 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.388862    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1-host-var-lib-kubelet\") pod \"multus-2f2qc\" (UID: \"752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1\") " pod="openshift-multus/multus-2f2qc"
Jul 30 17:00:38.389689 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.388905    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1-multus-conf-dir\") pod \"multus-2f2qc\" (UID: \"752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1\") " pod="openshift-multus/multus-2f2qc"
Jul 30 17:00:38.389689 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.388928    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/19e7774c-aa26-4558-a50c-0c5ff05d9767-run-systemd\") pod \"ovnkube-node-tblxl\" (UID: \"19e7774c-aa26-4558-a50c-0c5ff05d9767\") " pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:00:38.389689 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.388952    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19e7774c-aa26-4558-a50c-0c5ff05d9767-run-openvswitch\") pod \"ovnkube-node-tblxl\" (UID: \"19e7774c-aa26-4558-a50c-0c5ff05d9767\") " pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:00:38.390473 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.388994    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/408b59ca-0265-474f-994c-799e6e74dc90-tmp\") pod \"tuned-j4f5j\" (UID: \"408b59ca-0265-474f-994c-799e6e74dc90\") " pod="openshift-cluster-node-tuning-operator/tuned-j4f5j"
Jul 30 17:00:38.390473 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.389020    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f8b90e9-e705-4b9c-89f7-047151a2b098-metrics-certs\") pod \"network-metrics-daemon-kbsf2\" (UID: \"4f8b90e9-e705-4b9c-89f7-047151a2b098\") " pod="openshift-multus/network-metrics-daemon-kbsf2"
Jul 30 17:00:38.390473 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.389045    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1-multus-socket-dir-parent\") pod \"multus-2f2qc\" (UID: \"752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1\") " pod="openshift-multus/multus-2f2qc"
Jul 30 17:00:38.390473 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.389069    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/19e7774c-aa26-4558-a50c-0c5ff05d9767-host-cni-bin\") pod \"ovnkube-node-tblxl\" (UID: \"19e7774c-aa26-4558-a50c-0c5ff05d9767\") " pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:00:38.390473 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:38.389118    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 17:00:38.390473 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:38.389256    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f8b90e9-e705-4b9c-89f7-047151a2b098-metrics-certs podName:4f8b90e9-e705-4b9c-89f7-047151a2b098 nodeName:}" failed. No retries permitted until 2026-07-30 17:00:38.889236261 +0000 UTC m=+1.964536143 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4f8b90e9-e705-4b9c-89f7-047151a2b098-metrics-certs") pod "network-metrics-daemon-kbsf2" (UID: "4f8b90e9-e705-4b9c-89f7-047151a2b098") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 17:00:38.390473 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.389434    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/440bb960-b449-499c-b4b8-dabc4cc9d3cf-konnectivity-ca\") pod \"konnectivity-agent-d8nwh\" (UID: \"440bb960-b449-499c-b4b8-dabc4cc9d3cf\") " pod="kube-system/konnectivity-agent-d8nwh"
Jul 30 17:00:38.391697 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.391673    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/408b59ca-0265-474f-994c-799e6e74dc90-tmp\") pod \"tuned-j4f5j\" (UID: \"408b59ca-0265-474f-994c-799e6e74dc90\") " pod="openshift-cluster-node-tuning-operator/tuned-j4f5j"
Jul 30 17:00:38.391827 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.391709    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/408b59ca-0265-474f-994c-799e6e74dc90-etc-tuned\") pod \"tuned-j4f5j\" (UID: \"408b59ca-0265-474f-994c-799e6e74dc90\") " pod="openshift-cluster-node-tuning-operator/tuned-j4f5j"
Jul 30 17:00:38.391909 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.391889    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/440bb960-b449-499c-b4b8-dabc4cc9d3cf-agent-certs\") pod \"konnectivity-agent-d8nwh\" (UID: \"440bb960-b449-499c-b4b8-dabc4cc9d3cf\") " pod="kube-system/konnectivity-agent-d8nwh"
Jul 30 17:00:38.403960 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.403926    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92pcn\" (UniqueName: \"kubernetes.io/projected/4f8b90e9-e705-4b9c-89f7-047151a2b098-kube-api-access-92pcn\") pod \"network-metrics-daemon-kbsf2\" (UID: \"4f8b90e9-e705-4b9c-89f7-047151a2b098\") " pod="openshift-multus/network-metrics-daemon-kbsf2"
Jul 30 17:00:38.404084 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.403928    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l6nm\" (UniqueName: \"kubernetes.io/projected/408b59ca-0265-474f-994c-799e6e74dc90-kube-api-access-7l6nm\") pod \"tuned-j4f5j\" (UID: \"408b59ca-0265-474f-994c-799e6e74dc90\") " pod="openshift-cluster-node-tuning-operator/tuned-j4f5j"
Jul 30 17:00:38.489432 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.489385    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1-os-release\") pod \"multus-2f2qc\" (UID: \"752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1\") " pod="openshift-multus/multus-2f2qc"
Jul 30 17:00:38.489432 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.489426    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1-host-var-lib-kubelet\") pod \"multus-2f2qc\" (UID: \"752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1\") " pod="openshift-multus/multus-2f2qc"
Jul 30 17:00:38.489432 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.489443    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1-multus-conf-dir\") pod \"multus-2f2qc\" (UID: \"752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1\") " pod="openshift-multus/multus-2f2qc"
Jul 30 17:00:38.489727 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.489461    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/19e7774c-aa26-4558-a50c-0c5ff05d9767-run-systemd\") pod \"ovnkube-node-tblxl\" (UID: \"19e7774c-aa26-4558-a50c-0c5ff05d9767\") " pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:00:38.489727 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.489478    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19e7774c-aa26-4558-a50c-0c5ff05d9767-run-openvswitch\") pod \"ovnkube-node-tblxl\" (UID: \"19e7774c-aa26-4558-a50c-0c5ff05d9767\") " pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:00:38.489727 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.489509    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1-multus-socket-dir-parent\") pod \"multus-2f2qc\" (UID: \"752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1\") " pod="openshift-multus/multus-2f2qc"
Jul 30 17:00:38.489727 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.489521    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1-multus-conf-dir\") pod \"multus-2f2qc\" (UID: \"752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1\") " pod="openshift-multus/multus-2f2qc"
Jul 30 17:00:38.489727 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.489540    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/19e7774c-aa26-4558-a50c-0c5ff05d9767-run-systemd\") pod \"ovnkube-node-tblxl\" (UID: \"19e7774c-aa26-4558-a50c-0c5ff05d9767\") " pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:00:38.489727 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.489521    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1-host-var-lib-kubelet\") pod \"multus-2f2qc\" (UID: \"752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1\") " pod="openshift-multus/multus-2f2qc"
Jul 30 17:00:38.489727 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.489559    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19e7774c-aa26-4558-a50c-0c5ff05d9767-run-openvswitch\") pod \"ovnkube-node-tblxl\" (UID: \"19e7774c-aa26-4558-a50c-0c5ff05d9767\") " pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:00:38.489727 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.489571    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1-multus-socket-dir-parent\") pod \"multus-2f2qc\" (UID: \"752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1\") " pod="openshift-multus/multus-2f2qc"
Jul 30 17:00:38.489727 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.489532    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1-os-release\") pod \"multus-2f2qc\" (UID: \"752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1\") " pod="openshift-multus/multus-2f2qc"
Jul 30 17:00:38.489727 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.489583    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/19e7774c-aa26-4558-a50c-0c5ff05d9767-host-cni-bin\") pod \"ovnkube-node-tblxl\" (UID: \"19e7774c-aa26-4558-a50c-0c5ff05d9767\") " pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:00:38.489727 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.489614    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/38224201-2e3f-4863-b749-06ec97bf2c1d-registration-dir\") pod \"aws-ebs-csi-driver-node-fbxnv\" (UID: \"38224201-2e3f-4863-b749-06ec97bf2c1d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbxnv"
Jul 30 17:00:38.489727 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.489632    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1-host-var-lib-cni-bin\") pod \"multus-2f2qc\" (UID: \"752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1\") " pod="openshift-multus/multus-2f2qc"
Jul 30 17:00:38.489727 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.489650    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f366886f-2162-4153-a077-55dcd0fef1e8-system-cni-dir\") pod \"multus-additional-cni-plugins-mz6bl\" (UID: \"f366886f-2162-4153-a077-55dcd0fef1e8\") " pod="openshift-multus/multus-additional-cni-plugins-mz6bl"
Jul 30 17:00:38.489727 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.489673    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1-etc-kubernetes\") pod \"multus-2f2qc\" (UID: \"752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1\") " pod="openshift-multus/multus-2f2qc"
Jul 30 17:00:38.489727 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.489690    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1-host-var-lib-cni-bin\") pod \"multus-2f2qc\" (UID: \"752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1\") " pod="openshift-multus/multus-2f2qc"
Jul 30 17:00:38.489727 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.489701    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zd7m\" (UniqueName: \"kubernetes.io/projected/752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1-kube-api-access-5zd7m\") pod \"multus-2f2qc\" (UID: \"752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1\") " pod="openshift-multus/multus-2f2qc"
Jul 30 17:00:38.489727 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.489706    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f366886f-2162-4153-a077-55dcd0fef1e8-system-cni-dir\") pod \"multus-additional-cni-plugins-mz6bl\" (UID: \"f366886f-2162-4153-a077-55dcd0fef1e8\") " pod="openshift-multus/multus-additional-cni-plugins-mz6bl"
Jul 30 17:00:38.489727 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.489707    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/38224201-2e3f-4863-b749-06ec97bf2c1d-registration-dir\") pod \"aws-ebs-csi-driver-node-fbxnv\" (UID: \"38224201-2e3f-4863-b749-06ec97bf2c1d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbxnv"
Jul 30 17:00:38.490560 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.489723    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1-etc-kubernetes\") pod \"multus-2f2qc\" (UID: \"752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1\") " pod="openshift-multus/multus-2f2qc"
Jul 30 17:00:38.490560 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.489694    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/19e7774c-aa26-4558-a50c-0c5ff05d9767-host-cni-bin\") pod \"ovnkube-node-tblxl\" (UID: \"19e7774c-aa26-4558-a50c-0c5ff05d9767\") " pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:00:38.490560 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.489725    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/19e7774c-aa26-4558-a50c-0c5ff05d9767-run-ovn\") pod \"ovnkube-node-tblxl\" (UID: \"19e7774c-aa26-4558-a50c-0c5ff05d9767\") " pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:00:38.490560 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.489749    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/19e7774c-aa26-4558-a50c-0c5ff05d9767-run-ovn\") pod \"ovnkube-node-tblxl\" (UID: \"19e7774c-aa26-4558-a50c-0c5ff05d9767\") " pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:00:38.490560 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.489751    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f366886f-2162-4153-a077-55dcd0fef1e8-cni-binary-copy\") pod \"multus-additional-cni-plugins-mz6bl\" (UID: \"f366886f-2162-4153-a077-55dcd0fef1e8\") " pod="openshift-multus/multus-additional-cni-plugins-mz6bl"
Jul 30 17:00:38.490560 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.489771    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f366886f-2162-4153-a077-55dcd0fef1e8-cnibin\") pod \"multus-additional-cni-plugins-mz6bl\" (UID: \"f366886f-2162-4153-a077-55dcd0fef1e8\") " pod="openshift-multus/multus-additional-cni-plugins-mz6bl"
Jul 30 17:00:38.490560 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.489786    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1-host-var-lib-cni-multus\") pod \"multus-2f2qc\" (UID: \"752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1\") " pod="openshift-multus/multus-2f2qc"
Jul 30 17:00:38.490560 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.489802    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/19e7774c-aa26-4558-a50c-0c5ff05d9767-systemd-units\") pod \"ovnkube-node-tblxl\" (UID: \"19e7774c-aa26-4558-a50c-0c5ff05d9767\") " pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:00:38.490560 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.489818    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4c8x\" (UniqueName: \"kubernetes.io/projected/19e7774c-aa26-4558-a50c-0c5ff05d9767-kube-api-access-z4c8x\") pod \"ovnkube-node-tblxl\" (UID: \"19e7774c-aa26-4558-a50c-0c5ff05d9767\") " pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:00:38.490560 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.489833    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgq7j\" (UniqueName: \"kubernetes.io/projected/38224201-2e3f-4863-b749-06ec97bf2c1d-kube-api-access-hgq7j\") pod \"aws-ebs-csi-driver-node-fbxnv\" (UID: \"38224201-2e3f-4863-b749-06ec97bf2c1d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbxnv"
Jul 30 17:00:38.490560 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.489835    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1-host-var-lib-cni-multus\") pod \"multus-2f2qc\" (UID: \"752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1\") " pod="openshift-multus/multus-2f2qc"
Jul 30 17:00:38.490560 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.489850    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ad2d9b30-7d70-4236-9398-93ff51843c30-kubelet-config\") pod \"global-pull-secret-syncer-k7lnc\" (UID: \"ad2d9b30-7d70-4236-9398-93ff51843c30\") " pod="kube-system/global-pull-secret-syncer-k7lnc"
Jul 30 17:00:38.490560 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.489865    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1-multus-daemon-config\") pod \"multus-2f2qc\" (UID: \"752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1\") " pod="openshift-multus/multus-2f2qc"
Jul 30 17:00:38.490560 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.489869    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f366886f-2162-4153-a077-55dcd0fef1e8-cnibin\") pod \"multus-additional-cni-plugins-mz6bl\" (UID: \"f366886f-2162-4153-a077-55dcd0fef1e8\") " pod="openshift-multus/multus-additional-cni-plugins-mz6bl"
Jul 30 17:00:38.490560 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.489885    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a47afe28-69d9-4b1b-8e40-b28645745766-iptables-alerter-script\") pod \"iptables-alerter-46vf4\" (UID: \"a47afe28-69d9-4b1b-8e40-b28645745766\") " pod="openshift-network-operator/iptables-alerter-46vf4"
Jul 30 17:00:38.490560 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.489893    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/19e7774c-aa26-4558-a50c-0c5ff05d9767-systemd-units\") pod \"ovnkube-node-tblxl\" (UID: \"19e7774c-aa26-4558-a50c-0c5ff05d9767\") " pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:00:38.490560 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.489910    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1-host-run-k8s-cni-cncf-io\") pod \"multus-2f2qc\" (UID: \"752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1\") " pod="openshift-multus/multus-2f2qc"
Jul 30 17:00:38.491370 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.489934    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/19e7774c-aa26-4558-a50c-0c5ff05d9767-node-log\") pod \"ovnkube-node-tblxl\" (UID: \"19e7774c-aa26-4558-a50c-0c5ff05d9767\") " pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:00:38.491370 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.489957    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/38224201-2e3f-4863-b749-06ec97bf2c1d-etc-selinux\") pod \"aws-ebs-csi-driver-node-fbxnv\" (UID: \"38224201-2e3f-4863-b749-06ec97bf2c1d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbxnv"
Jul 30 17:00:38.491370 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.489980    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ad2d9b30-7d70-4236-9398-93ff51843c30-original-pull-secret\") pod \"global-pull-secret-syncer-k7lnc\" (UID: \"ad2d9b30-7d70-4236-9398-93ff51843c30\") " pod="kube-system/global-pull-secret-syncer-k7lnc"
Jul 30 17:00:38.491370 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.490004    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1-system-cni-dir\") pod \"multus-2f2qc\" (UID: \"752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1\") " pod="openshift-multus/multus-2f2qc"
Jul 30 17:00:38.491370 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.489930    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ad2d9b30-7d70-4236-9398-93ff51843c30-kubelet-config\") pod \"global-pull-secret-syncer-k7lnc\" (UID: \"ad2d9b30-7d70-4236-9398-93ff51843c30\") " pod="kube-system/global-pull-secret-syncer-k7lnc"
Jul 30 17:00:38.491370 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.490029    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1-cni-binary-copy\") pod \"multus-2f2qc\" (UID: \"752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1\") " pod="openshift-multus/multus-2f2qc"
Jul 30 17:00:38.491370 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.490052    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1-hostroot\") pod \"multus-2f2qc\" (UID: \"752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1\") " pod="openshift-multus/multus-2f2qc"
Jul 30 17:00:38.491370 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.490068    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1-host-run-k8s-cni-cncf-io\") pod \"multus-2f2qc\" (UID: \"752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1\") " pod="openshift-multus/multus-2f2qc"
Jul 30 17:00:38.491370 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.490074    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1-host-run-multus-certs\") pod \"multus-2f2qc\" (UID: \"752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1\") " pod="openshift-multus/multus-2f2qc"
Jul 30 17:00:38.491370 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.490105    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1-host-run-multus-certs\") pod \"multus-2f2qc\" (UID: \"752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1\") " pod="openshift-multus/multus-2f2qc"
Jul 30 17:00:38.491370 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:38.490115    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 30 17:00:38.491370 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.490123    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svhd5\" (UniqueName: \"kubernetes.io/projected/33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e-kube-api-access-svhd5\") pod \"network-check-target-ltj2t\" (UID: \"33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e\") " pod="openshift-network-diagnostics/network-check-target-ltj2t"
Jul 30 17:00:38.491370 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.490157    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1-system-cni-dir\") pod \"multus-2f2qc\" (UID: \"752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1\") " pod="openshift-multus/multus-2f2qc"
Jul 30 17:00:38.491370 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.490161    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19e7774c-aa26-4558-a50c-0c5ff05d9767-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tblxl\" (UID: \"19e7774c-aa26-4558-a50c-0c5ff05d9767\") " pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:00:38.491370 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:38.490203    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad2d9b30-7d70-4236-9398-93ff51843c30-original-pull-secret podName:ad2d9b30-7d70-4236-9398-93ff51843c30 nodeName:}" failed. No retries permitted until 2026-07-30 17:00:38.990163155 +0000 UTC m=+2.065463042 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ad2d9b30-7d70-4236-9398-93ff51843c30-original-pull-secret") pod "global-pull-secret-syncer-k7lnc" (UID: "ad2d9b30-7d70-4236-9398-93ff51843c30") : object "kube-system"/"original-pull-secret" not registered
Jul 30 17:00:38.491370 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.490220    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1-hostroot\") pod \"multus-2f2qc\" (UID: \"752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1\") " pod="openshift-multus/multus-2f2qc"
Jul 30 17:00:38.491370 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.490313    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/38224201-2e3f-4863-b749-06ec97bf2c1d-etc-selinux\") pod \"aws-ebs-csi-driver-node-fbxnv\" (UID: \"38224201-2e3f-4863-b749-06ec97bf2c1d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbxnv"
Jul 30 17:00:38.492066 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.490355    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19e7774c-aa26-4558-a50c-0c5ff05d9767-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tblxl\" (UID: \"19e7774c-aa26-4558-a50c-0c5ff05d9767\") " pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:00:38.492066 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.490358    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ad2d9b30-7d70-4236-9398-93ff51843c30-dbus\") pod \"global-pull-secret-syncer-k7lnc\" (UID: \"ad2d9b30-7d70-4236-9398-93ff51843c30\") " pod="kube-system/global-pull-secret-syncer-k7lnc"
Jul 30 17:00:38.492066 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.490396    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1-host-run-netns\") pod \"multus-2f2qc\" (UID: \"752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1\") " pod="openshift-multus/multus-2f2qc"
Jul 30 17:00:38.492066 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.490422    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/19e7774c-aa26-4558-a50c-0c5ff05d9767-host-run-netns\") pod \"ovnkube-node-tblxl\" (UID: \"19e7774c-aa26-4558-a50c-0c5ff05d9767\") " pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:00:38.492066 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.490218    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/19e7774c-aa26-4558-a50c-0c5ff05d9767-node-log\") pod \"ovnkube-node-tblxl\" (UID: \"19e7774c-aa26-4558-a50c-0c5ff05d9767\") " pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:00:38.492066 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.490445    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/19e7774c-aa26-4558-a50c-0c5ff05d9767-log-socket\") pod \"ovnkube-node-tblxl\" (UID: \"19e7774c-aa26-4558-a50c-0c5ff05d9767\") " pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:00:38.492066 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.490456    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f366886f-2162-4153-a077-55dcd0fef1e8-cni-binary-copy\") pod \"multus-additional-cni-plugins-mz6bl\" (UID: \"f366886f-2162-4153-a077-55dcd0fef1e8\") " pod="openshift-multus/multus-additional-cni-plugins-mz6bl"
Jul 30 17:00:38.492066 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.490486    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/19e7774c-aa26-4558-a50c-0c5ff05d9767-host-run-netns\") pod \"ovnkube-node-tblxl\" (UID: \"19e7774c-aa26-4558-a50c-0c5ff05d9767\") " pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:00:38.492066 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.490491    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1-host-run-netns\") pod \"multus-2f2qc\" (UID: \"752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1\") " pod="openshift-multus/multus-2f2qc"
Jul 30 17:00:38.492066 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.490498    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/19e7774c-aa26-4558-a50c-0c5ff05d9767-host-cni-netd\") pod \"ovnkube-node-tblxl\" (UID: \"19e7774c-aa26-4558-a50c-0c5ff05d9767\") " pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:00:38.492066 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.490491    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1-multus-daemon-config\") pod \"multus-2f2qc\" (UID: \"752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1\") " pod="openshift-multus/multus-2f2qc"
Jul 30 17:00:38.492066 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.490523    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ad2d9b30-7d70-4236-9398-93ff51843c30-dbus\") pod \"global-pull-secret-syncer-k7lnc\" (UID: \"ad2d9b30-7d70-4236-9398-93ff51843c30\") " pod="kube-system/global-pull-secret-syncer-k7lnc"
Jul 30 17:00:38.492066 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.490533    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f366886f-2162-4153-a077-55dcd0fef1e8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mz6bl\" (UID: \"f366886f-2162-4153-a077-55dcd0fef1e8\") " pod="openshift-multus/multus-additional-cni-plugins-mz6bl"
Jul 30 17:00:38.492066 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.490534    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/19e7774c-aa26-4558-a50c-0c5ff05d9767-host-cni-netd\") pod \"ovnkube-node-tblxl\" (UID: \"19e7774c-aa26-4558-a50c-0c5ff05d9767\") " pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:00:38.492066 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.490568    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/19e7774c-aa26-4558-a50c-0c5ff05d9767-log-socket\") pod \"ovnkube-node-tblxl\" (UID: \"19e7774c-aa26-4558-a50c-0c5ff05d9767\") " pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:00:38.492066 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.490566    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a47afe28-69d9-4b1b-8e40-b28645745766-iptables-alerter-script\") pod \"iptables-alerter-46vf4\" (UID: \"a47afe28-69d9-4b1b-8e40-b28645745766\") " pod="openshift-network-operator/iptables-alerter-46vf4"
Jul 30 17:00:38.492066 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.490562    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7sm5\" (UniqueName: \"kubernetes.io/projected/e4e1630f-f153-4be6-883a-1fbeca875a69-kube-api-access-c7sm5\") pod \"node-resolver-c77l8\" (UID: \"e4e1630f-f153-4be6-883a-1fbeca875a69\") " pod="openshift-dns/node-resolver-c77l8"
Jul 30 17:00:38.492575 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.490618    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/adf2f8d3-a0f4-4514-b6e3-b1507fa0c0bb-host\") pod \"node-ca-tdkrg\" (UID: \"adf2f8d3-a0f4-4514-b6e3-b1507fa0c0bb\") " pod="openshift-image-registry/node-ca-tdkrg"
Jul 30 17:00:38.492575 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.490649    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j89dv\" (UniqueName: \"kubernetes.io/projected/f366886f-2162-4153-a077-55dcd0fef1e8-kube-api-access-j89dv\") pod \"multus-additional-cni-plugins-mz6bl\" (UID: \"f366886f-2162-4153-a077-55dcd0fef1e8\") " pod="openshift-multus/multus-additional-cni-plugins-mz6bl"
Jul 30 17:00:38.492575 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.490678    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4e1630f-f153-4be6-883a-1fbeca875a69-tmp-dir\") pod \"node-resolver-c77l8\" (UID: \"e4e1630f-f153-4be6-883a-1fbeca875a69\") " pod="openshift-dns/node-resolver-c77l8"
Jul 30 17:00:38.492575 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.490702    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1-multus-cni-dir\") pod \"multus-2f2qc\" (UID: \"752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1\") " pod="openshift-multus/multus-2f2qc"
Jul 30 17:00:38.492575 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.490728    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1-cni-binary-copy\") pod \"multus-2f2qc\" (UID: \"752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1\") " pod="openshift-multus/multus-2f2qc"
Jul 30 17:00:38.492575 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.490750    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/19e7774c-aa26-4558-a50c-0c5ff05d9767-env-overrides\") pod \"ovnkube-node-tblxl\" (UID: \"19e7774c-aa26-4558-a50c-0c5ff05d9767\") " pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:00:38.492575 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.490776    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19e7774c-aa26-4558-a50c-0c5ff05d9767-etc-openvswitch\") pod \"ovnkube-node-tblxl\" (UID: \"19e7774c-aa26-4558-a50c-0c5ff05d9767\") " pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:00:38.492575 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.490800    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/19e7774c-aa26-4558-a50c-0c5ff05d9767-ovn-node-metrics-cert\") pod \"ovnkube-node-tblxl\" (UID: \"19e7774c-aa26-4558-a50c-0c5ff05d9767\") " pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:00:38.492575 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.490797    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/adf2f8d3-a0f4-4514-b6e3-b1507fa0c0bb-host\") pod \"node-ca-tdkrg\" (UID: \"adf2f8d3-a0f4-4514-b6e3-b1507fa0c0bb\") " pod="openshift-image-registry/node-ca-tdkrg"
Jul 30 17:00:38.492575 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.490825    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/38224201-2e3f-4863-b749-06ec97bf2c1d-sys-fs\") pod \"aws-ebs-csi-driver-node-fbxnv\" (UID: \"38224201-2e3f-4863-b749-06ec97bf2c1d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbxnv"
Jul 30 17:00:38.492575 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.490865    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19e7774c-aa26-4558-a50c-0c5ff05d9767-etc-openvswitch\") pod \"ovnkube-node-tblxl\" (UID: \"19e7774c-aa26-4558-a50c-0c5ff05d9767\") " pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:00:38.492575 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.490888    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/38224201-2e3f-4863-b749-06ec97bf2c1d-sys-fs\") pod \"aws-ebs-csi-driver-node-fbxnv\" (UID: \"38224201-2e3f-4863-b749-06ec97bf2c1d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbxnv"
Jul 30 17:00:38.492575 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.490933    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1-multus-cni-dir\") pod \"multus-2f2qc\" (UID: \"752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1\") " pod="openshift-multus/multus-2f2qc"
Jul 30 17:00:38.492575 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.490936    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/adf2f8d3-a0f4-4514-b6e3-b1507fa0c0bb-serviceca\") pod \"node-ca-tdkrg\" (UID: \"adf2f8d3-a0f4-4514-b6e3-b1507fa0c0bb\") " pod="openshift-image-registry/node-ca-tdkrg"
Jul 30 17:00:38.492575 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.490974    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28dfd\" (UniqueName: \"kubernetes.io/projected/adf2f8d3-a0f4-4514-b6e3-b1507fa0c0bb-kube-api-access-28dfd\") pod \"node-ca-tdkrg\" (UID: \"adf2f8d3-a0f4-4514-b6e3-b1507fa0c0bb\") " pod="openshift-image-registry/node-ca-tdkrg"
Jul 30 17:00:38.492575 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.490987    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f366886f-2162-4153-a077-55dcd0fef1e8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mz6bl\" (UID: \"f366886f-2162-4153-a077-55dcd0fef1e8\") " pod="openshift-multus/multus-additional-cni-plugins-mz6bl"
Jul 30 17:00:38.492575 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.490995    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4e1630f-f153-4be6-883a-1fbeca875a69-tmp-dir\") pod \"node-resolver-c77l8\" (UID: \"e4e1630f-f153-4be6-883a-1fbeca875a69\") " pod="openshift-dns/node-resolver-c77l8"
Jul 30 17:00:38.492575 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.491003    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f366886f-2162-4153-a077-55dcd0fef1e8-os-release\") pod \"multus-additional-cni-plugins-mz6bl\" (UID: \"f366886f-2162-4153-a077-55dcd0fef1e8\") " pod="openshift-multus/multus-additional-cni-plugins-mz6bl"
Jul 30 17:00:38.493073 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.491032    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f366886f-2162-4153-a077-55dcd0fef1e8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mz6bl\" (UID: \"f366886f-2162-4153-a077-55dcd0fef1e8\") " pod="openshift-multus/multus-additional-cni-plugins-mz6bl"
Jul 30 17:00:38.493073 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.491054    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1-cnibin\") pod \"multus-2f2qc\" (UID: \"752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1\") " pod="openshift-multus/multus-2f2qc"
Jul 30 17:00:38.493073 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.491075    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a47afe28-69d9-4b1b-8e40-b28645745766-host-slash\") pod \"iptables-alerter-46vf4\" (UID: \"a47afe28-69d9-4b1b-8e40-b28645745766\") " pod="openshift-network-operator/iptables-alerter-46vf4"
Jul 30 17:00:38.493073 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.491099    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19e7774c-aa26-4558-a50c-0c5ff05d9767-var-lib-openvswitch\") pod \"ovnkube-node-tblxl\" (UID: \"19e7774c-aa26-4558-a50c-0c5ff05d9767\") " pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:00:38.493073 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.491127    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f366886f-2162-4153-a077-55dcd0fef1e8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mz6bl\" (UID: \"f366886f-2162-4153-a077-55dcd0fef1e8\") " pod="openshift-multus/multus-additional-cni-plugins-mz6bl"
Jul 30 17:00:38.493073 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.491155    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e4e1630f-f153-4be6-883a-1fbeca875a69-hosts-file\") pod \"node-resolver-c77l8\" (UID: \"e4e1630f-f153-4be6-883a-1fbeca875a69\") " pod="openshift-dns/node-resolver-c77l8"
Jul 30 17:00:38.493073 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.491197    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19e7774c-aa26-4558-a50c-0c5ff05d9767-host-run-ovn-kubernetes\") pod \"ovnkube-node-tblxl\" (UID: \"19e7774c-aa26-4558-a50c-0c5ff05d9767\") " pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:00:38.493073 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.491220    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/19e7774c-aa26-4558-a50c-0c5ff05d9767-ovnkube-config\") pod \"ovnkube-node-tblxl\" (UID: \"19e7774c-aa26-4558-a50c-0c5ff05d9767\") " pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:00:38.493073 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.491234    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f366886f-2162-4153-a077-55dcd0fef1e8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mz6bl\" (UID: \"f366886f-2162-4153-a077-55dcd0fef1e8\") " pod="openshift-multus/multus-additional-cni-plugins-mz6bl"
Jul 30 17:00:38.493073 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.491249    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/19e7774c-aa26-4558-a50c-0c5ff05d9767-ovnkube-script-lib\") pod \"ovnkube-node-tblxl\" (UID: \"19e7774c-aa26-4558-a50c-0c5ff05d9767\") " pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:00:38.493073 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.491258    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19e7774c-aa26-4558-a50c-0c5ff05d9767-var-lib-openvswitch\") pod \"ovnkube-node-tblxl\" (UID: \"19e7774c-aa26-4558-a50c-0c5ff05d9767\") " pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:00:38.493073 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.491279    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/38224201-2e3f-4863-b749-06ec97bf2c1d-device-dir\") pod \"aws-ebs-csi-driver-node-fbxnv\" (UID: \"38224201-2e3f-4863-b749-06ec97bf2c1d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbxnv"
Jul 30 17:00:38.493073 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.491297    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1-cnibin\") pod \"multus-2f2qc\" (UID: \"752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1\") " pod="openshift-multus/multus-2f2qc"
Jul 30 17:00:38.493073 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.491306    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19e7774c-aa26-4558-a50c-0c5ff05d9767-host-slash\") pod \"ovnkube-node-tblxl\" (UID: \"19e7774c-aa26-4558-a50c-0c5ff05d9767\") " pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:00:38.493073 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.491318    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a47afe28-69d9-4b1b-8e40-b28645745766-host-slash\") pod \"iptables-alerter-46vf4\" (UID: \"a47afe28-69d9-4b1b-8e40-b28645745766\") " pod="openshift-network-operator/iptables-alerter-46vf4"
Jul 30 17:00:38.493073 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.491336    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/38224201-2e3f-4863-b749-06ec97bf2c1d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fbxnv\" (UID: \"38224201-2e3f-4863-b749-06ec97bf2c1d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbxnv"
Jul 30 17:00:38.493073 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.491345    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19e7774c-aa26-4558-a50c-0c5ff05d9767-host-run-ovn-kubernetes\") pod \"ovnkube-node-tblxl\" (UID: \"19e7774c-aa26-4558-a50c-0c5ff05d9767\") " pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:00:38.493597 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.491361    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/38224201-2e3f-4863-b749-06ec97bf2c1d-socket-dir\") pod \"aws-ebs-csi-driver-node-fbxnv\" (UID: \"38224201-2e3f-4863-b749-06ec97bf2c1d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbxnv"
Jul 30 17:00:38.493597 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.491390    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxfsr\" (UniqueName: \"kubernetes.io/projected/a47afe28-69d9-4b1b-8e40-b28645745766-kube-api-access-bxfsr\") pod \"iptables-alerter-46vf4\" (UID: \"a47afe28-69d9-4b1b-8e40-b28645745766\") " pod="openshift-network-operator/iptables-alerter-46vf4"
Jul 30 17:00:38.493597 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.491402    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e4e1630f-f153-4be6-883a-1fbeca875a69-hosts-file\") pod \"node-resolver-c77l8\" (UID: \"e4e1630f-f153-4be6-883a-1fbeca875a69\") " pod="openshift-dns/node-resolver-c77l8"
Jul 30 17:00:38.493597 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.491416    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/19e7774c-aa26-4558-a50c-0c5ff05d9767-host-kubelet\") pod \"ovnkube-node-tblxl\" (UID: \"19e7774c-aa26-4558-a50c-0c5ff05d9767\") " pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:00:38.493597 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.491443    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19e7774c-aa26-4558-a50c-0c5ff05d9767-host-slash\") pod \"ovnkube-node-tblxl\" (UID: \"19e7774c-aa26-4558-a50c-0c5ff05d9767\") " pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:00:38.493597 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.491463    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/19e7774c-aa26-4558-a50c-0c5ff05d9767-env-overrides\") pod \"ovnkube-node-tblxl\" (UID: \"19e7774c-aa26-4558-a50c-0c5ff05d9767\") " pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:00:38.493597 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.491492    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/19e7774c-aa26-4558-a50c-0c5ff05d9767-host-kubelet\") pod \"ovnkube-node-tblxl\" (UID: \"19e7774c-aa26-4558-a50c-0c5ff05d9767\") " pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:00:38.493597 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.491511    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/adf2f8d3-a0f4-4514-b6e3-b1507fa0c0bb-serviceca\") pod \"node-ca-tdkrg\" (UID: \"adf2f8d3-a0f4-4514-b6e3-b1507fa0c0bb\") " pod="openshift-image-registry/node-ca-tdkrg"
Jul 30 17:00:38.493597 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.491591    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/38224201-2e3f-4863-b749-06ec97bf2c1d-socket-dir\") pod \"aws-ebs-csi-driver-node-fbxnv\" (UID: \"38224201-2e3f-4863-b749-06ec97bf2c1d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbxnv"
Jul 30 17:00:38.493597 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.491648    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/38224201-2e3f-4863-b749-06ec97bf2c1d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fbxnv\" (UID: \"38224201-2e3f-4863-b749-06ec97bf2c1d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbxnv"
Jul 30 17:00:38.493597 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.491702    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/38224201-2e3f-4863-b749-06ec97bf2c1d-device-dir\") pod \"aws-ebs-csi-driver-node-fbxnv\" (UID: \"38224201-2e3f-4863-b749-06ec97bf2c1d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbxnv"
Jul 30 17:00:38.493597 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.491802    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f366886f-2162-4153-a077-55dcd0fef1e8-os-release\") pod \"multus-additional-cni-plugins-mz6bl\" (UID: \"f366886f-2162-4153-a077-55dcd0fef1e8\") " pod="openshift-multus/multus-additional-cni-plugins-mz6bl"
Jul 30 17:00:38.493597 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.491819    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f366886f-2162-4153-a077-55dcd0fef1e8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mz6bl\" (UID: \"f366886f-2162-4153-a077-55dcd0fef1e8\") " pod="openshift-multus/multus-additional-cni-plugins-mz6bl"
Jul 30 17:00:38.493597 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.491899    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/19e7774c-aa26-4558-a50c-0c5ff05d9767-ovnkube-script-lib\") pod \"ovnkube-node-tblxl\" (UID: \"19e7774c-aa26-4558-a50c-0c5ff05d9767\") " pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:00:38.493597 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.491957    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/19e7774c-aa26-4558-a50c-0c5ff05d9767-ovnkube-config\") pod \"ovnkube-node-tblxl\" (UID: \"19e7774c-aa26-4558-a50c-0c5ff05d9767\") " pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:00:38.493597 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.493246    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/19e7774c-aa26-4558-a50c-0c5ff05d9767-ovn-node-metrics-cert\") pod \"ovnkube-node-tblxl\" (UID: \"19e7774c-aa26-4558-a50c-0c5ff05d9767\") " pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:00:38.498713 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:38.498689    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 30 17:00:38.498713 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:38.498715    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 30 17:00:38.498883 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:38.498730    2580 projected.go:196] Error preparing data for projected volume kube-api-access-svhd5 for pod openshift-network-diagnostics/network-check-target-ltj2t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 17:00:38.498883 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:38.498810    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e-kube-api-access-svhd5 podName:33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e nodeName:}" failed. No retries permitted until 2026-07-30 17:00:38.998789482 +0000 UTC m=+2.074089364 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-svhd5" (UniqueName: "kubernetes.io/projected/33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e-kube-api-access-svhd5") pod "network-check-target-ltj2t" (UID: "33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 17:00:38.501252 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.501219    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zd7m\" (UniqueName: \"kubernetes.io/projected/752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1-kube-api-access-5zd7m\") pod \"multus-2f2qc\" (UID: \"752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1\") " pod="openshift-multus/multus-2f2qc"
Jul 30 17:00:38.501810 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.501776    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4c8x\" (UniqueName: \"kubernetes.io/projected/19e7774c-aa26-4558-a50c-0c5ff05d9767-kube-api-access-z4c8x\") pod \"ovnkube-node-tblxl\" (UID: \"19e7774c-aa26-4558-a50c-0c5ff05d9767\") " pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:00:38.502456 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.502426    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28dfd\" (UniqueName: \"kubernetes.io/projected/adf2f8d3-a0f4-4514-b6e3-b1507fa0c0bb-kube-api-access-28dfd\") pod \"node-ca-tdkrg\" (UID: \"adf2f8d3-a0f4-4514-b6e3-b1507fa0c0bb\") " pod="openshift-image-registry/node-ca-tdkrg"
Jul 30 17:00:38.502556 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.502431    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j89dv\" (UniqueName: \"kubernetes.io/projected/f366886f-2162-4153-a077-55dcd0fef1e8-kube-api-access-j89dv\") pod \"multus-additional-cni-plugins-mz6bl\" (UID: \"f366886f-2162-4153-a077-55dcd0fef1e8\") " pod="openshift-multus/multus-additional-cni-plugins-mz6bl"
Jul 30 17:00:38.502757 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.502736    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7sm5\" (UniqueName: \"kubernetes.io/projected/e4e1630f-f153-4be6-883a-1fbeca875a69-kube-api-access-c7sm5\") pod \"node-resolver-c77l8\" (UID: \"e4e1630f-f153-4be6-883a-1fbeca875a69\") " pod="openshift-dns/node-resolver-c77l8"
Jul 30 17:00:38.502826 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.502742    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgq7j\" (UniqueName: \"kubernetes.io/projected/38224201-2e3f-4863-b749-06ec97bf2c1d-kube-api-access-hgq7j\") pod \"aws-ebs-csi-driver-node-fbxnv\" (UID: \"38224201-2e3f-4863-b749-06ec97bf2c1d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbxnv"
Jul 30 17:00:38.503001 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.502985    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxfsr\" (UniqueName: \"kubernetes.io/projected/a47afe28-69d9-4b1b-8e40-b28645745766-kube-api-access-bxfsr\") pod \"iptables-alerter-46vf4\" (UID: \"a47afe28-69d9-4b1b-8e40-b28645745766\") " pod="openshift-network-operator/iptables-alerter-46vf4"
Jul 30 17:00:38.609003 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.608906    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-j4f5j"
Jul 30 17:00:38.630076 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.630030    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-d8nwh"
Jul 30 17:00:38.647150 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.647114    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbxnv"
Jul 30 17:00:38.661430 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.661401    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tdkrg"
Jul 30 17:00:38.679648 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.679617    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-c77l8"
Jul 30 17:00:38.693053 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.693021    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2f2qc"
Jul 30 17:00:38.715093 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.715053    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mz6bl"
Jul 30 17:00:38.727217 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.727187    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-46vf4"
Jul 30 17:00:38.732719 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.732688    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:00:38.879356 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.879255    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 30 17:00:38.894079 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.894039    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f8b90e9-e705-4b9c-89f7-047151a2b098-metrics-certs\") pod \"network-metrics-daemon-kbsf2\" (UID: \"4f8b90e9-e705-4b9c-89f7-047151a2b098\") " pod="openshift-multus/network-metrics-daemon-kbsf2"
Jul 30 17:00:38.894244 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:38.894164    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 17:00:38.894244 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:38.894235    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f8b90e9-e705-4b9c-89f7-047151a2b098-metrics-certs podName:4f8b90e9-e705-4b9c-89f7-047151a2b098 nodeName:}" failed. No retries permitted until 2026-07-30 17:00:39.894219642 +0000 UTC m=+2.969519511 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4f8b90e9-e705-4b9c-89f7-047151a2b098-metrics-certs") pod "network-metrics-daemon-kbsf2" (UID: "4f8b90e9-e705-4b9c-89f7-047151a2b098") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 17:00:38.995143 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:38.995110    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ad2d9b30-7d70-4236-9398-93ff51843c30-original-pull-secret\") pod \"global-pull-secret-syncer-k7lnc\" (UID: \"ad2d9b30-7d70-4236-9398-93ff51843c30\") " pod="kube-system/global-pull-secret-syncer-k7lnc"
Jul 30 17:00:38.995341 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:38.995275    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 30 17:00:38.995341 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:38.995333    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad2d9b30-7d70-4236-9398-93ff51843c30-original-pull-secret podName:ad2d9b30-7d70-4236-9398-93ff51843c30 nodeName:}" failed. No retries permitted until 2026-07-30 17:00:39.995318202 +0000 UTC m=+3.070618071 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ad2d9b30-7d70-4236-9398-93ff51843c30-original-pull-secret") pod "global-pull-secret-syncer-k7lnc" (UID: "ad2d9b30-7d70-4236-9398-93ff51843c30") : object "kube-system"/"original-pull-secret" not registered
Jul 30 17:00:39.095601 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:39.095558    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svhd5\" (UniqueName: \"kubernetes.io/projected/33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e-kube-api-access-svhd5\") pod \"network-check-target-ltj2t\" (UID: \"33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e\") " pod="openshift-network-diagnostics/network-check-target-ltj2t"
Jul 30 17:00:39.095800 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:39.095724    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 30 17:00:39.095800 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:39.095744    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 30 17:00:39.095800 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:39.095760    2580 projected.go:196] Error preparing data for projected volume kube-api-access-svhd5 for pod openshift-network-diagnostics/network-check-target-ltj2t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 17:00:39.095972 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:39.095817    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e-kube-api-access-svhd5 podName:33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e nodeName:}" failed. No retries permitted until 2026-07-30 17:00:40.095800761 +0000 UTC m=+3.171100631 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-svhd5" (UniqueName: "kubernetes.io/projected/33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e-kube-api-access-svhd5") pod "network-check-target-ltj2t" (UID: "33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 17:00:39.263841 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:39.263804    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5609348d151739648e886175062b0fc.slice/crio-cb269aaf32475feb5263a5e366f5aee1cee28bbbfc6f830e8cb51b480540d36d WatchSource:0}: Error finding container cb269aaf32475feb5263a5e366f5aee1cee28bbbfc6f830e8cb51b480540d36d: Status 404 returned error can't find the container with id cb269aaf32475feb5263a5e366f5aee1cee28bbbfc6f830e8cb51b480540d36d
Jul 30 17:00:39.264611 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:39.264221    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4e1630f_f153_4be6_883a_1fbeca875a69.slice/crio-683c12dc310a222ed4f3b65d9b7ede0a23ca6ffa93b1e07c411f9856a9bfe552 WatchSource:0}: Error finding container 683c12dc310a222ed4f3b65d9b7ede0a23ca6ffa93b1e07c411f9856a9bfe552: Status 404 returned error can't find the container with id 683c12dc310a222ed4f3b65d9b7ede0a23ca6ffa93b1e07c411f9856a9bfe552
Jul 30 17:00:39.271162 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:39.271139    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 17:00:39.275006 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:39.274974    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod752fa9f3_f89c_4b80_ac71_c32c0e4ae5b1.slice/crio-ecac6b4a0255972a0c4d7cd37f87d85540b6a69170849fa49a72461d1a89be30 WatchSource:0}: Error finding container ecac6b4a0255972a0c4d7cd37f87d85540b6a69170849fa49a72461d1a89be30: Status 404 returned error can't find the container with id ecac6b4a0255972a0c4d7cd37f87d85540b6a69170849fa49a72461d1a89be30
Jul 30 17:00:39.297188 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:39.297131    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod440bb960_b449_499c_b4b8_dabc4cc9d3cf.slice/crio-7fb44dccb3dc7e6807920377136422546de0787a6d3afdb8fb5eb8b04ebac19c WatchSource:0}: Error finding container 7fb44dccb3dc7e6807920377136422546de0787a6d3afdb8fb5eb8b04ebac19c: Status 404 returned error can't find the container with id 7fb44dccb3dc7e6807920377136422546de0787a6d3afdb8fb5eb8b04ebac19c
Jul 30 17:00:39.313600 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:39.313565    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 30 17:00:39.324072 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:39.324013    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-29 16:55:38 +0000 UTC" deadline="2028-04-25 10:32:54.313455864 +0000 UTC"
Jul 30 17:00:39.324072 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:39.324063    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15233h32m14.989397341s"
Jul 30 17:00:39.329909 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:39.329870    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod408b59ca_0265_474f_994c_799e6e74dc90.slice/crio-77065171f5a55ac1536137213f53e68c208cf6468cd748fcf2df0a4fe0672380 WatchSource:0}: Error finding container 77065171f5a55ac1536137213f53e68c208cf6468cd748fcf2df0a4fe0672380: Status 404 returned error can't find the container with id 77065171f5a55ac1536137213f53e68c208cf6468cd748fcf2df0a4fe0672380
Jul 30 17:00:39.351509 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:39.351469    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadf2f8d3_a0f4_4514_b6e3_b1507fa0c0bb.slice/crio-726c6eb174a60b79553de460ec19b16aa6a352b1a3886495a905a10c1215f0f5 WatchSource:0}: Error finding container 726c6eb174a60b79553de460ec19b16aa6a352b1a3886495a905a10c1215f0f5: Status 404 returned error can't find the container with id 726c6eb174a60b79553de460ec19b16aa6a352b1a3886495a905a10c1215f0f5
Jul 30 17:00:39.386871 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:39.386832    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf366886f_2162_4153_a077_55dcd0fef1e8.slice/crio-857a7b100456e96ab632d7d463e0b7d7f9e6ca757c2a8e0be3de0d4611e7a8ea WatchSource:0}: Error finding container 857a7b100456e96ab632d7d463e0b7d7f9e6ca757c2a8e0be3de0d4611e7a8ea: Status 404 returned error can't find the container with id 857a7b100456e96ab632d7d463e0b7d7f9e6ca757c2a8e0be3de0d4611e7a8ea
Jul 30 17:00:39.471385 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:39.471346    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda47afe28_69d9_4b1b_8e40_b28645745766.slice/crio-c78cdf2615d2aaa7542cfb4a26bf14f6a0390dbd46c6da048b7f5493535454eb WatchSource:0}: Error finding container c78cdf2615d2aaa7542cfb4a26bf14f6a0390dbd46c6da048b7f5493535454eb: Status 404 returned error can't find the container with id c78cdf2615d2aaa7542cfb4a26bf14f6a0390dbd46c6da048b7f5493535454eb
Jul 30 17:00:39.476840 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:39.476813    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kbsf2"
Jul 30 17:00:39.476973 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:39.476813    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ltj2t"
Jul 30 17:00:39.476973 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:39.476952    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kbsf2" podUID="4f8b90e9-e705-4b9c-89f7-047151a2b098"
Jul 30 17:00:39.477086 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:39.477026    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ltj2t" podUID="33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e"
Jul 30 17:00:39.480693 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:39.480631    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-46vf4" event={"ID":"a47afe28-69d9-4b1b-8e40-b28645745766","Type":"ContainerStarted","Data":"c78cdf2615d2aaa7542cfb4a26bf14f6a0390dbd46c6da048b7f5493535454eb"}
Jul 30 17:00:39.481631 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:39.481608    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-j4f5j" event={"ID":"408b59ca-0265-474f-994c-799e6e74dc90","Type":"ContainerStarted","Data":"77065171f5a55ac1536137213f53e68c208cf6468cd748fcf2df0a4fe0672380"}
Jul 30 17:00:39.482539 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:39.482521    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-c77l8" event={"ID":"e4e1630f-f153-4be6-883a-1fbeca875a69","Type":"ContainerStarted","Data":"683c12dc310a222ed4f3b65d9b7ede0a23ca6ffa93b1e07c411f9856a9bfe552"}
Jul 30 17:00:39.483419 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:39.483387    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-68.ec2.internal" event={"ID":"74460565773fc50cdf3aee810c639568","Type":"ContainerStarted","Data":"6f1eca932ef756709b3f8a1af6ebca5e17cd755ad6c87b01d77a862cdb5660e4"}
Jul 30 17:00:39.484426 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:39.484404    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mz6bl" event={"ID":"f366886f-2162-4153-a077-55dcd0fef1e8","Type":"ContainerStarted","Data":"857a7b100456e96ab632d7d463e0b7d7f9e6ca757c2a8e0be3de0d4611e7a8ea"}
Jul 30 17:00:39.485413 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:39.485391    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tdkrg" event={"ID":"adf2f8d3-a0f4-4514-b6e3-b1507fa0c0bb","Type":"ContainerStarted","Data":"726c6eb174a60b79553de460ec19b16aa6a352b1a3886495a905a10c1215f0f5"}
Jul 30 17:00:39.486693 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:39.486672    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-d8nwh" event={"ID":"440bb960-b449-499c-b4b8-dabc4cc9d3cf","Type":"ContainerStarted","Data":"7fb44dccb3dc7e6807920377136422546de0787a6d3afdb8fb5eb8b04ebac19c"}
Jul 30 17:00:39.487890 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:39.487868    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2f2qc" event={"ID":"752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1","Type":"ContainerStarted","Data":"ecac6b4a0255972a0c4d7cd37f87d85540b6a69170849fa49a72461d1a89be30"}
Jul 30 17:00:39.489288 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:39.489264    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-68.ec2.internal" event={"ID":"c5609348d151739648e886175062b0fc","Type":"ContainerStarted","Data":"cb269aaf32475feb5263a5e366f5aee1cee28bbbfc6f830e8cb51b480540d36d"}
Jul 30 17:00:39.598064 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:39.598029    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38224201_2e3f_4863_b749_06ec97bf2c1d.slice/crio-2349972531ddca391c5bf006bc7a81fcc2ae3dd6415f36bc622b48977cb53809 WatchSource:0}: Error finding container 2349972531ddca391c5bf006bc7a81fcc2ae3dd6415f36bc622b48977cb53809: Status 404 returned error can't find the container with id 2349972531ddca391c5bf006bc7a81fcc2ae3dd6415f36bc622b48977cb53809
Jul 30 17:00:39.881759 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:00:39.881711    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19e7774c_aa26_4558_a50c_0c5ff05d9767.slice/crio-0a7c2e07f2d3e62e82da60d1eb8b3e0c96ab75030c5438115f60d38f741d2aa8 WatchSource:0}: Error finding container 0a7c2e07f2d3e62e82da60d1eb8b3e0c96ab75030c5438115f60d38f741d2aa8: Status 404 returned error can't find the container with id 0a7c2e07f2d3e62e82da60d1eb8b3e0c96ab75030c5438115f60d38f741d2aa8
Jul 30 17:00:39.901118 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:39.901069    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f8b90e9-e705-4b9c-89f7-047151a2b098-metrics-certs\") pod \"network-metrics-daemon-kbsf2\" (UID: \"4f8b90e9-e705-4b9c-89f7-047151a2b098\") " pod="openshift-multus/network-metrics-daemon-kbsf2"
Jul 30 17:00:39.901451 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:39.901430    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 17:00:39.901543 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:39.901516    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f8b90e9-e705-4b9c-89f7-047151a2b098-metrics-certs podName:4f8b90e9-e705-4b9c-89f7-047151a2b098 nodeName:}" failed. No retries permitted until 2026-07-30 17:00:41.901495079 +0000 UTC m=+4.976794951 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4f8b90e9-e705-4b9c-89f7-047151a2b098-metrics-certs") pod "network-metrics-daemon-kbsf2" (UID: "4f8b90e9-e705-4b9c-89f7-047151a2b098") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 17:00:40.001953 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:40.001907    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ad2d9b30-7d70-4236-9398-93ff51843c30-original-pull-secret\") pod \"global-pull-secret-syncer-k7lnc\" (UID: \"ad2d9b30-7d70-4236-9398-93ff51843c30\") " pod="kube-system/global-pull-secret-syncer-k7lnc"
Jul 30 17:00:40.002168 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:40.002116    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 30 17:00:40.002266 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:40.002190    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad2d9b30-7d70-4236-9398-93ff51843c30-original-pull-secret podName:ad2d9b30-7d70-4236-9398-93ff51843c30 nodeName:}" failed. No retries permitted until 2026-07-30 17:00:42.002167181 +0000 UTC m=+5.077467053 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ad2d9b30-7d70-4236-9398-93ff51843c30-original-pull-secret") pod "global-pull-secret-syncer-k7lnc" (UID: "ad2d9b30-7d70-4236-9398-93ff51843c30") : object "kube-system"/"original-pull-secret" not registered
Jul 30 17:00:40.103089 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:40.103033    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svhd5\" (UniqueName: \"kubernetes.io/projected/33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e-kube-api-access-svhd5\") pod \"network-check-target-ltj2t\" (UID: \"33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e\") " pod="openshift-network-diagnostics/network-check-target-ltj2t"
Jul 30 17:00:40.103319 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:40.103260    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 30 17:00:40.103319 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:40.103285    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 30 17:00:40.103319 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:40.103301    2580 projected.go:196] Error preparing data for projected volume kube-api-access-svhd5 for pod openshift-network-diagnostics/network-check-target-ltj2t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 17:00:40.103497 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:40.103369    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e-kube-api-access-svhd5 podName:33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e nodeName:}" failed. No retries permitted until 2026-07-30 17:00:42.103346066 +0000 UTC m=+5.178645960 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-svhd5" (UniqueName: "kubernetes.io/projected/33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e-kube-api-access-svhd5") pod "network-check-target-ltj2t" (UID: "33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 17:00:40.325485 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:40.325344    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-29 16:55:38 +0000 UTC" deadline="2028-03-29 20:40:00.542320042 +0000 UTC"
Jul 30 17:00:40.325485 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:40.325421    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14595h39m20.21690401s"
Jul 30 17:00:40.477177 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:40.477135    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k7lnc"
Jul 30 17:00:40.477409 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:40.477306    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k7lnc" podUID="ad2d9b30-7d70-4236-9398-93ff51843c30"
Jul 30 17:00:40.514282 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:40.514220    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbxnv" event={"ID":"38224201-2e3f-4863-b749-06ec97bf2c1d","Type":"ContainerStarted","Data":"2349972531ddca391c5bf006bc7a81fcc2ae3dd6415f36bc622b48977cb53809"}
Jul 30 17:00:40.517638 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:40.517595    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tblxl" event={"ID":"19e7774c-aa26-4558-a50c-0c5ff05d9767","Type":"ContainerStarted","Data":"0a7c2e07f2d3e62e82da60d1eb8b3e0c96ab75030c5438115f60d38f741d2aa8"}
Jul 30 17:00:40.626489 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:40.626360    2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 30 17:00:41.134526 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:41.134474    2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 30 17:00:41.477127 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:41.477086    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kbsf2"
Jul 30 17:00:41.477701 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:41.477268    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kbsf2" podUID="4f8b90e9-e705-4b9c-89f7-047151a2b098"
Jul 30 17:00:41.477791 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:41.477698    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ltj2t"
Jul 30 17:00:41.477851 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:41.477814    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ltj2t" podUID="33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e"
Jul 30 17:00:41.921872 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:41.921767    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f8b90e9-e705-4b9c-89f7-047151a2b098-metrics-certs\") pod \"network-metrics-daemon-kbsf2\" (UID: \"4f8b90e9-e705-4b9c-89f7-047151a2b098\") " pod="openshift-multus/network-metrics-daemon-kbsf2"
Jul 30 17:00:41.922068 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:41.921970    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 17:00:41.922068 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:41.922040    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f8b90e9-e705-4b9c-89f7-047151a2b098-metrics-certs podName:4f8b90e9-e705-4b9c-89f7-047151a2b098 nodeName:}" failed. No retries permitted until 2026-07-30 17:00:45.922021082 +0000 UTC m=+8.997320956 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4f8b90e9-e705-4b9c-89f7-047151a2b098-metrics-certs") pod "network-metrics-daemon-kbsf2" (UID: "4f8b90e9-e705-4b9c-89f7-047151a2b098") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 17:00:42.023145 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:42.023093    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ad2d9b30-7d70-4236-9398-93ff51843c30-original-pull-secret\") pod \"global-pull-secret-syncer-k7lnc\" (UID: \"ad2d9b30-7d70-4236-9398-93ff51843c30\") " pod="kube-system/global-pull-secret-syncer-k7lnc"
Jul 30 17:00:42.023387 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:42.023296    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 30 17:00:42.023387 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:42.023373    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad2d9b30-7d70-4236-9398-93ff51843c30-original-pull-secret podName:ad2d9b30-7d70-4236-9398-93ff51843c30 nodeName:}" failed. No retries permitted until 2026-07-30 17:00:46.023352935 +0000 UTC m=+9.098652825 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ad2d9b30-7d70-4236-9398-93ff51843c30-original-pull-secret") pod "global-pull-secret-syncer-k7lnc" (UID: "ad2d9b30-7d70-4236-9398-93ff51843c30") : object "kube-system"/"original-pull-secret" not registered
Jul 30 17:00:42.123579 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:42.123533    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svhd5\" (UniqueName: \"kubernetes.io/projected/33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e-kube-api-access-svhd5\") pod \"network-check-target-ltj2t\" (UID: \"33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e\") " pod="openshift-network-diagnostics/network-check-target-ltj2t"
Jul 30 17:00:42.123826 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:42.123776    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 30 17:00:42.123826 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:42.123799    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 30 17:00:42.123826 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:42.123814    2580 projected.go:196] Error preparing data for projected volume kube-api-access-svhd5 for pod openshift-network-diagnostics/network-check-target-ltj2t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 17:00:42.124020 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:42.123884    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e-kube-api-access-svhd5 podName:33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e nodeName:}" failed. No retries permitted until 2026-07-30 17:00:46.123862121 +0000 UTC m=+9.199161993 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-svhd5" (UniqueName: "kubernetes.io/projected/33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e-kube-api-access-svhd5") pod "network-check-target-ltj2t" (UID: "33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 17:00:42.476857 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:42.476782    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k7lnc"
Jul 30 17:00:42.477097 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:42.476969    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k7lnc" podUID="ad2d9b30-7d70-4236-9398-93ff51843c30"
Jul 30 17:00:43.478983 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:43.478370    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ltj2t"
Jul 30 17:00:43.478983 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:43.478371    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kbsf2"
Jul 30 17:00:43.478983 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:43.478530    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ltj2t" podUID="33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e"
Jul 30 17:00:43.478983 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:43.478626    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kbsf2" podUID="4f8b90e9-e705-4b9c-89f7-047151a2b098"
Jul 30 17:00:44.477306 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:44.477250    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k7lnc"
Jul 30 17:00:44.477574 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:44.477413    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k7lnc" podUID="ad2d9b30-7d70-4236-9398-93ff51843c30"
Jul 30 17:00:45.478157 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:45.477721    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kbsf2"
Jul 30 17:00:45.478157 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:45.477746    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ltj2t"
Jul 30 17:00:45.478157 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:45.477876    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kbsf2" podUID="4f8b90e9-e705-4b9c-89f7-047151a2b098"
Jul 30 17:00:45.478157 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:45.478009    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ltj2t" podUID="33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e"
Jul 30 17:00:45.959952 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:45.959857    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f8b90e9-e705-4b9c-89f7-047151a2b098-metrics-certs\") pod \"network-metrics-daemon-kbsf2\" (UID: \"4f8b90e9-e705-4b9c-89f7-047151a2b098\") " pod="openshift-multus/network-metrics-daemon-kbsf2"
Jul 30 17:00:45.960128 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:45.960073    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 17:00:45.960211 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:45.960184    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f8b90e9-e705-4b9c-89f7-047151a2b098-metrics-certs podName:4f8b90e9-e705-4b9c-89f7-047151a2b098 nodeName:}" failed. No retries permitted until 2026-07-30 17:00:53.960145253 +0000 UTC m=+17.035445125 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4f8b90e9-e705-4b9c-89f7-047151a2b098-metrics-certs") pod "network-metrics-daemon-kbsf2" (UID: "4f8b90e9-e705-4b9c-89f7-047151a2b098") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 17:00:46.060638 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:46.060530    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ad2d9b30-7d70-4236-9398-93ff51843c30-original-pull-secret\") pod \"global-pull-secret-syncer-k7lnc\" (UID: \"ad2d9b30-7d70-4236-9398-93ff51843c30\") " pod="kube-system/global-pull-secret-syncer-k7lnc"
Jul 30 17:00:46.060846 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:46.060750    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 30 17:00:46.060915 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:46.060854    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad2d9b30-7d70-4236-9398-93ff51843c30-original-pull-secret podName:ad2d9b30-7d70-4236-9398-93ff51843c30 nodeName:}" failed. No retries permitted until 2026-07-30 17:00:54.060832567 +0000 UTC m=+17.136132437 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ad2d9b30-7d70-4236-9398-93ff51843c30-original-pull-secret") pod "global-pull-secret-syncer-k7lnc" (UID: "ad2d9b30-7d70-4236-9398-93ff51843c30") : object "kube-system"/"original-pull-secret" not registered
Jul 30 17:00:46.161720 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:46.161130    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svhd5\" (UniqueName: \"kubernetes.io/projected/33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e-kube-api-access-svhd5\") pod \"network-check-target-ltj2t\" (UID: \"33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e\") " pod="openshift-network-diagnostics/network-check-target-ltj2t"
Jul 30 17:00:46.161720 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:46.161294    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 30 17:00:46.161720 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:46.161311    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 30 17:00:46.161720 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:46.161321    2580 projected.go:196] Error preparing data for projected volume kube-api-access-svhd5 for pod openshift-network-diagnostics/network-check-target-ltj2t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 17:00:46.161720 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:46.161373    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e-kube-api-access-svhd5 podName:33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e nodeName:}" failed. No retries permitted until 2026-07-30 17:00:54.161354169 +0000 UTC m=+17.236654036 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-svhd5" (UniqueName: "kubernetes.io/projected/33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e-kube-api-access-svhd5") pod "network-check-target-ltj2t" (UID: "33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 17:00:46.477623 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:46.477583    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k7lnc"
Jul 30 17:00:46.477808 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:46.477739    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k7lnc" podUID="ad2d9b30-7d70-4236-9398-93ff51843c30"
Jul 30 17:00:47.478414 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:47.478336    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ltj2t"
Jul 30 17:00:47.478880 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:47.478464    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ltj2t" podUID="33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e"
Jul 30 17:00:47.478880 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:47.478650    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kbsf2"
Jul 30 17:00:47.478880 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:47.478806    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kbsf2" podUID="4f8b90e9-e705-4b9c-89f7-047151a2b098"
Jul 30 17:00:48.477270 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:48.477228    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k7lnc"
Jul 30 17:00:48.477482 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:48.477403    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k7lnc" podUID="ad2d9b30-7d70-4236-9398-93ff51843c30"
Jul 30 17:00:49.478417 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:49.477717    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kbsf2"
Jul 30 17:00:49.478417 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:49.478123    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kbsf2" podUID="4f8b90e9-e705-4b9c-89f7-047151a2b098"
Jul 30 17:00:49.478949 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:49.477730    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ltj2t"
Jul 30 17:00:49.478949 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:49.478594    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ltj2t" podUID="33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e"
Jul 30 17:00:50.477773 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:50.477738    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k7lnc"
Jul 30 17:00:50.477966 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:50.477857    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k7lnc" podUID="ad2d9b30-7d70-4236-9398-93ff51843c30"
Jul 30 17:00:51.477444 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:51.477398    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kbsf2"
Jul 30 17:00:51.477857 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:51.477539    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kbsf2" podUID="4f8b90e9-e705-4b9c-89f7-047151a2b098"
Jul 30 17:00:51.477857 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:51.477591    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ltj2t"
Jul 30 17:00:51.477857 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:51.477715    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ltj2t" podUID="33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e"
Jul 30 17:00:52.476993 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:52.476892    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k7lnc"
Jul 30 17:00:52.477200 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:52.477028    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k7lnc" podUID="ad2d9b30-7d70-4236-9398-93ff51843c30"
Jul 30 17:00:53.477526 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:53.477480    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kbsf2"
Jul 30 17:00:53.478021 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:53.477483    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ltj2t"
Jul 30 17:00:53.478021 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:53.477639    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kbsf2" podUID="4f8b90e9-e705-4b9c-89f7-047151a2b098"
Jul 30 17:00:53.478021 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:53.477706    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ltj2t" podUID="33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e"
Jul 30 17:00:54.027562 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:54.027515    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f8b90e9-e705-4b9c-89f7-047151a2b098-metrics-certs\") pod \"network-metrics-daemon-kbsf2\" (UID: \"4f8b90e9-e705-4b9c-89f7-047151a2b098\") " pod="openshift-multus/network-metrics-daemon-kbsf2"
Jul 30 17:00:54.027800 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:54.027716    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 17:00:54.027853 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:54.027804    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f8b90e9-e705-4b9c-89f7-047151a2b098-metrics-certs podName:4f8b90e9-e705-4b9c-89f7-047151a2b098 nodeName:}" failed. No retries permitted until 2026-07-30 17:01:10.027786878 +0000 UTC m=+33.103086750 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4f8b90e9-e705-4b9c-89f7-047151a2b098-metrics-certs") pod "network-metrics-daemon-kbsf2" (UID: "4f8b90e9-e705-4b9c-89f7-047151a2b098") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 17:00:54.128701 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:54.128661    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ad2d9b30-7d70-4236-9398-93ff51843c30-original-pull-secret\") pod \"global-pull-secret-syncer-k7lnc\" (UID: \"ad2d9b30-7d70-4236-9398-93ff51843c30\") " pod="kube-system/global-pull-secret-syncer-k7lnc"
Jul 30 17:00:54.128912 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:54.128841    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 30 17:00:54.128968 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:54.128935    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad2d9b30-7d70-4236-9398-93ff51843c30-original-pull-secret podName:ad2d9b30-7d70-4236-9398-93ff51843c30 nodeName:}" failed. No retries permitted until 2026-07-30 17:01:10.128911131 +0000 UTC m=+33.204211000 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ad2d9b30-7d70-4236-9398-93ff51843c30-original-pull-secret") pod "global-pull-secret-syncer-k7lnc" (UID: "ad2d9b30-7d70-4236-9398-93ff51843c30") : object "kube-system"/"original-pull-secret" not registered
Jul 30 17:00:54.229200 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:54.229141    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svhd5\" (UniqueName: \"kubernetes.io/projected/33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e-kube-api-access-svhd5\") pod \"network-check-target-ltj2t\" (UID: \"33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e\") " pod="openshift-network-diagnostics/network-check-target-ltj2t"
Jul 30 17:00:54.229396 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:54.229331    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 30 17:00:54.229396 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:54.229363    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 30 17:00:54.229396 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:54.229378    2580 projected.go:196] Error preparing data for projected volume kube-api-access-svhd5 for pod openshift-network-diagnostics/network-check-target-ltj2t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 17:00:54.229528 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:54.229447    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e-kube-api-access-svhd5 podName:33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e nodeName:}" failed. No retries permitted until 2026-07-30 17:01:10.229424053 +0000 UTC m=+33.304723925 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-svhd5" (UniqueName: "kubernetes.io/projected/33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e-kube-api-access-svhd5") pod "network-check-target-ltj2t" (UID: "33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 17:00:54.477147 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:54.477039    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k7lnc"
Jul 30 17:00:54.477330 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:54.477191    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k7lnc" podUID="ad2d9b30-7d70-4236-9398-93ff51843c30"
Jul 30 17:00:55.477427 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:55.477382    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kbsf2"
Jul 30 17:00:55.478001 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:55.477395    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ltj2t"
Jul 30 17:00:55.478001 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:55.477556    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kbsf2" podUID="4f8b90e9-e705-4b9c-89f7-047151a2b098"
Jul 30 17:00:55.478001 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:55.477614    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ltj2t" podUID="33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e"
Jul 30 17:00:56.476895 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:56.476857    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k7lnc"
Jul 30 17:00:56.477110 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:56.476983    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k7lnc" podUID="ad2d9b30-7d70-4236-9398-93ff51843c30"
Jul 30 17:00:57.478297 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:57.478255    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kbsf2"
Jul 30 17:00:57.478714 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:57.478409    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kbsf2" podUID="4f8b90e9-e705-4b9c-89f7-047151a2b098"
Jul 30 17:00:57.478830 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:57.478255    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ltj2t"
Jul 30 17:00:57.478972 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:57.478921    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ltj2t" podUID="33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e"
Jul 30 17:00:58.477012 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:58.476962    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k7lnc"
Jul 30 17:00:58.477247 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:58.477098    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k7lnc" podUID="ad2d9b30-7d70-4236-9398-93ff51843c30"
Jul 30 17:00:59.477691 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:59.477637    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ltj2t"
Jul 30 17:00:59.478268 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:00:59.477636    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kbsf2"
Jul 30 17:00:59.478268 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:59.477794    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ltj2t" podUID="33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e"
Jul 30 17:00:59.478268 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:00:59.477835    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kbsf2" podUID="4f8b90e9-e705-4b9c-89f7-047151a2b098"
Jul 30 17:01:00.477182 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:00.477130    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k7lnc"
Jul 30 17:01:00.477417 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:00.477288    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k7lnc" podUID="ad2d9b30-7d70-4236-9398-93ff51843c30"
Jul 30 17:01:00.569822 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:00.569586    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tblxl" event={"ID":"19e7774c-aa26-4558-a50c-0c5ff05d9767","Type":"ContainerStarted","Data":"fe556d5f63679046edd90b45f7e2170cfc67c0bf81446243306bbb737fa8f17c"}
Jul 30 17:01:00.571151 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:00.571125    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-j4f5j" event={"ID":"408b59ca-0265-474f-994c-799e6e74dc90","Type":"ContainerStarted","Data":"fc18563623b5a3b7a39f631e7161a8cd4cf44465f4fcdd42f7c7629d513d7123"}
Jul 30 17:01:00.572932 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:00.572898    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-68.ec2.internal" event={"ID":"74460565773fc50cdf3aee810c639568","Type":"ContainerStarted","Data":"7a679c99fbf0bb2bbbadad308853b3cf490e7e5db4cc0cb557f83a0518c25a5d"}
Jul 30 17:01:00.575807 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:00.575744    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2f2qc" event={"ID":"752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1","Type":"ContainerStarted","Data":"bc7beb9ad1a3053994736615795339683d29f9c051c94fe3cdade8325e07e575"}
Jul 30 17:01:00.590102 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:00.590035    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-j4f5j" podStartSLOduration=2.651386591 podStartE2EDuration="23.59001089s" totalImagesPullingTime="20.938624299s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 17:00:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:00.589373316 +0000 UTC m=+23.664673207" watchObservedRunningTime="2026-07-30 17:01:00.59001089 +0000 UTC m=+23.665310782"
Jul 30 17:01:00.607702 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:00.607643    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-2f2qc" podStartSLOduration=1.603017503 podStartE2EDuration="22.607625892s" totalImagesPullingTime="21.004608389s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 17:00:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:00.606946681 +0000 UTC m=+23.682246571" watchObservedRunningTime="2026-07-30 17:01:00.607625892 +0000 UTC m=+23.682925783"
Jul 30 17:01:00.621877 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:00.621820    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-68.ec2.internal" podStartSLOduration=22.621793356 podStartE2EDuration="22.621793356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 17:00:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:00.621349345 +0000 UTC m=+23.696649237" watchObservedRunningTime="2026-07-30 17:01:00.621793356 +0000 UTC m=+23.697093246"
Jul 30 17:01:01.480876 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:01.480832    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ltj2t"
Jul 30 17:01:01.481046 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:01.480829    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kbsf2"
Jul 30 17:01:01.481046 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:01.480981    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ltj2t" podUID="33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e"
Jul 30 17:01:01.481046 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:01.481035    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kbsf2" podUID="4f8b90e9-e705-4b9c-89f7-047151a2b098"
Jul 30 17:01:01.579808 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:01.579712    2580 generic.go:356] "Generic (PLEG): container finished" podID="c5609348d151739648e886175062b0fc" containerID="1ffc308d3b0ad2c0483483bd535efeed035b69e3344444d115171b539742cac8" exitCode=0
Jul 30 17:01:01.580630 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:01.579802    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-68.ec2.internal" event={"ID":"c5609348d151739648e886175062b0fc","Type":"ContainerDied","Data":"1ffc308d3b0ad2c0483483bd535efeed035b69e3344444d115171b539742cac8"}
Jul 30 17:01:01.585035 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:01.585006    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tblxl" event={"ID":"19e7774c-aa26-4558-a50c-0c5ff05d9767","Type":"ContainerStarted","Data":"d7b399d4a59e1e6efbe11ba28df85a146debcf313466a7de52578b8ce00d6216"}
Jul 30 17:01:01.585220 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:01.585099    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tblxl" event={"ID":"19e7774c-aa26-4558-a50c-0c5ff05d9767","Type":"ContainerStarted","Data":"9d5e8c8ffbce8d66ca87c1c4aca54eb923a78217bbc6663933ea133ebcdc9e6f"}
Jul 30 17:01:01.585220 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:01.585141    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tblxl" event={"ID":"19e7774c-aa26-4558-a50c-0c5ff05d9767","Type":"ContainerStarted","Data":"dc2a107576855bb11a0252fe3a72ed5e6ca484a79eeba2b293216d9e6626dd5b"}
Jul 30 17:01:01.585220 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:01.585203    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tblxl" event={"ID":"19e7774c-aa26-4558-a50c-0c5ff05d9767","Type":"ContainerStarted","Data":"701d73de6f3d739b86e101efaaba8e892b2cdc17b16bbe05e42b412c7a4d92ce"}
Jul 30 17:01:01.585373 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:01.585240    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tblxl" event={"ID":"19e7774c-aa26-4558-a50c-0c5ff05d9767","Type":"ContainerStarted","Data":"e3e15dc5093b61b0632d6b5f0fcb3a740a904f893c3b1dbdc17c09525cad890c"}
Jul 30 17:01:01.586460 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:01.586434    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-c77l8" event={"ID":"e4e1630f-f153-4be6-883a-1fbeca875a69","Type":"ContainerStarted","Data":"076331d742200cc808d82657825fa626b0f660c22fb0ca0ef2642ba9a97c8ed2"}
Jul 30 17:01:01.587885 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:01.587859    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbxnv" event={"ID":"38224201-2e3f-4863-b749-06ec97bf2c1d","Type":"ContainerStarted","Data":"6c39ee00b4349ca367c9ba1543d08b0b61ae1f8750ea71afaf8f211332b25725"}
Jul 30 17:01:01.592383 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:01.592352    2580 generic.go:356] "Generic (PLEG): container finished" podID="f366886f-2162-4153-a077-55dcd0fef1e8" containerID="fe72501b18ede340577697fceca9dbfa98e79f4d07ed7dbeea22ac6af2b82700" exitCode=0
Jul 30 17:01:01.592504 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:01.592425    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mz6bl" event={"ID":"f366886f-2162-4153-a077-55dcd0fef1e8","Type":"ContainerDied","Data":"fe72501b18ede340577697fceca9dbfa98e79f4d07ed7dbeea22ac6af2b82700"}
Jul 30 17:01:01.594032 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:01.593940    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tdkrg" event={"ID":"adf2f8d3-a0f4-4514-b6e3-b1507fa0c0bb","Type":"ContainerStarted","Data":"1d69744dddbbe91e005f87e06cd9443ba80505bea8233c5515596ad1770ab0b5"}
Jul 30 17:01:01.595482 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:01.595455    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-d8nwh" event={"ID":"440bb960-b449-499c-b4b8-dabc4cc9d3cf","Type":"ContainerStarted","Data":"8240258c0727c22c1ed4ec255612b4e06dab6a1c8b6ef7fa9c7118fdf1d16845"}
Jul 30 17:01:01.610333 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:01.610003    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-c77l8" podStartSLOduration=2.629683374 podStartE2EDuration="23.60997851s" totalImagesPullingTime="20.980295136s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 17:00:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:01.609106934 +0000 UTC m=+24.684406824" watchObservedRunningTime="2026-07-30 17:01:01.60997851 +0000 UTC m=+24.685278401"
Jul 30 17:01:01.645679 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:01.645629    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-tdkrg" podStartSLOduration=2.725839055 podStartE2EDuration="23.64561352s" totalImagesPullingTime="20.919774465s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 17:00:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:01.645375667 +0000 UTC m=+24.720675558" watchObservedRunningTime="2026-07-30 17:01:01.64561352 +0000 UTC m=+24.720913410"
Jul 30 17:01:01.661574 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:01.661525    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-d8nwh" podStartSLOduration=8.017834726 podStartE2EDuration="24.661506155s" totalImagesPullingTime="16.643671429s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:00:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:01.66111465 +0000 UTC m=+24.736414552" watchObservedRunningTime="2026-07-30 17:01:01.661506155 +0000 UTC m=+24.736806045"
Jul 30 17:01:01.944974 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:01.944941    2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 30 17:01:02.365346 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:02.365245    2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-30T17:01:01.944960882Z","UUID":"78a4e5dd-9603-4a97-bad3-f0cb11cedbc6","Handler":null,"Name":"","Endpoint":""}
Jul 30 17:01:02.366967 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:02.366948    2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 30 17:01:02.367102 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:02.366984    2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 30 17:01:02.477677 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:02.477642    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k7lnc"
Jul 30 17:01:02.477920 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:02.477761    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k7lnc" podUID="ad2d9b30-7d70-4236-9398-93ff51843c30"
Jul 30 17:01:02.599269 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:02.599234    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-46vf4" event={"ID":"a47afe28-69d9-4b1b-8e40-b28645745766","Type":"ContainerStarted","Data":"30f348199c0b63195a3f6c19fdae6fd3caf6cd35814d72dfa43025fe1ca4ce9f"}
Jul 30 17:01:02.601019 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:02.600996    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbxnv" event={"ID":"38224201-2e3f-4863-b749-06ec97bf2c1d","Type":"ContainerStarted","Data":"a866fbb764d2a416cfa265215a1a6cde079e01f6b1fb796f6e886805e2086e5a"}
Jul 30 17:01:02.602767 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:02.602740    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-68.ec2.internal" event={"ID":"c5609348d151739648e886175062b0fc","Type":"ContainerStarted","Data":"f480ccdc11a9429912f05d25613cc06e94ebddb87fef3ea8f820aafa48dd0bbb"}
Jul 30 17:01:02.615906 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:02.615785    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-46vf4" podStartSLOduration=3.838046862 podStartE2EDuration="24.615766287s" totalImagesPullingTime="20.777719425s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 17:00:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:02.615565445 +0000 UTC m=+25.690865340" watchObservedRunningTime="2026-07-30 17:01:02.615766287 +0000 UTC m=+25.691066178"
Jul 30 17:01:02.633049 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:02.632991    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-68.ec2.internal" podStartSLOduration=24.632974053 podStartE2EDuration="24.632974053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 17:00:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:02.632428206 +0000 UTC m=+25.707728096" watchObservedRunningTime="2026-07-30 17:01:02.632974053 +0000 UTC m=+25.708273943"
Jul 30 17:01:03.480720 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:03.480680    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kbsf2"
Jul 30 17:01:03.480720 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:03.480704    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ltj2t"
Jul 30 17:01:03.481002 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:03.480820    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ltj2t" podUID="33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e"
Jul 30 17:01:03.481002 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:03.480975    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kbsf2" podUID="4f8b90e9-e705-4b9c-89f7-047151a2b098"
Jul 30 17:01:03.616917 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:03.616462    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tblxl" event={"ID":"19e7774c-aa26-4558-a50c-0c5ff05d9767","Type":"ContainerStarted","Data":"ba7a6f7edf6c0aa70b4c371d29078ce4c6d59925c20197d2098391ba6aee2b1c"}
Jul 30 17:01:04.122095 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:04.122055    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-d8nwh"
Jul 30 17:01:04.122743 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:04.122712    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-d8nwh"
Jul 30 17:01:04.477478 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:04.477392    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k7lnc"
Jul 30 17:01:04.477642 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:04.477528    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k7lnc" podUID="ad2d9b30-7d70-4236-9398-93ff51843c30"
Jul 30 17:01:04.621069 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:04.621020    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbxnv" event={"ID":"38224201-2e3f-4863-b749-06ec97bf2c1d","Type":"ContainerStarted","Data":"b63a1d0bc4fba57565891e96de49d05ef8b5f24167627889b2fc0f9f77756873"}
Jul 30 17:01:04.621578 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:04.621403    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-d8nwh"
Jul 30 17:01:04.622121 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:04.622099    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-d8nwh"
Jul 30 17:01:04.640030 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:04.639973    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbxnv" podStartSLOduration=4.66517568 podStartE2EDuration="27.639955677s" totalImagesPullingTime="22.974779997s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 17:00:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:04.638657259 +0000 UTC m=+27.713957172" watchObservedRunningTime="2026-07-30 17:01:04.639955677 +0000 UTC m=+27.715255567"
Jul 30 17:01:05.477193 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:05.477067    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kbsf2"
Jul 30 17:01:05.477193 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:05.477098    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ltj2t"
Jul 30 17:01:05.477435 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:05.477257    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kbsf2" podUID="4f8b90e9-e705-4b9c-89f7-047151a2b098"
Jul 30 17:01:05.477435 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:05.477365    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ltj2t" podUID="33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e"
Jul 30 17:01:06.477578 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:06.477540    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k7lnc"
Jul 30 17:01:06.478009 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:06.477679    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k7lnc" podUID="ad2d9b30-7d70-4236-9398-93ff51843c30"
Jul 30 17:01:07.478386 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:07.478288    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kbsf2"
Jul 30 17:01:07.479103 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:07.478410    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kbsf2" podUID="4f8b90e9-e705-4b9c-89f7-047151a2b098"
Jul 30 17:01:07.479103 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:07.478456    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ltj2t"
Jul 30 17:01:07.479103 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:07.478546    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ltj2t" podUID="33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e"
Jul 30 17:01:07.629498 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:07.629453    2580 generic.go:356] "Generic (PLEG): container finished" podID="f366886f-2162-4153-a077-55dcd0fef1e8" containerID="a18e21e8f0d66c8a3a87554216813fb60ddf6382c11a625cef01022a5308cbee" exitCode=0
Jul 30 17:01:07.629795 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:07.629548    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mz6bl" event={"ID":"f366886f-2162-4153-a077-55dcd0fef1e8","Type":"ContainerDied","Data":"a18e21e8f0d66c8a3a87554216813fb60ddf6382c11a625cef01022a5308cbee"}
Jul 30 17:01:07.633577 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:07.633542    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tblxl" event={"ID":"19e7774c-aa26-4558-a50c-0c5ff05d9767","Type":"ContainerStarted","Data":"642459374ba714141b8ae2e35d047c224f3703015612feea063fa181559331d4"}
Jul 30 17:01:07.633905 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:07.633888    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:01:07.634015 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:07.633997    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:01:07.634093 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:07.634076    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:01:07.649852 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:07.649821    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:01:07.650007 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:07.649926    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:01:07.681791 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:07.681731    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-tblxl" podStartSLOduration=9.181205465 podStartE2EDuration="29.681713677s" totalImagesPullingTime="20.500508212s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 17:00:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:07.681185813 +0000 UTC m=+30.756485694" watchObservedRunningTime="2026-07-30 17:01:07.681713677 +0000 UTC m=+30.757013568"
Jul 30 17:01:08.477201 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:08.477146    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k7lnc"
Jul 30 17:01:08.477409 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:08.477290    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k7lnc" podUID="ad2d9b30-7d70-4236-9398-93ff51843c30"
Jul 30 17:01:08.958882 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:08.958849    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-k7lnc"]
Jul 30 17:01:08.959489 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:08.958969    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k7lnc"
Jul 30 17:01:08.959489 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:08.959056    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k7lnc" podUID="ad2d9b30-7d70-4236-9398-93ff51843c30"
Jul 30 17:01:08.961142 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:08.961106    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ltj2t"]
Jul 30 17:01:08.961302 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:08.961281    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ltj2t"
Jul 30 17:01:08.961421 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:08.961402    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ltj2t" podUID="33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e"
Jul 30 17:01:08.977098 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:08.977065    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kbsf2"]
Jul 30 17:01:08.977305 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:08.977222    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kbsf2"
Jul 30 17:01:08.977360 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:08.977326    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kbsf2" podUID="4f8b90e9-e705-4b9c-89f7-047151a2b098"
Jul 30 17:01:10.048653 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:10.048620    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f8b90e9-e705-4b9c-89f7-047151a2b098-metrics-certs\") pod \"network-metrics-daemon-kbsf2\" (UID: \"4f8b90e9-e705-4b9c-89f7-047151a2b098\") " pod="openshift-multus/network-metrics-daemon-kbsf2"
Jul 30 17:01:10.049133 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:10.048746    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 17:01:10.049133 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:10.048807    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f8b90e9-e705-4b9c-89f7-047151a2b098-metrics-certs podName:4f8b90e9-e705-4b9c-89f7-047151a2b098 nodeName:}" failed. No retries permitted until 2026-07-30 17:01:42.048790585 +0000 UTC m=+65.124090456 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4f8b90e9-e705-4b9c-89f7-047151a2b098-metrics-certs") pod "network-metrics-daemon-kbsf2" (UID: "4f8b90e9-e705-4b9c-89f7-047151a2b098") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 17:01:10.149426 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:10.149377    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ad2d9b30-7d70-4236-9398-93ff51843c30-original-pull-secret\") pod \"global-pull-secret-syncer-k7lnc\" (UID: \"ad2d9b30-7d70-4236-9398-93ff51843c30\") " pod="kube-system/global-pull-secret-syncer-k7lnc"
Jul 30 17:01:10.149616 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:10.149526    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 30 17:01:10.149616 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:10.149605    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad2d9b30-7d70-4236-9398-93ff51843c30-original-pull-secret podName:ad2d9b30-7d70-4236-9398-93ff51843c30 nodeName:}" failed. No retries permitted until 2026-07-30 17:01:42.149583297 +0000 UTC m=+65.224883184 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ad2d9b30-7d70-4236-9398-93ff51843c30-original-pull-secret") pod "global-pull-secret-syncer-k7lnc" (UID: "ad2d9b30-7d70-4236-9398-93ff51843c30") : object "kube-system"/"original-pull-secret" not registered
Jul 30 17:01:10.250763 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:10.250724    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svhd5\" (UniqueName: \"kubernetes.io/projected/33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e-kube-api-access-svhd5\") pod \"network-check-target-ltj2t\" (UID: \"33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e\") " pod="openshift-network-diagnostics/network-check-target-ltj2t"
Jul 30 17:01:10.250980 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:10.250922    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 30 17:01:10.250980 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:10.250952    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 30 17:01:10.250980 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:10.250965    2580 projected.go:196] Error preparing data for projected volume kube-api-access-svhd5 for pod openshift-network-diagnostics/network-check-target-ltj2t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 17:01:10.251137 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:10.251035    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e-kube-api-access-svhd5 podName:33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e nodeName:}" failed. No retries permitted until 2026-07-30 17:01:42.251014617 +0000 UTC m=+65.326314488 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-svhd5" (UniqueName: "kubernetes.io/projected/33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e-kube-api-access-svhd5") pod "network-check-target-ltj2t" (UID: "33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 17:01:10.477198 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:10.477071    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k7lnc"
Jul 30 17:01:10.477198 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:10.477117    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kbsf2"
Jul 30 17:01:10.477198 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:10.477117    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ltj2t"
Jul 30 17:01:10.477491 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:10.477223    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k7lnc" podUID="ad2d9b30-7d70-4236-9398-93ff51843c30"
Jul 30 17:01:10.477491 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:10.477311    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kbsf2" podUID="4f8b90e9-e705-4b9c-89f7-047151a2b098"
Jul 30 17:01:10.477491 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:10.477395    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ltj2t" podUID="33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e"
Jul 30 17:01:10.642357 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:10.642324    2580 generic.go:356] "Generic (PLEG): container finished" podID="f366886f-2162-4153-a077-55dcd0fef1e8" containerID="87d338d41c8fc9420da1770d8c8c2c12612367423a796cf22e0e51c800a9e1f8" exitCode=0
Jul 30 17:01:10.642566 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:10.642392    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mz6bl" event={"ID":"f366886f-2162-4153-a077-55dcd0fef1e8","Type":"ContainerDied","Data":"87d338d41c8fc9420da1770d8c8c2c12612367423a796cf22e0e51c800a9e1f8"}
Jul 30 17:01:12.477305 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:12.477210    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kbsf2"
Jul 30 17:01:12.477305 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:12.477211    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ltj2t"
Jul 30 17:01:12.477707 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:12.477323    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k7lnc"
Jul 30 17:01:12.477707 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:12.477322    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kbsf2" podUID="4f8b90e9-e705-4b9c-89f7-047151a2b098"
Jul 30 17:01:12.477707 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:12.477397    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ltj2t" podUID="33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e"
Jul 30 17:01:12.477707 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:12.477448    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k7lnc" podUID="ad2d9b30-7d70-4236-9398-93ff51843c30"
Jul 30 17:01:12.652636 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:12.652587    2580 generic.go:356] "Generic (PLEG): container finished" podID="f366886f-2162-4153-a077-55dcd0fef1e8" containerID="c2b945dd3dd961a7d97edeb0fd63311ba9f9d1c43b96c786e464293ff709886d" exitCode=0
Jul 30 17:01:12.652810 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:12.652667    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mz6bl" event={"ID":"f366886f-2162-4153-a077-55dcd0fef1e8","Type":"ContainerDied","Data":"c2b945dd3dd961a7d97edeb0fd63311ba9f9d1c43b96c786e464293ff709886d"}
Jul 30 17:01:13.266920 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:13.266897    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-68.ec2.internal" event="NodeReady"
Jul 30 17:01:13.267161 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:13.267025    2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 30 17:01:13.320727 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:13.320685    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8l786"]
Jul 30 17:01:13.322814 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:13.322783    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-d2cnh"]
Jul 30 17:01:13.323005 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:13.322913    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8l786"
Jul 30 17:01:13.324521 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:13.324498    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-d2cnh"
Jul 30 17:01:13.325658 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:13.325635    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bcbk6\""
Jul 30 17:01:13.325778 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:13.325696    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 30 17:01:13.325778 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:13.325699    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 30 17:01:13.327046 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:13.327026    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 30 17:01:13.327127 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:13.327065    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-8ckwn\""
Jul 30 17:01:13.327127 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:13.327093    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 30 17:01:13.327127 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:13.327114    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 30 17:01:13.339721 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:13.339686    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-d2cnh"]
Jul 30 17:01:13.340247 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:13.340228    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8l786"]
Jul 30 17:01:13.473604 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:13.473550    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d7e1b15f-2f31-42d4-b8e6-56bee84ad252-config-volume\") pod \"dns-default-8l786\" (UID: \"d7e1b15f-2f31-42d4-b8e6-56bee84ad252\") " pod="openshift-dns/dns-default-8l786"
Jul 30 17:01:13.473801 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:13.473616    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d7e1b15f-2f31-42d4-b8e6-56bee84ad252-metrics-tls\") pod \"dns-default-8l786\" (UID: \"d7e1b15f-2f31-42d4-b8e6-56bee84ad252\") " pod="openshift-dns/dns-default-8l786"
Jul 30 17:01:13.473801 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:13.473677    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vzhx\" (UniqueName: \"kubernetes.io/projected/439955b8-2c26-4450-8a8c-ab8b9f7f0b3f-kube-api-access-6vzhx\") pod \"ingress-canary-d2cnh\" (UID: \"439955b8-2c26-4450-8a8c-ab8b9f7f0b3f\") " pod="openshift-ingress-canary/ingress-canary-d2cnh"
Jul 30 17:01:13.473801 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:13.473762    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7e1b15f-2f31-42d4-b8e6-56bee84ad252-tmp-dir\") pod \"dns-default-8l786\" (UID: \"d7e1b15f-2f31-42d4-b8e6-56bee84ad252\") " pod="openshift-dns/dns-default-8l786"
Jul 30 17:01:13.473801 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:13.473790    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/439955b8-2c26-4450-8a8c-ab8b9f7f0b3f-cert\") pod \"ingress-canary-d2cnh\" (UID: \"439955b8-2c26-4450-8a8c-ab8b9f7f0b3f\") " pod="openshift-ingress-canary/ingress-canary-d2cnh"
Jul 30 17:01:13.473965 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:13.473825    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvqnp\" (UniqueName: \"kubernetes.io/projected/d7e1b15f-2f31-42d4-b8e6-56bee84ad252-kube-api-access-hvqnp\") pod \"dns-default-8l786\" (UID: \"d7e1b15f-2f31-42d4-b8e6-56bee84ad252\") " pod="openshift-dns/dns-default-8l786"
Jul 30 17:01:13.574564 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:13.574463    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d7e1b15f-2f31-42d4-b8e6-56bee84ad252-config-volume\") pod \"dns-default-8l786\" (UID: \"d7e1b15f-2f31-42d4-b8e6-56bee84ad252\") " pod="openshift-dns/dns-default-8l786"
Jul 30 17:01:13.574564 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:13.574526    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d7e1b15f-2f31-42d4-b8e6-56bee84ad252-metrics-tls\") pod \"dns-default-8l786\" (UID: \"d7e1b15f-2f31-42d4-b8e6-56bee84ad252\") " pod="openshift-dns/dns-default-8l786"
Jul 30 17:01:13.574564 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:13.574558    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vzhx\" (UniqueName: \"kubernetes.io/projected/439955b8-2c26-4450-8a8c-ab8b9f7f0b3f-kube-api-access-6vzhx\") pod \"ingress-canary-d2cnh\" (UID: \"439955b8-2c26-4450-8a8c-ab8b9f7f0b3f\") " pod="openshift-ingress-canary/ingress-canary-d2cnh"
Jul 30 17:01:13.575140 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:13.574609    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7e1b15f-2f31-42d4-b8e6-56bee84ad252-tmp-dir\") pod \"dns-default-8l786\" (UID: \"d7e1b15f-2f31-42d4-b8e6-56bee84ad252\") " pod="openshift-dns/dns-default-8l786"
Jul 30 17:01:13.575140 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:13.574630    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/439955b8-2c26-4450-8a8c-ab8b9f7f0b3f-cert\") pod \"ingress-canary-d2cnh\" (UID: \"439955b8-2c26-4450-8a8c-ab8b9f7f0b3f\") " pod="openshift-ingress-canary/ingress-canary-d2cnh"
Jul 30 17:01:13.575140 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:13.574681    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvqnp\" (UniqueName: \"kubernetes.io/projected/d7e1b15f-2f31-42d4-b8e6-56bee84ad252-kube-api-access-hvqnp\") pod \"dns-default-8l786\" (UID: \"d7e1b15f-2f31-42d4-b8e6-56bee84ad252\") " pod="openshift-dns/dns-default-8l786"
Jul 30 17:01:13.575140 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:13.574718    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 30 17:01:13.575140 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:13.574796    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7e1b15f-2f31-42d4-b8e6-56bee84ad252-metrics-tls podName:d7e1b15f-2f31-42d4-b8e6-56bee84ad252 nodeName:}" failed. No retries permitted until 2026-07-30 17:01:14.074773437 +0000 UTC m=+37.150073305 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d7e1b15f-2f31-42d4-b8e6-56bee84ad252-metrics-tls") pod "dns-default-8l786" (UID: "d7e1b15f-2f31-42d4-b8e6-56bee84ad252") : secret "dns-default-metrics-tls" not found
Jul 30 17:01:13.575140 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:13.575135    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d7e1b15f-2f31-42d4-b8e6-56bee84ad252-config-volume\") pod \"dns-default-8l786\" (UID: \"d7e1b15f-2f31-42d4-b8e6-56bee84ad252\") " pod="openshift-dns/dns-default-8l786"
Jul 30 17:01:13.575455 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:13.575134    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7e1b15f-2f31-42d4-b8e6-56bee84ad252-tmp-dir\") pod \"dns-default-8l786\" (UID: \"d7e1b15f-2f31-42d4-b8e6-56bee84ad252\") " pod="openshift-dns/dns-default-8l786"
Jul 30 17:01:13.580855 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:13.580826    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/439955b8-2c26-4450-8a8c-ab8b9f7f0b3f-cert\") pod \"ingress-canary-d2cnh\" (UID: \"439955b8-2c26-4450-8a8c-ab8b9f7f0b3f\") " pod="openshift-ingress-canary/ingress-canary-d2cnh"
Jul 30 17:01:13.584338 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:13.584306    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvqnp\" (UniqueName: \"kubernetes.io/projected/d7e1b15f-2f31-42d4-b8e6-56bee84ad252-kube-api-access-hvqnp\") pod \"dns-default-8l786\" (UID: \"d7e1b15f-2f31-42d4-b8e6-56bee84ad252\") " pod="openshift-dns/dns-default-8l786"
Jul 30 17:01:13.584591 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:13.584563    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vzhx\" (UniqueName: \"kubernetes.io/projected/439955b8-2c26-4450-8a8c-ab8b9f7f0b3f-kube-api-access-6vzhx\") pod \"ingress-canary-d2cnh\" (UID: \"439955b8-2c26-4450-8a8c-ab8b9f7f0b3f\") " pod="openshift-ingress-canary/ingress-canary-d2cnh"
Jul 30 17:01:13.638592 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:13.638553    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-d2cnh"
Jul 30 17:01:13.807973 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:13.807755    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-d2cnh"]
Jul 30 17:01:13.813678 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:01:13.813634    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod439955b8_2c26_4450_8a8c_ab8b9f7f0b3f.slice/crio-18df223f43a0dab52ba8d3f9a7efce98b24930cdb9c9bbac4bbea3dfcdb4ef51 WatchSource:0}: Error finding container 18df223f43a0dab52ba8d3f9a7efce98b24930cdb9c9bbac4bbea3dfcdb4ef51: Status 404 returned error can't find the container with id 18df223f43a0dab52ba8d3f9a7efce98b24930cdb9c9bbac4bbea3dfcdb4ef51
Jul 30 17:01:14.078641 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:14.078598    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d7e1b15f-2f31-42d4-b8e6-56bee84ad252-metrics-tls\") pod \"dns-default-8l786\" (UID: \"d7e1b15f-2f31-42d4-b8e6-56bee84ad252\") " pod="openshift-dns/dns-default-8l786"
Jul 30 17:01:14.078844 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:14.078776    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 30 17:01:14.078896 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:14.078851    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7e1b15f-2f31-42d4-b8e6-56bee84ad252-metrics-tls podName:d7e1b15f-2f31-42d4-b8e6-56bee84ad252 nodeName:}" failed. No retries permitted until 2026-07-30 17:01:15.078832302 +0000 UTC m=+38.154132171 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d7e1b15f-2f31-42d4-b8e6-56bee84ad252-metrics-tls") pod "dns-default-8l786" (UID: "d7e1b15f-2f31-42d4-b8e6-56bee84ad252") : secret "dns-default-metrics-tls" not found
Jul 30 17:01:14.477163 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:14.476989    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k7lnc"
Jul 30 17:01:14.477163 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:14.477035    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ltj2t"
Jul 30 17:01:14.477163 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:14.477035    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kbsf2"
Jul 30 17:01:14.479824 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:14.479784    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 30 17:01:14.479982 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:14.479846    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 30 17:01:14.481132 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:14.480936    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 30 17:01:14.481132 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:14.481021    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dnsfv\""
Jul 30 17:01:14.481132 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:14.481118    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mcvkj\""
Jul 30 17:01:14.481417 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:14.481207    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 30 17:01:14.659908 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:14.659859    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-d2cnh" event={"ID":"439955b8-2c26-4450-8a8c-ab8b9f7f0b3f","Type":"ContainerStarted","Data":"18df223f43a0dab52ba8d3f9a7efce98b24930cdb9c9bbac4bbea3dfcdb4ef51"}
Jul 30 17:01:15.034278 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:15.034238    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-wqszn"]
Jul 30 17:01:15.036719 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:15.036693    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-wqszn"
Jul 30 17:01:15.039559 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:15.039525    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 30 17:01:15.039719 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:15.039592    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 30 17:01:15.039719 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:15.039530    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 30 17:01:15.039719 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:15.039644    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 30 17:01:15.041971 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:15.041851    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-jnnqf\""
Jul 30 17:01:15.047285 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:15.046954    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 30 17:01:15.051367 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:15.051339    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-wqszn"]
Jul 30 17:01:15.087781 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:15.087735    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d7e1b15f-2f31-42d4-b8e6-56bee84ad252-metrics-tls\") pod \"dns-default-8l786\" (UID: \"d7e1b15f-2f31-42d4-b8e6-56bee84ad252\") " pod="openshift-dns/dns-default-8l786"
Jul 30 17:01:15.087977 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:15.087925    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 30 17:01:15.088038 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:15.088012    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7e1b15f-2f31-42d4-b8e6-56bee84ad252-metrics-tls podName:d7e1b15f-2f31-42d4-b8e6-56bee84ad252 nodeName:}" failed. No retries permitted until 2026-07-30 17:01:17.087988174 +0000 UTC m=+40.163288047 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d7e1b15f-2f31-42d4-b8e6-56bee84ad252-metrics-tls") pod "dns-default-8l786" (UID: "d7e1b15f-2f31-42d4-b8e6-56bee84ad252") : secret "dns-default-metrics-tls" not found
Jul 30 17:01:15.137310 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:15.137266    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-6xrrb"]
Jul 30 17:01:15.139717 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:15.139688    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-6xrrb"
Jul 30 17:01:15.142839 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:15.142782    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 30 17:01:15.143070 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:15.143052    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-zb88h\""
Jul 30 17:01:15.143211 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:15.143189    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 30 17:01:15.143439 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:15.143392    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 30 17:01:15.143587 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:15.143456    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 30 17:01:15.154712 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:15.154673    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-6xrrb"]
Jul 30 17:01:15.188679 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:15.188642    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm8k9\" (UniqueName: \"kubernetes.io/projected/f231a916-1fda-4059-9130-d4112e738ee1-kube-api-access-gm8k9\") pod \"console-operator-584dcb6b54-wqszn\" (UID: \"f231a916-1fda-4059-9130-d4112e738ee1\") " pod="openshift-console-operator/console-operator-584dcb6b54-wqszn"
Jul 30 17:01:15.188893 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:15.188735    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f231a916-1fda-4059-9130-d4112e738ee1-serving-cert\") pod \"console-operator-584dcb6b54-wqszn\" (UID: \"f231a916-1fda-4059-9130-d4112e738ee1\") " pod="openshift-console-operator/console-operator-584dcb6b54-wqszn"
Jul 30 17:01:15.188893 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:15.188807    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f231a916-1fda-4059-9130-d4112e738ee1-trusted-ca\") pod \"console-operator-584dcb6b54-wqszn\" (UID: \"f231a916-1fda-4059-9130-d4112e738ee1\") " pod="openshift-console-operator/console-operator-584dcb6b54-wqszn"
Jul 30 17:01:15.188893 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:15.188847    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f231a916-1fda-4059-9130-d4112e738ee1-config\") pod \"console-operator-584dcb6b54-wqszn\" (UID: \"f231a916-1fda-4059-9130-d4112e738ee1\") " pod="openshift-console-operator/console-operator-584dcb6b54-wqszn"
Jul 30 17:01:15.290369 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:15.290266    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm8k9\" (UniqueName: \"kubernetes.io/projected/f231a916-1fda-4059-9130-d4112e738ee1-kube-api-access-gm8k9\") pod \"console-operator-584dcb6b54-wqszn\" (UID: \"f231a916-1fda-4059-9130-d4112e738ee1\") " pod="openshift-console-operator/console-operator-584dcb6b54-wqszn"
Jul 30 17:01:15.290369 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:15.290341    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/badf66a2-ec5d-4590-b910-2f6cddef1caf-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-6xrrb\" (UID: \"badf66a2-ec5d-4590-b910-2f6cddef1caf\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-6xrrb"
Jul 30 17:01:15.290609 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:15.290382    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr8zc\" (UniqueName: \"kubernetes.io/projected/badf66a2-ec5d-4590-b910-2f6cddef1caf-kube-api-access-cr8zc\") pod \"cluster-monitoring-operator-548b489cb8-6xrrb\" (UID: \"badf66a2-ec5d-4590-b910-2f6cddef1caf\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-6xrrb"
Jul 30 17:01:15.290609 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:15.290424    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f231a916-1fda-4059-9130-d4112e738ee1-serving-cert\") pod \"console-operator-584dcb6b54-wqszn\" (UID: \"f231a916-1fda-4059-9130-d4112e738ee1\") " pod="openshift-console-operator/console-operator-584dcb6b54-wqszn"
Jul 30 17:01:15.290609 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:15.290477    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/badf66a2-ec5d-4590-b910-2f6cddef1caf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-6xrrb\" (UID: \"badf66a2-ec5d-4590-b910-2f6cddef1caf\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-6xrrb"
Jul 30 17:01:15.290609 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:15.290535    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f231a916-1fda-4059-9130-d4112e738ee1-trusted-ca\") pod \"console-operator-584dcb6b54-wqszn\" (UID: \"f231a916-1fda-4059-9130-d4112e738ee1\") " pod="openshift-console-operator/console-operator-584dcb6b54-wqszn"
Jul 30 17:01:15.290840 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:15.290663    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f231a916-1fda-4059-9130-d4112e738ee1-config\") pod \"console-operator-584dcb6b54-wqszn\" (UID: \"f231a916-1fda-4059-9130-d4112e738ee1\") " pod="openshift-console-operator/console-operator-584dcb6b54-wqszn"
Jul 30 17:01:15.291548 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:15.291490    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f231a916-1fda-4059-9130-d4112e738ee1-config\") pod \"console-operator-584dcb6b54-wqszn\" (UID: \"f231a916-1fda-4059-9130-d4112e738ee1\") " pod="openshift-console-operator/console-operator-584dcb6b54-wqszn"
Jul 30 17:01:15.291881 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:15.291856    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f231a916-1fda-4059-9130-d4112e738ee1-trusted-ca\") pod \"console-operator-584dcb6b54-wqszn\" (UID: \"f231a916-1fda-4059-9130-d4112e738ee1\") " pod="openshift-console-operator/console-operator-584dcb6b54-wqszn"
Jul 30 17:01:15.295366 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:15.295344    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f231a916-1fda-4059-9130-d4112e738ee1-serving-cert\") pod \"console-operator-584dcb6b54-wqszn\" (UID: \"f231a916-1fda-4059-9130-d4112e738ee1\") " pod="openshift-console-operator/console-operator-584dcb6b54-wqszn"
Jul 30 17:01:15.299923 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:15.299894    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm8k9\" (UniqueName: \"kubernetes.io/projected/f231a916-1fda-4059-9130-d4112e738ee1-kube-api-access-gm8k9\") pod \"console-operator-584dcb6b54-wqszn\" (UID: \"f231a916-1fda-4059-9130-d4112e738ee1\") " pod="openshift-console-operator/console-operator-584dcb6b54-wqszn"
Jul 30 17:01:15.350508 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:15.350463    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-wqszn"
Jul 30 17:01:15.392121 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:15.392084    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/badf66a2-ec5d-4590-b910-2f6cddef1caf-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-6xrrb\" (UID: \"badf66a2-ec5d-4590-b910-2f6cddef1caf\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-6xrrb"
Jul 30 17:01:15.392344 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:15.392128    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr8zc\" (UniqueName: \"kubernetes.io/projected/badf66a2-ec5d-4590-b910-2f6cddef1caf-kube-api-access-cr8zc\") pod \"cluster-monitoring-operator-548b489cb8-6xrrb\" (UID: \"badf66a2-ec5d-4590-b910-2f6cddef1caf\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-6xrrb"
Jul 30 17:01:15.392344 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:15.392211    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/badf66a2-ec5d-4590-b910-2f6cddef1caf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-6xrrb\" (UID: \"badf66a2-ec5d-4590-b910-2f6cddef1caf\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-6xrrb"
Jul 30 17:01:15.392344 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:15.392340    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 30 17:01:15.392516 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:15.392404    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/badf66a2-ec5d-4590-b910-2f6cddef1caf-cluster-monitoring-operator-tls podName:badf66a2-ec5d-4590-b910-2f6cddef1caf nodeName:}" failed. No retries permitted until 2026-07-30 17:01:15.892383395 +0000 UTC m=+38.967683263 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/badf66a2-ec5d-4590-b910-2f6cddef1caf-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-6xrrb" (UID: "badf66a2-ec5d-4590-b910-2f6cddef1caf") : secret "cluster-monitoring-operator-tls" not found
Jul 30 17:01:15.393014 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:15.392982    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/badf66a2-ec5d-4590-b910-2f6cddef1caf-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-6xrrb\" (UID: \"badf66a2-ec5d-4590-b910-2f6cddef1caf\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-6xrrb"
Jul 30 17:01:15.404290 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:15.404241    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr8zc\" (UniqueName: \"kubernetes.io/projected/badf66a2-ec5d-4590-b910-2f6cddef1caf-kube-api-access-cr8zc\") pod \"cluster-monitoring-operator-548b489cb8-6xrrb\" (UID: \"badf66a2-ec5d-4590-b910-2f6cddef1caf\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-6xrrb"
Jul 30 17:01:15.897055 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:15.897010    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/badf66a2-ec5d-4590-b910-2f6cddef1caf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-6xrrb\" (UID: \"badf66a2-ec5d-4590-b910-2f6cddef1caf\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-6xrrb"
Jul 30 17:01:15.897824 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:15.897200    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 30 17:01:15.897824 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:15.897303    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/badf66a2-ec5d-4590-b910-2f6cddef1caf-cluster-monitoring-operator-tls podName:badf66a2-ec5d-4590-b910-2f6cddef1caf nodeName:}" failed. No retries permitted until 2026-07-30 17:01:16.897282748 +0000 UTC m=+39.972582633 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/badf66a2-ec5d-4590-b910-2f6cddef1caf-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-6xrrb" (UID: "badf66a2-ec5d-4590-b910-2f6cddef1caf") : secret "cluster-monitoring-operator-tls" not found
Jul 30 17:01:16.905615 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:16.905534    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/badf66a2-ec5d-4590-b910-2f6cddef1caf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-6xrrb\" (UID: \"badf66a2-ec5d-4590-b910-2f6cddef1caf\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-6xrrb"
Jul 30 17:01:16.906158 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:16.905816    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 30 17:01:16.906158 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:16.905915    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/badf66a2-ec5d-4590-b910-2f6cddef1caf-cluster-monitoring-operator-tls podName:badf66a2-ec5d-4590-b910-2f6cddef1caf nodeName:}" failed. No retries permitted until 2026-07-30 17:01:18.905890072 +0000 UTC m=+41.981189948 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/badf66a2-ec5d-4590-b910-2f6cddef1caf-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-6xrrb" (UID: "badf66a2-ec5d-4590-b910-2f6cddef1caf") : secret "cluster-monitoring-operator-tls" not found
Jul 30 17:01:17.107949 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:17.107909    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d7e1b15f-2f31-42d4-b8e6-56bee84ad252-metrics-tls\") pod \"dns-default-8l786\" (UID: \"d7e1b15f-2f31-42d4-b8e6-56bee84ad252\") " pod="openshift-dns/dns-default-8l786"
Jul 30 17:01:17.108136 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:17.108050    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 30 17:01:17.108231 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:17.108140    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7e1b15f-2f31-42d4-b8e6-56bee84ad252-metrics-tls podName:d7e1b15f-2f31-42d4-b8e6-56bee84ad252 nodeName:}" failed. No retries permitted until 2026-07-30 17:01:21.108116416 +0000 UTC m=+44.183416309 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d7e1b15f-2f31-42d4-b8e6-56bee84ad252-metrics-tls") pod "dns-default-8l786" (UID: "d7e1b15f-2f31-42d4-b8e6-56bee84ad252") : secret "dns-default-metrics-tls" not found
Jul 30 17:01:18.023963 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:18.023925    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-wqszn"]
Jul 30 17:01:18.042814 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:01:18.042779    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf231a916_1fda_4059_9130_d4112e738ee1.slice/crio-279baf16c3fb20d30c49c9194067ac60548c80e8cc3a960319dc3c2af1788154 WatchSource:0}: Error finding container 279baf16c3fb20d30c49c9194067ac60548c80e8cc3a960319dc3c2af1788154: Status 404 returned error can't find the container with id 279baf16c3fb20d30c49c9194067ac60548c80e8cc3a960319dc3c2af1788154
Jul 30 17:01:18.672383 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:18.672340    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-d2cnh" event={"ID":"439955b8-2c26-4450-8a8c-ab8b9f7f0b3f","Type":"ContainerStarted","Data":"f57ce4bf9c7858d348acc845c1924e69b00fae021d07fcc8aa85afb7e3c00156"}
Jul 30 17:01:18.673681 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:18.673652    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-wqszn" event={"ID":"f231a916-1fda-4059-9130-d4112e738ee1","Type":"ContainerStarted","Data":"279baf16c3fb20d30c49c9194067ac60548c80e8cc3a960319dc3c2af1788154"}
Jul 30 17:01:18.676848 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:18.676815    2580 generic.go:356] "Generic (PLEG): container finished" podID="f366886f-2162-4153-a077-55dcd0fef1e8" containerID="0bae13369f9f976ad83543f959fe8bdaa29611165476143561e3eb544c2629b6" exitCode=0
Jul 30 17:01:18.677005 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:18.676892    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mz6bl" event={"ID":"f366886f-2162-4153-a077-55dcd0fef1e8","Type":"ContainerDied","Data":"0bae13369f9f976ad83543f959fe8bdaa29611165476143561e3eb544c2629b6"}
Jul 30 17:01:18.691401 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:18.691328    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-d2cnh" podStartSLOduration=1.622344715 podStartE2EDuration="5.691305481s" totalImagesPullingTime="4.068960766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:01:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:18.690980086 +0000 UTC m=+41.766279987" watchObservedRunningTime="2026-07-30 17:01:18.691305481 +0000 UTC m=+41.766605372"
Jul 30 17:01:18.922019 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:18.921974    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/badf66a2-ec5d-4590-b910-2f6cddef1caf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-6xrrb\" (UID: \"badf66a2-ec5d-4590-b910-2f6cddef1caf\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-6xrrb"
Jul 30 17:01:18.922215 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:18.922153    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 30 17:01:18.922291 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:18.922230    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/badf66a2-ec5d-4590-b910-2f6cddef1caf-cluster-monitoring-operator-tls podName:badf66a2-ec5d-4590-b910-2f6cddef1caf nodeName:}" failed. No retries permitted until 2026-07-30 17:01:22.922209279 +0000 UTC m=+45.997509147 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/badf66a2-ec5d-4590-b910-2f6cddef1caf-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-6xrrb" (UID: "badf66a2-ec5d-4590-b910-2f6cddef1caf") : secret "cluster-monitoring-operator-tls" not found
Jul 30 17:01:19.685857 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:19.685821    2580 generic.go:356] "Generic (PLEG): container finished" podID="f366886f-2162-4153-a077-55dcd0fef1e8" containerID="3330f994d585c7b944e52c0a30954692e96a424a4f4009ab49d34f097b5d848d" exitCode=0
Jul 30 17:01:19.686357 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:19.685907    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mz6bl" event={"ID":"f366886f-2162-4153-a077-55dcd0fef1e8","Type":"ContainerDied","Data":"3330f994d585c7b944e52c0a30954692e96a424a4f4009ab49d34f097b5d848d"}
Jul 30 17:01:20.691850 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:20.691813    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mz6bl" event={"ID":"f366886f-2162-4153-a077-55dcd0fef1e8","Type":"ContainerStarted","Data":"9828266a7fd2f0eeadb88d0762fc0238486b05f1047cfa94175d5936fa6beaa2"}
Jul 30 17:01:20.714378 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:20.714324    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-mz6bl" podStartSLOduration=7.389609861 podStartE2EDuration="42.714306315s" totalImagesPullingTime="35.324696454s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 17:00:38 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:20.713441466 +0000 UTC m=+43.788741357" watchObservedRunningTime="2026-07-30 17:01:20.714306315 +0000 UTC m=+43.789606183"
Jul 30 17:01:21.139968 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:21.139920    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d7e1b15f-2f31-42d4-b8e6-56bee84ad252-metrics-tls\") pod \"dns-default-8l786\" (UID: \"d7e1b15f-2f31-42d4-b8e6-56bee84ad252\") " pod="openshift-dns/dns-default-8l786"
Jul 30 17:01:21.140163 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:21.140051    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 30 17:01:21.140163 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:21.140110    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7e1b15f-2f31-42d4-b8e6-56bee84ad252-metrics-tls podName:d7e1b15f-2f31-42d4-b8e6-56bee84ad252 nodeName:}" failed. No retries permitted until 2026-07-30 17:01:29.140095988 +0000 UTC m=+52.215395856 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d7e1b15f-2f31-42d4-b8e6-56bee84ad252-metrics-tls") pod "dns-default-8l786" (UID: "d7e1b15f-2f31-42d4-b8e6-56bee84ad252") : secret "dns-default-metrics-tls" not found
Jul 30 17:01:21.695888 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:21.695851    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-wqszn_f231a916-1fda-4059-9130-d4112e738ee1/console-operator/0.log"
Jul 30 17:01:21.696340 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:21.695896    2580 generic.go:356] "Generic (PLEG): container finished" podID="f231a916-1fda-4059-9130-d4112e738ee1" containerID="c08834f036cd377a9ab923351e438c0902aeab294b0132eebdcec051f75aad81" exitCode=255
Jul 30 17:01:21.696340 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:21.695996    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-wqszn" event={"ID":"f231a916-1fda-4059-9130-d4112e738ee1","Type":"ContainerDied","Data":"c08834f036cd377a9ab923351e438c0902aeab294b0132eebdcec051f75aad81"}
Jul 30 17:01:21.696340 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:21.696204    2580 scope.go:117] "RemoveContainer" containerID="c08834f036cd377a9ab923351e438c0902aeab294b0132eebdcec051f75aad81"
Jul 30 17:01:22.699759 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:22.699729    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-wqszn_f231a916-1fda-4059-9130-d4112e738ee1/console-operator/1.log"
Jul 30 17:01:22.700214 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:22.700198    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-wqszn_f231a916-1fda-4059-9130-d4112e738ee1/console-operator/0.log"
Jul 30 17:01:22.700267 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:22.700233    2580 generic.go:356] "Generic (PLEG): container finished" podID="f231a916-1fda-4059-9130-d4112e738ee1" containerID="cf46b58e24c35eba1d021fa48dd7a639183ec1da242b21d27a77fabd5dd175d7" exitCode=255
Jul 30 17:01:22.700310 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:22.700261    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-wqszn" event={"ID":"f231a916-1fda-4059-9130-d4112e738ee1","Type":"ContainerDied","Data":"cf46b58e24c35eba1d021fa48dd7a639183ec1da242b21d27a77fabd5dd175d7"}
Jul 30 17:01:22.700310 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:22.700305    2580 scope.go:117] "RemoveContainer" containerID="c08834f036cd377a9ab923351e438c0902aeab294b0132eebdcec051f75aad81"
Jul 30 17:01:22.700561 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:22.700541    2580 scope.go:117] "RemoveContainer" containerID="cf46b58e24c35eba1d021fa48dd7a639183ec1da242b21d27a77fabd5dd175d7"
Jul 30 17:01:22.700760 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:22.700741    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-wqszn_openshift-console-operator(f231a916-1fda-4059-9130-d4112e738ee1)\"" pod="openshift-console-operator/console-operator-584dcb6b54-wqszn" podUID="f231a916-1fda-4059-9130-d4112e738ee1"
Jul 30 17:01:22.891070 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:22.891022    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-b65z5"]
Jul 30 17:01:22.921200 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:22.921108    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-b65z5"]
Jul 30 17:01:22.921395 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:22.921260    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-b65z5"
Jul 30 17:01:22.923747 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:22.923721    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 30 17:01:22.923897 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:22.923802    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-gvccl\""
Jul 30 17:01:22.923983 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:22.923969    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 30 17:01:22.955552 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:22.955448    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/badf66a2-ec5d-4590-b910-2f6cddef1caf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-6xrrb\" (UID: \"badf66a2-ec5d-4590-b910-2f6cddef1caf\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-6xrrb"
Jul 30 17:01:22.955725 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:22.955573    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 30 17:01:22.955725 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:22.955644    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/badf66a2-ec5d-4590-b910-2f6cddef1caf-cluster-monitoring-operator-tls podName:badf66a2-ec5d-4590-b910-2f6cddef1caf nodeName:}" failed. No retries permitted until 2026-07-30 17:01:30.955628959 +0000 UTC m=+54.030928827 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/badf66a2-ec5d-4590-b910-2f6cddef1caf-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-6xrrb" (UID: "badf66a2-ec5d-4590-b910-2f6cddef1caf") : secret "cluster-monitoring-operator-tls" not found
Jul 30 17:01:23.056038 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:23.056004    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khz65\" (UniqueName: \"kubernetes.io/projected/25af7388-ce5d-49e0-8ecb-c4aa3c1f91be-kube-api-access-khz65\") pod \"migrator-6865c9967-b65z5\" (UID: \"25af7388-ce5d-49e0-8ecb-c4aa3c1f91be\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-b65z5"
Jul 30 17:01:23.120457 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:23.120421    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-bjfhv"]
Jul 30 17:01:23.143606 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:23.143565    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bjfhv"]
Jul 30 17:01:23.143792 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:23.143692    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bjfhv"
Jul 30 17:01:23.147103 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:23.147079    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 30 17:01:23.147282 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:23.147079    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 30 17:01:23.147282 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:23.147217    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 30 17:01:23.147548 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:23.147531    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 30 17:01:23.147708 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:23.147695    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-brp5b\""
Jul 30 17:01:23.156835 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:23.156805    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khz65\" (UniqueName: \"kubernetes.io/projected/25af7388-ce5d-49e0-8ecb-c4aa3c1f91be-kube-api-access-khz65\") pod \"migrator-6865c9967-b65z5\" (UID: \"25af7388-ce5d-49e0-8ecb-c4aa3c1f91be\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-b65z5"
Jul 30 17:01:23.165540 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:23.165500    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khz65\" (UniqueName: \"kubernetes.io/projected/25af7388-ce5d-49e0-8ecb-c4aa3c1f91be-kube-api-access-khz65\") pod \"migrator-6865c9967-b65z5\" (UID: \"25af7388-ce5d-49e0-8ecb-c4aa3c1f91be\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-b65z5"
Jul 30 17:01:23.231703 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:23.231604    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-b65z5"
Jul 30 17:01:23.257361 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:23.257331    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zng2k\" (UniqueName: \"kubernetes.io/projected/3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61-kube-api-access-zng2k\") pod \"insights-runtime-extractor-bjfhv\" (UID: \"3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61\") " pod="openshift-insights/insights-runtime-extractor-bjfhv"
Jul 30 17:01:23.257508 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:23.257386    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61-data-volume\") pod \"insights-runtime-extractor-bjfhv\" (UID: \"3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61\") " pod="openshift-insights/insights-runtime-extractor-bjfhv"
Jul 30 17:01:23.257508 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:23.257441    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bjfhv\" (UID: \"3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61\") " pod="openshift-insights/insights-runtime-extractor-bjfhv"
Jul 30 17:01:23.257508 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:23.257475    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bjfhv\" (UID: \"3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61\") " pod="openshift-insights/insights-runtime-extractor-bjfhv"
Jul 30 17:01:23.257619 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:23.257536    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61-crio-socket\") pod \"insights-runtime-extractor-bjfhv\" (UID: \"3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61\") " pod="openshift-insights/insights-runtime-extractor-bjfhv"
Jul 30 17:01:23.358891 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:23.358851    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zng2k\" (UniqueName: \"kubernetes.io/projected/3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61-kube-api-access-zng2k\") pod \"insights-runtime-extractor-bjfhv\" (UID: \"3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61\") " pod="openshift-insights/insights-runtime-extractor-bjfhv"
Jul 30 17:01:23.359067 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:23.358929    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61-data-volume\") pod \"insights-runtime-extractor-bjfhv\" (UID: \"3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61\") " pod="openshift-insights/insights-runtime-extractor-bjfhv"
Jul 30 17:01:23.359067 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:23.359046    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bjfhv\" (UID: \"3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61\") " pod="openshift-insights/insights-runtime-extractor-bjfhv"
Jul 30 17:01:23.359203 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:23.359076    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bjfhv\" (UID: \"3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61\") " pod="openshift-insights/insights-runtime-extractor-bjfhv"
Jul 30 17:01:23.359203 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:23.359142    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61-crio-socket\") pod \"insights-runtime-extractor-bjfhv\" (UID: \"3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61\") " pod="openshift-insights/insights-runtime-extractor-bjfhv"
Jul 30 17:01:23.359203 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:23.359162    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 30 17:01:23.359350 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:23.359283    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 30 17:01:23.359350 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:23.359325    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61-data-volume\") pod \"insights-runtime-extractor-bjfhv\" (UID: \"3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61\") " pod="openshift-insights/insights-runtime-extractor-bjfhv"
Jul 30 17:01:23.359440 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:23.359286    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61-kube-rbac-proxy-cm podName:3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61 nodeName:}" failed. No retries permitted until 2026-07-30 17:01:23.859265741 +0000 UTC m=+46.934565624 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bjfhv" (UID: "3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61") : configmap "kube-rbac-proxy" not found
Jul 30 17:01:23.359440 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:23.359391    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61-insights-runtime-extractor-tls podName:3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61 nodeName:}" failed. No retries permitted until 2026-07-30 17:01:23.859368189 +0000 UTC m=+46.934668072 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bjfhv" (UID: "3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61") : secret "insights-runtime-extractor-tls" not found
Jul 30 17:01:23.359440 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:23.359392    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61-crio-socket\") pod \"insights-runtime-extractor-bjfhv\" (UID: \"3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61\") " pod="openshift-insights/insights-runtime-extractor-bjfhv"
Jul 30 17:01:23.367157 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:23.367120    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zng2k\" (UniqueName: \"kubernetes.io/projected/3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61-kube-api-access-zng2k\") pod \"insights-runtime-extractor-bjfhv\" (UID: \"3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61\") " pod="openshift-insights/insights-runtime-extractor-bjfhv"
Jul 30 17:01:23.413412 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:23.413377    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-b65z5"]
Jul 30 17:01:23.416397 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:01:23.416360    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25af7388_ce5d_49e0_8ecb_c4aa3c1f91be.slice/crio-c6ea10451f844570c3fb6bb2e9c9006ae2c1bbb6c414b5944c27644ae1707e70 WatchSource:0}: Error finding container c6ea10451f844570c3fb6bb2e9c9006ae2c1bbb6c414b5944c27644ae1707e70: Status 404 returned error can't find the container with id c6ea10451f844570c3fb6bb2e9c9006ae2c1bbb6c414b5944c27644ae1707e70
Jul 30 17:01:23.703844 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:23.703805    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-b65z5" event={"ID":"25af7388-ce5d-49e0-8ecb-c4aa3c1f91be","Type":"ContainerStarted","Data":"c6ea10451f844570c3fb6bb2e9c9006ae2c1bbb6c414b5944c27644ae1707e70"}
Jul 30 17:01:23.705353 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:23.705331    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-wqszn_f231a916-1fda-4059-9130-d4112e738ee1/console-operator/1.log"
Jul 30 17:01:23.705716 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:23.705698    2580 scope.go:117] "RemoveContainer" containerID="cf46b58e24c35eba1d021fa48dd7a639183ec1da242b21d27a77fabd5dd175d7"
Jul 30 17:01:23.705915 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:23.705893    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-wqszn_openshift-console-operator(f231a916-1fda-4059-9130-d4112e738ee1)\"" pod="openshift-console-operator/console-operator-584dcb6b54-wqszn" podUID="f231a916-1fda-4059-9130-d4112e738ee1"
Jul 30 17:01:23.864723 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:23.864679    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bjfhv\" (UID: \"3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61\") " pod="openshift-insights/insights-runtime-extractor-bjfhv"
Jul 30 17:01:23.864943 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:23.864753    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bjfhv\" (UID: \"3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61\") " pod="openshift-insights/insights-runtime-extractor-bjfhv"
Jul 30 17:01:23.864943 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:23.864829    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 30 17:01:23.864943 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:23.864922    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61-kube-rbac-proxy-cm podName:3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61 nodeName:}" failed. No retries permitted until 2026-07-30 17:01:24.864899329 +0000 UTC m=+47.940199199 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bjfhv" (UID: "3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61") : configmap "kube-rbac-proxy" not found
Jul 30 17:01:23.864943 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:23.864925    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 30 17:01:23.865157 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:23.864988    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61-insights-runtime-extractor-tls podName:3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61 nodeName:}" failed. No retries permitted until 2026-07-30 17:01:24.864969906 +0000 UTC m=+47.940269773 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bjfhv" (UID: "3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61") : secret "insights-runtime-extractor-tls" not found
Jul 30 17:01:24.872396 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:24.872356    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bjfhv\" (UID: \"3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61\") " pod="openshift-insights/insights-runtime-extractor-bjfhv"
Jul 30 17:01:24.872396 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:24.872400    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bjfhv\" (UID: \"3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61\") " pod="openshift-insights/insights-runtime-extractor-bjfhv"
Jul 30 17:01:24.872924 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:24.872492    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 30 17:01:24.872924 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:24.872523    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 30 17:01:24.872924 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:24.872562    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61-kube-rbac-proxy-cm podName:3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61 nodeName:}" failed. No retries permitted until 2026-07-30 17:01:26.872545186 +0000 UTC m=+49.947845053 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bjfhv" (UID: "3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61") : configmap "kube-rbac-proxy" not found
Jul 30 17:01:24.872924 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:24.872577    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61-insights-runtime-extractor-tls podName:3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61 nodeName:}" failed. No retries permitted until 2026-07-30 17:01:26.872571335 +0000 UTC m=+49.947871202 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bjfhv" (UID: "3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61") : secret "insights-runtime-extractor-tls" not found
Jul 30 17:01:25.350909 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:25.350874    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-wqszn"
Jul 30 17:01:25.351126 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:25.350934    2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-wqszn"
Jul 30 17:01:25.351384 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:25.351368    2580 scope.go:117] "RemoveContainer" containerID="cf46b58e24c35eba1d021fa48dd7a639183ec1da242b21d27a77fabd5dd175d7"
Jul 30 17:01:25.351577 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:25.351556    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-wqszn_openshift-console-operator(f231a916-1fda-4059-9130-d4112e738ee1)\"" pod="openshift-console-operator/console-operator-584dcb6b54-wqszn" podUID="f231a916-1fda-4059-9130-d4112e738ee1"
Jul 30 17:01:25.712700 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:25.712587    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-b65z5" event={"ID":"25af7388-ce5d-49e0-8ecb-c4aa3c1f91be","Type":"ContainerStarted","Data":"d6dae78799506693f6c8f0c3a4cc9a53432c87f1bff8a88fee84e50fe4c66e63"}
Jul 30 17:01:25.712700 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:25.712687    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-b65z5" event={"ID":"25af7388-ce5d-49e0-8ecb-c4aa3c1f91be","Type":"ContainerStarted","Data":"c55f7c5bee0f1f02ff7fcf9b8ef6a596a3205cb86e8ee4ec01fcf3cbc3ed6502"}
Jul 30 17:01:25.734732 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:25.734666    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-b65z5" podStartSLOduration=2.274745596 podStartE2EDuration="3.734648183s" totalImagesPullingTime="1.459902587s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:01:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:25.733055289 +0000 UTC m=+48.808355178" watchObservedRunningTime="2026-07-30 17:01:25.734648183 +0000 UTC m=+48.809948073"
Jul 30 17:01:25.918850 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:25.918821    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-c77l8_e4e1630f-f153-4be6-883a-1fbeca875a69/dns-node-resolver/0.log"
Jul 30 17:01:26.711425 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:26.711393    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tdkrg_adf2f8d3-a0f4-4514-b6e3-b1507fa0c0bb/node-ca/0.log"
Jul 30 17:01:26.890834 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:26.890783    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bjfhv\" (UID: \"3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61\") " pod="openshift-insights/insights-runtime-extractor-bjfhv"
Jul 30 17:01:26.890834 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:26.890832    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bjfhv\" (UID: \"3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61\") " pod="openshift-insights/insights-runtime-extractor-bjfhv"
Jul 30 17:01:26.891117 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:26.890917    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 30 17:01:26.891117 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:26.890986    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61-kube-rbac-proxy-cm podName:3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61 nodeName:}" failed. No retries permitted until 2026-07-30 17:01:30.89096658 +0000 UTC m=+53.966266463 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bjfhv" (UID: "3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61") : configmap "kube-rbac-proxy" not found
Jul 30 17:01:26.891117 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:26.891040    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 30 17:01:26.891284 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:26.891121    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61-insights-runtime-extractor-tls podName:3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61 nodeName:}" failed. No retries permitted until 2026-07-30 17:01:30.891098848 +0000 UTC m=+53.966398731 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bjfhv" (UID: "3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61") : secret "insights-runtime-extractor-tls" not found
Jul 30 17:01:27.509916 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:27.509879    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-d2cnh_439955b8-2c26-4450-8a8c-ab8b9f7f0b3f/serve-healthcheck-canary/0.log"
Jul 30 17:01:27.910597 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:27.910515    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-b65z5_25af7388-ce5d-49e0-8ecb-c4aa3c1f91be/migrator/0.log"
Jul 30 17:01:28.110290 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:28.110258    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-b65z5_25af7388-ce5d-49e0-8ecb-c4aa3c1f91be/graceful-termination/0.log"
Jul 30 17:01:29.211150 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:29.211106    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d7e1b15f-2f31-42d4-b8e6-56bee84ad252-metrics-tls\") pod \"dns-default-8l786\" (UID: \"d7e1b15f-2f31-42d4-b8e6-56bee84ad252\") " pod="openshift-dns/dns-default-8l786"
Jul 30 17:01:29.213576 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:29.213545    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d7e1b15f-2f31-42d4-b8e6-56bee84ad252-metrics-tls\") pod \"dns-default-8l786\" (UID: \"d7e1b15f-2f31-42d4-b8e6-56bee84ad252\") " pod="openshift-dns/dns-default-8l786"
Jul 30 17:01:29.233733 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:29.233698    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8l786"
Jul 30 17:01:29.383851 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:29.383818    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8l786"]
Jul 30 17:01:29.387407 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:01:29.387368    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7e1b15f_2f31_42d4_b8e6_56bee84ad252.slice/crio-6d6a8f9e811e80db3745c8639b546d9861600fb5b1d53141d970d54b4263f77a WatchSource:0}: Error finding container 6d6a8f9e811e80db3745c8639b546d9861600fb5b1d53141d970d54b4263f77a: Status 404 returned error can't find the container with id 6d6a8f9e811e80db3745c8639b546d9861600fb5b1d53141d970d54b4263f77a
Jul 30 17:01:29.724102 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:29.724061    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8l786" event={"ID":"d7e1b15f-2f31-42d4-b8e6-56bee84ad252","Type":"ContainerStarted","Data":"6d6a8f9e811e80db3745c8639b546d9861600fb5b1d53141d970d54b4263f77a"}
Jul 30 17:01:30.923868 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:30.923829    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bjfhv\" (UID: \"3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61\") " pod="openshift-insights/insights-runtime-extractor-bjfhv"
Jul 30 17:01:30.924356 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:30.923875    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bjfhv\" (UID: \"3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61\") " pod="openshift-insights/insights-runtime-extractor-bjfhv"
Jul 30 17:01:30.924356 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:30.923986    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 30 17:01:30.924356 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:30.924017    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 30 17:01:30.924356 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:30.924076    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61-kube-rbac-proxy-cm podName:3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61 nodeName:}" failed. No retries permitted until 2026-07-30 17:01:38.924054284 +0000 UTC m=+61.999354168 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bjfhv" (UID: "3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61") : configmap "kube-rbac-proxy" not found
Jul 30 17:01:30.924356 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:30.924100    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61-insights-runtime-extractor-tls podName:3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61 nodeName:}" failed. No retries permitted until 2026-07-30 17:01:38.924089144 +0000 UTC m=+61.999389014 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bjfhv" (UID: "3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61") : secret "insights-runtime-extractor-tls" not found
Jul 30 17:01:31.024296 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:31.024242    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/badf66a2-ec5d-4590-b910-2f6cddef1caf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-6xrrb\" (UID: \"badf66a2-ec5d-4590-b910-2f6cddef1caf\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-6xrrb"
Jul 30 17:01:31.024548 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:31.024434    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 30 17:01:31.024548 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:31.024518    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/badf66a2-ec5d-4590-b910-2f6cddef1caf-cluster-monitoring-operator-tls podName:badf66a2-ec5d-4590-b910-2f6cddef1caf nodeName:}" failed. No retries permitted until 2026-07-30 17:01:47.024496487 +0000 UTC m=+70.099796357 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/badf66a2-ec5d-4590-b910-2f6cddef1caf-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-6xrrb" (UID: "badf66a2-ec5d-4590-b910-2f6cddef1caf") : secret "cluster-monitoring-operator-tls" not found
Jul 30 17:01:32.735356 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:32.735260    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8l786" event={"ID":"d7e1b15f-2f31-42d4-b8e6-56bee84ad252","Type":"ContainerStarted","Data":"9e07c949d0bc1d6843247202f8fe38eace813da8969daa0f2ffe3e907c1126e5"}
Jul 30 17:01:32.735356 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:32.735328    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8l786" event={"ID":"d7e1b15f-2f31-42d4-b8e6-56bee84ad252","Type":"ContainerStarted","Data":"8c8a8a206442409bc6b46a2405bc12820823fd3b6df360b073a27a0f228ad662"}
Jul 30 17:01:32.735786 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:32.735412    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8l786"
Jul 30 17:01:32.754708 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:32.754647    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-8l786" podStartSLOduration=17.15148118 podStartE2EDuration="19.754629072s" totalImagesPullingTime="2.603147892s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:01:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:32.753716275 +0000 UTC m=+55.829016164" watchObservedRunningTime="2026-07-30 17:01:32.754629072 +0000 UTC m=+55.829928961"
Jul 30 17:01:38.989311 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:38.989253    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bjfhv\" (UID: \"3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61\") " pod="openshift-insights/insights-runtime-extractor-bjfhv"
Jul 30 17:01:38.989311 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:38.989307    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bjfhv\" (UID: \"3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61\") " pod="openshift-insights/insights-runtime-extractor-bjfhv"
Jul 30 17:01:38.989885 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:38.989832    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bjfhv\" (UID: \"3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61\") " pod="openshift-insights/insights-runtime-extractor-bjfhv"
Jul 30 17:01:38.991735 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:38.991717    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bjfhv\" (UID: \"3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61\") " pod="openshift-insights/insights-runtime-extractor-bjfhv"
Jul 30 17:01:39.056529 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:39.056489    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-brp5b\""
Jul 30 17:01:39.063811 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:39.063782    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bjfhv"
Jul 30 17:01:39.229535 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:39.229498    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bjfhv"]
Jul 30 17:01:39.232947 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:01:39.232910    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3263a9ff_a9b4_4eb3_9af6_e03df8dc4c61.slice/crio-7cd5a5df1bf3ac5fce4d86418d9afe83ab75da71a392a41fd3475ec9abd7c52f WatchSource:0}: Error finding container 7cd5a5df1bf3ac5fce4d86418d9afe83ab75da71a392a41fd3475ec9abd7c52f: Status 404 returned error can't find the container with id 7cd5a5df1bf3ac5fce4d86418d9afe83ab75da71a392a41fd3475ec9abd7c52f
Jul 30 17:01:39.477746 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:39.477714    2580 scope.go:117] "RemoveContainer" containerID="cf46b58e24c35eba1d021fa48dd7a639183ec1da242b21d27a77fabd5dd175d7"
Jul 30 17:01:39.652433 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:39.652398    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tblxl"
Jul 30 17:01:39.762159 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:39.762130    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-wqszn_f231a916-1fda-4059-9130-d4112e738ee1/console-operator/1.log"
Jul 30 17:01:39.762366 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:39.762245    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-wqszn" event={"ID":"f231a916-1fda-4059-9130-d4112e738ee1","Type":"ContainerStarted","Data":"3ee414c008d29a360137082d7dd707ee1ee939b24089b9fdee3d86f4a1ebf32b"}
Jul 30 17:01:39.762588 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:39.762533    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-wqszn"
Jul 30 17:01:39.763527 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:39.763506    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bjfhv" event={"ID":"3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61","Type":"ContainerStarted","Data":"7cd5a5df1bf3ac5fce4d86418d9afe83ab75da71a392a41fd3475ec9abd7c52f"}
Jul 30 17:01:39.781950 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:39.781888    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-wqszn" podStartSLOduration=22.161640201 podStartE2EDuration="24.781867739s" totalImagesPullingTime="2.620227538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:01:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:39.781492657 +0000 UTC m=+62.856792550" watchObservedRunningTime="2026-07-30 17:01:39.781867739 +0000 UTC m=+62.857167607"
Jul 30 17:01:40.147633 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:40.147541    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-wqszn"
Jul 30 17:01:42.116712 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:42.116662    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f8b90e9-e705-4b9c-89f7-047151a2b098-metrics-certs\") pod \"network-metrics-daemon-kbsf2\" (UID: \"4f8b90e9-e705-4b9c-89f7-047151a2b098\") " pod="openshift-multus/network-metrics-daemon-kbsf2"
Jul 30 17:01:42.119665 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:42.119635    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 30 17:01:42.130797 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:42.130751    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f8b90e9-e705-4b9c-89f7-047151a2b098-metrics-certs\") pod \"network-metrics-daemon-kbsf2\" (UID: \"4f8b90e9-e705-4b9c-89f7-047151a2b098\") " pod="openshift-multus/network-metrics-daemon-kbsf2"
Jul 30 17:01:42.217346 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:42.217296    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ad2d9b30-7d70-4236-9398-93ff51843c30-original-pull-secret\") pod \"global-pull-secret-syncer-k7lnc\" (UID: \"ad2d9b30-7d70-4236-9398-93ff51843c30\") " pod="kube-system/global-pull-secret-syncer-k7lnc"
Jul 30 17:01:42.220303 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:42.220277    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 30 17:01:42.231238 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:42.231202    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ad2d9b30-7d70-4236-9398-93ff51843c30-original-pull-secret\") pod \"global-pull-secret-syncer-k7lnc\" (UID: \"ad2d9b30-7d70-4236-9398-93ff51843c30\") " pod="kube-system/global-pull-secret-syncer-k7lnc"
Jul 30 17:01:42.317783 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:42.317742    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svhd5\" (UniqueName: \"kubernetes.io/projected/33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e-kube-api-access-svhd5\") pod \"network-check-target-ltj2t\" (UID: \"33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e\") " pod="openshift-network-diagnostics/network-check-target-ltj2t"
Jul 30 17:01:42.320722 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:42.320693    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 30 17:01:42.331529 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:42.331494    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 30 17:01:42.342192 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:42.342137    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svhd5\" (UniqueName: \"kubernetes.io/projected/33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e-kube-api-access-svhd5\") pod \"network-check-target-ltj2t\" (UID: \"33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e\") " pod="openshift-network-diagnostics/network-check-target-ltj2t"
Jul 30 17:01:42.392954 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:42.392844    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k7lnc"
Jul 30 17:01:42.404496 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:42.404449    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mcvkj\""
Jul 30 17:01:42.410396 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:42.410368    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dnsfv\""
Jul 30 17:01:42.411382 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:42.411359    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ltj2t"
Jul 30 17:01:42.418687 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:42.418650    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kbsf2"
Jul 30 17:01:42.742547 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:42.742458    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8l786"
Jul 30 17:01:44.516588 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:44.516557    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-k7lnc"]
Jul 30 17:01:44.519679 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:01:44.519649    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad2d9b30_7d70_4236_9398_93ff51843c30.slice/crio-4bb1d9e9d357db78d86a5ed877878b0019fbaf4d2d393903e6b6e4bfb4dd0a11 WatchSource:0}: Error finding container 4bb1d9e9d357db78d86a5ed877878b0019fbaf4d2d393903e6b6e4bfb4dd0a11: Status 404 returned error can't find the container with id 4bb1d9e9d357db78d86a5ed877878b0019fbaf4d2d393903e6b6e4bfb4dd0a11
Jul 30 17:01:44.733224 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:44.733116    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ltj2t"]
Jul 30 17:01:44.736031 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:44.736002    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kbsf2"]
Jul 30 17:01:44.737094 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:01:44.737066    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33ffc7a5_96d7_4a2a_b6c2_ec922b4de57e.slice/crio-06c372a652a14b622cc261c6c41a09e299e79cc73be3b0e9b5af2037fdd3f3fc WatchSource:0}: Error finding container 06c372a652a14b622cc261c6c41a09e299e79cc73be3b0e9b5af2037fdd3f3fc: Status 404 returned error can't find the container with id 06c372a652a14b622cc261c6c41a09e299e79cc73be3b0e9b5af2037fdd3f3fc
Jul 30 17:01:44.739407 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:01:44.739375    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f8b90e9_e705_4b9c_89f7_047151a2b098.slice/crio-f75514d7512b7f3abb4e08d9b12587f17e570610c67e6e2a1871cdba05f3335e WatchSource:0}: Error finding container f75514d7512b7f3abb4e08d9b12587f17e570610c67e6e2a1871cdba05f3335e: Status 404 returned error can't find the container with id f75514d7512b7f3abb4e08d9b12587f17e570610c67e6e2a1871cdba05f3335e
Jul 30 17:01:44.785942 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:44.785894    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bjfhv" event={"ID":"3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61","Type":"ContainerStarted","Data":"c506b2fd90e3defebfd27fc53886d609434f91f2bcd8e6814d6149de19d5b425"}
Jul 30 17:01:44.787142 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:44.787112    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-k7lnc" event={"ID":"ad2d9b30-7d70-4236-9398-93ff51843c30","Type":"ContainerStarted","Data":"4bb1d9e9d357db78d86a5ed877878b0019fbaf4d2d393903e6b6e4bfb4dd0a11"}
Jul 30 17:01:44.788288 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:44.788262    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ltj2t" event={"ID":"33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e","Type":"ContainerStarted","Data":"06c372a652a14b622cc261c6c41a09e299e79cc73be3b0e9b5af2037fdd3f3fc"}
Jul 30 17:01:44.789307 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:44.789281    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kbsf2" event={"ID":"4f8b90e9-e705-4b9c-89f7-047151a2b098","Type":"ContainerStarted","Data":"f75514d7512b7f3abb4e08d9b12587f17e570610c67e6e2a1871cdba05f3335e"}
Jul 30 17:01:45.316629 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:45.316588    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6684865766-kwh46"]
Jul 30 17:01:45.319410 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:45.319376    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6684865766-kwh46"
Jul 30 17:01:45.320796 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:45.320763    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55469b5866-mlwg5"]
Jul 30 17:01:45.323523 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:45.323496    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55469b5866-mlwg5"
Jul 30 17:01:45.329097 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:45.329064    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 30 17:01:45.329288 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:45.329269    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-j72kw\""
Jul 30 17:01:45.331487 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:45.331037    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 30 17:01:45.331487 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:45.331315    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 30 17:01:45.332785 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:45.332748    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 30 17:01:45.333436 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:45.333330    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 30 17:01:45.352934 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:45.350542    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6684865766-kwh46"]
Jul 30 17:01:45.378892 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:45.378803    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55469b5866-mlwg5"]
Jul 30 17:01:45.457840 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:45.457491    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkw9s\" (UniqueName: \"kubernetes.io/projected/50113ea3-26e6-44c8-b840-7002b59e2df4-kube-api-access-fkw9s\") pod \"klusterlet-addon-workmgr-6684865766-kwh46\" (UID: \"50113ea3-26e6-44c8-b840-7002b59e2df4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6684865766-kwh46"
Jul 30 17:01:45.457840 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:45.457561    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/50113ea3-26e6-44c8-b840-7002b59e2df4-klusterlet-config\") pod \"klusterlet-addon-workmgr-6684865766-kwh46\" (UID: \"50113ea3-26e6-44c8-b840-7002b59e2df4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6684865766-kwh46"
Jul 30 17:01:45.457840 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:45.457604    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50113ea3-26e6-44c8-b840-7002b59e2df4-tmp\") pod \"klusterlet-addon-workmgr-6684865766-kwh46\" (UID: \"50113ea3-26e6-44c8-b840-7002b59e2df4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6684865766-kwh46"
Jul 30 17:01:45.457840 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:45.457634    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6f1d960d-3a0c-41d8-ac96-b57484b1ebd7-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-55469b5866-mlwg5\" (UID: \"6f1d960d-3a0c-41d8-ac96-b57484b1ebd7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55469b5866-mlwg5"
Jul 30 17:01:45.457840 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:45.457732    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kcjj\" (UniqueName: \"kubernetes.io/projected/6f1d960d-3a0c-41d8-ac96-b57484b1ebd7-kube-api-access-5kcjj\") pod \"managed-serviceaccount-addon-agent-55469b5866-mlwg5\" (UID: \"6f1d960d-3a0c-41d8-ac96-b57484b1ebd7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55469b5866-mlwg5"
Jul 30 17:01:45.559300 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:45.558286    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kcjj\" (UniqueName: \"kubernetes.io/projected/6f1d960d-3a0c-41d8-ac96-b57484b1ebd7-kube-api-access-5kcjj\") pod \"managed-serviceaccount-addon-agent-55469b5866-mlwg5\" (UID: \"6f1d960d-3a0c-41d8-ac96-b57484b1ebd7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55469b5866-mlwg5"
Jul 30 17:01:45.559300 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:45.558425    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkw9s\" (UniqueName: \"kubernetes.io/projected/50113ea3-26e6-44c8-b840-7002b59e2df4-kube-api-access-fkw9s\") pod \"klusterlet-addon-workmgr-6684865766-kwh46\" (UID: \"50113ea3-26e6-44c8-b840-7002b59e2df4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6684865766-kwh46"
Jul 30 17:01:45.559300 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:45.558474    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/50113ea3-26e6-44c8-b840-7002b59e2df4-klusterlet-config\") pod \"klusterlet-addon-workmgr-6684865766-kwh46\" (UID: \"50113ea3-26e6-44c8-b840-7002b59e2df4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6684865766-kwh46"
Jul 30 17:01:45.559300 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:45.558524    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50113ea3-26e6-44c8-b840-7002b59e2df4-tmp\") pod \"klusterlet-addon-workmgr-6684865766-kwh46\" (UID: \"50113ea3-26e6-44c8-b840-7002b59e2df4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6684865766-kwh46"
Jul 30 17:01:45.559300 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:45.558552    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6f1d960d-3a0c-41d8-ac96-b57484b1ebd7-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-55469b5866-mlwg5\" (UID: \"6f1d960d-3a0c-41d8-ac96-b57484b1ebd7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55469b5866-mlwg5"
Jul 30 17:01:45.560051 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:45.560021    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50113ea3-26e6-44c8-b840-7002b59e2df4-tmp\") pod \"klusterlet-addon-workmgr-6684865766-kwh46\" (UID: \"50113ea3-26e6-44c8-b840-7002b59e2df4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6684865766-kwh46"
Jul 30 17:01:45.571890 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:45.571794    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/50113ea3-26e6-44c8-b840-7002b59e2df4-klusterlet-config\") pod \"klusterlet-addon-workmgr-6684865766-kwh46\" (UID: \"50113ea3-26e6-44c8-b840-7002b59e2df4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6684865766-kwh46"
Jul 30 17:01:45.572958 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:45.572839    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkw9s\" (UniqueName: \"kubernetes.io/projected/50113ea3-26e6-44c8-b840-7002b59e2df4-kube-api-access-fkw9s\") pod \"klusterlet-addon-workmgr-6684865766-kwh46\" (UID: \"50113ea3-26e6-44c8-b840-7002b59e2df4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6684865766-kwh46"
Jul 30 17:01:45.573660 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:45.573605    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kcjj\" (UniqueName: \"kubernetes.io/projected/6f1d960d-3a0c-41d8-ac96-b57484b1ebd7-kube-api-access-5kcjj\") pod \"managed-serviceaccount-addon-agent-55469b5866-mlwg5\" (UID: \"6f1d960d-3a0c-41d8-ac96-b57484b1ebd7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55469b5866-mlwg5"
Jul 30 17:01:45.582471 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:45.582406    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6f1d960d-3a0c-41d8-ac96-b57484b1ebd7-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-55469b5866-mlwg5\" (UID: \"6f1d960d-3a0c-41d8-ac96-b57484b1ebd7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55469b5866-mlwg5"
Jul 30 17:01:45.638390 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:45.638346    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6684865766-kwh46"
Jul 30 17:01:45.657188 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:45.657134    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55469b5866-mlwg5"
Jul 30 17:01:46.256809 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:46.256709    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6684865766-kwh46"]
Jul 30 17:01:46.265477 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:01:46.265436    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50113ea3_26e6_44c8_b840_7002b59e2df4.slice/crio-fe9628c3b8712271e7e8809186a9fadd77032678ceb0641145f4060f39daaed2 WatchSource:0}: Error finding container fe9628c3b8712271e7e8809186a9fadd77032678ceb0641145f4060f39daaed2: Status 404 returned error can't find the container with id fe9628c3b8712271e7e8809186a9fadd77032678ceb0641145f4060f39daaed2
Jul 30 17:01:46.283422 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:46.282817    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55469b5866-mlwg5"]
Jul 30 17:01:46.287348 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:01:46.287305    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6f1d960d_3a0c_41d8_ac96_b57484b1ebd7.slice/crio-61293eb056858735f6afd3daf492caeee80a5e097530cc39d7873f629fae4144 WatchSource:0}: Error finding container 61293eb056858735f6afd3daf492caeee80a5e097530cc39d7873f629fae4144: Status 404 returned error can't find the container with id 61293eb056858735f6afd3daf492caeee80a5e097530cc39d7873f629fae4144
Jul 30 17:01:46.802068 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:46.801948    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6684865766-kwh46" event={"ID":"50113ea3-26e6-44c8-b840-7002b59e2df4","Type":"ContainerStarted","Data":"fe9628c3b8712271e7e8809186a9fadd77032678ceb0641145f4060f39daaed2"}
Jul 30 17:01:46.805853 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:46.805253    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kbsf2" event={"ID":"4f8b90e9-e705-4b9c-89f7-047151a2b098","Type":"ContainerStarted","Data":"b23e4f052af421f31a650930ec5047443008c80deded61f3d483ae6976c09fbc"}
Jul 30 17:01:46.805853 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:46.805342    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kbsf2" event={"ID":"4f8b90e9-e705-4b9c-89f7-047151a2b098","Type":"ContainerStarted","Data":"5fef059b1f6ece1cabe801b70505af475976446ff306af76ab7f9b40e13c142b"}
Jul 30 17:01:46.808486 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:46.808448    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55469b5866-mlwg5" event={"ID":"6f1d960d-3a0c-41d8-ac96-b57484b1ebd7","Type":"ContainerStarted","Data":"61293eb056858735f6afd3daf492caeee80a5e097530cc39d7873f629fae4144"}
Jul 30 17:01:46.823990 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:46.823926    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-kbsf2" podStartSLOduration=68.465051013 podStartE2EDuration="1m9.823906572s" totalImagesPullingTime="1.358855559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:00:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:46.822625405 +0000 UTC m=+69.897925297" watchObservedRunningTime="2026-07-30 17:01:46.823906572 +0000 UTC m=+69.899206440"
Jul 30 17:01:47.080151 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:47.079493    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/badf66a2-ec5d-4590-b910-2f6cddef1caf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-6xrrb\" (UID: \"badf66a2-ec5d-4590-b910-2f6cddef1caf\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-6xrrb"
Jul 30 17:01:47.097581 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:47.094478    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/badf66a2-ec5d-4590-b910-2f6cddef1caf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-6xrrb\" (UID: \"badf66a2-ec5d-4590-b910-2f6cddef1caf\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-6xrrb"
Jul 30 17:01:47.259593 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:47.259548    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-zb88h\""
Jul 30 17:01:47.267827 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:47.267781    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-6xrrb"
Jul 30 17:01:55.373034 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:55.372976    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-6xrrb"]
Jul 30 17:01:55.377782 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:01:55.377739    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbadf66a2_ec5d_4590_b910_2f6cddef1caf.slice/crio-32b0a81e4a0a4d3e1094845674c79cce4dfb712753112f2baafc91c0dfeb6b34 WatchSource:0}: Error finding container 32b0a81e4a0a4d3e1094845674c79cce4dfb712753112f2baafc91c0dfeb6b34: Status 404 returned error can't find the container with id 32b0a81e4a0a4d3e1094845674c79cce4dfb712753112f2baafc91c0dfeb6b34
Jul 30 17:01:55.847235 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:55.847190    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bjfhv" event={"ID":"3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61","Type":"ContainerStarted","Data":"c5cbaf603472de2407978cc1c472a3ba4e2d07f702421c79f79ae3f56dfbbf8e"}
Jul 30 17:01:55.848945 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:55.848911    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-k7lnc" event={"ID":"ad2d9b30-7d70-4236-9398-93ff51843c30","Type":"ContainerStarted","Data":"9453c4182f60d529dcf16bba0589b7131ed13c187f9c05178bf56070b25e7e59"}
Jul 30 17:01:55.850335 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:55.850301    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6684865766-kwh46" event={"ID":"50113ea3-26e6-44c8-b840-7002b59e2df4","Type":"ContainerStarted","Data":"8c3798b760db11b93f05e9df2ad6520273a3d0c2349c402afaf277562a16174c"}
Jul 30 17:01:55.850489 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:55.850477    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6684865766-kwh46"
Jul 30 17:01:55.851872 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:55.851833    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ltj2t" event={"ID":"33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e","Type":"ContainerStarted","Data":"388da57505ff03a806bb2fc8c0496c60420c15869ee22e84953f116fd5e01c4b"}
Jul 30 17:01:55.852009 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:55.851903    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ltj2t"
Jul 30 17:01:55.852525 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:55.852502    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6684865766-kwh46"
Jul 30 17:01:55.853287 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:55.853269    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-6xrrb" event={"ID":"badf66a2-ec5d-4590-b910-2f6cddef1caf","Type":"ContainerStarted","Data":"32b0a81e4a0a4d3e1094845674c79cce4dfb712753112f2baafc91c0dfeb6b34"}
Jul 30 17:01:55.854675 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:55.854655    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55469b5866-mlwg5" event={"ID":"6f1d960d-3a0c-41d8-ac96-b57484b1ebd7","Type":"ContainerStarted","Data":"32338302ed8b38529b837e47917489b9441407bdce0c9651f1c1228d1363fa81"}
Jul 30 17:01:55.865962 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:55.865910    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-k7lnc" podStartSLOduration=67.231696883 podStartE2EDuration="1m17.865889733s" totalImagesPullingTime="10.63419285s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 17:00:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:55.864104281 +0000 UTC m=+78.939404169" watchObservedRunningTime="2026-07-30 17:01:55.865889733 +0000 UTC m=+78.941189623"
Jul 30 17:01:55.879988 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:55.879910    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55469b5866-mlwg5" podStartSLOduration=1.970168514 podStartE2EDuration="10.879886519s" totalImagesPullingTime="8.909718005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:01:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:55.879195721 +0000 UTC m=+78.954495609" watchObservedRunningTime="2026-07-30 17:01:55.879886519 +0000 UTC m=+78.955186388"
Jul 30 17:01:55.895771 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:55.895714    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6684865766-kwh46" podStartSLOduration=1.9648059660000001 podStartE2EDuration="10.895696273s" totalImagesPullingTime="8.930890307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:01:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:55.894860941 +0000 UTC m=+78.970160843" watchObservedRunningTime="2026-07-30 17:01:55.895696273 +0000 UTC m=+78.970996162"
Jul 30 17:01:55.911796 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:55.911738    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-ltj2t" podStartSLOduration=67.495160025 podStartE2EDuration="1m17.911715582s" totalImagesPullingTime="10.416555557s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:00:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:55.911299717 +0000 UTC m=+78.986599609" watchObservedRunningTime="2026-07-30 17:01:55.911715582 +0000 UTC m=+78.987015476"
Jul 30 17:01:57.865051 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:57.865007    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-6xrrb" event={"ID":"badf66a2-ec5d-4590-b910-2f6cddef1caf","Type":"ContainerStarted","Data":"4409c6fb3b0bd8a3083e9938cafdd8225c3f1cde991201ee448049763eff43fe"}
Jul 30 17:01:57.888963 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:57.887838    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-qgll2"]
Jul 30 17:01:57.892113 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:57.892080    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-qgll2"
Jul 30 17:01:57.894678 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:57.894650    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 30 17:01:57.894851 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:57.894693    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-zcqkk\""
Jul 30 17:01:57.902933 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:57.902878    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-6xrrb" podStartSLOduration=40.893205363 podStartE2EDuration="42.902855648s" totalImagesPullingTime="2.009650285s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:01:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:57.889058214 +0000 UTC m=+80.964358105" watchObservedRunningTime="2026-07-30 17:01:57.902855648 +0000 UTC m=+80.978155538"
Jul 30 17:01:57.904129 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:57.904095    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-qgll2"]
Jul 30 17:01:57.968831 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:57.968665    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/abad2709-49a5-41f9-bc4d-25ff3db9e554-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-qgll2\" (UID: \"abad2709-49a5-41f9-bc4d-25ff3db9e554\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-qgll2"
Jul 30 17:01:58.070007 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:58.069966    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/abad2709-49a5-41f9-bc4d-25ff3db9e554-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-qgll2\" (UID: \"abad2709-49a5-41f9-bc4d-25ff3db9e554\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-qgll2"
Jul 30 17:01:58.070229 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:58.070141    2580 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 30 17:01:58.070308 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:01:58.070245    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abad2709-49a5-41f9-bc4d-25ff3db9e554-tls-certificates podName:abad2709-49a5-41f9-bc4d-25ff3db9e554 nodeName:}" failed. No retries permitted until 2026-07-30 17:01:58.570220933 +0000 UTC m=+81.645520805 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/abad2709-49a5-41f9-bc4d-25ff3db9e554-tls-certificates") pod "prometheus-operator-admission-webhook-7f8d78f54b-qgll2" (UID: "abad2709-49a5-41f9-bc4d-25ff3db9e554") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 30 17:01:58.575429 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:58.575382    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/abad2709-49a5-41f9-bc4d-25ff3db9e554-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-qgll2\" (UID: \"abad2709-49a5-41f9-bc4d-25ff3db9e554\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-qgll2"
Jul 30 17:01:58.578473 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:58.578430    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/abad2709-49a5-41f9-bc4d-25ff3db9e554-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-qgll2\" (UID: \"abad2709-49a5-41f9-bc4d-25ff3db9e554\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-qgll2"
Jul 30 17:01:58.806427 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:58.806373    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-qgll2"
Jul 30 17:01:59.314401 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:59.314367    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-qgll2"]
Jul 30 17:01:59.318007 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:01:59.317969    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabad2709_49a5_41f9_bc4d_25ff3db9e554.slice/crio-4caff13cccaefc71efe22771509ff496f88a0cf8805570bc39ef16fded117e0d WatchSource:0}: Error finding container 4caff13cccaefc71efe22771509ff496f88a0cf8805570bc39ef16fded117e0d: Status 404 returned error can't find the container with id 4caff13cccaefc71efe22771509ff496f88a0cf8805570bc39ef16fded117e0d
Jul 30 17:01:59.880032 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:59.879991    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bjfhv" event={"ID":"3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61","Type":"ContainerStarted","Data":"b57b882f7c773ad5d9cf1f9338855e2a075514b4517b92097499d3ea37596550"}
Jul 30 17:01:59.881359 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:59.881332    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-qgll2" event={"ID":"abad2709-49a5-41f9-bc4d-25ff3db9e554","Type":"ContainerStarted","Data":"4caff13cccaefc71efe22771509ff496f88a0cf8805570bc39ef16fded117e0d"}
Jul 30 17:01:59.901917 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:01:59.901851    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-bjfhv" podStartSLOduration=17.116601426 podStartE2EDuration="36.901828154s" totalImagesPullingTime="19.785226728s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 17:01:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:59.900242049 +0000 UTC m=+82.975541945" watchObservedRunningTime="2026-07-30 17:01:59.901828154 +0000 UTC m=+82.977128035"
Jul 30 17:02:00.886668 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:00.886626    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-qgll2" event={"ID":"abad2709-49a5-41f9-bc4d-25ff3db9e554","Type":"ContainerStarted","Data":"ffaf4a35116d4a7d85d2069817de69b58084f00865a7e90af1cd2d195f294033"}
Jul 30 17:02:00.905225 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:00.905152    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-qgll2" podStartSLOduration=2.791260406 podStartE2EDuration="3.90513142s" totalImagesPullingTime="1.113871014s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:01:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:02:00.90447401 +0000 UTC m=+83.979773899" watchObservedRunningTime="2026-07-30 17:02:00.90513142 +0000 UTC m=+83.980431309"
Jul 30 17:02:01.890835 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:01.890799    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-qgll2"
Jul 30 17:02:01.895757 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:01.895725    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-qgll2"
Jul 30 17:02:02.962590 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:02.962556    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-69625"]
Jul 30 17:02:02.965539 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:02.965519    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-69625"
Jul 30 17:02:02.968155 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:02.968129    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-sfwf4\""
Jul 30 17:02:02.968324 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:02.968222    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 30 17:02:02.968324 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:02.968266    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 30 17:02:02.969273 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:02.969256    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 30 17:02:02.975865 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:02.975838    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-69625"]
Jul 30 17:02:03.111476 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:03.111419    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e497af4-651c-4078-a0b5-bf1b7697e869-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-69625\" (UID: \"5e497af4-651c-4078-a0b5-bf1b7697e869\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-69625"
Jul 30 17:02:03.111476 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:03.111484    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e497af4-651c-4078-a0b5-bf1b7697e869-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-69625\" (UID: \"5e497af4-651c-4078-a0b5-bf1b7697e869\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-69625"
Jul 30 17:02:03.111721 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:03.111512    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e497af4-651c-4078-a0b5-bf1b7697e869-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-69625\" (UID: \"5e497af4-651c-4078-a0b5-bf1b7697e869\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-69625"
Jul 30 17:02:03.111721 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:03.111534    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98rt7\" (UniqueName: \"kubernetes.io/projected/5e497af4-651c-4078-a0b5-bf1b7697e869-kube-api-access-98rt7\") pod \"prometheus-operator-84d69f794d-69625\" (UID: \"5e497af4-651c-4078-a0b5-bf1b7697e869\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-69625"
Jul 30 17:02:03.211976 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:03.211936    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e497af4-651c-4078-a0b5-bf1b7697e869-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-69625\" (UID: \"5e497af4-651c-4078-a0b5-bf1b7697e869\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-69625"
Jul 30 17:02:03.211976 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:03.211985    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e497af4-651c-4078-a0b5-bf1b7697e869-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-69625\" (UID: \"5e497af4-651c-4078-a0b5-bf1b7697e869\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-69625"
Jul 30 17:02:03.212435 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:03.212012    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98rt7\" (UniqueName: \"kubernetes.io/projected/5e497af4-651c-4078-a0b5-bf1b7697e869-kube-api-access-98rt7\") pod \"prometheus-operator-84d69f794d-69625\" (UID: \"5e497af4-651c-4078-a0b5-bf1b7697e869\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-69625"
Jul 30 17:02:03.212435 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:03.212092    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e497af4-651c-4078-a0b5-bf1b7697e869-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-69625\" (UID: \"5e497af4-651c-4078-a0b5-bf1b7697e869\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-69625"
Jul 30 17:02:03.212767 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:03.212689    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e497af4-651c-4078-a0b5-bf1b7697e869-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-69625\" (UID: \"5e497af4-651c-4078-a0b5-bf1b7697e869\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-69625"
Jul 30 17:02:03.214527 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:03.214491    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e497af4-651c-4078-a0b5-bf1b7697e869-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-69625\" (UID: \"5e497af4-651c-4078-a0b5-bf1b7697e869\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-69625"
Jul 30 17:02:03.214666 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:03.214565    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e497af4-651c-4078-a0b5-bf1b7697e869-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-69625\" (UID: \"5e497af4-651c-4078-a0b5-bf1b7697e869\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-69625"
Jul 30 17:02:03.221014 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:03.220987    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98rt7\" (UniqueName: \"kubernetes.io/projected/5e497af4-651c-4078-a0b5-bf1b7697e869-kube-api-access-98rt7\") pod \"prometheus-operator-84d69f794d-69625\" (UID: \"5e497af4-651c-4078-a0b5-bf1b7697e869\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-69625"
Jul 30 17:02:03.275104 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:03.275061    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-69625"
Jul 30 17:02:03.427449 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:03.427408    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-69625"]
Jul 30 17:02:03.430303 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:02:03.430272    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e497af4_651c_4078_a0b5_bf1b7697e869.slice/crio-7f3be43607355866145d8b9d662381fba00a625f7e3303de01ee4373242968f0 WatchSource:0}: Error finding container 7f3be43607355866145d8b9d662381fba00a625f7e3303de01ee4373242968f0: Status 404 returned error can't find the container with id 7f3be43607355866145d8b9d662381fba00a625f7e3303de01ee4373242968f0
Jul 30 17:02:03.898854 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:03.898810    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-69625" event={"ID":"5e497af4-651c-4078-a0b5-bf1b7697e869","Type":"ContainerStarted","Data":"7f3be43607355866145d8b9d662381fba00a625f7e3303de01ee4373242968f0"}
Jul 30 17:02:05.908673 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:05.908640    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-69625" event={"ID":"5e497af4-651c-4078-a0b5-bf1b7697e869","Type":"ContainerStarted","Data":"826941dfa82aef7c1db3a6639902c19ab6af7bb8e3d75cc86b2f83c6e0ebbe29"}
Jul 30 17:02:05.909106 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:05.908696    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-69625" event={"ID":"5e497af4-651c-4078-a0b5-bf1b7697e869","Type":"ContainerStarted","Data":"633fec4def3b8084906514a7402639b0a2e0f5f8d6f35c10c33231158d9e30b8"}
Jul 30 17:02:05.927590 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:05.927535    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-69625" podStartSLOduration=2.190033475 podStartE2EDuration="3.927516691s" totalImagesPullingTime="1.737483216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:02:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:02:05.925892817 +0000 UTC m=+89.001192698" watchObservedRunningTime="2026-07-30 17:02:05.927516691 +0000 UTC m=+89.002816580"
Jul 30 17:02:08.349236 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:08.349197    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-99w2g"]
Jul 30 17:02:08.355955 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:08.355921    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-99w2g"
Jul 30 17:02:08.358913 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:08.358786    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 30 17:02:08.359219 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:08.359198    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-skn5h\""
Jul 30 17:02:08.359788 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:08.359771    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 30 17:02:08.359872 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:08.359776    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 30 17:02:08.360070 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:08.360052    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-l7cvg"]
Jul 30 17:02:08.362405 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:08.362382    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-l7cvg"
Jul 30 17:02:08.364852 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:08.364829    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-99w2g"]
Jul 30 17:02:08.365063 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:08.365040    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-kdk5k\""
Jul 30 17:02:08.365291 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:08.365073    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 30 17:02:08.365392 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:08.365098    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 30 17:02:08.365392 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:08.365153    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 30 17:02:08.454997 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:08.454954    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9b737c7c-d8a8-4567-86cd-bbbbe37b1a36-sys\") pod \"node-exporter-l7cvg\" (UID: \"9b737c7c-d8a8-4567-86cd-bbbbe37b1a36\") " pod="openshift-monitoring/node-exporter-l7cvg"
Jul 30 17:02:08.454997 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:08.455008    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b737c7c-d8a8-4567-86cd-bbbbe37b1a36-metrics-client-ca\") pod \"node-exporter-l7cvg\" (UID: \"9b737c7c-d8a8-4567-86cd-bbbbe37b1a36\") " pod="openshift-monitoring/node-exporter-l7cvg"
Jul 30 17:02:08.455288 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:08.455034    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlvmp\" (UniqueName: \"kubernetes.io/projected/f775eb31-353d-410e-add2-e2aa791f31c8-kube-api-access-xlvmp\") pod \"kube-state-metrics-56d9f944b4-99w2g\" (UID: \"f775eb31-353d-410e-add2-e2aa791f31c8\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-99w2g"
Jul 30 17:02:08.455288 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:08.455102    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9b737c7c-d8a8-4567-86cd-bbbbe37b1a36-root\") pod \"node-exporter-l7cvg\" (UID: \"9b737c7c-d8a8-4567-86cd-bbbbe37b1a36\") " pod="openshift-monitoring/node-exporter-l7cvg"
Jul 30 17:02:08.455288 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:08.455155    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9b737c7c-d8a8-4567-86cd-bbbbe37b1a36-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l7cvg\" (UID: \"9b737c7c-d8a8-4567-86cd-bbbbe37b1a36\") " pod="openshift-monitoring/node-exporter-l7cvg"
Jul 30 17:02:08.455288 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:08.455217    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9b737c7c-d8a8-4567-86cd-bbbbe37b1a36-node-exporter-wtmp\") pod \"node-exporter-l7cvg\" (UID: \"9b737c7c-d8a8-4567-86cd-bbbbe37b1a36\") " pod="openshift-monitoring/node-exporter-l7cvg"
Jul 30 17:02:08.455288 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:08.455244    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f775eb31-353d-410e-add2-e2aa791f31c8-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-99w2g\" (UID: \"f775eb31-353d-410e-add2-e2aa791f31c8\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-99w2g"
Jul 30 17:02:08.455288 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:08.455262    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f775eb31-353d-410e-add2-e2aa791f31c8-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-99w2g\" (UID: \"f775eb31-353d-410e-add2-e2aa791f31c8\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-99w2g"
Jul 30 17:02:08.455474 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:08.455294    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f775eb31-353d-410e-add2-e2aa791f31c8-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-99w2g\" (UID: \"f775eb31-353d-410e-add2-e2aa791f31c8\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-99w2g"
Jul 30 17:02:08.455474 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:08.455334    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f775eb31-353d-410e-add2-e2aa791f31c8-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-99w2g\" (UID: \"f775eb31-353d-410e-add2-e2aa791f31c8\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-99w2g"
Jul 30 17:02:08.455474 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:08.455357    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9b737c7c-d8a8-4567-86cd-bbbbe37b1a36-node-exporter-accelerators-collector-config\") pod \"node-exporter-l7cvg\" (UID: \"9b737c7c-d8a8-4567-86cd-bbbbe37b1a36\") " pod="openshift-monitoring/node-exporter-l7cvg"
Jul 30 17:02:08.455474 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:08.455393    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f775eb31-353d-410e-add2-e2aa791f31c8-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-99w2g\" (UID: \"f775eb31-353d-410e-add2-e2aa791f31c8\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-99w2g"
Jul 30 17:02:08.455474 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:08.455414    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9b737c7c-d8a8-4567-86cd-bbbbe37b1a36-node-exporter-tls\") pod \"node-exporter-l7cvg\" (UID: \"9b737c7c-d8a8-4567-86cd-bbbbe37b1a36\") " pod="openshift-monitoring/node-exporter-l7cvg"
Jul 30 17:02:08.455474 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:08.455440    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22pb8\" (UniqueName: \"kubernetes.io/projected/9b737c7c-d8a8-4567-86cd-bbbbe37b1a36-kube-api-access-22pb8\") pod \"node-exporter-l7cvg\" (UID: \"9b737c7c-d8a8-4567-86cd-bbbbe37b1a36\") " pod="openshift-monitoring/node-exporter-l7cvg"
Jul 30 17:02:08.455474 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:08.455464    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9b737c7c-d8a8-4567-86cd-bbbbe37b1a36-node-exporter-textfile\") pod \"node-exporter-l7cvg\" (UID: \"9b737c7c-d8a8-4567-86cd-bbbbe37b1a36\") " pod="openshift-monitoring/node-exporter-l7cvg"
Jul 30 17:02:08.556107 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:08.556059    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f775eb31-353d-410e-add2-e2aa791f31c8-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-99w2g\" (UID: \"f775eb31-353d-410e-add2-e2aa791f31c8\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-99w2g"
Jul 30 17:02:08.556107 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:08.556104    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f775eb31-353d-410e-add2-e2aa791f31c8-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-99w2g\" (UID: \"f775eb31-353d-410e-add2-e2aa791f31c8\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-99w2g"
Jul 30 17:02:08.556371 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:08.556126    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9b737c7c-d8a8-4567-86cd-bbbbe37b1a36-node-exporter-accelerators-collector-config\") pod \"node-exporter-l7cvg\" (UID: \"9b737c7c-d8a8-4567-86cd-bbbbe37b1a36\") " pod="openshift-monitoring/node-exporter-l7cvg"
Jul 30 17:02:08.556371 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:08.556146    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f775eb31-353d-410e-add2-e2aa791f31c8-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-99w2g\" (UID: \"f775eb31-353d-410e-add2-e2aa791f31c8\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-99w2g"
Jul 30 17:02:08.556371 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:08.556296    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9b737c7c-d8a8-4567-86cd-bbbbe37b1a36-node-exporter-tls\") pod \"node-exporter-l7cvg\" (UID: \"9b737c7c-d8a8-4567-86cd-bbbbe37b1a36\") " pod="openshift-monitoring/node-exporter-l7cvg"
Jul 30 17:02:08.556371 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:08.556356    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22pb8\" (UniqueName: \"kubernetes.io/projected/9b737c7c-d8a8-4567-86cd-bbbbe37b1a36-kube-api-access-22pb8\") pod \"node-exporter-l7cvg\" (UID: \"9b737c7c-d8a8-4567-86cd-bbbbe37b1a36\") " pod="openshift-monitoring/node-exporter-l7cvg"
Jul 30 17:02:08.556570 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:08.556384    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9b737c7c-d8a8-4567-86cd-bbbbe37b1a36-node-exporter-textfile\") pod \"node-exporter-l7cvg\" (UID: \"9b737c7c-d8a8-4567-86cd-bbbbe37b1a36\") " pod="openshift-monitoring/node-exporter-l7cvg"
Jul 30 17:02:08.556570 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:08.556428    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9b737c7c-d8a8-4567-86cd-bbbbe37b1a36-sys\") pod \"node-exporter-l7cvg\" (UID: \"9b737c7c-d8a8-4567-86cd-bbbbe37b1a36\") " pod="openshift-monitoring/node-exporter-l7cvg"
Jul 30 17:02:08.556570 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:02:08.556446    2580 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 30 17:02:08.556570 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:08.556473    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b737c7c-d8a8-4567-86cd-bbbbe37b1a36-metrics-client-ca\") pod \"node-exporter-l7cvg\" (UID: \"9b737c7c-d8a8-4567-86cd-bbbbe37b1a36\") " pod="openshift-monitoring/node-exporter-l7cvg"
Jul 30 17:02:08.556570 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:08.556509    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlvmp\" (UniqueName: \"kubernetes.io/projected/f775eb31-353d-410e-add2-e2aa791f31c8-kube-api-access-xlvmp\") pod \"kube-state-metrics-56d9f944b4-99w2g\" (UID: \"f775eb31-353d-410e-add2-e2aa791f31c8\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-99w2g"
Jul 30 17:02:08.556570 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:02:08.556531    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b737c7c-d8a8-4567-86cd-bbbbe37b1a36-node-exporter-tls podName:9b737c7c-d8a8-4567-86cd-bbbbe37b1a36 nodeName:}" failed. No retries permitted until 2026-07-30 17:02:09.056506809 +0000 UTC m=+92.131806692 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/9b737c7c-d8a8-4567-86cd-bbbbe37b1a36-node-exporter-tls") pod "node-exporter-l7cvg" (UID: "9b737c7c-d8a8-4567-86cd-bbbbe37b1a36") : secret "node-exporter-tls" not found
Jul 30 17:02:08.556877 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:08.556574    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f775eb31-353d-410e-add2-e2aa791f31c8-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-99w2g\" (UID: \"f775eb31-353d-410e-add2-e2aa791f31c8\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-99w2g"
Jul 30 17:02:08.556877 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:08.556689    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9b737c7c-d8a8-4567-86cd-bbbbe37b1a36-root\") pod \"node-exporter-l7cvg\" (UID: \"9b737c7c-d8a8-4567-86cd-bbbbe37b1a36\") " pod="openshift-monitoring/node-exporter-l7cvg"
Jul 30 17:02:08.556877 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:08.556694    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9b737c7c-d8a8-4567-86cd-bbbbe37b1a36-sys\") pod \"node-exporter-l7cvg\" (UID: \"9b737c7c-d8a8-4567-86cd-bbbbe37b1a36\") " pod="openshift-monitoring/node-exporter-l7cvg"
Jul 30 17:02:08.556877 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:08.556748    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9b737c7c-d8a8-4567-86cd-bbbbe37b1a36-root\") pod \"node-exporter-l7cvg\" (UID: \"9b737c7c-d8a8-4567-86cd-bbbbe37b1a36\") " pod="openshift-monitoring/node-exporter-l7cvg"
Jul 30 17:02:08.556877 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:08.556782    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9b737c7c-d8a8-4567-86cd-bbbbe37b1a36-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l7cvg\" (UID: \"9b737c7c-d8a8-4567-86cd-bbbbe37b1a36\") " pod="openshift-monitoring/node-exporter-l7cvg"
Jul 30 17:02:08.556877 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:08.556806    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9b737c7c-d8a8-4567-86cd-bbbbe37b1a36-node-exporter-textfile\") pod \"node-exporter-l7cvg\" (UID: \"9b737c7c-d8a8-4567-86cd-bbbbe37b1a36\") " pod="openshift-monitoring/node-exporter-l7cvg"
Jul 30 17:02:08.556877 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:08.556825    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9b737c7c-d8a8-4567-86cd-bbbbe37b1a36-node-exporter-wtmp\") pod \"node-exporter-l7cvg\" (UID: \"9b737c7c-d8a8-4567-86cd-bbbbe37b1a36\") " pod="openshift-monitoring/node-exporter-l7cvg"
Jul 30 17:02:08.556877 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:08.556834    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9b737c7c-d8a8-4567-86cd-bbbbe37b1a36-node-exporter-accelerators-collector-config\") pod \"node-exporter-l7cvg\" (UID: \"9b737c7c-d8a8-4567-86cd-bbbbe37b1a36\") " pod="openshift-monitoring/node-exporter-l7cvg"
Jul 30 17:02:08.556877 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:08.556860    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f775eb31-353d-410e-add2-e2aa791f31c8-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-99w2g\" (UID: \"f775eb31-353d-410e-add2-e2aa791f31c8\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-99w2g"
Jul 30 17:02:08.557310 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:08.556887    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f775eb31-353d-410e-add2-e2aa791f31c8-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-99w2g\" (UID: \"f775eb31-353d-410e-add2-e2aa791f31c8\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-99w2g"
Jul 30 17:02:08.557310 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:02:08.556975    2580 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 30 17:02:08.557310 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:08.556997    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9b737c7c-d8a8-4567-86cd-bbbbe37b1a36-node-exporter-wtmp\") pod \"node-exporter-l7cvg\" (UID: \"9b737c7c-d8a8-4567-86cd-bbbbe37b1a36\") " pod="openshift-monitoring/node-exporter-l7cvg"
Jul 30 17:02:08.557310 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:02:08.557029    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f775eb31-353d-410e-add2-e2aa791f31c8-kube-state-metrics-tls podName:f775eb31-353d-410e-add2-e2aa791f31c8 nodeName:}" failed. No retries permitted until 2026-07-30 17:02:09.057009464 +0000 UTC m=+92.132309342 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/f775eb31-353d-410e-add2-e2aa791f31c8-kube-state-metrics-tls") pod "kube-state-metrics-56d9f944b4-99w2g" (UID: "f775eb31-353d-410e-add2-e2aa791f31c8") : secret "kube-state-metrics-tls" not found
Jul 30 17:02:08.557310 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:08.557073    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b737c7c-d8a8-4567-86cd-bbbbe37b1a36-metrics-client-ca\") pod \"node-exporter-l7cvg\" (UID: \"9b737c7c-d8a8-4567-86cd-bbbbe37b1a36\") " pod="openshift-monitoring/node-exporter-l7cvg"
Jul 30 17:02:08.557563 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:08.557454    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f775eb31-353d-410e-add2-e2aa791f31c8-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-99w2g\" (UID: \"f775eb31-353d-410e-add2-e2aa791f31c8\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-99w2g"
Jul 30 17:02:08.557563 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:08.557473    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f775eb31-353d-410e-add2-e2aa791f31c8-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-99w2g\" (UID: \"f775eb31-353d-410e-add2-e2aa791f31c8\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-99w2g"
Jul 30 17:02:08.558776 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:08.558751    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f775eb31-353d-410e-add2-e2aa791f31c8-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-99w2g\" (UID: \"f775eb31-353d-410e-add2-e2aa791f31c8\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-99w2g"
Jul 30 17:02:08.559001 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:08.558982    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9b737c7c-d8a8-4567-86cd-bbbbe37b1a36-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l7cvg\" (UID: \"9b737c7c-d8a8-4567-86cd-bbbbe37b1a36\") " pod="openshift-monitoring/node-exporter-l7cvg"
Jul 30 17:02:08.566995 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:08.566958    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22pb8\" (UniqueName: \"kubernetes.io/projected/9b737c7c-d8a8-4567-86cd-bbbbe37b1a36-kube-api-access-22pb8\") pod \"node-exporter-l7cvg\" (UID: \"9b737c7c-d8a8-4567-86cd-bbbbe37b1a36\") " pod="openshift-monitoring/node-exporter-l7cvg"
Jul 30 17:02:08.571359 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:08.571326    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlvmp\" (UniqueName: \"kubernetes.io/projected/f775eb31-353d-410e-add2-e2aa791f31c8-kube-api-access-xlvmp\") pod \"kube-state-metrics-56d9f944b4-99w2g\" (UID: \"f775eb31-353d-410e-add2-e2aa791f31c8\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-99w2g"
Jul 30 17:02:09.060920 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:09.060876    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f775eb31-353d-410e-add2-e2aa791f31c8-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-99w2g\" (UID: \"f775eb31-353d-410e-add2-e2aa791f31c8\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-99w2g"
Jul 30 17:02:09.060920 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:09.060929    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9b737c7c-d8a8-4567-86cd-bbbbe37b1a36-node-exporter-tls\") pod \"node-exporter-l7cvg\" (UID: \"9b737c7c-d8a8-4567-86cd-bbbbe37b1a36\") " pod="openshift-monitoring/node-exporter-l7cvg"
Jul 30 17:02:09.061234 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:02:09.061070    2580 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 30 17:02:09.061234 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:02:09.061148    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b737c7c-d8a8-4567-86cd-bbbbe37b1a36-node-exporter-tls podName:9b737c7c-d8a8-4567-86cd-bbbbe37b1a36 nodeName:}" failed. No retries permitted until 2026-07-30 17:02:10.06113098 +0000 UTC m=+93.136430863 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/9b737c7c-d8a8-4567-86cd-bbbbe37b1a36-node-exporter-tls") pod "node-exporter-l7cvg" (UID: "9b737c7c-d8a8-4567-86cd-bbbbe37b1a36") : secret "node-exporter-tls" not found
Jul 30 17:02:09.063429 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:09.063407    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f775eb31-353d-410e-add2-e2aa791f31c8-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-99w2g\" (UID: \"f775eb31-353d-410e-add2-e2aa791f31c8\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-99w2g"
Jul 30 17:02:09.266939 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:09.266899    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-99w2g"
Jul 30 17:02:09.426707 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:09.426663    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-99w2g"]
Jul 30 17:02:09.430359 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:02:09.430318    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf775eb31_353d_410e_add2_e2aa791f31c8.slice/crio-f9e57fbf15c2066264b389cad27680851fd2126a2b924a3e0433b398511d3ccd WatchSource:0}: Error finding container f9e57fbf15c2066264b389cad27680851fd2126a2b924a3e0433b398511d3ccd: Status 404 returned error can't find the container with id f9e57fbf15c2066264b389cad27680851fd2126a2b924a3e0433b398511d3ccd
Jul 30 17:02:09.925369 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:09.925326    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-99w2g" event={"ID":"f775eb31-353d-410e-add2-e2aa791f31c8","Type":"ContainerStarted","Data":"f9e57fbf15c2066264b389cad27680851fd2126a2b924a3e0433b398511d3ccd"}
Jul 30 17:02:10.070401 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:10.070358    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9b737c7c-d8a8-4567-86cd-bbbbe37b1a36-node-exporter-tls\") pod \"node-exporter-l7cvg\" (UID: \"9b737c7c-d8a8-4567-86cd-bbbbe37b1a36\") " pod="openshift-monitoring/node-exporter-l7cvg"
Jul 30 17:02:10.073307 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:10.073275    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9b737c7c-d8a8-4567-86cd-bbbbe37b1a36-node-exporter-tls\") pod \"node-exporter-l7cvg\" (UID: \"9b737c7c-d8a8-4567-86cd-bbbbe37b1a36\") " pod="openshift-monitoring/node-exporter-l7cvg"
Jul 30 17:02:10.173495 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:10.173456    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-l7cvg"
Jul 30 17:02:10.185372 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:02:10.185335    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b737c7c_d8a8_4567_86cd_bbbbe37b1a36.slice/crio-c8a94b121ba06e94e5262de9928843414574aadf42e7660e0c4e7b798b71c563 WatchSource:0}: Error finding container c8a94b121ba06e94e5262de9928843414574aadf42e7660e0c4e7b798b71c563: Status 404 returned error can't find the container with id c8a94b121ba06e94e5262de9928843414574aadf42e7660e0c4e7b798b71c563
Jul 30 17:02:10.932190 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:10.932069    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l7cvg" event={"ID":"9b737c7c-d8a8-4567-86cd-bbbbe37b1a36","Type":"ContainerStarted","Data":"c8a94b121ba06e94e5262de9928843414574aadf42e7660e0c4e7b798b71c563"}
Jul 30 17:02:10.935121 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:10.935088    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-99w2g" event={"ID":"f775eb31-353d-410e-add2-e2aa791f31c8","Type":"ContainerStarted","Data":"37a4888cb6955a7653fdafc6b33403b9640e6b3fde235014c24e4e4cec7459e3"}
Jul 30 17:02:10.935307 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:10.935152    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-99w2g" event={"ID":"f775eb31-353d-410e-add2-e2aa791f31c8","Type":"ContainerStarted","Data":"b49a0d8052a767644594c8821e09182a65cdb14f535a08101d30fa0345578c7a"}
Jul 30 17:02:10.935307 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:10.935200    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-99w2g" event={"ID":"f775eb31-353d-410e-add2-e2aa791f31c8","Type":"ContainerStarted","Data":"e3cd5cf27f9f8209dc62128b91475f9d47166d17c5b0706536f8e3424b519da4"}
Jul 30 17:02:10.957305 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:10.957007    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-99w2g" podStartSLOduration=1.7659023550000001 podStartE2EDuration="2.956981129s" totalImagesPullingTime="1.191078774s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:02:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:02:10.956677274 +0000 UTC m=+94.031977211" watchObservedRunningTime="2026-07-30 17:02:10.956981129 +0000 UTC m=+94.032281018"
Jul 30 17:02:11.417336 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:11.417299    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5989fc4c8b-gdxvq"]
Jul 30 17:02:11.421646 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:11.421620    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5989fc4c8b-gdxvq"
Jul 30 17:02:11.424519 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:11.424487    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-44l5qutjfs4dg\""
Jul 30 17:02:11.424519 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:11.424519    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 30 17:02:11.424729 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:11.424519    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 30 17:02:11.424729 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:11.424567    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 30 17:02:11.424967 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:11.424940    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-4nf5v\""
Jul 30 17:02:11.425077 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:11.425020    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 30 17:02:11.425077 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:11.425041    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 30 17:02:11.433280 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:11.433247    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5989fc4c8b-gdxvq"]
Jul 30 17:02:11.584792 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:11.584744    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e8869c6-d815-4f74-b761-b5309510e276-metrics-client-ca\") pod \"thanos-querier-5989fc4c8b-gdxvq\" (UID: \"8e8869c6-d815-4f74-b761-b5309510e276\") " pod="openshift-monitoring/thanos-querier-5989fc4c8b-gdxvq"
Jul 30 17:02:11.585015 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:11.584809    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8e8869c6-d815-4f74-b761-b5309510e276-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5989fc4c8b-gdxvq\" (UID: \"8e8869c6-d815-4f74-b761-b5309510e276\") " pod="openshift-monitoring/thanos-querier-5989fc4c8b-gdxvq"
Jul 30 17:02:11.585015 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:11.584835    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22t58\" (UniqueName: \"kubernetes.io/projected/8e8869c6-d815-4f74-b761-b5309510e276-kube-api-access-22t58\") pod \"thanos-querier-5989fc4c8b-gdxvq\" (UID: \"8e8869c6-d815-4f74-b761-b5309510e276\") " pod="openshift-monitoring/thanos-querier-5989fc4c8b-gdxvq"
Jul 30 17:02:11.585015 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:11.584860    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8e8869c6-d815-4f74-b761-b5309510e276-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5989fc4c8b-gdxvq\" (UID: \"8e8869c6-d815-4f74-b761-b5309510e276\") " pod="openshift-monitoring/thanos-querier-5989fc4c8b-gdxvq"
Jul 30 17:02:11.585015 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:11.584963    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8e8869c6-d815-4f74-b761-b5309510e276-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5989fc4c8b-gdxvq\" (UID: \"8e8869c6-d815-4f74-b761-b5309510e276\") " pod="openshift-monitoring/thanos-querier-5989fc4c8b-gdxvq"
Jul 30 17:02:11.585269 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:11.585017    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8e8869c6-d815-4f74-b761-b5309510e276-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5989fc4c8b-gdxvq\" (UID: \"8e8869c6-d815-4f74-b761-b5309510e276\") " pod="openshift-monitoring/thanos-querier-5989fc4c8b-gdxvq"
Jul 30 17:02:11.585269 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:11.585061    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8e8869c6-d815-4f74-b761-b5309510e276-secret-grpc-tls\") pod \"thanos-querier-5989fc4c8b-gdxvq\" (UID: \"8e8869c6-d815-4f74-b761-b5309510e276\") " pod="openshift-monitoring/thanos-querier-5989fc4c8b-gdxvq"
Jul 30 17:02:11.585269 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:11.585125    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8e8869c6-d815-4f74-b761-b5309510e276-secret-thanos-querier-tls\") pod \"thanos-querier-5989fc4c8b-gdxvq\" (UID: \"8e8869c6-d815-4f74-b761-b5309510e276\") " pod="openshift-monitoring/thanos-querier-5989fc4c8b-gdxvq"
Jul 30 17:02:11.685611 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:11.685504    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22t58\" (UniqueName: \"kubernetes.io/projected/8e8869c6-d815-4f74-b761-b5309510e276-kube-api-access-22t58\") pod \"thanos-querier-5989fc4c8b-gdxvq\" (UID: \"8e8869c6-d815-4f74-b761-b5309510e276\") " pod="openshift-monitoring/thanos-querier-5989fc4c8b-gdxvq"
Jul 30 17:02:11.685611 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:11.685552    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8e8869c6-d815-4f74-b761-b5309510e276-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5989fc4c8b-gdxvq\" (UID: \"8e8869c6-d815-4f74-b761-b5309510e276\") " pod="openshift-monitoring/thanos-querier-5989fc4c8b-gdxvq"
Jul 30 17:02:11.685611 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:11.685576    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8e8869c6-d815-4f74-b761-b5309510e276-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5989fc4c8b-gdxvq\" (UID: \"8e8869c6-d815-4f74-b761-b5309510e276\") " pod="openshift-monitoring/thanos-querier-5989fc4c8b-gdxvq"
Jul 30 17:02:11.685903 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:11.685752    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8e8869c6-d815-4f74-b761-b5309510e276-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5989fc4c8b-gdxvq\" (UID: \"8e8869c6-d815-4f74-b761-b5309510e276\") " pod="openshift-monitoring/thanos-querier-5989fc4c8b-gdxvq"
Jul 30 17:02:11.685903 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:11.685808    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8e8869c6-d815-4f74-b761-b5309510e276-secret-grpc-tls\") pod \"thanos-querier-5989fc4c8b-gdxvq\" (UID: \"8e8869c6-d815-4f74-b761-b5309510e276\") " pod="openshift-monitoring/thanos-querier-5989fc4c8b-gdxvq"
Jul 30 17:02:11.685903 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:11.685860    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8e8869c6-d815-4f74-b761-b5309510e276-secret-thanos-querier-tls\") pod \"thanos-querier-5989fc4c8b-gdxvq\" (UID: \"8e8869c6-d815-4f74-b761-b5309510e276\") " pod="openshift-monitoring/thanos-querier-5989fc4c8b-gdxvq"
Jul 30 17:02:11.686065 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:11.685913    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e8869c6-d815-4f74-b761-b5309510e276-metrics-client-ca\") pod \"thanos-querier-5989fc4c8b-gdxvq\" (UID: \"8e8869c6-d815-4f74-b761-b5309510e276\") " pod="openshift-monitoring/thanos-querier-5989fc4c8b-gdxvq"
Jul 30 17:02:11.686065 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:11.685963    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8e8869c6-d815-4f74-b761-b5309510e276-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5989fc4c8b-gdxvq\" (UID: \"8e8869c6-d815-4f74-b761-b5309510e276\") " pod="openshift-monitoring/thanos-querier-5989fc4c8b-gdxvq"
Jul 30 17:02:11.686814 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:11.686783    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e8869c6-d815-4f74-b761-b5309510e276-metrics-client-ca\") pod \"thanos-querier-5989fc4c8b-gdxvq\" (UID: \"8e8869c6-d815-4f74-b761-b5309510e276\") " pod="openshift-monitoring/thanos-querier-5989fc4c8b-gdxvq"
Jul 30 17:02:11.688337 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:11.688304    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8e8869c6-d815-4f74-b761-b5309510e276-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5989fc4c8b-gdxvq\" (UID: \"8e8869c6-d815-4f74-b761-b5309510e276\") " pod="openshift-monitoring/thanos-querier-5989fc4c8b-gdxvq"
Jul 30 17:02:11.688552 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:11.688531    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8e8869c6-d815-4f74-b761-b5309510e276-secret-grpc-tls\") pod \"thanos-querier-5989fc4c8b-gdxvq\" (UID: \"8e8869c6-d815-4f74-b761-b5309510e276\") " pod="openshift-monitoring/thanos-querier-5989fc4c8b-gdxvq"
Jul 30 17:02:11.688626 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:11.688573    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8e8869c6-d815-4f74-b761-b5309510e276-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5989fc4c8b-gdxvq\" (UID: \"8e8869c6-d815-4f74-b761-b5309510e276\") " pod="openshift-monitoring/thanos-querier-5989fc4c8b-gdxvq"
Jul 30 17:02:11.688792 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:11.688771    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8e8869c6-d815-4f74-b761-b5309510e276-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5989fc4c8b-gdxvq\" (UID: \"8e8869c6-d815-4f74-b761-b5309510e276\") " pod="openshift-monitoring/thanos-querier-5989fc4c8b-gdxvq"
Jul 30 17:02:11.688856 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:11.688776    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8e8869c6-d815-4f74-b761-b5309510e276-secret-thanos-querier-tls\") pod \"thanos-querier-5989fc4c8b-gdxvq\" (UID: \"8e8869c6-d815-4f74-b761-b5309510e276\") " pod="openshift-monitoring/thanos-querier-5989fc4c8b-gdxvq"
Jul 30 17:02:11.688905 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:11.688850    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8e8869c6-d815-4f74-b761-b5309510e276-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5989fc4c8b-gdxvq\" (UID: \"8e8869c6-d815-4f74-b761-b5309510e276\") " pod="openshift-monitoring/thanos-querier-5989fc4c8b-gdxvq"
Jul 30 17:02:11.693382 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:11.693357    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22t58\" (UniqueName: \"kubernetes.io/projected/8e8869c6-d815-4f74-b761-b5309510e276-kube-api-access-22t58\") pod \"thanos-querier-5989fc4c8b-gdxvq\" (UID: \"8e8869c6-d815-4f74-b761-b5309510e276\") " pod="openshift-monitoring/thanos-querier-5989fc4c8b-gdxvq"
Jul 30 17:02:11.746139 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:11.746100    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5989fc4c8b-gdxvq"
Jul 30 17:02:11.897336 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:11.897301    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5989fc4c8b-gdxvq"]
Jul 30 17:02:11.900427 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:02:11.900390    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e8869c6_d815_4f74_b761_b5309510e276.slice/crio-ea5762336582019f8b82165934224358226fceea9e47c79a7324ab227dbccd81 WatchSource:0}: Error finding container ea5762336582019f8b82165934224358226fceea9e47c79a7324ab227dbccd81: Status 404 returned error can't find the container with id ea5762336582019f8b82165934224358226fceea9e47c79a7324ab227dbccd81
Jul 30 17:02:11.940842 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:11.940744    2580 generic.go:356] "Generic (PLEG): container finished" podID="9b737c7c-d8a8-4567-86cd-bbbbe37b1a36" containerID="9046c92a18fffa79dfc18b0ed0216781b2f2f091d796d9d0edc4006615599409" exitCode=0
Jul 30 17:02:11.941304 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:11.940830    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l7cvg" event={"ID":"9b737c7c-d8a8-4567-86cd-bbbbe37b1a36","Type":"ContainerDied","Data":"9046c92a18fffa79dfc18b0ed0216781b2f2f091d796d9d0edc4006615599409"}
Jul 30 17:02:11.942105 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:11.942083    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5989fc4c8b-gdxvq" event={"ID":"8e8869c6-d815-4f74-b761-b5309510e276","Type":"ContainerStarted","Data":"ea5762336582019f8b82165934224358226fceea9e47c79a7324ab227dbccd81"}
Jul 30 17:02:12.829397 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:12.829355    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-545cd44b95-ndcrt"]
Jul 30 17:02:12.831971 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:12.831944    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-545cd44b95-ndcrt"
Jul 30 17:02:12.834466 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:12.834437    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-4mgk8\""
Jul 30 17:02:12.834681 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:12.834437    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-770vv1nqltmag\""
Jul 30 17:02:12.834681 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:12.834489    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 30 17:02:12.835469 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:12.835450    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 30 17:02:12.835578 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:12.835450    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 30 17:02:12.835578 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:12.835491    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 30 17:02:12.842866 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:12.842836    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-545cd44b95-ndcrt"]
Jul 30 17:02:12.948721 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:12.948684    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l7cvg" event={"ID":"9b737c7c-d8a8-4567-86cd-bbbbe37b1a36","Type":"ContainerStarted","Data":"e5e43bd5258986fbefdd2bb9eac0be5151a6179d863e7074805ff8eccb41e98c"}
Jul 30 17:02:12.949289 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:12.948763    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l7cvg" event={"ID":"9b737c7c-d8a8-4567-86cd-bbbbe37b1a36","Type":"ContainerStarted","Data":"c7895963416f1711a4a1f718812719cf2a48fe0c58d979d482d08287907d916f"}
Jul 30 17:02:12.970866 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:12.970787    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-l7cvg" podStartSLOduration=3.688737524 podStartE2EDuration="4.970763775s" totalImagesPullingTime="1.211522705s" totalInitContainerRuntime="70.503546ms" isStatefulPod=true podCreationTimestamp="2026-07-30 17:02:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:02:12.96830158 +0000 UTC m=+96.043601470" watchObservedRunningTime="2026-07-30 17:02:12.970763775 +0000 UTC m=+96.046063667"
Jul 30 17:02:12.998202 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:12.998142    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5f6b791b-3b27-4b3a-8cc0-f2cf0f3163c5-secret-metrics-server-tls\") pod \"metrics-server-545cd44b95-ndcrt\" (UID: \"5f6b791b-3b27-4b3a-8cc0-f2cf0f3163c5\") " pod="openshift-monitoring/metrics-server-545cd44b95-ndcrt"
Jul 30 17:02:12.998389 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:12.998222    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5f6b791b-3b27-4b3a-8cc0-f2cf0f3163c5-audit-log\") pod \"metrics-server-545cd44b95-ndcrt\" (UID: \"5f6b791b-3b27-4b3a-8cc0-f2cf0f3163c5\") " pod="openshift-monitoring/metrics-server-545cd44b95-ndcrt"
Jul 30 17:02:12.998389 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:12.998256    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5f6b791b-3b27-4b3a-8cc0-f2cf0f3163c5-metrics-server-audit-profiles\") pod \"metrics-server-545cd44b95-ndcrt\" (UID: \"5f6b791b-3b27-4b3a-8cc0-f2cf0f3163c5\") " pod="openshift-monitoring/metrics-server-545cd44b95-ndcrt"
Jul 30 17:02:12.998389 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:12.998286    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5f6b791b-3b27-4b3a-8cc0-f2cf0f3163c5-secret-metrics-server-client-certs\") pod \"metrics-server-545cd44b95-ndcrt\" (UID: \"5f6b791b-3b27-4b3a-8cc0-f2cf0f3163c5\") " pod="openshift-monitoring/metrics-server-545cd44b95-ndcrt"
Jul 30 17:02:12.998389 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:12.998322    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25gk6\" (UniqueName: \"kubernetes.io/projected/5f6b791b-3b27-4b3a-8cc0-f2cf0f3163c5-kube-api-access-25gk6\") pod \"metrics-server-545cd44b95-ndcrt\" (UID: \"5f6b791b-3b27-4b3a-8cc0-f2cf0f3163c5\") " pod="openshift-monitoring/metrics-server-545cd44b95-ndcrt"
Jul 30 17:02:12.998389 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:12.998348    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5f6b791b-3b27-4b3a-8cc0-f2cf0f3163c5-client-ca-bundle\") pod \"metrics-server-545cd44b95-ndcrt\" (UID: \"5f6b791b-3b27-4b3a-8cc0-f2cf0f3163c5\") " pod="openshift-monitoring/metrics-server-545cd44b95-ndcrt"
Jul 30 17:02:12.998389 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:12.998380    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f6b791b-3b27-4b3a-8cc0-f2cf0f3163c5-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-545cd44b95-ndcrt\" (UID: \"5f6b791b-3b27-4b3a-8cc0-f2cf0f3163c5\") " pod="openshift-monitoring/metrics-server-545cd44b95-ndcrt"
Jul 30 17:02:13.099636 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:13.099526    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5f6b791b-3b27-4b3a-8cc0-f2cf0f3163c5-secret-metrics-server-tls\") pod \"metrics-server-545cd44b95-ndcrt\" (UID: \"5f6b791b-3b27-4b3a-8cc0-f2cf0f3163c5\") " pod="openshift-monitoring/metrics-server-545cd44b95-ndcrt"
Jul 30 17:02:13.099636 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:13.099627    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5f6b791b-3b27-4b3a-8cc0-f2cf0f3163c5-audit-log\") pod \"metrics-server-545cd44b95-ndcrt\" (UID: \"5f6b791b-3b27-4b3a-8cc0-f2cf0f3163c5\") " pod="openshift-monitoring/metrics-server-545cd44b95-ndcrt"
Jul 30 17:02:13.099885 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:13.099734    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5f6b791b-3b27-4b3a-8cc0-f2cf0f3163c5-metrics-server-audit-profiles\") pod \"metrics-server-545cd44b95-ndcrt\" (UID: \"5f6b791b-3b27-4b3a-8cc0-f2cf0f3163c5\") " pod="openshift-monitoring/metrics-server-545cd44b95-ndcrt"
Jul 30 17:02:13.099885 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:13.099792    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5f6b791b-3b27-4b3a-8cc0-f2cf0f3163c5-secret-metrics-server-client-certs\") pod \"metrics-server-545cd44b95-ndcrt\" (UID: \"5f6b791b-3b27-4b3a-8cc0-f2cf0f3163c5\") " pod="openshift-monitoring/metrics-server-545cd44b95-ndcrt"
Jul 30 17:02:13.099885 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:13.099842    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25gk6\" (UniqueName: \"kubernetes.io/projected/5f6b791b-3b27-4b3a-8cc0-f2cf0f3163c5-kube-api-access-25gk6\") pod \"metrics-server-545cd44b95-ndcrt\" (UID: \"5f6b791b-3b27-4b3a-8cc0-f2cf0f3163c5\") " pod="openshift-monitoring/metrics-server-545cd44b95-ndcrt"
Jul 30 17:02:13.100038 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:13.099904    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5f6b791b-3b27-4b3a-8cc0-f2cf0f3163c5-client-ca-bundle\") pod \"metrics-server-545cd44b95-ndcrt\" (UID: \"5f6b791b-3b27-4b3a-8cc0-f2cf0f3163c5\") " pod="openshift-monitoring/metrics-server-545cd44b95-ndcrt"
Jul 30 17:02:13.100038 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:13.099956    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f6b791b-3b27-4b3a-8cc0-f2cf0f3163c5-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-545cd44b95-ndcrt\" (UID: \"5f6b791b-3b27-4b3a-8cc0-f2cf0f3163c5\") " pod="openshift-monitoring/metrics-server-545cd44b95-ndcrt"
Jul 30 17:02:13.101140 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:13.100979    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5f6b791b-3b27-4b3a-8cc0-f2cf0f3163c5-audit-log\") pod \"metrics-server-545cd44b95-ndcrt\" (UID: \"5f6b791b-3b27-4b3a-8cc0-f2cf0f3163c5\") " pod="openshift-monitoring/metrics-server-545cd44b95-ndcrt"
Jul 30 17:02:13.101140 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:13.101121    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f6b791b-3b27-4b3a-8cc0-f2cf0f3163c5-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-545cd44b95-ndcrt\" (UID: \"5f6b791b-3b27-4b3a-8cc0-f2cf0f3163c5\") " pod="openshift-monitoring/metrics-server-545cd44b95-ndcrt"
Jul 30 17:02:13.101952 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:13.101919    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5f6b791b-3b27-4b3a-8cc0-f2cf0f3163c5-metrics-server-audit-profiles\") pod \"metrics-server-545cd44b95-ndcrt\" (UID: \"5f6b791b-3b27-4b3a-8cc0-f2cf0f3163c5\") " pod="openshift-monitoring/metrics-server-545cd44b95-ndcrt"
Jul 30 17:02:13.105316 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:13.102686    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5f6b791b-3b27-4b3a-8cc0-f2cf0f3163c5-secret-metrics-server-tls\") pod \"metrics-server-545cd44b95-ndcrt\" (UID: \"5f6b791b-3b27-4b3a-8cc0-f2cf0f3163c5\") " pod="openshift-monitoring/metrics-server-545cd44b95-ndcrt"
Jul 30 17:02:13.106424 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:13.106358    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5f6b791b-3b27-4b3a-8cc0-f2cf0f3163c5-secret-metrics-server-client-certs\") pod \"metrics-server-545cd44b95-ndcrt\" (UID: \"5f6b791b-3b27-4b3a-8cc0-f2cf0f3163c5\") " pod="openshift-monitoring/metrics-server-545cd44b95-ndcrt"
Jul 30 17:02:13.108651 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:13.108627    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5f6b791b-3b27-4b3a-8cc0-f2cf0f3163c5-client-ca-bundle\") pod \"metrics-server-545cd44b95-ndcrt\" (UID: \"5f6b791b-3b27-4b3a-8cc0-f2cf0f3163c5\") " pod="openshift-monitoring/metrics-server-545cd44b95-ndcrt"
Jul 30 17:02:13.115298 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:13.115262    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7d6db5dd88-zbch2"]
Jul 30 17:02:13.117901 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:13.117865    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25gk6\" (UniqueName: \"kubernetes.io/projected/5f6b791b-3b27-4b3a-8cc0-f2cf0f3163c5-kube-api-access-25gk6\") pod \"metrics-server-545cd44b95-ndcrt\" (UID: \"5f6b791b-3b27-4b3a-8cc0-f2cf0f3163c5\") " pod="openshift-monitoring/metrics-server-545cd44b95-ndcrt"
Jul 30 17:02:13.118901 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:13.118873    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d6db5dd88-zbch2"
Jul 30 17:02:13.121615 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:13.121584    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 30 17:02:13.121937 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:13.121902    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 30 17:02:13.121937 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:13.121909    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 30 17:02:13.122090 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:13.121946    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 30 17:02:13.122090 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:13.121974    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 30 17:02:13.122395 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:13.122368    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-wrh2v\""
Jul 30 17:02:13.122583 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:13.122455    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 30 17:02:13.122713 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:13.122696    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 30 17:02:13.128547 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:13.128515    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 30 17:02:13.131764 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:13.131728    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d6db5dd88-zbch2"]
Jul 30 17:02:13.143356 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:13.143321    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-545cd44b95-ndcrt"
Jul 30 17:02:13.301668 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:13.301624    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b340605c-3681-4020-81e0-745042dd3dd7-console-oauth-config\") pod \"console-7d6db5dd88-zbch2\" (UID: \"b340605c-3681-4020-81e0-745042dd3dd7\") " pod="openshift-console/console-7d6db5dd88-zbch2"
Jul 30 17:02:13.301891 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:13.301687    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5rrs\" (UniqueName: \"kubernetes.io/projected/b340605c-3681-4020-81e0-745042dd3dd7-kube-api-access-g5rrs\") pod \"console-7d6db5dd88-zbch2\" (UID: \"b340605c-3681-4020-81e0-745042dd3dd7\") " pod="openshift-console/console-7d6db5dd88-zbch2"
Jul 30 17:02:13.301891 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:13.301736    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b340605c-3681-4020-81e0-745042dd3dd7-console-config\") pod \"console-7d6db5dd88-zbch2\" (UID: \"b340605c-3681-4020-81e0-745042dd3dd7\") " pod="openshift-console/console-7d6db5dd88-zbch2"
Jul 30 17:02:13.301891 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:13.301812    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b340605c-3681-4020-81e0-745042dd3dd7-trusted-ca-bundle\") pod \"console-7d6db5dd88-zbch2\" (UID: \"b340605c-3681-4020-81e0-745042dd3dd7\") " pod="openshift-console/console-7d6db5dd88-zbch2"
Jul 30 17:02:13.301891 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:13.301839    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b340605c-3681-4020-81e0-745042dd3dd7-service-ca\") pod \"console-7d6db5dd88-zbch2\" (UID: \"b340605c-3681-4020-81e0-745042dd3dd7\") " pod="openshift-console/console-7d6db5dd88-zbch2"
Jul 30 17:02:13.301891 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:13.301864    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b340605c-3681-4020-81e0-745042dd3dd7-oauth-serving-cert\") pod \"console-7d6db5dd88-zbch2\" (UID: \"b340605c-3681-4020-81e0-745042dd3dd7\") " pod="openshift-console/console-7d6db5dd88-zbch2"
Jul 30 17:02:13.301891 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:13.301890    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b340605c-3681-4020-81e0-745042dd3dd7-console-serving-cert\") pod \"console-7d6db5dd88-zbch2\" (UID: \"b340605c-3681-4020-81e0-745042dd3dd7\") " pod="openshift-console/console-7d6db5dd88-zbch2"
Jul 30 17:02:13.311616 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:13.311577    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-545cd44b95-ndcrt"]
Jul 30 17:02:13.317591 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:02:13.317557    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f6b791b_3b27_4b3a_8cc0_f2cf0f3163c5.slice/crio-7ad8e8d6521d9bf432304c888fb613f9bc8c3a05480e919cc1892ef1f41b4456 WatchSource:0}: Error finding container 7ad8e8d6521d9bf432304c888fb613f9bc8c3a05480e919cc1892ef1f41b4456: Status 404 returned error can't find the container with id 7ad8e8d6521d9bf432304c888fb613f9bc8c3a05480e919cc1892ef1f41b4456
Jul 30 17:02:13.402718 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:13.402611    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b340605c-3681-4020-81e0-745042dd3dd7-console-oauth-config\") pod \"console-7d6db5dd88-zbch2\" (UID: \"b340605c-3681-4020-81e0-745042dd3dd7\") " pod="openshift-console/console-7d6db5dd88-zbch2"
Jul 30 17:02:13.402718 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:13.402667    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5rrs\" (UniqueName: \"kubernetes.io/projected/b340605c-3681-4020-81e0-745042dd3dd7-kube-api-access-g5rrs\") pod \"console-7d6db5dd88-zbch2\" (UID: \"b340605c-3681-4020-81e0-745042dd3dd7\") " pod="openshift-console/console-7d6db5dd88-zbch2"
Jul 30 17:02:13.402718 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:13.402709    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b340605c-3681-4020-81e0-745042dd3dd7-console-config\") pod \"console-7d6db5dd88-zbch2\" (UID: \"b340605c-3681-4020-81e0-745042dd3dd7\") " pod="openshift-console/console-7d6db5dd88-zbch2"
Jul 30 17:02:13.403030 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:13.402753    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b340605c-3681-4020-81e0-745042dd3dd7-trusted-ca-bundle\") pod \"console-7d6db5dd88-zbch2\" (UID: \"b340605c-3681-4020-81e0-745042dd3dd7\") " pod="openshift-console/console-7d6db5dd88-zbch2"
Jul 30 17:02:13.403030 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:13.402778    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b340605c-3681-4020-81e0-745042dd3dd7-service-ca\") pod \"console-7d6db5dd88-zbch2\" (UID: \"b340605c-3681-4020-81e0-745042dd3dd7\") " pod="openshift-console/console-7d6db5dd88-zbch2"
Jul 30 17:02:13.403030 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:13.402798    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b340605c-3681-4020-81e0-745042dd3dd7-oauth-serving-cert\") pod \"console-7d6db5dd88-zbch2\" (UID: \"b340605c-3681-4020-81e0-745042dd3dd7\") " pod="openshift-console/console-7d6db5dd88-zbch2"
Jul 30 17:02:13.403030 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:13.402933    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b340605c-3681-4020-81e0-745042dd3dd7-console-serving-cert\") pod \"console-7d6db5dd88-zbch2\" (UID: \"b340605c-3681-4020-81e0-745042dd3dd7\") " pod="openshift-console/console-7d6db5dd88-zbch2"
Jul 30 17:02:13.403739 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:13.403710    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b340605c-3681-4020-81e0-745042dd3dd7-oauth-serving-cert\") pod \"console-7d6db5dd88-zbch2\" (UID: \"b340605c-3681-4020-81e0-745042dd3dd7\") " pod="openshift-console/console-7d6db5dd88-zbch2"
Jul 30 17:02:13.403854 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:13.403755    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b340605c-3681-4020-81e0-745042dd3dd7-console-config\") pod \"console-7d6db5dd88-zbch2\" (UID: \"b340605c-3681-4020-81e0-745042dd3dd7\") " pod="openshift-console/console-7d6db5dd88-zbch2"
Jul 30 17:02:13.404087 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:13.404039    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b340605c-3681-4020-81e0-745042dd3dd7-trusted-ca-bundle\") pod \"console-7d6db5dd88-zbch2\" (UID: \"b340605c-3681-4020-81e0-745042dd3dd7\") " pod="openshift-console/console-7d6db5dd88-zbch2"
Jul 30 17:02:13.404483 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:13.404456    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b340605c-3681-4020-81e0-745042dd3dd7-service-ca\") pod \"console-7d6db5dd88-zbch2\" (UID: \"b340605c-3681-4020-81e0-745042dd3dd7\") " pod="openshift-console/console-7d6db5dd88-zbch2"
Jul 30 17:02:13.405828 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:13.405804    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b340605c-3681-4020-81e0-745042dd3dd7-console-oauth-config\") pod \"console-7d6db5dd88-zbch2\" (UID: \"b340605c-3681-4020-81e0-745042dd3dd7\") " pod="openshift-console/console-7d6db5dd88-zbch2"
Jul 30 17:02:13.405925 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:13.405804    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b340605c-3681-4020-81e0-745042dd3dd7-console-serving-cert\") pod \"console-7d6db5dd88-zbch2\" (UID: \"b340605c-3681-4020-81e0-745042dd3dd7\") " pod="openshift-console/console-7d6db5dd88-zbch2"
Jul 30 17:02:13.416805 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:13.416758    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5rrs\" (UniqueName: \"kubernetes.io/projected/b340605c-3681-4020-81e0-745042dd3dd7-kube-api-access-g5rrs\") pod \"console-7d6db5dd88-zbch2\" (UID: \"b340605c-3681-4020-81e0-745042dd3dd7\") " pod="openshift-console/console-7d6db5dd88-zbch2"
Jul 30 17:02:13.437416 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:13.437372    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d6db5dd88-zbch2"
Jul 30 17:02:13.604878 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:13.604760    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d6db5dd88-zbch2"]
Jul 30 17:02:13.608582 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:02:13.608540    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb340605c_3681_4020_81e0_745042dd3dd7.slice/crio-5a79ae3768a1f150e9c0b0d00e4c7bd3196fa661dcec663a6df10fd48df9a53a WatchSource:0}: Error finding container 5a79ae3768a1f150e9c0b0d00e4c7bd3196fa661dcec663a6df10fd48df9a53a: Status 404 returned error can't find the container with id 5a79ae3768a1f150e9c0b0d00e4c7bd3196fa661dcec663a6df10fd48df9a53a
Jul 30 17:02:13.959999 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:13.959950    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-545cd44b95-ndcrt" event={"ID":"5f6b791b-3b27-4b3a-8cc0-f2cf0f3163c5","Type":"ContainerStarted","Data":"7ad8e8d6521d9bf432304c888fb613f9bc8c3a05480e919cc1892ef1f41b4456"}
Jul 30 17:02:13.963085 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:13.962979    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d6db5dd88-zbch2" event={"ID":"b340605c-3681-4020-81e0-745042dd3dd7","Type":"ContainerStarted","Data":"5a79ae3768a1f150e9c0b0d00e4c7bd3196fa661dcec663a6df10fd48df9a53a"}
Jul 30 17:02:15.981637 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:15.981519    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5989fc4c8b-gdxvq" event={"ID":"8e8869c6-d815-4f74-b761-b5309510e276","Type":"ContainerStarted","Data":"fa2f16c903d7c848f4d6a5e47a8cdb66fa02c7407aa416aa124946a8992de847"}
Jul 30 17:02:15.982117 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:15.981656    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5989fc4c8b-gdxvq" event={"ID":"8e8869c6-d815-4f74-b761-b5309510e276","Type":"ContainerStarted","Data":"6baedb76d7d86e6e3f3eaaade384c6aa0df69faf31bc7e104af0b889802b2f14"}
Jul 30 17:02:15.983477 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:15.983439    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-545cd44b95-ndcrt" event={"ID":"5f6b791b-3b27-4b3a-8cc0-f2cf0f3163c5","Type":"ContainerStarted","Data":"b2bf70467939ad34e4d9ad9d803d09a51fbf7cf9fceff1efd2fbf87b0e8998a0"}
Jul 30 17:02:16.002915 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:16.002858    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-545cd44b95-ndcrt" podStartSLOduration=1.779870758 podStartE2EDuration="4.002839762s" totalImagesPullingTime="2.222969004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:02:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:02:16.00199415 +0000 UTC m=+99.077294041" watchObservedRunningTime="2026-07-30 17:02:16.002839762 +0000 UTC m=+99.078139652"
Jul 30 17:02:17.993911 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:17.993865    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5989fc4c8b-gdxvq" event={"ID":"8e8869c6-d815-4f74-b761-b5309510e276","Type":"ContainerStarted","Data":"22bb41ced64d070758cb4526273c39235e41b459fc1f19f86e085e44d1c7e76a"}
Jul 30 17:02:17.995310 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:17.995280    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d6db5dd88-zbch2" event={"ID":"b340605c-3681-4020-81e0-745042dd3dd7","Type":"ContainerStarted","Data":"4dd7396615a74b850414ee11a2be605cd1885b5ed2d515c055ea5e4ea3807963"}
Jul 30 17:02:18.015037 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:18.014975    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7d6db5dd88-zbch2" podStartSLOduration=1.4279262209999999 podStartE2EDuration="5.014956847s" totalImagesPullingTime="3.587030626s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:02:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:02:18.013478163 +0000 UTC m=+101.088778055" watchObservedRunningTime="2026-07-30 17:02:18.014956847 +0000 UTC m=+101.090256736"
Jul 30 17:02:18.601564 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:18.601119    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d6db5dd88-zbch2"]
Jul 30 17:02:19.005013 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:19.004828    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5989fc4c8b-gdxvq" event={"ID":"8e8869c6-d815-4f74-b761-b5309510e276","Type":"ContainerStarted","Data":"f4e7f5e7da018b0a17f5fc47b1d22516fb0c05833ef64e4e4d2a3370f4c8067a"}
Jul 30 17:02:19.009877 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:19.009797    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5989fc4c8b-gdxvq" event={"ID":"8e8869c6-d815-4f74-b761-b5309510e276","Type":"ContainerStarted","Data":"f353d3808c58b8ebc9c92dc7d424ca647b43bce241500806ae3cc3391909eb36"}
Jul 30 17:02:19.010161 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:19.010135    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5989fc4c8b-gdxvq" event={"ID":"8e8869c6-d815-4f74-b761-b5309510e276","Type":"ContainerStarted","Data":"4aac3d3b077029314e67a154dcbc5a3eed0d00187d8f5d3a511d8d16d11afbdc"}
Jul 30 17:02:19.038068 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:19.037999    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5989fc4c8b-gdxvq" podStartSLOduration=3.135686701 podStartE2EDuration="8.037978328s" totalImagesPullingTime="4.902291627s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:02:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:02:19.036762674 +0000 UTC m=+102.112062568" watchObservedRunningTime="2026-07-30 17:02:19.037978328 +0000 UTC m=+102.113278218"
Jul 30 17:02:20.012425 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:20.012382    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5989fc4c8b-gdxvq"
Jul 30 17:02:20.018195 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:20.018151    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5989fc4c8b-gdxvq"
Jul 30 17:02:23.438452 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:23.438404    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7d6db5dd88-zbch2"
Jul 30 17:02:26.862657 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:26.862622    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ltj2t"
Jul 30 17:02:33.144225 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:33.144152    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-545cd44b95-ndcrt"
Jul 30 17:02:33.144676 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:33.144255    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-545cd44b95-ndcrt"
Jul 30 17:02:45.033102 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:45.033030    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7d6db5dd88-zbch2" podUID="b340605c-3681-4020-81e0-745042dd3dd7" containerName="console" containerID="cri-o://4dd7396615a74b850414ee11a2be605cd1885b5ed2d515c055ea5e4ea3807963" gracePeriod=15
Jul 30 17:02:45.290828 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:45.290756    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d6db5dd88-zbch2"
Jul 30 17:02:45.399092 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:45.399050    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b340605c-3681-4020-81e0-745042dd3dd7-console-config\") pod \"b340605c-3681-4020-81e0-745042dd3dd7\" (UID: \"b340605c-3681-4020-81e0-745042dd3dd7\") "
Jul 30 17:02:45.399342 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:45.399115    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b340605c-3681-4020-81e0-745042dd3dd7-trusted-ca-bundle\") pod \"b340605c-3681-4020-81e0-745042dd3dd7\" (UID: \"b340605c-3681-4020-81e0-745042dd3dd7\") "
Jul 30 17:02:45.399342 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:45.399165    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g5rrs\" (UniqueName: \"kubernetes.io/projected/b340605c-3681-4020-81e0-745042dd3dd7-kube-api-access-g5rrs\") pod \"b340605c-3681-4020-81e0-745042dd3dd7\" (UID: \"b340605c-3681-4020-81e0-745042dd3dd7\") "
Jul 30 17:02:45.399342 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:45.399222    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b340605c-3681-4020-81e0-745042dd3dd7-console-serving-cert\") pod \"b340605c-3681-4020-81e0-745042dd3dd7\" (UID: \"b340605c-3681-4020-81e0-745042dd3dd7\") "
Jul 30 17:02:45.399342 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:45.399294    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b340605c-3681-4020-81e0-745042dd3dd7-console-oauth-config\") pod \"b340605c-3681-4020-81e0-745042dd3dd7\" (UID: \"b340605c-3681-4020-81e0-745042dd3dd7\") "
Jul 30 17:02:45.399342 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:45.399317    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b340605c-3681-4020-81e0-745042dd3dd7-service-ca\") pod \"b340605c-3681-4020-81e0-745042dd3dd7\" (UID: \"b340605c-3681-4020-81e0-745042dd3dd7\") "
Jul 30 17:02:45.399584 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:45.399349    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b340605c-3681-4020-81e0-745042dd3dd7-oauth-serving-cert\") pod \"b340605c-3681-4020-81e0-745042dd3dd7\" (UID: \"b340605c-3681-4020-81e0-745042dd3dd7\") "
Jul 30 17:02:45.399667 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:45.399634    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b340605c-3681-4020-81e0-745042dd3dd7-console-config" (OuterVolumeSpecName: "console-config") pod "b340605c-3681-4020-81e0-745042dd3dd7" (UID: "b340605c-3681-4020-81e0-745042dd3dd7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:02:45.399833 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:45.399803    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b340605c-3681-4020-81e0-745042dd3dd7-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b340605c-3681-4020-81e0-745042dd3dd7" (UID: "b340605c-3681-4020-81e0-745042dd3dd7"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:02:45.399921 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:45.399829    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b340605c-3681-4020-81e0-745042dd3dd7-service-ca" (OuterVolumeSpecName: "service-ca") pod "b340605c-3681-4020-81e0-745042dd3dd7" (UID: "b340605c-3681-4020-81e0-745042dd3dd7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:02:45.399921 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:45.399862    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b340605c-3681-4020-81e0-745042dd3dd7-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b340605c-3681-4020-81e0-745042dd3dd7" (UID: "b340605c-3681-4020-81e0-745042dd3dd7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:02:45.401762 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:45.401732    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b340605c-3681-4020-81e0-745042dd3dd7-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b340605c-3681-4020-81e0-745042dd3dd7" (UID: "b340605c-3681-4020-81e0-745042dd3dd7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:02:45.402024 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:45.402000    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b340605c-3681-4020-81e0-745042dd3dd7-kube-api-access-g5rrs" (OuterVolumeSpecName: "kube-api-access-g5rrs") pod "b340605c-3681-4020-81e0-745042dd3dd7" (UID: "b340605c-3681-4020-81e0-745042dd3dd7"). InnerVolumeSpecName "kube-api-access-g5rrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:02:45.402099 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:45.402001    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b340605c-3681-4020-81e0-745042dd3dd7-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b340605c-3681-4020-81e0-745042dd3dd7" (UID: "b340605c-3681-4020-81e0-745042dd3dd7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:02:45.500476 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:45.500436    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b340605c-3681-4020-81e0-745042dd3dd7-console-oauth-config\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:02:45.500476 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:45.500485    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b340605c-3681-4020-81e0-745042dd3dd7-service-ca\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:02:45.500679 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:45.500497    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b340605c-3681-4020-81e0-745042dd3dd7-oauth-serving-cert\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:02:45.500679 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:45.500506    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b340605c-3681-4020-81e0-745042dd3dd7-console-config\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:02:45.500679 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:45.500515    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b340605c-3681-4020-81e0-745042dd3dd7-trusted-ca-bundle\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:02:45.500679 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:45.500524    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5rrs\" (UniqueName: \"kubernetes.io/projected/b340605c-3681-4020-81e0-745042dd3dd7-kube-api-access-g5rrs\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:02:45.500679 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:45.500533    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b340605c-3681-4020-81e0-745042dd3dd7-console-serving-cert\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:02:46.114835 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:46.114798    2580 generic.go:356] "Generic (PLEG): container finished" podID="b340605c-3681-4020-81e0-745042dd3dd7" containerID="4dd7396615a74b850414ee11a2be605cd1885b5ed2d515c055ea5e4ea3807963" exitCode=0
Jul 30 17:02:46.115315 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:46.114854    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d6db5dd88-zbch2"
Jul 30 17:02:46.115315 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:46.114895    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d6db5dd88-zbch2" event={"ID":"b340605c-3681-4020-81e0-745042dd3dd7","Type":"ContainerDied","Data":"4dd7396615a74b850414ee11a2be605cd1885b5ed2d515c055ea5e4ea3807963"}
Jul 30 17:02:46.115315 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:46.114955    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d6db5dd88-zbch2" event={"ID":"b340605c-3681-4020-81e0-745042dd3dd7","Type":"ContainerDied","Data":"5a79ae3768a1f150e9c0b0d00e4c7bd3196fa661dcec663a6df10fd48df9a53a"}
Jul 30 17:02:46.115315 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:46.114980    2580 scope.go:117] "RemoveContainer" containerID="4dd7396615a74b850414ee11a2be605cd1885b5ed2d515c055ea5e4ea3807963"
Jul 30 17:02:46.123478 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:46.123455    2580 scope.go:117] "RemoveContainer" containerID="4dd7396615a74b850414ee11a2be605cd1885b5ed2d515c055ea5e4ea3807963"
Jul 30 17:02:46.123787 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:02:46.123766    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dd7396615a74b850414ee11a2be605cd1885b5ed2d515c055ea5e4ea3807963\": container with ID starting with 4dd7396615a74b850414ee11a2be605cd1885b5ed2d515c055ea5e4ea3807963 not found: ID does not exist" containerID="4dd7396615a74b850414ee11a2be605cd1885b5ed2d515c055ea5e4ea3807963"
Jul 30 17:02:46.123841 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:46.123795    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dd7396615a74b850414ee11a2be605cd1885b5ed2d515c055ea5e4ea3807963"} err="failed to get container status \"4dd7396615a74b850414ee11a2be605cd1885b5ed2d515c055ea5e4ea3807963\": rpc error: code = NotFound desc = could not find container \"4dd7396615a74b850414ee11a2be605cd1885b5ed2d515c055ea5e4ea3807963\": container with ID starting with 4dd7396615a74b850414ee11a2be605cd1885b5ed2d515c055ea5e4ea3807963 not found: ID does not exist"
Jul 30 17:02:46.134451 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:46.134312    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d6db5dd88-zbch2"]
Jul 30 17:02:46.136015 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:46.135984    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7d6db5dd88-zbch2"]
Jul 30 17:02:47.482542 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:47.482500    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b340605c-3681-4020-81e0-745042dd3dd7" path="/var/lib/kubelet/pods/b340605c-3681-4020-81e0-745042dd3dd7/volumes"
Jul 30 17:02:53.149203 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:53.149147    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-545cd44b95-ndcrt"
Jul 30 17:02:53.152928 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:02:53.152903    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-545cd44b95-ndcrt"
Jul 30 17:03:31.072324 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:03:31.072243    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6df97869cf-qbhcv"]
Jul 30 17:03:31.073400 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:03:31.073368    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b340605c-3681-4020-81e0-745042dd3dd7" containerName="console"
Jul 30 17:03:31.073400 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:03:31.073399    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b340605c-3681-4020-81e0-745042dd3dd7" containerName="console"
Jul 30 17:03:31.073587 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:03:31.073484    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b340605c-3681-4020-81e0-745042dd3dd7" containerName="console"
Jul 30 17:03:31.075700 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:03:31.075672    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6df97869cf-qbhcv"
Jul 30 17:03:31.081701 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:03:31.081679    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 30 17:03:31.081839 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:03:31.081819    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 30 17:03:31.081911 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:03:31.081819    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 30 17:03:31.081972 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:03:31.081910    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 30 17:03:31.082036 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:03:31.082022    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-wrh2v\""
Jul 30 17:03:31.082130 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:03:31.082105    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 30 17:03:31.082214 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:03:31.082156    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 30 17:03:31.082464 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:03:31.082396    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 30 17:03:31.088217 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:03:31.088198    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 30 17:03:31.090874 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:03:31.090852    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6df97869cf-qbhcv"]
Jul 30 17:03:31.147924 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:03:31.147899    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bfe08d08-0850-43c8-954e-98028612959e-console-oauth-config\") pod \"console-6df97869cf-qbhcv\" (UID: \"bfe08d08-0850-43c8-954e-98028612959e\") " pod="openshift-console/console-6df97869cf-qbhcv"
Jul 30 17:03:31.148023 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:03:31.147928    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mjkd\" (UniqueName: \"kubernetes.io/projected/bfe08d08-0850-43c8-954e-98028612959e-kube-api-access-5mjkd\") pod \"console-6df97869cf-qbhcv\" (UID: \"bfe08d08-0850-43c8-954e-98028612959e\") " pod="openshift-console/console-6df97869cf-qbhcv"
Jul 30 17:03:31.148023 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:03:31.147960    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bfe08d08-0850-43c8-954e-98028612959e-oauth-serving-cert\") pod \"console-6df97869cf-qbhcv\" (UID: \"bfe08d08-0850-43c8-954e-98028612959e\") " pod="openshift-console/console-6df97869cf-qbhcv"
Jul 30 17:03:31.148141 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:03:31.148022    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bfe08d08-0850-43c8-954e-98028612959e-console-config\") pod \"console-6df97869cf-qbhcv\" (UID: \"bfe08d08-0850-43c8-954e-98028612959e\") " pod="openshift-console/console-6df97869cf-qbhcv"
Jul 30 17:03:31.148141 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:03:31.148082    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bfe08d08-0850-43c8-954e-98028612959e-service-ca\") pod \"console-6df97869cf-qbhcv\" (UID: \"bfe08d08-0850-43c8-954e-98028612959e\") " pod="openshift-console/console-6df97869cf-qbhcv"
Jul 30 17:03:31.148234 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:03:31.148183    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfe08d08-0850-43c8-954e-98028612959e-trusted-ca-bundle\") pod \"console-6df97869cf-qbhcv\" (UID: \"bfe08d08-0850-43c8-954e-98028612959e\") " pod="openshift-console/console-6df97869cf-qbhcv"
Jul 30 17:03:31.148234 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:03:31.148215    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bfe08d08-0850-43c8-954e-98028612959e-console-serving-cert\") pod \"console-6df97869cf-qbhcv\" (UID: \"bfe08d08-0850-43c8-954e-98028612959e\") " pod="openshift-console/console-6df97869cf-qbhcv"
Jul 30 17:03:31.248934 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:03:31.248903    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bfe08d08-0850-43c8-954e-98028612959e-console-serving-cert\") pod \"console-6df97869cf-qbhcv\" (UID: \"bfe08d08-0850-43c8-954e-98028612959e\") " pod="openshift-console/console-6df97869cf-qbhcv"
Jul 30 17:03:31.249043 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:03:31.248957    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bfe08d08-0850-43c8-954e-98028612959e-console-oauth-config\") pod \"console-6df97869cf-qbhcv\" (UID: \"bfe08d08-0850-43c8-954e-98028612959e\") " pod="openshift-console/console-6df97869cf-qbhcv"
Jul 30 17:03:31.249043 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:03:31.248983    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mjkd\" (UniqueName: \"kubernetes.io/projected/bfe08d08-0850-43c8-954e-98028612959e-kube-api-access-5mjkd\") pod \"console-6df97869cf-qbhcv\" (UID: \"bfe08d08-0850-43c8-954e-98028612959e\") " pod="openshift-console/console-6df97869cf-qbhcv"
Jul 30 17:03:31.249043 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:03:31.249030    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bfe08d08-0850-43c8-954e-98028612959e-oauth-serving-cert\") pod \"console-6df97869cf-qbhcv\" (UID: \"bfe08d08-0850-43c8-954e-98028612959e\") " pod="openshift-console/console-6df97869cf-qbhcv"
Jul 30 17:03:31.249210 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:03:31.249155    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bfe08d08-0850-43c8-954e-98028612959e-console-config\") pod \"console-6df97869cf-qbhcv\" (UID: \"bfe08d08-0850-43c8-954e-98028612959e\") " pod="openshift-console/console-6df97869cf-qbhcv"
Jul 30 17:03:31.249210 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:03:31.249206    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bfe08d08-0850-43c8-954e-98028612959e-service-ca\") pod \"console-6df97869cf-qbhcv\" (UID: \"bfe08d08-0850-43c8-954e-98028612959e\") " pod="openshift-console/console-6df97869cf-qbhcv"
Jul 30 17:03:31.249329 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:03:31.249238    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfe08d08-0850-43c8-954e-98028612959e-trusted-ca-bundle\") pod \"console-6df97869cf-qbhcv\" (UID: \"bfe08d08-0850-43c8-954e-98028612959e\") " pod="openshift-console/console-6df97869cf-qbhcv"
Jul 30 17:03:31.249815 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:03:31.249792    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bfe08d08-0850-43c8-954e-98028612959e-console-config\") pod \"console-6df97869cf-qbhcv\" (UID: \"bfe08d08-0850-43c8-954e-98028612959e\") " pod="openshift-console/console-6df97869cf-qbhcv"
Jul 30 17:03:31.249923 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:03:31.249829    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bfe08d08-0850-43c8-954e-98028612959e-oauth-serving-cert\") pod \"console-6df97869cf-qbhcv\" (UID: \"bfe08d08-0850-43c8-954e-98028612959e\") " pod="openshift-console/console-6df97869cf-qbhcv"
Jul 30 17:03:31.249923 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:03:31.249899    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bfe08d08-0850-43c8-954e-98028612959e-service-ca\") pod \"console-6df97869cf-qbhcv\" (UID: \"bfe08d08-0850-43c8-954e-98028612959e\") " pod="openshift-console/console-6df97869cf-qbhcv"
Jul 30 17:03:31.250288 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:03:31.250270    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfe08d08-0850-43c8-954e-98028612959e-trusted-ca-bundle\") pod \"console-6df97869cf-qbhcv\" (UID: \"bfe08d08-0850-43c8-954e-98028612959e\") " pod="openshift-console/console-6df97869cf-qbhcv"
Jul 30 17:03:31.251382 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:03:31.251355    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bfe08d08-0850-43c8-954e-98028612959e-console-oauth-config\") pod \"console-6df97869cf-qbhcv\" (UID: \"bfe08d08-0850-43c8-954e-98028612959e\") " pod="openshift-console/console-6df97869cf-qbhcv"
Jul 30 17:03:31.251497 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:03:31.251478    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bfe08d08-0850-43c8-954e-98028612959e-console-serving-cert\") pod \"console-6df97869cf-qbhcv\" (UID: \"bfe08d08-0850-43c8-954e-98028612959e\") " pod="openshift-console/console-6df97869cf-qbhcv"
Jul 30 17:03:31.257355 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:03:31.257336    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mjkd\" (UniqueName: \"kubernetes.io/projected/bfe08d08-0850-43c8-954e-98028612959e-kube-api-access-5mjkd\") pod \"console-6df97869cf-qbhcv\" (UID: \"bfe08d08-0850-43c8-954e-98028612959e\") " pod="openshift-console/console-6df97869cf-qbhcv"
Jul 30 17:03:31.386739 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:03:31.386686    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6df97869cf-qbhcv"
Jul 30 17:03:31.512058 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:03:31.512029    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6df97869cf-qbhcv"]
Jul 30 17:03:31.516216 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:03:31.516194    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfe08d08_0850_43c8_954e_98028612959e.slice/crio-3eebce589397dfbe8ee111cd6f1845dd4d8c34d4d938e9b7d5f67701f9085ece WatchSource:0}: Error finding container 3eebce589397dfbe8ee111cd6f1845dd4d8c34d4d938e9b7d5f67701f9085ece: Status 404 returned error can't find the container with id 3eebce589397dfbe8ee111cd6f1845dd4d8c34d4d938e9b7d5f67701f9085ece
Jul 30 17:03:32.297286 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:03:32.297254    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6df97869cf-qbhcv" event={"ID":"bfe08d08-0850-43c8-954e-98028612959e","Type":"ContainerStarted","Data":"708d66b70c94b9843b26558b23e7af8720a4700f7b0dcc17d5e7f313b860172f"}
Jul 30 17:03:32.297727 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:03:32.297303    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6df97869cf-qbhcv" event={"ID":"bfe08d08-0850-43c8-954e-98028612959e","Type":"ContainerStarted","Data":"3eebce589397dfbe8ee111cd6f1845dd4d8c34d4d938e9b7d5f67701f9085ece"}
Jul 30 17:03:41.387664 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:03:41.387618    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6df97869cf-qbhcv"
Jul 30 17:03:41.388059 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:03:41.387685    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6df97869cf-qbhcv"
Jul 30 17:03:41.392731 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:03:41.392702    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6df97869cf-qbhcv"
Jul 30 17:03:41.412658 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:03:41.412576    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6df97869cf-qbhcv" podStartSLOduration=10.412551864 podStartE2EDuration="10.412551864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:03:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:03:32.316945707 +0000 UTC m=+175.392245598" watchObservedRunningTime="2026-07-30 17:03:41.412551864 +0000 UTC m=+184.487851756"
Jul 30 17:03:42.344475 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:03:42.344443    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6df97869cf-qbhcv"
Jul 30 17:05:37.287275 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:05:37.287240    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-wqszn_f231a916-1fda-4059-9130-d4112e738ee1/console-operator/1.log"
Jul 30 17:05:37.295623 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:05:37.295586    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-wqszn_f231a916-1fda-4059-9130-d4112e738ee1/console-operator/1.log"
Jul 30 17:05:37.296292 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:05:37.296265    2580 kubelet.go:1648] "Image garbage collection succeeded"
Jul 30 17:06:07.767729 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:06:07.767685    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-pxv7q"]
Jul 30 17:06:07.770151 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:06:07.770130    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-pxv7q"
Jul 30 17:06:07.773231 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:06:07.773204    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Jul 30 17:06:07.773377 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:06:07.773207    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 30 17:06:07.774646 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:06:07.774624    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 30 17:06:07.774776 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:06:07.774663    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 30 17:06:07.798828 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:06:07.798790    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-z5q6f\""
Jul 30 17:06:07.807564 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:06:07.807528    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-pxv7q"]
Jul 30 17:06:07.925507 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:06:07.925464    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbwpn\" (UniqueName: \"kubernetes.io/projected/88814294-70dd-4600-9e15-c1ccc1398887-kube-api-access-tbwpn\") pod \"keda-admission-56f9798bb4-pxv7q\" (UID: \"88814294-70dd-4600-9e15-c1ccc1398887\") " pod="openshift-keda/keda-admission-56f9798bb4-pxv7q"
Jul 30 17:06:07.925827 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:06:07.925808    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/88814294-70dd-4600-9e15-c1ccc1398887-certificates\") pod \"keda-admission-56f9798bb4-pxv7q\" (UID: \"88814294-70dd-4600-9e15-c1ccc1398887\") " pod="openshift-keda/keda-admission-56f9798bb4-pxv7q"
Jul 30 17:06:08.026943 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:06:08.026824    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbwpn\" (UniqueName: \"kubernetes.io/projected/88814294-70dd-4600-9e15-c1ccc1398887-kube-api-access-tbwpn\") pod \"keda-admission-56f9798bb4-pxv7q\" (UID: \"88814294-70dd-4600-9e15-c1ccc1398887\") " pod="openshift-keda/keda-admission-56f9798bb4-pxv7q"
Jul 30 17:06:08.026943 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:06:08.026886    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/88814294-70dd-4600-9e15-c1ccc1398887-certificates\") pod \"keda-admission-56f9798bb4-pxv7q\" (UID: \"88814294-70dd-4600-9e15-c1ccc1398887\") " pod="openshift-keda/keda-admission-56f9798bb4-pxv7q"
Jul 30 17:06:08.029715 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:06:08.029688    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/88814294-70dd-4600-9e15-c1ccc1398887-certificates\") pod \"keda-admission-56f9798bb4-pxv7q\" (UID: \"88814294-70dd-4600-9e15-c1ccc1398887\") " pod="openshift-keda/keda-admission-56f9798bb4-pxv7q"
Jul 30 17:06:08.044291 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:06:08.044259    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbwpn\" (UniqueName: \"kubernetes.io/projected/88814294-70dd-4600-9e15-c1ccc1398887-kube-api-access-tbwpn\") pod \"keda-admission-56f9798bb4-pxv7q\" (UID: \"88814294-70dd-4600-9e15-c1ccc1398887\") " pod="openshift-keda/keda-admission-56f9798bb4-pxv7q"
Jul 30 17:06:08.081187 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:06:08.081126    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-pxv7q"
Jul 30 17:06:08.263729 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:06:08.263689    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-pxv7q"]
Jul 30 17:06:08.267735 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:06:08.267680    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88814294_70dd_4600_9e15_c1ccc1398887.slice/crio-1dd9e884f9be05b044fdd2e79c10f2daed13c56e38ab2310515af0b7b04f34d5 WatchSource:0}: Error finding container 1dd9e884f9be05b044fdd2e79c10f2daed13c56e38ab2310515af0b7b04f34d5: Status 404 returned error can't find the container with id 1dd9e884f9be05b044fdd2e79c10f2daed13c56e38ab2310515af0b7b04f34d5
Jul 30 17:06:08.269285 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:06:08.269263    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 17:06:08.913960 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:06:08.913915    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-pxv7q" event={"ID":"88814294-70dd-4600-9e15-c1ccc1398887","Type":"ContainerStarted","Data":"1dd9e884f9be05b044fdd2e79c10f2daed13c56e38ab2310515af0b7b04f34d5"}
Jul 30 17:06:10.925512 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:06:10.925403    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-pxv7q" event={"ID":"88814294-70dd-4600-9e15-c1ccc1398887","Type":"ContainerStarted","Data":"aa3ae898fe210fc3c1b13ada768051a1080cd37cb33f7b23e2a459c55521d8f8"}
Jul 30 17:06:10.925512 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:06:10.925503    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-56f9798bb4-pxv7q"
Jul 30 17:06:10.951161 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:06:10.951099    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-56f9798bb4-pxv7q" podStartSLOduration=1.5606140979999998 podStartE2EDuration="3.95108141s" totalImagesPullingTime="2.390467312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:06:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:06:10.94729964 +0000 UTC m=+334.022599531" watchObservedRunningTime="2026-07-30 17:06:10.95108141 +0000 UTC m=+334.026381299"
Jul 30 17:06:31.932986 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:06:31.932898    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-56f9798bb4-pxv7q"
Jul 30 17:07:20.605114 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:20.605067    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-58c4fb9c59-zbljv"]
Jul 30 17:07:20.607841 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:20.607817    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58c4fb9c59-zbljv"
Jul 30 17:07:20.630490 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:20.630456    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58c4fb9c59-zbljv"]
Jul 30 17:07:20.653506 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:20.653460    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kwkt\" (UniqueName: \"kubernetes.io/projected/74b18cf4-f2b0-400e-b7df-87d89f76fe0b-kube-api-access-7kwkt\") pod \"console-58c4fb9c59-zbljv\" (UID: \"74b18cf4-f2b0-400e-b7df-87d89f76fe0b\") " pod="openshift-console/console-58c4fb9c59-zbljv"
Jul 30 17:07:20.653736 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:20.653529    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/74b18cf4-f2b0-400e-b7df-87d89f76fe0b-console-config\") pod \"console-58c4fb9c59-zbljv\" (UID: \"74b18cf4-f2b0-400e-b7df-87d89f76fe0b\") " pod="openshift-console/console-58c4fb9c59-zbljv"
Jul 30 17:07:20.653736 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:20.653576    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/74b18cf4-f2b0-400e-b7df-87d89f76fe0b-oauth-serving-cert\") pod \"console-58c4fb9c59-zbljv\" (UID: \"74b18cf4-f2b0-400e-b7df-87d89f76fe0b\") " pod="openshift-console/console-58c4fb9c59-zbljv"
Jul 30 17:07:20.653736 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:20.653658    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/74b18cf4-f2b0-400e-b7df-87d89f76fe0b-console-serving-cert\") pod \"console-58c4fb9c59-zbljv\" (UID: \"74b18cf4-f2b0-400e-b7df-87d89f76fe0b\") " pod="openshift-console/console-58c4fb9c59-zbljv"
Jul 30 17:07:20.653736 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:20.653694    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/74b18cf4-f2b0-400e-b7df-87d89f76fe0b-console-oauth-config\") pod \"console-58c4fb9c59-zbljv\" (UID: \"74b18cf4-f2b0-400e-b7df-87d89f76fe0b\") " pod="openshift-console/console-58c4fb9c59-zbljv"
Jul 30 17:07:20.653736 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:20.653738    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/74b18cf4-f2b0-400e-b7df-87d89f76fe0b-service-ca\") pod \"console-58c4fb9c59-zbljv\" (UID: \"74b18cf4-f2b0-400e-b7df-87d89f76fe0b\") " pod="openshift-console/console-58c4fb9c59-zbljv"
Jul 30 17:07:20.653925 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:20.653753    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74b18cf4-f2b0-400e-b7df-87d89f76fe0b-trusted-ca-bundle\") pod \"console-58c4fb9c59-zbljv\" (UID: \"74b18cf4-f2b0-400e-b7df-87d89f76fe0b\") " pod="openshift-console/console-58c4fb9c59-zbljv"
Jul 30 17:07:20.755027 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:20.754923    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/74b18cf4-f2b0-400e-b7df-87d89f76fe0b-console-serving-cert\") pod \"console-58c4fb9c59-zbljv\" (UID: \"74b18cf4-f2b0-400e-b7df-87d89f76fe0b\") " pod="openshift-console/console-58c4fb9c59-zbljv"
Jul 30 17:07:20.755299 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:20.755064    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/74b18cf4-f2b0-400e-b7df-87d89f76fe0b-console-oauth-config\") pod \"console-58c4fb9c59-zbljv\" (UID: \"74b18cf4-f2b0-400e-b7df-87d89f76fe0b\") " pod="openshift-console/console-58c4fb9c59-zbljv"
Jul 30 17:07:20.755299 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:20.755132    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/74b18cf4-f2b0-400e-b7df-87d89f76fe0b-service-ca\") pod \"console-58c4fb9c59-zbljv\" (UID: \"74b18cf4-f2b0-400e-b7df-87d89f76fe0b\") " pod="openshift-console/console-58c4fb9c59-zbljv"
Jul 30 17:07:20.755299 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:20.755153    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74b18cf4-f2b0-400e-b7df-87d89f76fe0b-trusted-ca-bundle\") pod \"console-58c4fb9c59-zbljv\" (UID: \"74b18cf4-f2b0-400e-b7df-87d89f76fe0b\") " pod="openshift-console/console-58c4fb9c59-zbljv"
Jul 30 17:07:20.755299 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:20.755228    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kwkt\" (UniqueName: \"kubernetes.io/projected/74b18cf4-f2b0-400e-b7df-87d89f76fe0b-kube-api-access-7kwkt\") pod \"console-58c4fb9c59-zbljv\" (UID: \"74b18cf4-f2b0-400e-b7df-87d89f76fe0b\") " pod="openshift-console/console-58c4fb9c59-zbljv"
Jul 30 17:07:20.755513 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:20.755322    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/74b18cf4-f2b0-400e-b7df-87d89f76fe0b-console-config\") pod \"console-58c4fb9c59-zbljv\" (UID: \"74b18cf4-f2b0-400e-b7df-87d89f76fe0b\") " pod="openshift-console/console-58c4fb9c59-zbljv"
Jul 30 17:07:20.755513 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:20.755347    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/74b18cf4-f2b0-400e-b7df-87d89f76fe0b-oauth-serving-cert\") pod \"console-58c4fb9c59-zbljv\" (UID: \"74b18cf4-f2b0-400e-b7df-87d89f76fe0b\") " pod="openshift-console/console-58c4fb9c59-zbljv"
Jul 30 17:07:20.755976 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:20.755943    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/74b18cf4-f2b0-400e-b7df-87d89f76fe0b-service-ca\") pod \"console-58c4fb9c59-zbljv\" (UID: \"74b18cf4-f2b0-400e-b7df-87d89f76fe0b\") " pod="openshift-console/console-58c4fb9c59-zbljv"
Jul 30 17:07:20.756102 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:20.755998    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/74b18cf4-f2b0-400e-b7df-87d89f76fe0b-console-config\") pod \"console-58c4fb9c59-zbljv\" (UID: \"74b18cf4-f2b0-400e-b7df-87d89f76fe0b\") " pod="openshift-console/console-58c4fb9c59-zbljv"
Jul 30 17:07:20.756102 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:20.756077    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74b18cf4-f2b0-400e-b7df-87d89f76fe0b-trusted-ca-bundle\") pod \"console-58c4fb9c59-zbljv\" (UID: \"74b18cf4-f2b0-400e-b7df-87d89f76fe0b\") " pod="openshift-console/console-58c4fb9c59-zbljv"
Jul 30 17:07:20.756216 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:20.756160    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/74b18cf4-f2b0-400e-b7df-87d89f76fe0b-oauth-serving-cert\") pod \"console-58c4fb9c59-zbljv\" (UID: \"74b18cf4-f2b0-400e-b7df-87d89f76fe0b\") " pod="openshift-console/console-58c4fb9c59-zbljv"
Jul 30 17:07:20.757760 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:20.757729    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/74b18cf4-f2b0-400e-b7df-87d89f76fe0b-console-oauth-config\") pod \"console-58c4fb9c59-zbljv\" (UID: \"74b18cf4-f2b0-400e-b7df-87d89f76fe0b\") " pod="openshift-console/console-58c4fb9c59-zbljv"
Jul 30 17:07:20.757848 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:20.757780    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/74b18cf4-f2b0-400e-b7df-87d89f76fe0b-console-serving-cert\") pod \"console-58c4fb9c59-zbljv\" (UID: \"74b18cf4-f2b0-400e-b7df-87d89f76fe0b\") " pod="openshift-console/console-58c4fb9c59-zbljv"
Jul 30 17:07:20.768909 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:20.768878    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kwkt\" (UniqueName: \"kubernetes.io/projected/74b18cf4-f2b0-400e-b7df-87d89f76fe0b-kube-api-access-7kwkt\") pod \"console-58c4fb9c59-zbljv\" (UID: \"74b18cf4-f2b0-400e-b7df-87d89f76fe0b\") " pod="openshift-console/console-58c4fb9c59-zbljv"
Jul 30 17:07:20.919485 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:20.919361    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58c4fb9c59-zbljv"
Jul 30 17:07:21.069330 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:21.069235    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58c4fb9c59-zbljv"]
Jul 30 17:07:21.072201 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:07:21.072148    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74b18cf4_f2b0_400e_b7df_87d89f76fe0b.slice/crio-96cddb3bfcbf071686a74e289e0cbf05caf870ad6cb9e1e710a7b5bec2226811 WatchSource:0}: Error finding container 96cddb3bfcbf071686a74e289e0cbf05caf870ad6cb9e1e710a7b5bec2226811: Status 404 returned error can't find the container with id 96cddb3bfcbf071686a74e289e0cbf05caf870ad6cb9e1e710a7b5bec2226811
Jul 30 17:07:21.201661 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:21.201556    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58c4fb9c59-zbljv" event={"ID":"74b18cf4-f2b0-400e-b7df-87d89f76fe0b","Type":"ContainerStarted","Data":"608427e86673d389cad64d12467cb6b141e37d0b63b5b2e867f8f85514fc3159"}
Jul 30 17:07:21.201661 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:21.201611    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58c4fb9c59-zbljv" event={"ID":"74b18cf4-f2b0-400e-b7df-87d89f76fe0b","Type":"ContainerStarted","Data":"96cddb3bfcbf071686a74e289e0cbf05caf870ad6cb9e1e710a7b5bec2226811"}
Jul 30 17:07:21.221111 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:21.221048    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-58c4fb9c59-zbljv" podStartSLOduration=1.221027538 podStartE2EDuration="1.221027538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:07:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:07:21.220396242 +0000 UTC m=+404.295696155" watchObservedRunningTime="2026-07-30 17:07:21.221027538 +0000 UTC m=+404.296327513"
Jul 30 17:07:27.101077 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:27.101032    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-69jnn"]
Jul 30 17:07:27.103902 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:27.103875    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-654c557856-69jnn"
Jul 30 17:07:27.106940 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:27.106918    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 30 17:07:27.107081 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:27.106918    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 30 17:07:27.107239 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:27.107227    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 30 17:07:27.108204 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:27.108163    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-275t9\""
Jul 30 17:07:27.114915 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:27.114881    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-69jnn"]
Jul 30 17:07:27.214544 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:27.214497    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e74064b8-5845-4d74-af33-e9c18de02def-cert\") pod \"llmisvc-controller-manager-654c557856-69jnn\" (UID: \"e74064b8-5845-4d74-af33-e9c18de02def\") " pod="kserve/llmisvc-controller-manager-654c557856-69jnn"
Jul 30 17:07:27.214755 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:27.214634    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4kbf\" (UniqueName: \"kubernetes.io/projected/e74064b8-5845-4d74-af33-e9c18de02def-kube-api-access-q4kbf\") pod \"llmisvc-controller-manager-654c557856-69jnn\" (UID: \"e74064b8-5845-4d74-af33-e9c18de02def\") " pod="kserve/llmisvc-controller-manager-654c557856-69jnn"
Jul 30 17:07:27.315578 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:27.315532    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4kbf\" (UniqueName: \"kubernetes.io/projected/e74064b8-5845-4d74-af33-e9c18de02def-kube-api-access-q4kbf\") pod \"llmisvc-controller-manager-654c557856-69jnn\" (UID: \"e74064b8-5845-4d74-af33-e9c18de02def\") " pod="kserve/llmisvc-controller-manager-654c557856-69jnn"
Jul 30 17:07:27.315792 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:27.315595    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e74064b8-5845-4d74-af33-e9c18de02def-cert\") pod \"llmisvc-controller-manager-654c557856-69jnn\" (UID: \"e74064b8-5845-4d74-af33-e9c18de02def\") " pod="kserve/llmisvc-controller-manager-654c557856-69jnn"
Jul 30 17:07:27.318103 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:27.318078    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e74064b8-5845-4d74-af33-e9c18de02def-cert\") pod \"llmisvc-controller-manager-654c557856-69jnn\" (UID: \"e74064b8-5845-4d74-af33-e9c18de02def\") " pod="kserve/llmisvc-controller-manager-654c557856-69jnn"
Jul 30 17:07:27.324842 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:27.324813    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4kbf\" (UniqueName: \"kubernetes.io/projected/e74064b8-5845-4d74-af33-e9c18de02def-kube-api-access-q4kbf\") pod \"llmisvc-controller-manager-654c557856-69jnn\" (UID: \"e74064b8-5845-4d74-af33-e9c18de02def\") " pod="kserve/llmisvc-controller-manager-654c557856-69jnn"
Jul 30 17:07:27.416867 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:27.416746    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-654c557856-69jnn"
Jul 30 17:07:27.565310 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:27.565285    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-69jnn"]
Jul 30 17:07:27.568276 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:07:27.568250    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode74064b8_5845_4d74_af33_e9c18de02def.slice/crio-87548aeabcb9b4fe9ce6a235dd27fd73227b462cfebdfb54fa8849b77270348e WatchSource:0}: Error finding container 87548aeabcb9b4fe9ce6a235dd27fd73227b462cfebdfb54fa8849b77270348e: Status 404 returned error can't find the container with id 87548aeabcb9b4fe9ce6a235dd27fd73227b462cfebdfb54fa8849b77270348e
Jul 30 17:07:28.236135 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:28.236100    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-654c557856-69jnn" event={"ID":"e74064b8-5845-4d74-af33-e9c18de02def","Type":"ContainerStarted","Data":"87548aeabcb9b4fe9ce6a235dd27fd73227b462cfebdfb54fa8849b77270348e"}
Jul 30 17:07:30.245560 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:30.245516    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-654c557856-69jnn" event={"ID":"e74064b8-5845-4d74-af33-e9c18de02def","Type":"ContainerStarted","Data":"841e44b538f004cf95ede37ecb5d87d8170d529478d6550d112da81306bfea88"}
Jul 30 17:07:30.246072 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:30.245648    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-654c557856-69jnn"
Jul 30 17:07:30.266791 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:30.266728    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-654c557856-69jnn" podStartSLOduration=0.670728508 podStartE2EDuration="3.266709607s" totalImagesPullingTime="2.595981099s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:07:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:07:30.264574683 +0000 UTC m=+413.339874573" watchObservedRunningTime="2026-07-30 17:07:30.266709607 +0000 UTC m=+413.342009497"
Jul 30 17:07:30.919684 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:30.919637    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-58c4fb9c59-zbljv"
Jul 30 17:07:30.919917 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:30.919715    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-58c4fb9c59-zbljv"
Jul 30 17:07:30.924959 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:30.924928    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-58c4fb9c59-zbljv"
Jul 30 17:07:31.254314 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:31.254282    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-58c4fb9c59-zbljv"
Jul 30 17:07:31.304343 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:31.304300    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6df97869cf-qbhcv"]
Jul 30 17:07:56.326303 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:56.326234    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6df97869cf-qbhcv" podUID="bfe08d08-0850-43c8-954e-98028612959e" containerName="console" containerID="cri-o://708d66b70c94b9843b26558b23e7af8720a4700f7b0dcc17d5e7f313b860172f" gracePeriod=15
Jul 30 17:07:56.577399 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:56.577328    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6df97869cf-qbhcv"
Jul 30 17:07:56.585952 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:56.585914    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bfe08d08-0850-43c8-954e-98028612959e-console-config\") pod \"bfe08d08-0850-43c8-954e-98028612959e\" (UID: \"bfe08d08-0850-43c8-954e-98028612959e\") "
Jul 30 17:07:56.586151 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:56.585991    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bfe08d08-0850-43c8-954e-98028612959e-console-oauth-config\") pod \"bfe08d08-0850-43c8-954e-98028612959e\" (UID: \"bfe08d08-0850-43c8-954e-98028612959e\") "
Jul 30 17:07:56.586151 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:56.586032    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfe08d08-0850-43c8-954e-98028612959e-trusted-ca-bundle\") pod \"bfe08d08-0850-43c8-954e-98028612959e\" (UID: \"bfe08d08-0850-43c8-954e-98028612959e\") "
Jul 30 17:07:56.586151 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:56.586060    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bfe08d08-0850-43c8-954e-98028612959e-oauth-serving-cert\") pod \"bfe08d08-0850-43c8-954e-98028612959e\" (UID: \"bfe08d08-0850-43c8-954e-98028612959e\") "
Jul 30 17:07:56.586365 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:56.586219    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5mjkd\" (UniqueName: \"kubernetes.io/projected/bfe08d08-0850-43c8-954e-98028612959e-kube-api-access-5mjkd\") pod \"bfe08d08-0850-43c8-954e-98028612959e\" (UID: \"bfe08d08-0850-43c8-954e-98028612959e\") "
Jul 30 17:07:56.586365 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:56.586278    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bfe08d08-0850-43c8-954e-98028612959e-service-ca\") pod \"bfe08d08-0850-43c8-954e-98028612959e\" (UID: \"bfe08d08-0850-43c8-954e-98028612959e\") "
Jul 30 17:07:56.586365 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:56.586326    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bfe08d08-0850-43c8-954e-98028612959e-console-serving-cert\") pod \"bfe08d08-0850-43c8-954e-98028612959e\" (UID: \"bfe08d08-0850-43c8-954e-98028612959e\") "
Jul 30 17:07:56.586513 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:56.586377    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfe08d08-0850-43c8-954e-98028612959e-console-config" (OuterVolumeSpecName: "console-config") pod "bfe08d08-0850-43c8-954e-98028612959e" (UID: "bfe08d08-0850-43c8-954e-98028612959e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:07:56.586513 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:56.586474    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfe08d08-0850-43c8-954e-98028612959e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "bfe08d08-0850-43c8-954e-98028612959e" (UID: "bfe08d08-0850-43c8-954e-98028612959e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:07:56.586600 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:56.586566    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bfe08d08-0850-43c8-954e-98028612959e-console-config\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:07:56.586600 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:56.586581    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bfe08d08-0850-43c8-954e-98028612959e-oauth-serving-cert\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:07:56.586600 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:56.586561    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfe08d08-0850-43c8-954e-98028612959e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "bfe08d08-0850-43c8-954e-98028612959e" (UID: "bfe08d08-0850-43c8-954e-98028612959e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:07:56.586736 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:56.586615    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfe08d08-0850-43c8-954e-98028612959e-service-ca" (OuterVolumeSpecName: "service-ca") pod "bfe08d08-0850-43c8-954e-98028612959e" (UID: "bfe08d08-0850-43c8-954e-98028612959e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:07:56.588435 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:56.588393    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfe08d08-0850-43c8-954e-98028612959e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "bfe08d08-0850-43c8-954e-98028612959e" (UID: "bfe08d08-0850-43c8-954e-98028612959e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:07:56.588558 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:56.588490    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfe08d08-0850-43c8-954e-98028612959e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "bfe08d08-0850-43c8-954e-98028612959e" (UID: "bfe08d08-0850-43c8-954e-98028612959e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:07:56.588558 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:56.588507    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfe08d08-0850-43c8-954e-98028612959e-kube-api-access-5mjkd" (OuterVolumeSpecName: "kube-api-access-5mjkd") pod "bfe08d08-0850-43c8-954e-98028612959e" (UID: "bfe08d08-0850-43c8-954e-98028612959e"). InnerVolumeSpecName "kube-api-access-5mjkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:07:56.686976 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:56.686928    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bfe08d08-0850-43c8-954e-98028612959e-console-oauth-config\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:07:56.686976 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:56.686965    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfe08d08-0850-43c8-954e-98028612959e-trusted-ca-bundle\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:07:56.686976 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:56.686975    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mjkd\" (UniqueName: \"kubernetes.io/projected/bfe08d08-0850-43c8-954e-98028612959e-kube-api-access-5mjkd\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:07:56.686976 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:56.686985    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bfe08d08-0850-43c8-954e-98028612959e-service-ca\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:07:56.687308 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:56.686994    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bfe08d08-0850-43c8-954e-98028612959e-console-serving-cert\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:07:57.354584 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:57.354542    2580 generic.go:356] "Generic (PLEG): container finished" podID="bfe08d08-0850-43c8-954e-98028612959e" containerID="708d66b70c94b9843b26558b23e7af8720a4700f7b0dcc17d5e7f313b860172f" exitCode=0
Jul 30 17:07:57.354584 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:57.354590    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6df97869cf-qbhcv"
Jul 30 17:07:57.355069 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:57.354632    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6df97869cf-qbhcv" event={"ID":"bfe08d08-0850-43c8-954e-98028612959e","Type":"ContainerDied","Data":"708d66b70c94b9843b26558b23e7af8720a4700f7b0dcc17d5e7f313b860172f"}
Jul 30 17:07:57.355069 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:57.354690    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6df97869cf-qbhcv" event={"ID":"bfe08d08-0850-43c8-954e-98028612959e","Type":"ContainerDied","Data":"3eebce589397dfbe8ee111cd6f1845dd4d8c34d4d938e9b7d5f67701f9085ece"}
Jul 30 17:07:57.355069 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:57.354716    2580 scope.go:117] "RemoveContainer" containerID="708d66b70c94b9843b26558b23e7af8720a4700f7b0dcc17d5e7f313b860172f"
Jul 30 17:07:57.363987 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:57.363967    2580 scope.go:117] "RemoveContainer" containerID="708d66b70c94b9843b26558b23e7af8720a4700f7b0dcc17d5e7f313b860172f"
Jul 30 17:07:57.364316 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:07:57.364291    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"708d66b70c94b9843b26558b23e7af8720a4700f7b0dcc17d5e7f313b860172f\": container with ID starting with 708d66b70c94b9843b26558b23e7af8720a4700f7b0dcc17d5e7f313b860172f not found: ID does not exist" containerID="708d66b70c94b9843b26558b23e7af8720a4700f7b0dcc17d5e7f313b860172f"
Jul 30 17:07:57.364373 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:57.364327    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"708d66b70c94b9843b26558b23e7af8720a4700f7b0dcc17d5e7f313b860172f"} err="failed to get container status \"708d66b70c94b9843b26558b23e7af8720a4700f7b0dcc17d5e7f313b860172f\": rpc error: code = NotFound desc = could not find container \"708d66b70c94b9843b26558b23e7af8720a4700f7b0dcc17d5e7f313b860172f\": container with ID starting with 708d66b70c94b9843b26558b23e7af8720a4700f7b0dcc17d5e7f313b860172f not found: ID does not exist"
Jul 30 17:07:57.379482 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:57.379009    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6df97869cf-qbhcv"]
Jul 30 17:07:57.385039 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:57.385004    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6df97869cf-qbhcv"]
Jul 30 17:07:57.481592 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:07:57.481560    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfe08d08-0850-43c8-954e-98028612959e" path="/var/lib/kubelet/pods/bfe08d08-0850-43c8-954e-98028612959e/volumes"
Jul 30 17:08:01.253467 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:08:01.253388    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-654c557856-69jnn"
Jul 30 17:10:30.620430 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:10:30.620391    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg"]
Jul 30 17:10:30.620948 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:10:30.620734    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfe08d08-0850-43c8-954e-98028612959e" containerName="console"
Jul 30 17:10:30.620948 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:10:30.620746    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfe08d08-0850-43c8-954e-98028612959e" containerName="console"
Jul 30 17:10:30.620948 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:10:30.620796    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfe08d08-0850-43c8-954e-98028612959e" containerName="console"
Jul 30 17:10:30.623111 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:10:30.623088    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg"
Jul 30 17:10:30.628092 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:10:30.628061    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-fbbbd-kube-rbac-proxy-sar-config\""
Jul 30 17:10:30.628349 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:10:30.628329    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 30 17:10:30.628424 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:10:30.628329    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 30 17:10:30.629061 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:10:30.629039    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-qz9j4\""
Jul 30 17:10:30.629222 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:10:30.629080    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 30 17:10:30.629509 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:10:30.629491    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 30 17:10:30.630746 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:10:30.630727    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-fbbbd-predictor-serving-cert\""
Jul 30 17:10:30.642065 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:10:30.642033    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg"]
Jul 30 17:10:30.773952 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:10:30.773907    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3aa38bf3-1bcd-474c-a184-4a4ef95950c2-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg\" (UID: \"3aa38bf3-1bcd-474c-a184-4a4ef95950c2\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg"
Jul 30 17:10:30.773952 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:10:30.773964    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3aa38bf3-1bcd-474c-a184-4a4ef95950c2-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg\" (UID: \"3aa38bf3-1bcd-474c-a184-4a4ef95950c2\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg"
Jul 30 17:10:30.774215 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:10:30.774039    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3aa38bf3-1bcd-474c-a184-4a4ef95950c2-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg\" (UID: \"3aa38bf3-1bcd-474c-a184-4a4ef95950c2\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg"
Jul 30 17:10:30.774215 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:10:30.774110    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3aa38bf3-1bcd-474c-a184-4a4ef95950c2-proxy-tls\") pod \"isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg\" (UID: \"3aa38bf3-1bcd-474c-a184-4a4ef95950c2\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg"
Jul 30 17:10:30.774215 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:10:30.774163    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t77w\" (UniqueName: \"kubernetes.io/projected/3aa38bf3-1bcd-474c-a184-4a4ef95950c2-kube-api-access-4t77w\") pod \"isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg\" (UID: \"3aa38bf3-1bcd-474c-a184-4a4ef95950c2\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg"
Jul 30 17:10:30.875345 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:10:30.875224    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3aa38bf3-1bcd-474c-a184-4a4ef95950c2-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg\" (UID: \"3aa38bf3-1bcd-474c-a184-4a4ef95950c2\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg"
Jul 30 17:10:30.875345 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:10:30.875300    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3aa38bf3-1bcd-474c-a184-4a4ef95950c2-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg\" (UID: \"3aa38bf3-1bcd-474c-a184-4a4ef95950c2\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg"
Jul 30 17:10:30.875345 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:10:30.875331    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3aa38bf3-1bcd-474c-a184-4a4ef95950c2-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg\" (UID: \"3aa38bf3-1bcd-474c-a184-4a4ef95950c2\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg"
Jul 30 17:10:30.875672 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:10:30.875360    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3aa38bf3-1bcd-474c-a184-4a4ef95950c2-proxy-tls\") pod \"isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg\" (UID: \"3aa38bf3-1bcd-474c-a184-4a4ef95950c2\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg"
Jul 30 17:10:30.875672 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:10:30.875415    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t77w\" (UniqueName: \"kubernetes.io/projected/3aa38bf3-1bcd-474c-a184-4a4ef95950c2-kube-api-access-4t77w\") pod \"isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg\" (UID: \"3aa38bf3-1bcd-474c-a184-4a4ef95950c2\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg"
Jul 30 17:10:30.875822 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:10:30.875795    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3aa38bf3-1bcd-474c-a184-4a4ef95950c2-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg\" (UID: \"3aa38bf3-1bcd-474c-a184-4a4ef95950c2\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg"
Jul 30 17:10:30.876074 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:10:30.876038    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3aa38bf3-1bcd-474c-a184-4a4ef95950c2-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg\" (UID: \"3aa38bf3-1bcd-474c-a184-4a4ef95950c2\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg"
Jul 30 17:10:30.876074 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:10:30.876063    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3aa38bf3-1bcd-474c-a184-4a4ef95950c2-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg\" (UID: \"3aa38bf3-1bcd-474c-a184-4a4ef95950c2\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg"
Jul 30 17:10:30.878085 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:10:30.878061    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3aa38bf3-1bcd-474c-a184-4a4ef95950c2-proxy-tls\") pod \"isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg\" (UID: \"3aa38bf3-1bcd-474c-a184-4a4ef95950c2\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg"
Jul 30 17:10:30.885103 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:10:30.885074    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t77w\" (UniqueName: \"kubernetes.io/projected/3aa38bf3-1bcd-474c-a184-4a4ef95950c2-kube-api-access-4t77w\") pod \"isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg\" (UID: \"3aa38bf3-1bcd-474c-a184-4a4ef95950c2\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg"
Jul 30 17:10:30.934092 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:10:30.934054    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg"
Jul 30 17:10:31.080037 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:10:31.079931    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg"]
Jul 30 17:10:31.082907 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:10:31.082873    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aa38bf3_1bcd_474c_a184_4a4ef95950c2.slice/crio-84dba860bb7e775a3f0315510a1033b89791ab00bf4eb478247d242b9311461f WatchSource:0}: Error finding container 84dba860bb7e775a3f0315510a1033b89791ab00bf4eb478247d242b9311461f: Status 404 returned error can't find the container with id 84dba860bb7e775a3f0315510a1033b89791ab00bf4eb478247d242b9311461f
Jul 30 17:10:31.985197 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:10:31.985093    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg" event={"ID":"3aa38bf3-1bcd-474c-a184-4a4ef95950c2","Type":"ContainerStarted","Data":"84dba860bb7e775a3f0315510a1033b89791ab00bf4eb478247d242b9311461f"}
Jul 30 17:10:35.003253 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:10:35.003213    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg" event={"ID":"3aa38bf3-1bcd-474c-a184-4a4ef95950c2","Type":"ContainerStarted","Data":"2bdce362f6b0452a54d99a62174ae87571a6f9e3cc024c57c7535134471e8101"}
Jul 30 17:10:37.320699 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:10:37.320662    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-wqszn_f231a916-1fda-4059-9130-d4112e738ee1/console-operator/1.log"
Jul 30 17:10:37.325219 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:10:37.325187    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-wqszn_f231a916-1fda-4059-9130-d4112e738ee1/console-operator/1.log"
Jul 30 17:10:39.021112 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:10:39.021074    2580 generic.go:356] "Generic (PLEG): container finished" podID="3aa38bf3-1bcd-474c-a184-4a4ef95950c2" containerID="2bdce362f6b0452a54d99a62174ae87571a6f9e3cc024c57c7535134471e8101" exitCode=0
Jul 30 17:10:39.021579 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:10:39.021131    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg" event={"ID":"3aa38bf3-1bcd-474c-a184-4a4ef95950c2","Type":"ContainerDied","Data":"2bdce362f6b0452a54d99a62174ae87571a6f9e3cc024c57c7535134471e8101"}
Jul 30 17:10:52.092117 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:10:52.092075    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg" event={"ID":"3aa38bf3-1bcd-474c-a184-4a4ef95950c2","Type":"ContainerStarted","Data":"2ea5a1afb4a5fd3a5d5fa10f6d660b21af13954ed52af1c10da58d23138f02f3"}
Jul 30 17:10:55.108796 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:10:55.108748    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg" event={"ID":"3aa38bf3-1bcd-474c-a184-4a4ef95950c2","Type":"ContainerStarted","Data":"3ffe49ce4401cdedc229b7fc382fc86fe289ab9fa82d340d2119913720dc4b9c"}
Jul 30 17:10:57.120814 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:10:57.120777    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg" event={"ID":"3aa38bf3-1bcd-474c-a184-4a4ef95950c2","Type":"ContainerStarted","Data":"7e4335daba2d1efe1f3285902037a10242986fbc9216b1826fe09b55a4df2e97"}
Jul 30 17:10:57.121334 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:10:57.120934    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg"
Jul 30 17:10:57.144100 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:10:57.144040    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg" podStartSLOduration=2.7205540409999998 podStartE2EDuration="27.144020638s" totalImagesPullingTime="20.496979018s" totalInitContainerRuntime="3.926487579s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:10:30 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:10:57.143640626 +0000 UTC m=+620.218940530" watchObservedRunningTime="2026-07-30 17:10:57.144020638 +0000 UTC m=+620.219320529"
Jul 30 17:10:58.125669 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:10:58.125581    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg"
Jul 30 17:10:58.125669 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:10:58.125651    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg"
Jul 30 17:10:58.126975 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:10:58.126925    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg" podUID="3aa38bf3-1bcd-474c-a184-4a4ef95950c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 30 17:10:58.127631 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:10:58.127604    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg" podUID="3aa38bf3-1bcd-474c-a184-4a4ef95950c2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 17:10:59.135878 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:10:59.135827    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg" podUID="3aa38bf3-1bcd-474c-a184-4a4ef95950c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 30 17:10:59.136374 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:10:59.136227    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg" podUID="3aa38bf3-1bcd-474c-a184-4a4ef95950c2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 17:10:59.139997 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:10:59.139971    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg"
Jul 30 17:11:00.139913 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:11:00.139869    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg" podUID="3aa38bf3-1bcd-474c-a184-4a4ef95950c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 30 17:11:00.140374 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:11:00.140327    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg" podUID="3aa38bf3-1bcd-474c-a184-4a4ef95950c2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 17:11:10.139986 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:11:10.139925    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg" podUID="3aa38bf3-1bcd-474c-a184-4a4ef95950c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 30 17:11:10.260303 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:11:10.140458    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg" podUID="3aa38bf3-1bcd-474c-a184-4a4ef95950c2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 17:11:20.140654 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:11:20.140610    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg" podUID="3aa38bf3-1bcd-474c-a184-4a4ef95950c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 30 17:11:20.141207 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:11:20.141150    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg" podUID="3aa38bf3-1bcd-474c-a184-4a4ef95950c2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 17:11:30.140929 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:11:30.140864    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg" podUID="3aa38bf3-1bcd-474c-a184-4a4ef95950c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 30 17:11:30.141400 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:11:30.141320    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg" podUID="3aa38bf3-1bcd-474c-a184-4a4ef95950c2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 17:11:40.140491 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:11:40.140435    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg" podUID="3aa38bf3-1bcd-474c-a184-4a4ef95950c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 30 17:11:40.141101 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:11:40.140993    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg" podUID="3aa38bf3-1bcd-474c-a184-4a4ef95950c2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 17:11:50.140421 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:11:50.140370    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg" podUID="3aa38bf3-1bcd-474c-a184-4a4ef95950c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 30 17:11:50.140952 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:11:50.140888    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg" podUID="3aa38bf3-1bcd-474c-a184-4a4ef95950c2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 17:12:00.140059 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:00.140002    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg" podUID="3aa38bf3-1bcd-474c-a184-4a4ef95950c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 30 17:12:00.140554 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:00.140515    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg" podUID="3aa38bf3-1bcd-474c-a184-4a4ef95950c2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 17:12:07.479411 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:07.479356    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg" podUID="3aa38bf3-1bcd-474c-a184-4a4ef95950c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 30 17:12:07.479922 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:07.479732    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg" podUID="3aa38bf3-1bcd-474c-a184-4a4ef95950c2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 17:12:17.483089 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:17.483056    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg"
Jul 30 17:12:17.483618 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:17.483136    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg"
Jul 30 17:12:25.699091 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:25.699052    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg"]
Jul 30 17:12:25.699647 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:25.699418    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg" podUID="3aa38bf3-1bcd-474c-a184-4a4ef95950c2" containerName="kserve-container" containerID="cri-o://2ea5a1afb4a5fd3a5d5fa10f6d660b21af13954ed52af1c10da58d23138f02f3" gracePeriod=30
Jul 30 17:12:25.699647 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:25.699497    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg" podUID="3aa38bf3-1bcd-474c-a184-4a4ef95950c2" containerName="agent" containerID="cri-o://7e4335daba2d1efe1f3285902037a10242986fbc9216b1826fe09b55a4df2e97" gracePeriod=30
Jul 30 17:12:25.699647 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:25.699499    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg" podUID="3aa38bf3-1bcd-474c-a184-4a4ef95950c2" containerName="kube-rbac-proxy" containerID="cri-o://3ffe49ce4401cdedc229b7fc382fc86fe289ab9fa82d340d2119913720dc4b9c" gracePeriod=30
Jul 30 17:12:25.747695 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:25.747651    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz"]
Jul 30 17:12:25.750541 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:25.750516    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz"
Jul 30 17:12:25.753099 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:25.753069    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-5f73e-predictor-serving-cert\""
Jul 30 17:12:25.753270 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:25.753094    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-5f73e-kube-rbac-proxy-sar-config\""
Jul 30 17:12:25.762449 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:25.762419    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz"]
Jul 30 17:12:25.808462 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:25.808418    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ceb692e6-ac6d-4ceb-b514-777eddf7c439-proxy-tls\") pod \"isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz\" (UID: \"ceb692e6-ac6d-4ceb-b514-777eddf7c439\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz"
Jul 30 17:12:25.808654 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:25.808532    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ceb692e6-ac6d-4ceb-b514-777eddf7c439-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz\" (UID: \"ceb692e6-ac6d-4ceb-b514-777eddf7c439\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz"
Jul 30 17:12:25.808654 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:25.808611    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ceb692e6-ac6d-4ceb-b514-777eddf7c439-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz\" (UID: \"ceb692e6-ac6d-4ceb-b514-777eddf7c439\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz"
Jul 30 17:12:25.808654 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:25.808636    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ceb692e6-ac6d-4ceb-b514-777eddf7c439-cabundle-cert\") pod \"isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz\" (UID: \"ceb692e6-ac6d-4ceb-b514-777eddf7c439\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz"
Jul 30 17:12:25.808821 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:25.808677    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d68r5\" (UniqueName: \"kubernetes.io/projected/ceb692e6-ac6d-4ceb-b514-777eddf7c439-kube-api-access-d68r5\") pod \"isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz\" (UID: \"ceb692e6-ac6d-4ceb-b514-777eddf7c439\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz"
Jul 30 17:12:25.853051 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:25.853006    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz"]
Jul 30 17:12:25.855789 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:25.855765    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz"
Jul 30 17:12:25.858108 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:25.858085    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-5f73e-predictor-serving-cert\""
Jul 30 17:12:25.858264 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:25.858243    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-5f73e-kube-rbac-proxy-sar-config\""
Jul 30 17:12:25.866241 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:25.866157    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz"]
Jul 30 17:12:25.909906 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:25.909862    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ceb692e6-ac6d-4ceb-b514-777eddf7c439-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz\" (UID: \"ceb692e6-ac6d-4ceb-b514-777eddf7c439\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz"
Jul 30 17:12:25.909906 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:25.909906    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ceb692e6-ac6d-4ceb-b514-777eddf7c439-cabundle-cert\") pod \"isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz\" (UID: \"ceb692e6-ac6d-4ceb-b514-777eddf7c439\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz"
Jul 30 17:12:25.910217 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:25.909943    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d68r5\" (UniqueName: \"kubernetes.io/projected/ceb692e6-ac6d-4ceb-b514-777eddf7c439-kube-api-access-d68r5\") pod \"isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz\" (UID: \"ceb692e6-ac6d-4ceb-b514-777eddf7c439\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz"
Jul 30 17:12:25.910217 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:25.909971    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22be6187-f972-45d4-a410-3f4978fb4425-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz\" (UID: \"22be6187-f972-45d4-a410-3f4978fb4425\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz"
Jul 30 17:12:25.910217 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:25.909998    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22be6187-f972-45d4-a410-3f4978fb4425-proxy-tls\") pod \"isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz\" (UID: \"22be6187-f972-45d4-a410-3f4978fb4425\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz"
Jul 30 17:12:25.910217 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:25.910092    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22be6187-f972-45d4-a410-3f4978fb4425-cabundle-cert\") pod \"isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz\" (UID: \"22be6187-f972-45d4-a410-3f4978fb4425\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz"
Jul 30 17:12:25.910217 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:25.910196    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22be6187-f972-45d4-a410-3f4978fb4425-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz\" (UID: \"22be6187-f972-45d4-a410-3f4978fb4425\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz"
Jul 30 17:12:25.910485 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:25.910233    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk4rc\" (UniqueName: \"kubernetes.io/projected/22be6187-f972-45d4-a410-3f4978fb4425-kube-api-access-kk4rc\") pod \"isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz\" (UID: \"22be6187-f972-45d4-a410-3f4978fb4425\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz"
Jul 30 17:12:25.910485 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:25.910324    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ceb692e6-ac6d-4ceb-b514-777eddf7c439-proxy-tls\") pod \"isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz\" (UID: \"ceb692e6-ac6d-4ceb-b514-777eddf7c439\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz"
Jul 30 17:12:25.910485 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:25.910373    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ceb692e6-ac6d-4ceb-b514-777eddf7c439-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz\" (UID: \"ceb692e6-ac6d-4ceb-b514-777eddf7c439\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz"
Jul 30 17:12:25.910678 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:25.910658    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ceb692e6-ac6d-4ceb-b514-777eddf7c439-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz\" (UID: \"ceb692e6-ac6d-4ceb-b514-777eddf7c439\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz"
Jul 30 17:12:25.910711 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:25.910699    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ceb692e6-ac6d-4ceb-b514-777eddf7c439-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz\" (UID: \"ceb692e6-ac6d-4ceb-b514-777eddf7c439\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz"
Jul 30 17:12:25.910766 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:25.910750    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ceb692e6-ac6d-4ceb-b514-777eddf7c439-cabundle-cert\") pod \"isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz\" (UID: \"ceb692e6-ac6d-4ceb-b514-777eddf7c439\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz"
Jul 30 17:12:25.912837 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:25.912814    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ceb692e6-ac6d-4ceb-b514-777eddf7c439-proxy-tls\") pod \"isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz\" (UID: \"ceb692e6-ac6d-4ceb-b514-777eddf7c439\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz"
Jul 30 17:12:25.918079 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:25.918050    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d68r5\" (UniqueName: \"kubernetes.io/projected/ceb692e6-ac6d-4ceb-b514-777eddf7c439-kube-api-access-d68r5\") pod \"isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz\" (UID: \"ceb692e6-ac6d-4ceb-b514-777eddf7c439\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz"
Jul 30 17:12:26.010816 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:26.010780    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22be6187-f972-45d4-a410-3f4978fb4425-cabundle-cert\") pod \"isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz\" (UID: \"22be6187-f972-45d4-a410-3f4978fb4425\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz"
Jul 30 17:12:26.011063 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:26.010836    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22be6187-f972-45d4-a410-3f4978fb4425-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz\" (UID: \"22be6187-f972-45d4-a410-3f4978fb4425\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz"
Jul 30 17:12:26.011063 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:26.010865    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk4rc\" (UniqueName: \"kubernetes.io/projected/22be6187-f972-45d4-a410-3f4978fb4425-kube-api-access-kk4rc\") pod \"isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz\" (UID: \"22be6187-f972-45d4-a410-3f4978fb4425\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz"
Jul 30 17:12:26.011063 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:26.010963    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22be6187-f972-45d4-a410-3f4978fb4425-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz\" (UID: \"22be6187-f972-45d4-a410-3f4978fb4425\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz"
Jul 30 17:12:26.011063 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:26.010993    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22be6187-f972-45d4-a410-3f4978fb4425-proxy-tls\") pod \"isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz\" (UID: \"22be6187-f972-45d4-a410-3f4978fb4425\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz"
Jul 30 17:12:26.011331 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:12:26.011097    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-5f73e-predictor-serving-cert: secret "isvc-xgboost-graph-raw-5f73e-predictor-serving-cert" not found
Jul 30 17:12:26.011331 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:12:26.011201    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22be6187-f972-45d4-a410-3f4978fb4425-proxy-tls podName:22be6187-f972-45d4-a410-3f4978fb4425 nodeName:}" failed. No retries permitted until 2026-07-30 17:12:26.511142689 +0000 UTC m=+709.586442558 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/22be6187-f972-45d4-a410-3f4978fb4425-proxy-tls") pod "isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz" (UID: "22be6187-f972-45d4-a410-3f4978fb4425") : secret "isvc-xgboost-graph-raw-5f73e-predictor-serving-cert" not found
Jul 30 17:12:26.011418 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:26.011389    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22be6187-f972-45d4-a410-3f4978fb4425-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz\" (UID: \"22be6187-f972-45d4-a410-3f4978fb4425\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz"
Jul 30 17:12:26.011653 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:26.011635    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22be6187-f972-45d4-a410-3f4978fb4425-cabundle-cert\") pod \"isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz\" (UID: \"22be6187-f972-45d4-a410-3f4978fb4425\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz"
Jul 30 17:12:26.011688 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:26.011649    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22be6187-f972-45d4-a410-3f4978fb4425-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz\" (UID: \"22be6187-f972-45d4-a410-3f4978fb4425\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz"
Jul 30 17:12:26.021055 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:26.021023    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk4rc\" (UniqueName: \"kubernetes.io/projected/22be6187-f972-45d4-a410-3f4978fb4425-kube-api-access-kk4rc\") pod \"isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz\" (UID: \"22be6187-f972-45d4-a410-3f4978fb4425\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz"
Jul 30 17:12:26.061843 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:26.061802    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz"
Jul 30 17:12:26.206233 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:26.206199    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz"]
Jul 30 17:12:26.208743 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:12:26.208714    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceb692e6_ac6d_4ceb_b514_777eddf7c439.slice/crio-54f6ef98cbd04845992f06e56f0aa8439bd726c9fb21472d1ccfb977c0d4cf99 WatchSource:0}: Error finding container 54f6ef98cbd04845992f06e56f0aa8439bd726c9fb21472d1ccfb977c0d4cf99: Status 404 returned error can't find the container with id 54f6ef98cbd04845992f06e56f0aa8439bd726c9fb21472d1ccfb977c0d4cf99
Jul 30 17:12:26.509089 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:26.509053    2580 generic.go:356] "Generic (PLEG): container finished" podID="3aa38bf3-1bcd-474c-a184-4a4ef95950c2" containerID="3ffe49ce4401cdedc229b7fc382fc86fe289ab9fa82d340d2119913720dc4b9c" exitCode=2
Jul 30 17:12:26.509306 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:26.509128    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg" event={"ID":"3aa38bf3-1bcd-474c-a184-4a4ef95950c2","Type":"ContainerDied","Data":"3ffe49ce4401cdedc229b7fc382fc86fe289ab9fa82d340d2119913720dc4b9c"}
Jul 30 17:12:26.510737 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:26.510708    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz" event={"ID":"ceb692e6-ac6d-4ceb-b514-777eddf7c439","Type":"ContainerStarted","Data":"7190dffb0197a4fd97d2397b002e66957b0130b90cc98b9ea9d506a3e1461287"}
Jul 30 17:12:26.510883 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:26.510760    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz" event={"ID":"ceb692e6-ac6d-4ceb-b514-777eddf7c439","Type":"ContainerStarted","Data":"54f6ef98cbd04845992f06e56f0aa8439bd726c9fb21472d1ccfb977c0d4cf99"}
Jul 30 17:12:26.516337 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:26.516314    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22be6187-f972-45d4-a410-3f4978fb4425-proxy-tls\") pod \"isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz\" (UID: \"22be6187-f972-45d4-a410-3f4978fb4425\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz"
Jul 30 17:12:26.518785 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:26.518752    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22be6187-f972-45d4-a410-3f4978fb4425-proxy-tls\") pod \"isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz\" (UID: \"22be6187-f972-45d4-a410-3f4978fb4425\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz"
Jul 30 17:12:26.767380 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:26.767272    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz"
Jul 30 17:12:26.924393 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:26.924355    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz"]
Jul 30 17:12:26.927395 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:12:26.927355    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22be6187_f972_45d4_a410_3f4978fb4425.slice/crio-3d3e4dc8285a2d9634e6642e5792996affb75d85a95e84cd5ce4da881aadc803 WatchSource:0}: Error finding container 3d3e4dc8285a2d9634e6642e5792996affb75d85a95e84cd5ce4da881aadc803: Status 404 returned error can't find the container with id 3d3e4dc8285a2d9634e6642e5792996affb75d85a95e84cd5ce4da881aadc803
Jul 30 17:12:27.480600 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:27.480554    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg" podUID="3aa38bf3-1bcd-474c-a184-4a4ef95950c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 30 17:12:27.480974 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:27.480935    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg" podUID="3aa38bf3-1bcd-474c-a184-4a4ef95950c2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 17:12:27.517124 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:27.517081    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz" event={"ID":"22be6187-f972-45d4-a410-3f4978fb4425","Type":"ContainerStarted","Data":"348505a5aa587af376bb29f193d20cfad4ba7476ed2e607c2d7c8a98a6fad876"}
Jul 30 17:12:27.517342 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:27.517159    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz" event={"ID":"22be6187-f972-45d4-a410-3f4978fb4425","Type":"ContainerStarted","Data":"3d3e4dc8285a2d9634e6642e5792996affb75d85a95e84cd5ce4da881aadc803"}
Jul 30 17:12:29.136424 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:29.136314    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg" podUID="3aa38bf3-1bcd-474c-a184-4a4ef95950c2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jul 30 17:12:30.534088 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:30.534052    2580 generic.go:356] "Generic (PLEG): container finished" podID="ceb692e6-ac6d-4ceb-b514-777eddf7c439" containerID="7190dffb0197a4fd97d2397b002e66957b0130b90cc98b9ea9d506a3e1461287" exitCode=0
Jul 30 17:12:30.534545 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:30.534104    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz" event={"ID":"ceb692e6-ac6d-4ceb-b514-777eddf7c439","Type":"ContainerDied","Data":"7190dffb0197a4fd97d2397b002e66957b0130b90cc98b9ea9d506a3e1461287"}
Jul 30 17:12:31.541215 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:31.541153    2580 generic.go:356] "Generic (PLEG): container finished" podID="3aa38bf3-1bcd-474c-a184-4a4ef95950c2" containerID="2ea5a1afb4a5fd3a5d5fa10f6d660b21af13954ed52af1c10da58d23138f02f3" exitCode=0
Jul 30 17:12:31.541652 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:31.541238    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg" event={"ID":"3aa38bf3-1bcd-474c-a184-4a4ef95950c2","Type":"ContainerDied","Data":"2ea5a1afb4a5fd3a5d5fa10f6d660b21af13954ed52af1c10da58d23138f02f3"}
Jul 30 17:12:31.543387 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:31.543355    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz" event={"ID":"ceb692e6-ac6d-4ceb-b514-777eddf7c439","Type":"ContainerStarted","Data":"b1523b48d22438231e2a873e072a03d13e64acabb9a340ab1e39c0dae544ea8e"}
Jul 30 17:12:31.543546 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:31.543421    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz" event={"ID":"ceb692e6-ac6d-4ceb-b514-777eddf7c439","Type":"ContainerStarted","Data":"d9494acfd405b887aa67776ad56ba833161e25702ed40d85f4eb3eff676a5cbd"}
Jul 30 17:12:31.543700 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:31.543681    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz"
Jul 30 17:12:31.544819 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:31.544797    2580 generic.go:356] "Generic (PLEG): container finished" podID="22be6187-f972-45d4-a410-3f4978fb4425" containerID="348505a5aa587af376bb29f193d20cfad4ba7476ed2e607c2d7c8a98a6fad876" exitCode=0
Jul 30 17:12:31.544943 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:31.544831    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz" event={"ID":"22be6187-f972-45d4-a410-3f4978fb4425","Type":"ContainerDied","Data":"348505a5aa587af376bb29f193d20cfad4ba7476ed2e607c2d7c8a98a6fad876"}
Jul 30 17:12:31.546159 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:31.546143    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 17:12:31.568466 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:31.568403    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz" podStartSLOduration=2.614186241 podStartE2EDuration="6.568382733s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.954196492s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:12:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:12:31.568274336 +0000 UTC m=+714.643574232" watchObservedRunningTime="2026-07-30 17:12:31.568382733 +0000 UTC m=+714.643682620"
Jul 30 17:12:32.550485 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:32.550390    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz"
Jul 30 17:12:32.551988 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:32.551952    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz" podUID="ceb692e6-ac6d-4ceb-b514-777eddf7c439" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 30 17:12:33.557008 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:33.556945    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz" podUID="ceb692e6-ac6d-4ceb-b514-777eddf7c439" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 30 17:12:34.136453 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:34.136400    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg" podUID="3aa38bf3-1bcd-474c-a184-4a4ef95950c2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jul 30 17:12:37.480282 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:37.480048    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg" podUID="3aa38bf3-1bcd-474c-a184-4a4ef95950c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 30 17:12:37.480884 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:37.480568    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg" podUID="3aa38bf3-1bcd-474c-a184-4a4ef95950c2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 17:12:38.562822 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:38.562788    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz"
Jul 30 17:12:38.563418 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:38.563385    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz" podUID="ceb692e6-ac6d-4ceb-b514-777eddf7c439" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 30 17:12:39.136484 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:39.136419    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg" podUID="3aa38bf3-1bcd-474c-a184-4a4ef95950c2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jul 30 17:12:39.136755 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:39.136646    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg"
Jul 30 17:12:44.136509 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:44.136454    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg" podUID="3aa38bf3-1bcd-474c-a184-4a4ef95950c2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jul 30 17:12:47.479632 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:47.479576    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg" podUID="3aa38bf3-1bcd-474c-a184-4a4ef95950c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 30 17:12:47.480154 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:47.479987    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg" podUID="3aa38bf3-1bcd-474c-a184-4a4ef95950c2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 17:12:47.483835 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:47.483807    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg"
Jul 30 17:12:47.484040 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:47.483897    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg"
Jul 30 17:12:48.564277 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:48.564223    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz" podUID="ceb692e6-ac6d-4ceb-b514-777eddf7c439" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 30 17:12:49.136847 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:49.136792    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg" podUID="3aa38bf3-1bcd-474c-a184-4a4ef95950c2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jul 30 17:12:53.656234 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:53.656192    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz" event={"ID":"22be6187-f972-45d4-a410-3f4978fb4425","Type":"ContainerStarted","Data":"68a58546d158aa34955c9440a47172e424e4e8841c5d870be011640294c09abd"}
Jul 30 17:12:53.656682 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:53.656260    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz" event={"ID":"22be6187-f972-45d4-a410-3f4978fb4425","Type":"ContainerStarted","Data":"d87f5ffcdafbca58ca46e2ec1211a13f510a9fb0d960e16140b2a8a755daf89f"}
Jul 30 17:12:53.656682 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:53.656523    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz"
Jul 30 17:12:53.676954 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:53.676888    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz" podStartSLOduration=3.348418408 podStartE2EDuration="28.676870912s" totalImagesPullingTime="21.425237835s" totalInitContainerRuntime="3.903214669s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:12:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:12:53.675743585 +0000 UTC m=+736.751043477" watchObservedRunningTime="2026-07-30 17:12:53.676870912 +0000 UTC m=+736.752170802"
Jul 30 17:12:54.136839 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:54.136790    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg" podUID="3aa38bf3-1bcd-474c-a184-4a4ef95950c2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jul 30 17:12:54.660749 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:54.660716    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz"
Jul 30 17:12:54.662052 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:54.662016    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz" podUID="22be6187-f972-45d4-a410-3f4978fb4425" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 30 17:12:55.665875 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:55.665830    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz" podUID="22be6187-f972-45d4-a410-3f4978fb4425" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 30 17:12:56.397008 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:56.396970    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg"
Jul 30 17:12:56.499699 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:56.499664    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3aa38bf3-1bcd-474c-a184-4a4ef95950c2-proxy-tls\") pod \"3aa38bf3-1bcd-474c-a184-4a4ef95950c2\" (UID: \"3aa38bf3-1bcd-474c-a184-4a4ef95950c2\") "
Jul 30 17:12:56.499898 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:56.499728    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4t77w\" (UniqueName: \"kubernetes.io/projected/3aa38bf3-1bcd-474c-a184-4a4ef95950c2-kube-api-access-4t77w\") pod \"3aa38bf3-1bcd-474c-a184-4a4ef95950c2\" (UID: \"3aa38bf3-1bcd-474c-a184-4a4ef95950c2\") "
Jul 30 17:12:56.499898 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:56.499762    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3aa38bf3-1bcd-474c-a184-4a4ef95950c2-kube-rbac-proxy-sar-config\") pod \"3aa38bf3-1bcd-474c-a184-4a4ef95950c2\" (UID: \"3aa38bf3-1bcd-474c-a184-4a4ef95950c2\") "
Jul 30 17:12:56.499898 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:56.499784    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3aa38bf3-1bcd-474c-a184-4a4ef95950c2-cabundle-cert\") pod \"3aa38bf3-1bcd-474c-a184-4a4ef95950c2\" (UID: \"3aa38bf3-1bcd-474c-a184-4a4ef95950c2\") "
Jul 30 17:12:56.499898 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:56.499830    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3aa38bf3-1bcd-474c-a184-4a4ef95950c2-kserve-provision-location\") pod \"3aa38bf3-1bcd-474c-a184-4a4ef95950c2\" (UID: \"3aa38bf3-1bcd-474c-a184-4a4ef95950c2\") "
Jul 30 17:12:56.500251 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:56.500219    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3aa38bf3-1bcd-474c-a184-4a4ef95950c2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3aa38bf3-1bcd-474c-a184-4a4ef95950c2" (UID: "3aa38bf3-1bcd-474c-a184-4a4ef95950c2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:12:56.500251 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:56.500241    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3aa38bf3-1bcd-474c-a184-4a4ef95950c2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3aa38bf3-1bcd-474c-a184-4a4ef95950c2" (UID: "3aa38bf3-1bcd-474c-a184-4a4ef95950c2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:12:56.500396 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:56.500240    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aa38bf3-1bcd-474c-a184-4a4ef95950c2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3aa38bf3-1bcd-474c-a184-4a4ef95950c2" (UID: "3aa38bf3-1bcd-474c-a184-4a4ef95950c2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 17:12:56.502207 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:56.502156    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3aa38bf3-1bcd-474c-a184-4a4ef95950c2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3aa38bf3-1bcd-474c-a184-4a4ef95950c2" (UID: "3aa38bf3-1bcd-474c-a184-4a4ef95950c2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:12:56.502299 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:56.502235    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aa38bf3-1bcd-474c-a184-4a4ef95950c2-kube-api-access-4t77w" (OuterVolumeSpecName: "kube-api-access-4t77w") pod "3aa38bf3-1bcd-474c-a184-4a4ef95950c2" (UID: "3aa38bf3-1bcd-474c-a184-4a4ef95950c2"). InnerVolumeSpecName "kube-api-access-4t77w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:12:56.600697 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:56.600646    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t77w\" (UniqueName: \"kubernetes.io/projected/3aa38bf3-1bcd-474c-a184-4a4ef95950c2-kube-api-access-4t77w\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:12:56.600697 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:56.600685    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3aa38bf3-1bcd-474c-a184-4a4ef95950c2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:12:56.600697 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:56.600697    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3aa38bf3-1bcd-474c-a184-4a4ef95950c2-cabundle-cert\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:12:56.600697 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:56.600707    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3aa38bf3-1bcd-474c-a184-4a4ef95950c2-kserve-provision-location\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:12:56.601003 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:56.600716    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3aa38bf3-1bcd-474c-a184-4a4ef95950c2-proxy-tls\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:12:56.672323 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:56.672284    2580 generic.go:356] "Generic (PLEG): container finished" podID="3aa38bf3-1bcd-474c-a184-4a4ef95950c2" containerID="7e4335daba2d1efe1f3285902037a10242986fbc9216b1826fe09b55a4df2e97" exitCode=0
Jul 30 17:12:56.672750 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:56.672352    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg"
Jul 30 17:12:56.672750 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:56.672377    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg" event={"ID":"3aa38bf3-1bcd-474c-a184-4a4ef95950c2","Type":"ContainerDied","Data":"7e4335daba2d1efe1f3285902037a10242986fbc9216b1826fe09b55a4df2e97"}
Jul 30 17:12:56.672750 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:56.672447    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg" event={"ID":"3aa38bf3-1bcd-474c-a184-4a4ef95950c2","Type":"ContainerDied","Data":"84dba860bb7e775a3f0315510a1033b89791ab00bf4eb478247d242b9311461f"}
Jul 30 17:12:56.672750 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:56.672484    2580 scope.go:117] "RemoveContainer" containerID="7e4335daba2d1efe1f3285902037a10242986fbc9216b1826fe09b55a4df2e97"
Jul 30 17:12:56.682043 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:56.682019    2580 scope.go:117] "RemoveContainer" containerID="3ffe49ce4401cdedc229b7fc382fc86fe289ab9fa82d340d2119913720dc4b9c"
Jul 30 17:12:56.690398 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:56.690372    2580 scope.go:117] "RemoveContainer" containerID="2ea5a1afb4a5fd3a5d5fa10f6d660b21af13954ed52af1c10da58d23138f02f3"
Jul 30 17:12:56.695721 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:56.695693    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg"]
Jul 30 17:12:56.699433 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:56.699399    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fbbbd-predictor-777bc5d7bb-4n6qg"]
Jul 30 17:12:56.699822 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:56.699805    2580 scope.go:117] "RemoveContainer" containerID="2bdce362f6b0452a54d99a62174ae87571a6f9e3cc024c57c7535134471e8101"
Jul 30 17:12:56.708332 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:56.708222    2580 scope.go:117] "RemoveContainer" containerID="7e4335daba2d1efe1f3285902037a10242986fbc9216b1826fe09b55a4df2e97"
Jul 30 17:12:56.708598 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:12:56.708570    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e4335daba2d1efe1f3285902037a10242986fbc9216b1826fe09b55a4df2e97\": container with ID starting with 7e4335daba2d1efe1f3285902037a10242986fbc9216b1826fe09b55a4df2e97 not found: ID does not exist" containerID="7e4335daba2d1efe1f3285902037a10242986fbc9216b1826fe09b55a4df2e97"
Jul 30 17:12:56.708667 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:56.708610    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e4335daba2d1efe1f3285902037a10242986fbc9216b1826fe09b55a4df2e97"} err="failed to get container status \"7e4335daba2d1efe1f3285902037a10242986fbc9216b1826fe09b55a4df2e97\": rpc error: code = NotFound desc = could not find container \"7e4335daba2d1efe1f3285902037a10242986fbc9216b1826fe09b55a4df2e97\": container with ID starting with 7e4335daba2d1efe1f3285902037a10242986fbc9216b1826fe09b55a4df2e97 not found: ID does not exist"
Jul 30 17:12:56.708667 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:56.708632    2580 scope.go:117] "RemoveContainer" containerID="3ffe49ce4401cdedc229b7fc382fc86fe289ab9fa82d340d2119913720dc4b9c"
Jul 30 17:12:56.708908 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:12:56.708879    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ffe49ce4401cdedc229b7fc382fc86fe289ab9fa82d340d2119913720dc4b9c\": container with ID starting with 3ffe49ce4401cdedc229b7fc382fc86fe289ab9fa82d340d2119913720dc4b9c not found: ID does not exist" containerID="3ffe49ce4401cdedc229b7fc382fc86fe289ab9fa82d340d2119913720dc4b9c"
Jul 30 17:12:56.708950 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:56.708914    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ffe49ce4401cdedc229b7fc382fc86fe289ab9fa82d340d2119913720dc4b9c"} err="failed to get container status \"3ffe49ce4401cdedc229b7fc382fc86fe289ab9fa82d340d2119913720dc4b9c\": rpc error: code = NotFound desc = could not find container \"3ffe49ce4401cdedc229b7fc382fc86fe289ab9fa82d340d2119913720dc4b9c\": container with ID starting with 3ffe49ce4401cdedc229b7fc382fc86fe289ab9fa82d340d2119913720dc4b9c not found: ID does not exist"
Jul 30 17:12:56.708950 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:56.708931    2580 scope.go:117] "RemoveContainer" containerID="2ea5a1afb4a5fd3a5d5fa10f6d660b21af13954ed52af1c10da58d23138f02f3"
Jul 30 17:12:56.709158 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:12:56.709142    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ea5a1afb4a5fd3a5d5fa10f6d660b21af13954ed52af1c10da58d23138f02f3\": container with ID starting with 2ea5a1afb4a5fd3a5d5fa10f6d660b21af13954ed52af1c10da58d23138f02f3 not found: ID does not exist" containerID="2ea5a1afb4a5fd3a5d5fa10f6d660b21af13954ed52af1c10da58d23138f02f3"
Jul 30 17:12:56.709273 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:56.709160    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ea5a1afb4a5fd3a5d5fa10f6d660b21af13954ed52af1c10da58d23138f02f3"} err="failed to get container status \"2ea5a1afb4a5fd3a5d5fa10f6d660b21af13954ed52af1c10da58d23138f02f3\": rpc error: code = NotFound desc = could not find container \"2ea5a1afb4a5fd3a5d5fa10f6d660b21af13954ed52af1c10da58d23138f02f3\": container with ID starting with 2ea5a1afb4a5fd3a5d5fa10f6d660b21af13954ed52af1c10da58d23138f02f3 not found: ID does not exist"
Jul 30 17:12:56.709273 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:56.709265    2580 scope.go:117] "RemoveContainer" containerID="2bdce362f6b0452a54d99a62174ae87571a6f9e3cc024c57c7535134471e8101"
Jul 30 17:12:56.709554 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:12:56.709523    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bdce362f6b0452a54d99a62174ae87571a6f9e3cc024c57c7535134471e8101\": container with ID starting with 2bdce362f6b0452a54d99a62174ae87571a6f9e3cc024c57c7535134471e8101 not found: ID does not exist" containerID="2bdce362f6b0452a54d99a62174ae87571a6f9e3cc024c57c7535134471e8101"
Jul 30 17:12:56.709637 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:56.709549    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bdce362f6b0452a54d99a62174ae87571a6f9e3cc024c57c7535134471e8101"} err="failed to get container status \"2bdce362f6b0452a54d99a62174ae87571a6f9e3cc024c57c7535134471e8101\": rpc error: code = NotFound desc = could not find container \"2bdce362f6b0452a54d99a62174ae87571a6f9e3cc024c57c7535134471e8101\": container with ID starting with 2bdce362f6b0452a54d99a62174ae87571a6f9e3cc024c57c7535134471e8101 not found: ID does not exist"
Jul 30 17:12:57.482226 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:57.482191    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3aa38bf3-1bcd-474c-a184-4a4ef95950c2" path="/var/lib/kubelet/pods/3aa38bf3-1bcd-474c-a184-4a4ef95950c2/volumes"
Jul 30 17:12:58.563992 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:12:58.563949    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz" podUID="ceb692e6-ac6d-4ceb-b514-777eddf7c439" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 30 17:13:00.670867 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:13:00.670835    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz"
Jul 30 17:13:00.671434 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:13:00.671405    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz" podUID="22be6187-f972-45d4-a410-3f4978fb4425" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 30 17:13:08.564286 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:13:08.564242    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz" podUID="ceb692e6-ac6d-4ceb-b514-777eddf7c439" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 30 17:13:10.671917 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:13:10.671870    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz" podUID="22be6187-f972-45d4-a410-3f4978fb4425" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 30 17:13:18.563913 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:13:18.563866    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz" podUID="ceb692e6-ac6d-4ceb-b514-777eddf7c439" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 30 17:13:20.671814 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:13:20.671770    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz" podUID="22be6187-f972-45d4-a410-3f4978fb4425" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 30 17:13:28.564151 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:13:28.564109    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz" podUID="ceb692e6-ac6d-4ceb-b514-777eddf7c439" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 30 17:13:30.671650 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:13:30.671603    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz" podUID="22be6187-f972-45d4-a410-3f4978fb4425" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 30 17:13:38.563748 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:13:38.563701    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz" podUID="ceb692e6-ac6d-4ceb-b514-777eddf7c439" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 30 17:13:40.671749 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:13:40.671698    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz" podUID="22be6187-f972-45d4-a410-3f4978fb4425" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 30 17:13:46.477618 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:13:46.477573    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz" podUID="ceb692e6-ac6d-4ceb-b514-777eddf7c439" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 30 17:13:50.671880 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:13:50.671837    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz" podUID="22be6187-f972-45d4-a410-3f4978fb4425" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 30 17:13:56.478237 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:13:56.478199    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz"
Jul 30 17:14:00.672414 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:00.672321    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz"
Jul 30 17:14:25.999389 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:25.999350    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz"]
Jul 30 17:14:25.999968 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:25.999671    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz" podUID="ceb692e6-ac6d-4ceb-b514-777eddf7c439" containerName="kserve-container" containerID="cri-o://d9494acfd405b887aa67776ad56ba833161e25702ed40d85f4eb3eff676a5cbd" gracePeriod=30
Jul 30 17:14:25.999968 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:25.999706    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz" podUID="ceb692e6-ac6d-4ceb-b514-777eddf7c439" containerName="kube-rbac-proxy" containerID="cri-o://b1523b48d22438231e2a873e072a03d13e64acabb9a340ab1e39c0dae544ea8e" gracePeriod=30
Jul 30 17:14:26.061463 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:26.061428    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb"]
Jul 30 17:14:26.061868 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:26.061850    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3aa38bf3-1bcd-474c-a184-4a4ef95950c2" containerName="storage-initializer"
Jul 30 17:14:26.061972 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:26.061871    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3aa38bf3-1bcd-474c-a184-4a4ef95950c2" containerName="storage-initializer"
Jul 30 17:14:26.061972 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:26.061885    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3aa38bf3-1bcd-474c-a184-4a4ef95950c2" containerName="agent"
Jul 30 17:14:26.061972 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:26.061893    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3aa38bf3-1bcd-474c-a184-4a4ef95950c2" containerName="agent"
Jul 30 17:14:26.061972 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:26.061907    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3aa38bf3-1bcd-474c-a184-4a4ef95950c2" containerName="kserve-container"
Jul 30 17:14:26.061972 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:26.061915    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3aa38bf3-1bcd-474c-a184-4a4ef95950c2" containerName="kserve-container"
Jul 30 17:14:26.061972 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:26.061947    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3aa38bf3-1bcd-474c-a184-4a4ef95950c2" containerName="kube-rbac-proxy"
Jul 30 17:14:26.061972 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:26.061955    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3aa38bf3-1bcd-474c-a184-4a4ef95950c2" containerName="kube-rbac-proxy"
Jul 30 17:14:26.062326 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:26.062039    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3aa38bf3-1bcd-474c-a184-4a4ef95950c2" containerName="kube-rbac-proxy"
Jul 30 17:14:26.062326 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:26.062055    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3aa38bf3-1bcd-474c-a184-4a4ef95950c2" containerName="agent"
Jul 30 17:14:26.062326 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:26.062064    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3aa38bf3-1bcd-474c-a184-4a4ef95950c2" containerName="kserve-container"
Jul 30 17:14:26.064803 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:26.064779    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb"
Jul 30 17:14:26.067352 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:26.067301    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-32669-kube-rbac-proxy-sar-config\""
Jul 30 17:14:26.067509 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:26.067301    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-32669-predictor-serving-cert\""
Jul 30 17:14:26.077830 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:26.077797    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb"]
Jul 30 17:14:26.102589 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:26.102542    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-928c6\" (UniqueName: \"kubernetes.io/projected/89210773-9748-4652-8d4c-80f1c870dfc4-kube-api-access-928c6\") pod \"isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb\" (UID: \"89210773-9748-4652-8d4c-80f1c870dfc4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb"
Jul 30 17:14:26.102816 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:26.102653    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89210773-9748-4652-8d4c-80f1c870dfc4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb\" (UID: \"89210773-9748-4652-8d4c-80f1c870dfc4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb"
Jul 30 17:14:26.102816 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:26.102704    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89210773-9748-4652-8d4c-80f1c870dfc4-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb\" (UID: \"89210773-9748-4652-8d4c-80f1c870dfc4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb"
Jul 30 17:14:26.102816 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:26.102743    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89210773-9748-4652-8d4c-80f1c870dfc4-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb\" (UID: \"89210773-9748-4652-8d4c-80f1c870dfc4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb"
Jul 30 17:14:26.102945 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:26.102829    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89210773-9748-4652-8d4c-80f1c870dfc4-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb\" (UID: \"89210773-9748-4652-8d4c-80f1c870dfc4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb"
Jul 30 17:14:26.143297 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:26.143251    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc"]
Jul 30 17:14:26.146115 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:26.146094    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc"
Jul 30 17:14:26.148916 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:26.148880    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-32669-predictor-serving-cert\""
Jul 30 17:14:26.148916 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:26.148890    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-32669-kube-rbac-proxy-sar-config\""
Jul 30 17:14:26.166192 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:26.166126    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc"]
Jul 30 17:14:26.190194 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:26.190133    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz"]
Jul 30 17:14:26.190490 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:26.190453    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz" podUID="22be6187-f972-45d4-a410-3f4978fb4425" containerName="kserve-container" containerID="cri-o://d87f5ffcdafbca58ca46e2ec1211a13f510a9fb0d960e16140b2a8a755daf89f" gracePeriod=30
Jul 30 17:14:26.190608 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:26.190455    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz" podUID="22be6187-f972-45d4-a410-3f4978fb4425" containerName="kube-rbac-proxy" containerID="cri-o://68a58546d158aa34955c9440a47172e424e4e8841c5d870be011640294c09abd" gracePeriod=30
Jul 30 17:14:26.203428 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:26.203398    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00498391-6658-44f7-8284-19244f9c36a9-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc\" (UID: \"00498391-6658-44f7-8284-19244f9c36a9\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc"
Jul 30 17:14:26.203526 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:26.203506    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89210773-9748-4652-8d4c-80f1c870dfc4-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb\" (UID: \"89210773-9748-4652-8d4c-80f1c870dfc4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb"
Jul 30 17:14:26.203584 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:26.203548    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00498391-6658-44f7-8284-19244f9c36a9-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc\" (UID: \"00498391-6658-44f7-8284-19244f9c36a9\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc"
Jul 30 17:14:26.203584 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:26.203576    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-928c6\" (UniqueName: \"kubernetes.io/projected/89210773-9748-4652-8d4c-80f1c870dfc4-kube-api-access-928c6\") pod \"isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb\" (UID: \"89210773-9748-4652-8d4c-80f1c870dfc4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb"
Jul 30 17:14:26.203667 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:26.203610    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghj4d\" (UniqueName: \"kubernetes.io/projected/00498391-6658-44f7-8284-19244f9c36a9-kube-api-access-ghj4d\") pod \"isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc\" (UID: \"00498391-6658-44f7-8284-19244f9c36a9\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc"
Jul 30 17:14:26.203667 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:26.203651    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/00498391-6658-44f7-8284-19244f9c36a9-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc\" (UID: \"00498391-6658-44f7-8284-19244f9c36a9\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc"
Jul 30 17:14:26.203758 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:26.203738    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00498391-6658-44f7-8284-19244f9c36a9-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc\" (UID: \"00498391-6658-44f7-8284-19244f9c36a9\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc"
Jul 30 17:14:26.203798 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:26.203766    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89210773-9748-4652-8d4c-80f1c870dfc4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb\" (UID: \"89210773-9748-4652-8d4c-80f1c870dfc4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb"
Jul 30 17:14:26.203798 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:26.203793    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89210773-9748-4652-8d4c-80f1c870dfc4-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb\" (UID: \"89210773-9748-4652-8d4c-80f1c870dfc4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb"
Jul 30 17:14:26.203895 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:26.203827    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89210773-9748-4652-8d4c-80f1c870dfc4-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb\" (UID: \"89210773-9748-4652-8d4c-80f1c870dfc4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb"
Jul 30 17:14:26.203950 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:26.203918    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89210773-9748-4652-8d4c-80f1c870dfc4-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb\" (UID: \"89210773-9748-4652-8d4c-80f1c870dfc4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb"
Jul 30 17:14:26.204004 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:14:26.203952    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-32669-predictor-serving-cert: secret "isvc-sklearn-graph-raw-hpa-32669-predictor-serving-cert" not found
Jul 30 17:14:26.204056 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:14:26.204048    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89210773-9748-4652-8d4c-80f1c870dfc4-proxy-tls podName:89210773-9748-4652-8d4c-80f1c870dfc4 nodeName:}" failed. No retries permitted until 2026-07-30 17:14:26.704027119 +0000 UTC m=+829.779326987 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/89210773-9748-4652-8d4c-80f1c870dfc4-proxy-tls") pod "isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb" (UID: "89210773-9748-4652-8d4c-80f1c870dfc4") : secret "isvc-sklearn-graph-raw-hpa-32669-predictor-serving-cert" not found
Jul 30 17:14:26.204484 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:26.204461    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89210773-9748-4652-8d4c-80f1c870dfc4-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb\" (UID: \"89210773-9748-4652-8d4c-80f1c870dfc4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb"
Jul 30 17:14:26.204587 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:26.204555    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89210773-9748-4652-8d4c-80f1c870dfc4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb\" (UID: \"89210773-9748-4652-8d4c-80f1c870dfc4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb"
Jul 30 17:14:26.217875 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:26.217845    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-928c6\" (UniqueName: \"kubernetes.io/projected/89210773-9748-4652-8d4c-80f1c870dfc4-kube-api-access-928c6\") pod \"isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb\" (UID: \"89210773-9748-4652-8d4c-80f1c870dfc4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb"
Jul 30 17:14:26.305499 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:26.305408    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00498391-6658-44f7-8284-19244f9c36a9-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc\" (UID: \"00498391-6658-44f7-8284-19244f9c36a9\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc"
Jul 30 17:14:26.305653 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:26.305507    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00498391-6658-44f7-8284-19244f9c36a9-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc\" (UID: \"00498391-6658-44f7-8284-19244f9c36a9\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc"
Jul 30 17:14:26.305653 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:26.305561    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghj4d\" (UniqueName: \"kubernetes.io/projected/00498391-6658-44f7-8284-19244f9c36a9-kube-api-access-ghj4d\") pod \"isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc\" (UID: \"00498391-6658-44f7-8284-19244f9c36a9\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc"
Jul 30 17:14:26.305764 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:26.305683    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/00498391-6658-44f7-8284-19244f9c36a9-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc\" (UID: \"00498391-6658-44f7-8284-19244f9c36a9\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc"
Jul 30 17:14:26.305764 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:26.305749    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00498391-6658-44f7-8284-19244f9c36a9-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc\" (UID: \"00498391-6658-44f7-8284-19244f9c36a9\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc"
Jul 30 17:14:26.306234 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:26.306204    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00498391-6658-44f7-8284-19244f9c36a9-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc\" (UID: \"00498391-6658-44f7-8284-19244f9c36a9\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc"
Jul 30 17:14:26.306458 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:26.306435    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00498391-6658-44f7-8284-19244f9c36a9-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc\" (UID: \"00498391-6658-44f7-8284-19244f9c36a9\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc"
Jul 30 17:14:26.306691 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:26.306666    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/00498391-6658-44f7-8284-19244f9c36a9-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc\" (UID: \"00498391-6658-44f7-8284-19244f9c36a9\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc"
Jul 30 17:14:26.307829 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:26.307810    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00498391-6658-44f7-8284-19244f9c36a9-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc\" (UID: \"00498391-6658-44f7-8284-19244f9c36a9\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc"
Jul 30 17:14:26.314367 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:26.314334    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghj4d\" (UniqueName: \"kubernetes.io/projected/00498391-6658-44f7-8284-19244f9c36a9-kube-api-access-ghj4d\") pod \"isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc\" (UID: \"00498391-6658-44f7-8284-19244f9c36a9\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc"
Jul 30 17:14:26.457203 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:26.457134    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc"
Jul 30 17:14:26.477681 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:26.477637    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz" podUID="ceb692e6-ac6d-4ceb-b514-777eddf7c439" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 30 17:14:26.604761 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:26.604730    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc"]
Jul 30 17:14:26.607575 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:14:26.607543    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00498391_6658_44f7_8284_19244f9c36a9.slice/crio-2baaaa95dbb4d82825c66117ee2cf0ac50d3e99435ebdef1178321cc904361cb WatchSource:0}: Error finding container 2baaaa95dbb4d82825c66117ee2cf0ac50d3e99435ebdef1178321cc904361cb: Status 404 returned error can't find the container with id 2baaaa95dbb4d82825c66117ee2cf0ac50d3e99435ebdef1178321cc904361cb
Jul 30 17:14:26.709064 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:26.709020    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89210773-9748-4652-8d4c-80f1c870dfc4-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb\" (UID: \"89210773-9748-4652-8d4c-80f1c870dfc4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb"
Jul 30 17:14:26.711499 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:26.711470    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89210773-9748-4652-8d4c-80f1c870dfc4-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb\" (UID: \"89210773-9748-4652-8d4c-80f1c870dfc4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb"
Jul 30 17:14:26.978158 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:26.978043    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb"
Jul 30 17:14:27.049626 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:27.049519    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc" event={"ID":"00498391-6658-44f7-8284-19244f9c36a9","Type":"ContainerStarted","Data":"1fa7b3bc4c868ae0da587852b8c98b73a15b71dc030246de9212c2f0935f4f30"}
Jul 30 17:14:27.049626 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:27.049591    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc" event={"ID":"00498391-6658-44f7-8284-19244f9c36a9","Type":"ContainerStarted","Data":"2baaaa95dbb4d82825c66117ee2cf0ac50d3e99435ebdef1178321cc904361cb"}
Jul 30 17:14:27.052119 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:27.052084    2580 generic.go:356] "Generic (PLEG): container finished" podID="ceb692e6-ac6d-4ceb-b514-777eddf7c439" containerID="b1523b48d22438231e2a873e072a03d13e64acabb9a340ab1e39c0dae544ea8e" exitCode=2
Jul 30 17:14:27.052370 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:27.052148    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz" event={"ID":"ceb692e6-ac6d-4ceb-b514-777eddf7c439","Type":"ContainerDied","Data":"b1523b48d22438231e2a873e072a03d13e64acabb9a340ab1e39c0dae544ea8e"}
Jul 30 17:14:27.055119 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:27.055090    2580 generic.go:356] "Generic (PLEG): container finished" podID="22be6187-f972-45d4-a410-3f4978fb4425" containerID="68a58546d158aa34955c9440a47172e424e4e8841c5d870be011640294c09abd" exitCode=2
Jul 30 17:14:27.055345 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:27.055156    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz" event={"ID":"22be6187-f972-45d4-a410-3f4978fb4425","Type":"ContainerDied","Data":"68a58546d158aa34955c9440a47172e424e4e8841c5d870be011640294c09abd"}
Jul 30 17:14:27.131107 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:27.130907    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb"]
Jul 30 17:14:27.133967 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:14:27.133930    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89210773_9748_4652_8d4c_80f1c870dfc4.slice/crio-7a5bc412298f74957a9142726430d1cbd2b898b3bc9387d326d4e11dbdf16105 WatchSource:0}: Error finding container 7a5bc412298f74957a9142726430d1cbd2b898b3bc9387d326d4e11dbdf16105: Status 404 returned error can't find the container with id 7a5bc412298f74957a9142726430d1cbd2b898b3bc9387d326d4e11dbdf16105
Jul 30 17:14:28.061563 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:28.061521    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb" event={"ID":"89210773-9748-4652-8d4c-80f1c870dfc4","Type":"ContainerStarted","Data":"0b7ad9764a3bb1b0bd6ecf511b82f82cefe89d257ab2218de73c7a5cd11da335"}
Jul 30 17:14:28.061995 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:28.061597    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb" event={"ID":"89210773-9748-4652-8d4c-80f1c870dfc4","Type":"ContainerStarted","Data":"7a5bc412298f74957a9142726430d1cbd2b898b3bc9387d326d4e11dbdf16105"}
Jul 30 17:14:28.557643 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:28.557592    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz" podUID="ceb692e6-ac6d-4ceb-b514-777eddf7c439" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jul 30 17:14:30.752598 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:30.752570    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz"
Jul 30 17:14:30.850109 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:30.850063    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kk4rc\" (UniqueName: \"kubernetes.io/projected/22be6187-f972-45d4-a410-3f4978fb4425-kube-api-access-kk4rc\") pod \"22be6187-f972-45d4-a410-3f4978fb4425\" (UID: \"22be6187-f972-45d4-a410-3f4978fb4425\") "
Jul 30 17:14:30.850109 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:30.850110    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22be6187-f972-45d4-a410-3f4978fb4425-kube-rbac-proxy-sar-config\") pod \"22be6187-f972-45d4-a410-3f4978fb4425\" (UID: \"22be6187-f972-45d4-a410-3f4978fb4425\") "
Jul 30 17:14:30.850370 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:30.850144    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22be6187-f972-45d4-a410-3f4978fb4425-proxy-tls\") pod \"22be6187-f972-45d4-a410-3f4978fb4425\" (UID: \"22be6187-f972-45d4-a410-3f4978fb4425\") "
Jul 30 17:14:30.850370 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:30.850195    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22be6187-f972-45d4-a410-3f4978fb4425-kserve-provision-location\") pod \"22be6187-f972-45d4-a410-3f4978fb4425\" (UID: \"22be6187-f972-45d4-a410-3f4978fb4425\") "
Jul 30 17:14:30.850370 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:30.850221    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22be6187-f972-45d4-a410-3f4978fb4425-cabundle-cert\") pod \"22be6187-f972-45d4-a410-3f4978fb4425\" (UID: \"22be6187-f972-45d4-a410-3f4978fb4425\") "
Jul 30 17:14:30.850577 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:30.850547    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22be6187-f972-45d4-a410-3f4978fb4425-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "22be6187-f972-45d4-a410-3f4978fb4425" (UID: "22be6187-f972-45d4-a410-3f4978fb4425"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 17:14:30.850619 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:30.850586    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22be6187-f972-45d4-a410-3f4978fb4425-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "22be6187-f972-45d4-a410-3f4978fb4425" (UID: "22be6187-f972-45d4-a410-3f4978fb4425"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:14:30.850619 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:30.850606    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22be6187-f972-45d4-a410-3f4978fb4425-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "22be6187-f972-45d4-a410-3f4978fb4425" (UID: "22be6187-f972-45d4-a410-3f4978fb4425"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:14:30.852423 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:30.852394    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22be6187-f972-45d4-a410-3f4978fb4425-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "22be6187-f972-45d4-a410-3f4978fb4425" (UID: "22be6187-f972-45d4-a410-3f4978fb4425"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:14:30.852560 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:30.852457    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22be6187-f972-45d4-a410-3f4978fb4425-kube-api-access-kk4rc" (OuterVolumeSpecName: "kube-api-access-kk4rc") pod "22be6187-f972-45d4-a410-3f4978fb4425" (UID: "22be6187-f972-45d4-a410-3f4978fb4425"). InnerVolumeSpecName "kube-api-access-kk4rc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:14:30.951079 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:30.951036    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22be6187-f972-45d4-a410-3f4978fb4425-kserve-provision-location\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:14:30.951079 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:30.951069    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22be6187-f972-45d4-a410-3f4978fb4425-cabundle-cert\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:14:30.951079 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:30.951080    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kk4rc\" (UniqueName: \"kubernetes.io/projected/22be6187-f972-45d4-a410-3f4978fb4425-kube-api-access-kk4rc\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:14:30.951079 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:30.951089    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22be6187-f972-45d4-a410-3f4978fb4425-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:14:30.951403 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:30.951099    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22be6187-f972-45d4-a410-3f4978fb4425-proxy-tls\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:14:31.078123 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:31.077999    2580 generic.go:356] "Generic (PLEG): container finished" podID="22be6187-f972-45d4-a410-3f4978fb4425" containerID="d87f5ffcdafbca58ca46e2ec1211a13f510a9fb0d960e16140b2a8a755daf89f" exitCode=0
Jul 30 17:14:31.078123 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:31.078088    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz"
Jul 30 17:14:31.078123 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:31.078094    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz" event={"ID":"22be6187-f972-45d4-a410-3f4978fb4425","Type":"ContainerDied","Data":"d87f5ffcdafbca58ca46e2ec1211a13f510a9fb0d960e16140b2a8a755daf89f"}
Jul 30 17:14:31.078446 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:31.078214    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz" event={"ID":"22be6187-f972-45d4-a410-3f4978fb4425","Type":"ContainerDied","Data":"3d3e4dc8285a2d9634e6642e5792996affb75d85a95e84cd5ce4da881aadc803"}
Jul 30 17:14:31.078446 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:31.078248    2580 scope.go:117] "RemoveContainer" containerID="68a58546d158aa34955c9440a47172e424e4e8841c5d870be011640294c09abd"
Jul 30 17:14:31.080150 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:31.080120    2580 generic.go:356] "Generic (PLEG): container finished" podID="00498391-6658-44f7-8284-19244f9c36a9" containerID="1fa7b3bc4c868ae0da587852b8c98b73a15b71dc030246de9212c2f0935f4f30" exitCode=0
Jul 30 17:14:31.080310 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:31.080204    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc" event={"ID":"00498391-6658-44f7-8284-19244f9c36a9","Type":"ContainerDied","Data":"1fa7b3bc4c868ae0da587852b8c98b73a15b71dc030246de9212c2f0935f4f30"}
Jul 30 17:14:31.088474 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:31.088122    2580 scope.go:117] "RemoveContainer" containerID="d87f5ffcdafbca58ca46e2ec1211a13f510a9fb0d960e16140b2a8a755daf89f"
Jul 30 17:14:31.096460 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:31.096428    2580 scope.go:117] "RemoveContainer" containerID="348505a5aa587af376bb29f193d20cfad4ba7476ed2e607c2d7c8a98a6fad876"
Jul 30 17:14:31.106184 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:31.106147    2580 scope.go:117] "RemoveContainer" containerID="68a58546d158aa34955c9440a47172e424e4e8841c5d870be011640294c09abd"
Jul 30 17:14:31.106542 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:14:31.106513    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68a58546d158aa34955c9440a47172e424e4e8841c5d870be011640294c09abd\": container with ID starting with 68a58546d158aa34955c9440a47172e424e4e8841c5d870be011640294c09abd not found: ID does not exist" containerID="68a58546d158aa34955c9440a47172e424e4e8841c5d870be011640294c09abd"
Jul 30 17:14:31.106600 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:31.106557    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68a58546d158aa34955c9440a47172e424e4e8841c5d870be011640294c09abd"} err="failed to get container status \"68a58546d158aa34955c9440a47172e424e4e8841c5d870be011640294c09abd\": rpc error: code = NotFound desc = could not find container \"68a58546d158aa34955c9440a47172e424e4e8841c5d870be011640294c09abd\": container with ID starting with 68a58546d158aa34955c9440a47172e424e4e8841c5d870be011640294c09abd not found: ID does not exist"
Jul 30 17:14:31.106600 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:31.106583    2580 scope.go:117] "RemoveContainer" containerID="d87f5ffcdafbca58ca46e2ec1211a13f510a9fb0d960e16140b2a8a755daf89f"
Jul 30 17:14:31.106852 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:14:31.106834    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d87f5ffcdafbca58ca46e2ec1211a13f510a9fb0d960e16140b2a8a755daf89f\": container with ID starting with d87f5ffcdafbca58ca46e2ec1211a13f510a9fb0d960e16140b2a8a755daf89f not found: ID does not exist" containerID="d87f5ffcdafbca58ca46e2ec1211a13f510a9fb0d960e16140b2a8a755daf89f"
Jul 30 17:14:31.106921 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:31.106864    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d87f5ffcdafbca58ca46e2ec1211a13f510a9fb0d960e16140b2a8a755daf89f"} err="failed to get container status \"d87f5ffcdafbca58ca46e2ec1211a13f510a9fb0d960e16140b2a8a755daf89f\": rpc error: code = NotFound desc = could not find container \"d87f5ffcdafbca58ca46e2ec1211a13f510a9fb0d960e16140b2a8a755daf89f\": container with ID starting with d87f5ffcdafbca58ca46e2ec1211a13f510a9fb0d960e16140b2a8a755daf89f not found: ID does not exist"
Jul 30 17:14:31.106921 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:31.106887    2580 scope.go:117] "RemoveContainer" containerID="348505a5aa587af376bb29f193d20cfad4ba7476ed2e607c2d7c8a98a6fad876"
Jul 30 17:14:31.107143 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:14:31.107125    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"348505a5aa587af376bb29f193d20cfad4ba7476ed2e607c2d7c8a98a6fad876\": container with ID starting with 348505a5aa587af376bb29f193d20cfad4ba7476ed2e607c2d7c8a98a6fad876 not found: ID does not exist" containerID="348505a5aa587af376bb29f193d20cfad4ba7476ed2e607c2d7c8a98a6fad876"
Jul 30 17:14:31.107236 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:31.107151    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"348505a5aa587af376bb29f193d20cfad4ba7476ed2e607c2d7c8a98a6fad876"} err="failed to get container status \"348505a5aa587af376bb29f193d20cfad4ba7476ed2e607c2d7c8a98a6fad876\": rpc error: code = NotFound desc = could not find container \"348505a5aa587af376bb29f193d20cfad4ba7476ed2e607c2d7c8a98a6fad876\": container with ID starting with 348505a5aa587af376bb29f193d20cfad4ba7476ed2e607c2d7c8a98a6fad876 not found: ID does not exist"
Jul 30 17:14:31.115733 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:31.115698    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz"]
Jul 30 17:14:31.121268 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:31.121236    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz"]
Jul 30 17:14:31.482196 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:31.482090    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22be6187-f972-45d4-a410-3f4978fb4425" path="/var/lib/kubelet/pods/22be6187-f972-45d4-a410-3f4978fb4425/volumes"
Jul 30 17:14:31.666584 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:31.666533    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz" podUID="22be6187-f972-45d4-a410-3f4978fb4425" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": context deadline exceeded"
Jul 30 17:14:31.672485 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:31.672449    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5f73e-predictor-5f7bcf7969-qzkhz" podUID="22be6187-f972-45d4-a410-3f4978fb4425" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: i/o timeout"
Jul 30 17:14:32.059672 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:32.059646    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz"
Jul 30 17:14:32.087607 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:32.087525    2580 generic.go:356] "Generic (PLEG): container finished" podID="89210773-9748-4652-8d4c-80f1c870dfc4" containerID="0b7ad9764a3bb1b0bd6ecf511b82f82cefe89d257ab2218de73c7a5cd11da335" exitCode=0
Jul 30 17:14:32.087756 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:32.087601    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb" event={"ID":"89210773-9748-4652-8d4c-80f1c870dfc4","Type":"ContainerDied","Data":"0b7ad9764a3bb1b0bd6ecf511b82f82cefe89d257ab2218de73c7a5cd11da335"}
Jul 30 17:14:32.090291 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:32.090262    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc" event={"ID":"00498391-6658-44f7-8284-19244f9c36a9","Type":"ContainerStarted","Data":"73ef971cc014f9b49d7f134dbb910cb00098ddfb3e566cc3ad02003b4e0e8926"}
Jul 30 17:14:32.090423 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:32.090339    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc" event={"ID":"00498391-6658-44f7-8284-19244f9c36a9","Type":"ContainerStarted","Data":"1318865d608c3632b79ab5d16de0e880cdee35cc988676f66c60dbd9b72a87ef"}
Jul 30 17:14:32.090589 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:32.090562    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc"
Jul 30 17:14:32.092629 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:32.092601    2580 generic.go:356] "Generic (PLEG): container finished" podID="ceb692e6-ac6d-4ceb-b514-777eddf7c439" containerID="d9494acfd405b887aa67776ad56ba833161e25702ed40d85f4eb3eff676a5cbd" exitCode=0
Jul 30 17:14:32.092757 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:32.092656    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz"
Jul 30 17:14:32.092901 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:32.092658    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz" event={"ID":"ceb692e6-ac6d-4ceb-b514-777eddf7c439","Type":"ContainerDied","Data":"d9494acfd405b887aa67776ad56ba833161e25702ed40d85f4eb3eff676a5cbd"}
Jul 30 17:14:32.092901 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:32.092849    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz" event={"ID":"ceb692e6-ac6d-4ceb-b514-777eddf7c439","Type":"ContainerDied","Data":"54f6ef98cbd04845992f06e56f0aa8439bd726c9fb21472d1ccfb977c0d4cf99"}
Jul 30 17:14:32.092901 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:32.092882    2580 scope.go:117] "RemoveContainer" containerID="b1523b48d22438231e2a873e072a03d13e64acabb9a340ab1e39c0dae544ea8e"
Jul 30 17:14:32.102400 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:32.102378    2580 scope.go:117] "RemoveContainer" containerID="d9494acfd405b887aa67776ad56ba833161e25702ed40d85f4eb3eff676a5cbd"
Jul 30 17:14:32.120143 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:32.120120    2580 scope.go:117] "RemoveContainer" containerID="7190dffb0197a4fd97d2397b002e66957b0130b90cc98b9ea9d506a3e1461287"
Jul 30 17:14:32.126276 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:32.126195    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc" podStartSLOduration=2.306590897 podStartE2EDuration="6.126154854s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.819563957s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:14:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:14:32.125519573 +0000 UTC m=+835.200819464" watchObservedRunningTime="2026-07-30 17:14:32.126154854 +0000 UTC m=+835.201454748"
Jul 30 17:14:32.132564 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:32.132540    2580 scope.go:117] "RemoveContainer" containerID="b1523b48d22438231e2a873e072a03d13e64acabb9a340ab1e39c0dae544ea8e"
Jul 30 17:14:32.132955 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:14:32.132931    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1523b48d22438231e2a873e072a03d13e64acabb9a340ab1e39c0dae544ea8e\": container with ID starting with b1523b48d22438231e2a873e072a03d13e64acabb9a340ab1e39c0dae544ea8e not found: ID does not exist" containerID="b1523b48d22438231e2a873e072a03d13e64acabb9a340ab1e39c0dae544ea8e"
Jul 30 17:14:32.133040 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:32.132967    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1523b48d22438231e2a873e072a03d13e64acabb9a340ab1e39c0dae544ea8e"} err="failed to get container status \"b1523b48d22438231e2a873e072a03d13e64acabb9a340ab1e39c0dae544ea8e\": rpc error: code = NotFound desc = could not find container \"b1523b48d22438231e2a873e072a03d13e64acabb9a340ab1e39c0dae544ea8e\": container with ID starting with b1523b48d22438231e2a873e072a03d13e64acabb9a340ab1e39c0dae544ea8e not found: ID does not exist"
Jul 30 17:14:32.133040 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:32.132996    2580 scope.go:117] "RemoveContainer" containerID="d9494acfd405b887aa67776ad56ba833161e25702ed40d85f4eb3eff676a5cbd"
Jul 30 17:14:32.133365 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:14:32.133346    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9494acfd405b887aa67776ad56ba833161e25702ed40d85f4eb3eff676a5cbd\": container with ID starting with d9494acfd405b887aa67776ad56ba833161e25702ed40d85f4eb3eff676a5cbd not found: ID does not exist" containerID="d9494acfd405b887aa67776ad56ba833161e25702ed40d85f4eb3eff676a5cbd"
Jul 30 17:14:32.133429 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:32.133374    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9494acfd405b887aa67776ad56ba833161e25702ed40d85f4eb3eff676a5cbd"} err="failed to get container status \"d9494acfd405b887aa67776ad56ba833161e25702ed40d85f4eb3eff676a5cbd\": rpc error: code = NotFound desc = could not find container \"d9494acfd405b887aa67776ad56ba833161e25702ed40d85f4eb3eff676a5cbd\": container with ID starting with d9494acfd405b887aa67776ad56ba833161e25702ed40d85f4eb3eff676a5cbd not found: ID does not exist"
Jul 30 17:14:32.133429 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:32.133391    2580 scope.go:117] "RemoveContainer" containerID="7190dffb0197a4fd97d2397b002e66957b0130b90cc98b9ea9d506a3e1461287"
Jul 30 17:14:32.133716 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:14:32.133692    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7190dffb0197a4fd97d2397b002e66957b0130b90cc98b9ea9d506a3e1461287\": container with ID starting with 7190dffb0197a4fd97d2397b002e66957b0130b90cc98b9ea9d506a3e1461287 not found: ID does not exist" containerID="7190dffb0197a4fd97d2397b002e66957b0130b90cc98b9ea9d506a3e1461287"
Jul 30 17:14:32.133791 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:32.133716    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7190dffb0197a4fd97d2397b002e66957b0130b90cc98b9ea9d506a3e1461287"} err="failed to get container status \"7190dffb0197a4fd97d2397b002e66957b0130b90cc98b9ea9d506a3e1461287\": rpc error: code = NotFound desc = could not find container \"7190dffb0197a4fd97d2397b002e66957b0130b90cc98b9ea9d506a3e1461287\": container with ID starting with 7190dffb0197a4fd97d2397b002e66957b0130b90cc98b9ea9d506a3e1461287 not found: ID does not exist"
Jul 30 17:14:32.162696 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:32.162670    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ceb692e6-ac6d-4ceb-b514-777eddf7c439-cabundle-cert\") pod \"ceb692e6-ac6d-4ceb-b514-777eddf7c439\" (UID: \"ceb692e6-ac6d-4ceb-b514-777eddf7c439\") "
Jul 30 17:14:32.162817 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:32.162722    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ceb692e6-ac6d-4ceb-b514-777eddf7c439-kube-rbac-proxy-sar-config\") pod \"ceb692e6-ac6d-4ceb-b514-777eddf7c439\" (UID: \"ceb692e6-ac6d-4ceb-b514-777eddf7c439\") "
Jul 30 17:14:32.162817 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:32.162773    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ceb692e6-ac6d-4ceb-b514-777eddf7c439-proxy-tls\") pod \"ceb692e6-ac6d-4ceb-b514-777eddf7c439\" (UID: \"ceb692e6-ac6d-4ceb-b514-777eddf7c439\") "
Jul 30 17:14:32.162817 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:32.162800    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d68r5\" (UniqueName: \"kubernetes.io/projected/ceb692e6-ac6d-4ceb-b514-777eddf7c439-kube-api-access-d68r5\") pod \"ceb692e6-ac6d-4ceb-b514-777eddf7c439\" (UID: \"ceb692e6-ac6d-4ceb-b514-777eddf7c439\") "
Jul 30 17:14:32.162996 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:32.162934    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ceb692e6-ac6d-4ceb-b514-777eddf7c439-kserve-provision-location\") pod \"ceb692e6-ac6d-4ceb-b514-777eddf7c439\" (UID: \"ceb692e6-ac6d-4ceb-b514-777eddf7c439\") "
Jul 30 17:14:32.163614 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:32.163223    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceb692e6-ac6d-4ceb-b514-777eddf7c439-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ceb692e6-ac6d-4ceb-b514-777eddf7c439" (UID: "ceb692e6-ac6d-4ceb-b514-777eddf7c439"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:14:32.163614 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:32.163221    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceb692e6-ac6d-4ceb-b514-777eddf7c439-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ceb692e6-ac6d-4ceb-b514-777eddf7c439" (UID: "ceb692e6-ac6d-4ceb-b514-777eddf7c439"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:14:32.163614 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:32.163256    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceb692e6-ac6d-4ceb-b514-777eddf7c439-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ceb692e6-ac6d-4ceb-b514-777eddf7c439" (UID: "ceb692e6-ac6d-4ceb-b514-777eddf7c439"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 17:14:32.163614 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:32.163416    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ceb692e6-ac6d-4ceb-b514-777eddf7c439-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:14:32.163614 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:32.163436    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ceb692e6-ac6d-4ceb-b514-777eddf7c439-kserve-provision-location\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:14:32.163614 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:32.163452    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ceb692e6-ac6d-4ceb-b514-777eddf7c439-cabundle-cert\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:14:32.165525 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:32.165430    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ceb692e6-ac6d-4ceb-b514-777eddf7c439-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ceb692e6-ac6d-4ceb-b514-777eddf7c439" (UID: "ceb692e6-ac6d-4ceb-b514-777eddf7c439"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:14:32.165525 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:32.165453    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceb692e6-ac6d-4ceb-b514-777eddf7c439-kube-api-access-d68r5" (OuterVolumeSpecName: "kube-api-access-d68r5") pod "ceb692e6-ac6d-4ceb-b514-777eddf7c439" (UID: "ceb692e6-ac6d-4ceb-b514-777eddf7c439"). InnerVolumeSpecName "kube-api-access-d68r5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:14:32.264091 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:32.264046    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ceb692e6-ac6d-4ceb-b514-777eddf7c439-proxy-tls\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:14:32.264091 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:32.264080    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d68r5\" (UniqueName: \"kubernetes.io/projected/ceb692e6-ac6d-4ceb-b514-777eddf7c439-kube-api-access-d68r5\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:14:32.416373 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:32.416329    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz"]
Jul 30 17:14:32.418258 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:32.418227    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-5f73e-predictor-5d4b967878-whvsz"]
Jul 30 17:14:33.101652 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:33.101607    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb" event={"ID":"89210773-9748-4652-8d4c-80f1c870dfc4","Type":"ContainerStarted","Data":"e4ca173d80aa9c3077e9c0e7f81aa78a8f6172a01ed7bd1fcf49856bbb8681c9"}
Jul 30 17:14:33.102102 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:33.101687    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb" event={"ID":"89210773-9748-4652-8d4c-80f1c870dfc4","Type":"ContainerStarted","Data":"e0a79ebf9cd697d3b2801596aed5acfb2d500a6a1a2e9f2c1f88c2d3eb022e2a"}
Jul 30 17:14:33.102102 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:33.102048    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb"
Jul 30 17:14:33.102265 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:33.102206    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb"
Jul 30 17:14:33.103153 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:33.103127    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc"
Jul 30 17:14:33.103684 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:33.103661    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb" podUID="89210773-9748-4652-8d4c-80f1c870dfc4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 30 17:14:33.104197 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:33.104158    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc" podUID="00498391-6658-44f7-8284-19244f9c36a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 30 17:14:33.123062 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:33.122990    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb" podStartSLOduration=3.187779608 podStartE2EDuration="7.12296564s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.935186032s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:14:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:14:33.122770188 +0000 UTC m=+836.198070098" watchObservedRunningTime="2026-07-30 17:14:33.12296564 +0000 UTC m=+836.198265531"
Jul 30 17:14:33.482930 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:33.482840    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ceb692e6-ac6d-4ceb-b514-777eddf7c439" path="/var/lib/kubelet/pods/ceb692e6-ac6d-4ceb-b514-777eddf7c439/volumes"
Jul 30 17:14:34.107983 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:34.107928    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc" podUID="00498391-6658-44f7-8284-19244f9c36a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 30 17:14:34.108432 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:34.108039    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb" podUID="89210773-9748-4652-8d4c-80f1c870dfc4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 30 17:14:39.113343 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:39.113311    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb"
Jul 30 17:14:39.113907 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:39.113885    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc"
Jul 30 17:14:39.113998 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:39.113952    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb" podUID="89210773-9748-4652-8d4c-80f1c870dfc4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 30 17:14:39.114418 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:39.114386    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc" podUID="00498391-6658-44f7-8284-19244f9c36a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 30 17:14:49.114503 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:49.114456    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc" podUID="00498391-6658-44f7-8284-19244f9c36a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 30 17:14:49.114907 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:49.114455    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb" podUID="89210773-9748-4652-8d4c-80f1c870dfc4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 30 17:14:59.114580 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:59.114515    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb" podUID="89210773-9748-4652-8d4c-80f1c870dfc4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 30 17:14:59.115314 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:14:59.114515    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc" podUID="00498391-6658-44f7-8284-19244f9c36a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 30 17:15:09.114442 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:15:09.114385    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb" podUID="89210773-9748-4652-8d4c-80f1c870dfc4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 30 17:15:09.114442 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:15:09.114416    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc" podUID="00498391-6658-44f7-8284-19244f9c36a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 30 17:15:19.114827 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:15:19.114777    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb" podUID="89210773-9748-4652-8d4c-80f1c870dfc4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 30 17:15:19.115381 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:15:19.114777    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc" podUID="00498391-6658-44f7-8284-19244f9c36a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 30 17:15:29.114487 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:15:29.114437    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb" podUID="89210773-9748-4652-8d4c-80f1c870dfc4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 30 17:15:29.114487 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:15:29.114437    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc" podUID="00498391-6658-44f7-8284-19244f9c36a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 30 17:15:37.349607 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:15:37.349573    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-wqszn_f231a916-1fda-4059-9130-d4112e738ee1/console-operator/1.log"
Jul 30 17:15:37.351034 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:15:37.351010    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-wqszn_f231a916-1fda-4059-9130-d4112e738ee1/console-operator/1.log"
Jul 30 17:15:39.113999 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:15:39.113958    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb" podUID="89210773-9748-4652-8d4c-80f1c870dfc4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 30 17:15:39.115306 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:15:39.115277    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc"
Jul 30 17:15:40.477843 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:15:40.477793    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb" podUID="89210773-9748-4652-8d4c-80f1c870dfc4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 30 17:15:50.477802 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:15:50.477752    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb" podUID="89210773-9748-4652-8d4c-80f1c870dfc4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 30 17:16:00.478907 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:00.478875    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb"
Jul 30 17:16:26.359833 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:26.359791    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-94aa4-predictor-86778f57d5-2hz7w"]
Jul 30 17:16:26.362033 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:26.360410    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22be6187-f972-45d4-a410-3f4978fb4425" containerName="storage-initializer"
Jul 30 17:16:26.362033 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:26.360433    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="22be6187-f972-45d4-a410-3f4978fb4425" containerName="storage-initializer"
Jul 30 17:16:26.362033 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:26.360452    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22be6187-f972-45d4-a410-3f4978fb4425" containerName="kserve-container"
Jul 30 17:16:26.362033 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:26.360462    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="22be6187-f972-45d4-a410-3f4978fb4425" containerName="kserve-container"
Jul 30 17:16:26.362033 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:26.360472    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ceb692e6-ac6d-4ceb-b514-777eddf7c439" containerName="storage-initializer"
Jul 30 17:16:26.362033 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:26.360482    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ceb692e6-ac6d-4ceb-b514-777eddf7c439" containerName="storage-initializer"
Jul 30 17:16:26.362033 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:26.360502    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ceb692e6-ac6d-4ceb-b514-777eddf7c439" containerName="kserve-container"
Jul 30 17:16:26.362033 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:26.360511    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ceb692e6-ac6d-4ceb-b514-777eddf7c439" containerName="kserve-container"
Jul 30 17:16:26.362033 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:26.360524    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ceb692e6-ac6d-4ceb-b514-777eddf7c439" containerName="kube-rbac-proxy"
Jul 30 17:16:26.362033 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:26.360532    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ceb692e6-ac6d-4ceb-b514-777eddf7c439" containerName="kube-rbac-proxy"
Jul 30 17:16:26.362033 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:26.360545    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22be6187-f972-45d4-a410-3f4978fb4425" containerName="kube-rbac-proxy"
Jul 30 17:16:26.362033 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:26.360553    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="22be6187-f972-45d4-a410-3f4978fb4425" containerName="kube-rbac-proxy"
Jul 30 17:16:26.362033 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:26.360674    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ceb692e6-ac6d-4ceb-b514-777eddf7c439" containerName="kserve-container"
Jul 30 17:16:26.362033 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:26.360687    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ceb692e6-ac6d-4ceb-b514-777eddf7c439" containerName="kube-rbac-proxy"
Jul 30 17:16:26.362033 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:26.360697    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="22be6187-f972-45d4-a410-3f4978fb4425" containerName="kserve-container"
Jul 30 17:16:26.362033 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:26.360708    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="22be6187-f972-45d4-a410-3f4978fb4425" containerName="kube-rbac-proxy"
Jul 30 17:16:26.363317 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:26.363295    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-94aa4-predictor-86778f57d5-2hz7w"
Jul 30 17:16:26.365965 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:26.365939    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-94aa4-predictor-serving-cert\""
Jul 30 17:16:26.366108 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:26.365947    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-94aa4-kube-rbac-proxy-sar-config\""
Jul 30 17:16:26.372573 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:26.372549    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb"]
Jul 30 17:16:26.373077 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:26.372960    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb" podUID="89210773-9748-4652-8d4c-80f1c870dfc4" containerName="kserve-container" containerID="cri-o://e0a79ebf9cd697d3b2801596aed5acfb2d500a6a1a2e9f2c1f88c2d3eb022e2a" gracePeriod=30
Jul 30 17:16:26.373077 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:26.372995    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb" podUID="89210773-9748-4652-8d4c-80f1c870dfc4" containerName="kube-rbac-proxy" containerID="cri-o://e4ca173d80aa9c3077e9c0e7f81aa78a8f6172a01ed7bd1fcf49856bbb8681c9" gracePeriod=30
Jul 30 17:16:26.375511 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:26.375484    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-94aa4-predictor-86778f57d5-2hz7w"]
Jul 30 17:16:26.473567 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:26.473527    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc"]
Jul 30 17:16:26.473952 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:26.473805    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc" podUID="00498391-6658-44f7-8284-19244f9c36a9" containerName="kserve-container" containerID="cri-o://1318865d608c3632b79ab5d16de0e880cdee35cc988676f66c60dbd9b72a87ef" gracePeriod=30
Jul 30 17:16:26.473952 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:26.473823    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc" podUID="00498391-6658-44f7-8284-19244f9c36a9" containerName="kube-rbac-proxy" containerID="cri-o://73ef971cc014f9b49d7f134dbb910cb00098ddfb3e566cc3ad02003b4e0e8926" gracePeriod=30
Jul 30 17:16:26.486611 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:26.486577    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e864c7d3-57bd-4355-a494-2912f3f09f74-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-94aa4-predictor-86778f57d5-2hz7w\" (UID: \"e864c7d3-57bd-4355-a494-2912f3f09f74\") " pod="kserve-ci-e2e-test/message-dumper-raw-94aa4-predictor-86778f57d5-2hz7w"
Jul 30 17:16:26.486738 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:26.486718    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xttws\" (UniqueName: \"kubernetes.io/projected/e864c7d3-57bd-4355-a494-2912f3f09f74-kube-api-access-xttws\") pod \"message-dumper-raw-94aa4-predictor-86778f57d5-2hz7w\" (UID: \"e864c7d3-57bd-4355-a494-2912f3f09f74\") " pod="kserve-ci-e2e-test/message-dumper-raw-94aa4-predictor-86778f57d5-2hz7w"
Jul 30 17:16:26.486790 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:26.486758    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e864c7d3-57bd-4355-a494-2912f3f09f74-proxy-tls\") pod \"message-dumper-raw-94aa4-predictor-86778f57d5-2hz7w\" (UID: \"e864c7d3-57bd-4355-a494-2912f3f09f74\") " pod="kserve-ci-e2e-test/message-dumper-raw-94aa4-predictor-86778f57d5-2hz7w"
Jul 30 17:16:26.587662 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:26.587628    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e864c7d3-57bd-4355-a494-2912f3f09f74-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-94aa4-predictor-86778f57d5-2hz7w\" (UID: \"e864c7d3-57bd-4355-a494-2912f3f09f74\") " pod="kserve-ci-e2e-test/message-dumper-raw-94aa4-predictor-86778f57d5-2hz7w"
Jul 30 17:16:26.587831 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:26.587699    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xttws\" (UniqueName: \"kubernetes.io/projected/e864c7d3-57bd-4355-a494-2912f3f09f74-kube-api-access-xttws\") pod \"message-dumper-raw-94aa4-predictor-86778f57d5-2hz7w\" (UID: \"e864c7d3-57bd-4355-a494-2912f3f09f74\") " pod="kserve-ci-e2e-test/message-dumper-raw-94aa4-predictor-86778f57d5-2hz7w"
Jul 30 17:16:26.587831 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:26.587729    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e864c7d3-57bd-4355-a494-2912f3f09f74-proxy-tls\") pod \"message-dumper-raw-94aa4-predictor-86778f57d5-2hz7w\" (UID: \"e864c7d3-57bd-4355-a494-2912f3f09f74\") " pod="kserve-ci-e2e-test/message-dumper-raw-94aa4-predictor-86778f57d5-2hz7w"
Jul 30 17:16:26.587976 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:16:26.587857    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-raw-94aa4-predictor-serving-cert: secret "message-dumper-raw-94aa4-predictor-serving-cert" not found
Jul 30 17:16:26.587976 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:16:26.587923    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e864c7d3-57bd-4355-a494-2912f3f09f74-proxy-tls podName:e864c7d3-57bd-4355-a494-2912f3f09f74 nodeName:}" failed. No retries permitted until 2026-07-30 17:16:27.087900885 +0000 UTC m=+950.163200753 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e864c7d3-57bd-4355-a494-2912f3f09f74-proxy-tls") pod "message-dumper-raw-94aa4-predictor-86778f57d5-2hz7w" (UID: "e864c7d3-57bd-4355-a494-2912f3f09f74") : secret "message-dumper-raw-94aa4-predictor-serving-cert" not found
Jul 30 17:16:26.588450 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:26.588430    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e864c7d3-57bd-4355-a494-2912f3f09f74-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-94aa4-predictor-86778f57d5-2hz7w\" (UID: \"e864c7d3-57bd-4355-a494-2912f3f09f74\") " pod="kserve-ci-e2e-test/message-dumper-raw-94aa4-predictor-86778f57d5-2hz7w"
Jul 30 17:16:26.592290 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:26.592263    2580 generic.go:356] "Generic (PLEG): container finished" podID="89210773-9748-4652-8d4c-80f1c870dfc4" containerID="e4ca173d80aa9c3077e9c0e7f81aa78a8f6172a01ed7bd1fcf49856bbb8681c9" exitCode=2
Jul 30 17:16:26.592451 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:26.592331    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb" event={"ID":"89210773-9748-4652-8d4c-80f1c870dfc4","Type":"ContainerDied","Data":"e4ca173d80aa9c3077e9c0e7f81aa78a8f6172a01ed7bd1fcf49856bbb8681c9"}
Jul 30 17:16:26.594450 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:26.594426    2580 generic.go:356] "Generic (PLEG): container finished" podID="00498391-6658-44f7-8284-19244f9c36a9" containerID="73ef971cc014f9b49d7f134dbb910cb00098ddfb3e566cc3ad02003b4e0e8926" exitCode=2
Jul 30 17:16:26.594561 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:26.594495    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc" event={"ID":"00498391-6658-44f7-8284-19244f9c36a9","Type":"ContainerDied","Data":"73ef971cc014f9b49d7f134dbb910cb00098ddfb3e566cc3ad02003b4e0e8926"}
Jul 30 17:16:26.597563 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:26.597536    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xttws\" (UniqueName: \"kubernetes.io/projected/e864c7d3-57bd-4355-a494-2912f3f09f74-kube-api-access-xttws\") pod \"message-dumper-raw-94aa4-predictor-86778f57d5-2hz7w\" (UID: \"e864c7d3-57bd-4355-a494-2912f3f09f74\") " pod="kserve-ci-e2e-test/message-dumper-raw-94aa4-predictor-86778f57d5-2hz7w"
Jul 30 17:16:27.093185 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:27.093121    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e864c7d3-57bd-4355-a494-2912f3f09f74-proxy-tls\") pod \"message-dumper-raw-94aa4-predictor-86778f57d5-2hz7w\" (UID: \"e864c7d3-57bd-4355-a494-2912f3f09f74\") " pod="kserve-ci-e2e-test/message-dumper-raw-94aa4-predictor-86778f57d5-2hz7w"
Jul 30 17:16:27.095760 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:27.095732    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e864c7d3-57bd-4355-a494-2912f3f09f74-proxy-tls\") pod \"message-dumper-raw-94aa4-predictor-86778f57d5-2hz7w\" (UID: \"e864c7d3-57bd-4355-a494-2912f3f09f74\") " pod="kserve-ci-e2e-test/message-dumper-raw-94aa4-predictor-86778f57d5-2hz7w"
Jul 30 17:16:27.277296 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:27.277247    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-94aa4-predictor-86778f57d5-2hz7w"
Jul 30 17:16:27.422035 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:27.422008    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-94aa4-predictor-86778f57d5-2hz7w"]
Jul 30 17:16:27.424968 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:16:27.424929    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode864c7d3_57bd_4355_a494_2912f3f09f74.slice/crio-67845f2a852feb206f43c7e2346efb890c669c37dcb71cf9053526b2aa14f41b WatchSource:0}: Error finding container 67845f2a852feb206f43c7e2346efb890c669c37dcb71cf9053526b2aa14f41b: Status 404 returned error can't find the container with id 67845f2a852feb206f43c7e2346efb890c669c37dcb71cf9053526b2aa14f41b
Jul 30 17:16:27.600346 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:27.600300    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-94aa4-predictor-86778f57d5-2hz7w" event={"ID":"e864c7d3-57bd-4355-a494-2912f3f09f74","Type":"ContainerStarted","Data":"67845f2a852feb206f43c7e2346efb890c669c37dcb71cf9053526b2aa14f41b"}
Jul 30 17:16:29.108854 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:29.108802    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc" podUID="00498391-6658-44f7-8284-19244f9c36a9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Jul 30 17:16:29.109302 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:29.108802    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb" podUID="89210773-9748-4652-8d4c-80f1c870dfc4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 30 17:16:29.114707 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:29.114667    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc" podUID="00498391-6658-44f7-8284-19244f9c36a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 30 17:16:29.610959 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:29.610915    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-94aa4-predictor-86778f57d5-2hz7w" event={"ID":"e864c7d3-57bd-4355-a494-2912f3f09f74","Type":"ContainerStarted","Data":"4b6c228227898b0d9bfe4398bffc0cfb72f1cba7f4ddb7a8c702f4522e56670f"}
Jul 30 17:16:29.611207 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:29.610998    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-94aa4-predictor-86778f57d5-2hz7w" event={"ID":"e864c7d3-57bd-4355-a494-2912f3f09f74","Type":"ContainerStarted","Data":"2a1b63b89ee1fced3c1d0e2573a0150107d6a676fa4621436d0999a8dd8a19e9"}
Jul 30 17:16:29.611207 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:29.611026    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-94aa4-predictor-86778f57d5-2hz7w"
Jul 30 17:16:29.629578 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:29.629492    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-94aa4-predictor-86778f57d5-2hz7w" podStartSLOduration=2.363074355 podStartE2EDuration="3.629466804s" totalImagesPullingTime="1.266392449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:16:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:16:29.628677428 +0000 UTC m=+952.703977319" watchObservedRunningTime="2026-07-30 17:16:29.629466804 +0000 UTC m=+952.704766700"
Jul 30 17:16:30.478367 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:30.478313    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb" podUID="89210773-9748-4652-8d4c-80f1c870dfc4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 30 17:16:30.615927 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:30.615885    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-94aa4-predictor-86778f57d5-2hz7w"
Jul 30 17:16:30.617849 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:30.617824    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-94aa4-predictor-86778f57d5-2hz7w"
Jul 30 17:16:31.130409 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:31.130380    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc"
Jul 30 17:16:31.229139 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:31.229028    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ghj4d\" (UniqueName: \"kubernetes.io/projected/00498391-6658-44f7-8284-19244f9c36a9-kube-api-access-ghj4d\") pod \"00498391-6658-44f7-8284-19244f9c36a9\" (UID: \"00498391-6658-44f7-8284-19244f9c36a9\") "
Jul 30 17:16:31.229139 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:31.229072    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00498391-6658-44f7-8284-19244f9c36a9-proxy-tls\") pod \"00498391-6658-44f7-8284-19244f9c36a9\" (UID: \"00498391-6658-44f7-8284-19244f9c36a9\") "
Jul 30 17:16:31.229139 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:31.229092    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00498391-6658-44f7-8284-19244f9c36a9-kserve-provision-location\") pod \"00498391-6658-44f7-8284-19244f9c36a9\" (UID: \"00498391-6658-44f7-8284-19244f9c36a9\") "
Jul 30 17:16:31.229139 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:31.229138    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/00498391-6658-44f7-8284-19244f9c36a9-cabundle-cert\") pod \"00498391-6658-44f7-8284-19244f9c36a9\" (UID: \"00498391-6658-44f7-8284-19244f9c36a9\") "
Jul 30 17:16:31.229516 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:31.229204    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00498391-6658-44f7-8284-19244f9c36a9-kube-rbac-proxy-sar-config\") pod \"00498391-6658-44f7-8284-19244f9c36a9\" (UID: \"00498391-6658-44f7-8284-19244f9c36a9\") "
Jul 30 17:16:31.229516 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:31.229478    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00498391-6658-44f7-8284-19244f9c36a9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "00498391-6658-44f7-8284-19244f9c36a9" (UID: "00498391-6658-44f7-8284-19244f9c36a9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 17:16:31.229609 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:31.229522    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00498391-6658-44f7-8284-19244f9c36a9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "00498391-6658-44f7-8284-19244f9c36a9" (UID: "00498391-6658-44f7-8284-19244f9c36a9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:16:31.229645 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:31.229609    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00498391-6658-44f7-8284-19244f9c36a9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "00498391-6658-44f7-8284-19244f9c36a9" (UID: "00498391-6658-44f7-8284-19244f9c36a9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:16:31.231417 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:31.231384    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00498391-6658-44f7-8284-19244f9c36a9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "00498391-6658-44f7-8284-19244f9c36a9" (UID: "00498391-6658-44f7-8284-19244f9c36a9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:16:31.231417 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:31.231396    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00498391-6658-44f7-8284-19244f9c36a9-kube-api-access-ghj4d" (OuterVolumeSpecName: "kube-api-access-ghj4d") pod "00498391-6658-44f7-8284-19244f9c36a9" (UID: "00498391-6658-44f7-8284-19244f9c36a9"). InnerVolumeSpecName "kube-api-access-ghj4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:16:31.330697 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:31.330649    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghj4d\" (UniqueName: \"kubernetes.io/projected/00498391-6658-44f7-8284-19244f9c36a9-kube-api-access-ghj4d\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:16:31.330697 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:31.330687    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00498391-6658-44f7-8284-19244f9c36a9-proxy-tls\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:16:31.330697 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:31.330703    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00498391-6658-44f7-8284-19244f9c36a9-kserve-provision-location\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:16:31.331045 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:31.330716    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/00498391-6658-44f7-8284-19244f9c36a9-cabundle-cert\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:16:31.331045 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:31.330728    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00498391-6658-44f7-8284-19244f9c36a9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:16:31.621682 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:31.621644    2580 generic.go:356] "Generic (PLEG): container finished" podID="00498391-6658-44f7-8284-19244f9c36a9" containerID="1318865d608c3632b79ab5d16de0e880cdee35cc988676f66c60dbd9b72a87ef" exitCode=0
Jul 30 17:16:31.622153 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:31.621714    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc"
Jul 30 17:16:31.622153 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:31.621733    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc" event={"ID":"00498391-6658-44f7-8284-19244f9c36a9","Type":"ContainerDied","Data":"1318865d608c3632b79ab5d16de0e880cdee35cc988676f66c60dbd9b72a87ef"}
Jul 30 17:16:31.622153 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:31.621811    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc" event={"ID":"00498391-6658-44f7-8284-19244f9c36a9","Type":"ContainerDied","Data":"2baaaa95dbb4d82825c66117ee2cf0ac50d3e99435ebdef1178321cc904361cb"}
Jul 30 17:16:31.622153 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:31.621850    2580 scope.go:117] "RemoveContainer" containerID="73ef971cc014f9b49d7f134dbb910cb00098ddfb3e566cc3ad02003b4e0e8926"
Jul 30 17:16:31.631648 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:31.631620    2580 scope.go:117] "RemoveContainer" containerID="1318865d608c3632b79ab5d16de0e880cdee35cc988676f66c60dbd9b72a87ef"
Jul 30 17:16:31.640672 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:31.640643    2580 scope.go:117] "RemoveContainer" containerID="1fa7b3bc4c868ae0da587852b8c98b73a15b71dc030246de9212c2f0935f4f30"
Jul 30 17:16:31.641903 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:31.641870    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc"]
Jul 30 17:16:31.647774 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:31.647748    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-32669-predictor-587f74cb5c-887sc"]
Jul 30 17:16:31.649869 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:31.649850    2580 scope.go:117] "RemoveContainer" containerID="73ef971cc014f9b49d7f134dbb910cb00098ddfb3e566cc3ad02003b4e0e8926"
Jul 30 17:16:31.650214 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:16:31.650194    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73ef971cc014f9b49d7f134dbb910cb00098ddfb3e566cc3ad02003b4e0e8926\": container with ID starting with 73ef971cc014f9b49d7f134dbb910cb00098ddfb3e566cc3ad02003b4e0e8926 not found: ID does not exist" containerID="73ef971cc014f9b49d7f134dbb910cb00098ddfb3e566cc3ad02003b4e0e8926"
Jul 30 17:16:31.650274 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:31.650226    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73ef971cc014f9b49d7f134dbb910cb00098ddfb3e566cc3ad02003b4e0e8926"} err="failed to get container status \"73ef971cc014f9b49d7f134dbb910cb00098ddfb3e566cc3ad02003b4e0e8926\": rpc error: code = NotFound desc = could not find container \"73ef971cc014f9b49d7f134dbb910cb00098ddfb3e566cc3ad02003b4e0e8926\": container with ID starting with 73ef971cc014f9b49d7f134dbb910cb00098ddfb3e566cc3ad02003b4e0e8926 not found: ID does not exist"
Jul 30 17:16:31.650274 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:31.650247    2580 scope.go:117] "RemoveContainer" containerID="1318865d608c3632b79ab5d16de0e880cdee35cc988676f66c60dbd9b72a87ef"
Jul 30 17:16:31.650539 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:16:31.650522    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1318865d608c3632b79ab5d16de0e880cdee35cc988676f66c60dbd9b72a87ef\": container with ID starting with 1318865d608c3632b79ab5d16de0e880cdee35cc988676f66c60dbd9b72a87ef not found: ID does not exist" containerID="1318865d608c3632b79ab5d16de0e880cdee35cc988676f66c60dbd9b72a87ef"
Jul 30 17:16:31.650575 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:31.650550    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1318865d608c3632b79ab5d16de0e880cdee35cc988676f66c60dbd9b72a87ef"} err="failed to get container status \"1318865d608c3632b79ab5d16de0e880cdee35cc988676f66c60dbd9b72a87ef\": rpc error: code = NotFound desc = could not find container \"1318865d608c3632b79ab5d16de0e880cdee35cc988676f66c60dbd9b72a87ef\": container with ID starting with 1318865d608c3632b79ab5d16de0e880cdee35cc988676f66c60dbd9b72a87ef not found: ID does not exist"
Jul 30 17:16:31.650575 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:31.650568    2580 scope.go:117] "RemoveContainer" containerID="1fa7b3bc4c868ae0da587852b8c98b73a15b71dc030246de9212c2f0935f4f30"
Jul 30 17:16:31.650814 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:16:31.650797    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fa7b3bc4c868ae0da587852b8c98b73a15b71dc030246de9212c2f0935f4f30\": container with ID starting with 1fa7b3bc4c868ae0da587852b8c98b73a15b71dc030246de9212c2f0935f4f30 not found: ID does not exist" containerID="1fa7b3bc4c868ae0da587852b8c98b73a15b71dc030246de9212c2f0935f4f30"
Jul 30 17:16:31.650862 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:31.650818    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fa7b3bc4c868ae0da587852b8c98b73a15b71dc030246de9212c2f0935f4f30"} err="failed to get container status \"1fa7b3bc4c868ae0da587852b8c98b73a15b71dc030246de9212c2f0935f4f30\": rpc error: code = NotFound desc = could not find container \"1fa7b3bc4c868ae0da587852b8c98b73a15b71dc030246de9212c2f0935f4f30\": container with ID starting with 1fa7b3bc4c868ae0da587852b8c98b73a15b71dc030246de9212c2f0935f4f30 not found: ID does not exist"
Jul 30 17:16:32.222878 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:32.222851    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb"
Jul 30 17:16:32.340222 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:32.340077    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89210773-9748-4652-8d4c-80f1c870dfc4-kserve-provision-location\") pod \"89210773-9748-4652-8d4c-80f1c870dfc4\" (UID: \"89210773-9748-4652-8d4c-80f1c870dfc4\") "
Jul 30 17:16:32.340430 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:32.340231    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89210773-9748-4652-8d4c-80f1c870dfc4-proxy-tls\") pod \"89210773-9748-4652-8d4c-80f1c870dfc4\" (UID: \"89210773-9748-4652-8d4c-80f1c870dfc4\") "
Jul 30 17:16:32.340430 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:32.340287    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-928c6\" (UniqueName: \"kubernetes.io/projected/89210773-9748-4652-8d4c-80f1c870dfc4-kube-api-access-928c6\") pod \"89210773-9748-4652-8d4c-80f1c870dfc4\" (UID: \"89210773-9748-4652-8d4c-80f1c870dfc4\") "
Jul 30 17:16:32.340430 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:32.340320    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89210773-9748-4652-8d4c-80f1c870dfc4-kube-rbac-proxy-sar-config\") pod \"89210773-9748-4652-8d4c-80f1c870dfc4\" (UID: \"89210773-9748-4652-8d4c-80f1c870dfc4\") "
Jul 30 17:16:32.340430 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:32.340354    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89210773-9748-4652-8d4c-80f1c870dfc4-cabundle-cert\") pod \"89210773-9748-4652-8d4c-80f1c870dfc4\" (UID: \"89210773-9748-4652-8d4c-80f1c870dfc4\") "
Jul 30 17:16:32.340641 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:32.340620    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89210773-9748-4652-8d4c-80f1c870dfc4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "89210773-9748-4652-8d4c-80f1c870dfc4" (UID: "89210773-9748-4652-8d4c-80f1c870dfc4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 17:16:32.340751 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:32.340721    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89210773-9748-4652-8d4c-80f1c870dfc4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "89210773-9748-4652-8d4c-80f1c870dfc4" (UID: "89210773-9748-4652-8d4c-80f1c870dfc4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:16:32.340751 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:32.340742    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89210773-9748-4652-8d4c-80f1c870dfc4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "89210773-9748-4652-8d4c-80f1c870dfc4" (UID: "89210773-9748-4652-8d4c-80f1c870dfc4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:16:32.342502 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:32.342480    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89210773-9748-4652-8d4c-80f1c870dfc4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "89210773-9748-4652-8d4c-80f1c870dfc4" (UID: "89210773-9748-4652-8d4c-80f1c870dfc4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:16:32.342612 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:32.342530    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89210773-9748-4652-8d4c-80f1c870dfc4-kube-api-access-928c6" (OuterVolumeSpecName: "kube-api-access-928c6") pod "89210773-9748-4652-8d4c-80f1c870dfc4" (UID: "89210773-9748-4652-8d4c-80f1c870dfc4"). InnerVolumeSpecName "kube-api-access-928c6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:16:32.441915 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:32.441866    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89210773-9748-4652-8d4c-80f1c870dfc4-proxy-tls\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:16:32.441915 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:32.441902    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-928c6\" (UniqueName: \"kubernetes.io/projected/89210773-9748-4652-8d4c-80f1c870dfc4-kube-api-access-928c6\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:16:32.441915 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:32.441914    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89210773-9748-4652-8d4c-80f1c870dfc4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:16:32.441915 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:32.441924    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89210773-9748-4652-8d4c-80f1c870dfc4-cabundle-cert\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:16:32.442240 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:32.441933    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89210773-9748-4652-8d4c-80f1c870dfc4-kserve-provision-location\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:16:32.628608 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:32.628498    2580 generic.go:356] "Generic (PLEG): container finished" podID="89210773-9748-4652-8d4c-80f1c870dfc4" containerID="e0a79ebf9cd697d3b2801596aed5acfb2d500a6a1a2e9f2c1f88c2d3eb022e2a" exitCode=0
Jul 30 17:16:32.628608 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:32.628593    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb"
Jul 30 17:16:32.629200 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:32.628589    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb" event={"ID":"89210773-9748-4652-8d4c-80f1c870dfc4","Type":"ContainerDied","Data":"e0a79ebf9cd697d3b2801596aed5acfb2d500a6a1a2e9f2c1f88c2d3eb022e2a"}
Jul 30 17:16:32.629200 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:32.628752    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb" event={"ID":"89210773-9748-4652-8d4c-80f1c870dfc4","Type":"ContainerDied","Data":"7a5bc412298f74957a9142726430d1cbd2b898b3bc9387d326d4e11dbdf16105"}
Jul 30 17:16:32.629200 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:32.628797    2580 scope.go:117] "RemoveContainer" containerID="e4ca173d80aa9c3077e9c0e7f81aa78a8f6172a01ed7bd1fcf49856bbb8681c9"
Jul 30 17:16:32.638643 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:32.638482    2580 scope.go:117] "RemoveContainer" containerID="e0a79ebf9cd697d3b2801596aed5acfb2d500a6a1a2e9f2c1f88c2d3eb022e2a"
Jul 30 17:16:32.647490 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:32.647469    2580 scope.go:117] "RemoveContainer" containerID="0b7ad9764a3bb1b0bd6ecf511b82f82cefe89d257ab2218de73c7a5cd11da335"
Jul 30 17:16:32.652530 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:32.652502    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb"]
Jul 30 17:16:32.656301 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:32.656271    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-32669-predictor-74ff88df46-gp2rb"]
Jul 30 17:16:32.656531 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:32.656515    2580 scope.go:117] "RemoveContainer" containerID="e4ca173d80aa9c3077e9c0e7f81aa78a8f6172a01ed7bd1fcf49856bbb8681c9"
Jul 30 17:16:32.656846 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:16:32.656819    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4ca173d80aa9c3077e9c0e7f81aa78a8f6172a01ed7bd1fcf49856bbb8681c9\": container with ID starting with e4ca173d80aa9c3077e9c0e7f81aa78a8f6172a01ed7bd1fcf49856bbb8681c9 not found: ID does not exist" containerID="e4ca173d80aa9c3077e9c0e7f81aa78a8f6172a01ed7bd1fcf49856bbb8681c9"
Jul 30 17:16:32.656911 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:32.656856    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4ca173d80aa9c3077e9c0e7f81aa78a8f6172a01ed7bd1fcf49856bbb8681c9"} err="failed to get container status \"e4ca173d80aa9c3077e9c0e7f81aa78a8f6172a01ed7bd1fcf49856bbb8681c9\": rpc error: code = NotFound desc = could not find container \"e4ca173d80aa9c3077e9c0e7f81aa78a8f6172a01ed7bd1fcf49856bbb8681c9\": container with ID starting with e4ca173d80aa9c3077e9c0e7f81aa78a8f6172a01ed7bd1fcf49856bbb8681c9 not found: ID does not exist"
Jul 30 17:16:32.656911 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:32.656876    2580 scope.go:117] "RemoveContainer" containerID="e0a79ebf9cd697d3b2801596aed5acfb2d500a6a1a2e9f2c1f88c2d3eb022e2a"
Jul 30 17:16:32.657118 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:16:32.657102    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0a79ebf9cd697d3b2801596aed5acfb2d500a6a1a2e9f2c1f88c2d3eb022e2a\": container with ID starting with e0a79ebf9cd697d3b2801596aed5acfb2d500a6a1a2e9f2c1f88c2d3eb022e2a not found: ID does not exist" containerID="e0a79ebf9cd697d3b2801596aed5acfb2d500a6a1a2e9f2c1f88c2d3eb022e2a"
Jul 30 17:16:32.657160 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:32.657121    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0a79ebf9cd697d3b2801596aed5acfb2d500a6a1a2e9f2c1f88c2d3eb022e2a"} err="failed to get container status \"e0a79ebf9cd697d3b2801596aed5acfb2d500a6a1a2e9f2c1f88c2d3eb022e2a\": rpc error: code = NotFound desc = could not find container \"e0a79ebf9cd697d3b2801596aed5acfb2d500a6a1a2e9f2c1f88c2d3eb022e2a\": container with ID starting with e0a79ebf9cd697d3b2801596aed5acfb2d500a6a1a2e9f2c1f88c2d3eb022e2a not found: ID does not exist"
Jul 30 17:16:32.657160 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:32.657135    2580 scope.go:117] "RemoveContainer" containerID="0b7ad9764a3bb1b0bd6ecf511b82f82cefe89d257ab2218de73c7a5cd11da335"
Jul 30 17:16:32.657464 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:16:32.657447    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b7ad9764a3bb1b0bd6ecf511b82f82cefe89d257ab2218de73c7a5cd11da335\": container with ID starting with 0b7ad9764a3bb1b0bd6ecf511b82f82cefe89d257ab2218de73c7a5cd11da335 not found: ID does not exist" containerID="0b7ad9764a3bb1b0bd6ecf511b82f82cefe89d257ab2218de73c7a5cd11da335"
Jul 30 17:16:32.657517 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:32.657469    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b7ad9764a3bb1b0bd6ecf511b82f82cefe89d257ab2218de73c7a5cd11da335"} err="failed to get container status \"0b7ad9764a3bb1b0bd6ecf511b82f82cefe89d257ab2218de73c7a5cd11da335\": rpc error: code = NotFound desc = could not find container \"0b7ad9764a3bb1b0bd6ecf511b82f82cefe89d257ab2218de73c7a5cd11da335\": container with ID starting with 0b7ad9764a3bb1b0bd6ecf511b82f82cefe89d257ab2218de73c7a5cd11da335 not found: ID does not exist"
Jul 30 17:16:33.482383 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:33.482348    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00498391-6658-44f7-8284-19244f9c36a9" path="/var/lib/kubelet/pods/00498391-6658-44f7-8284-19244f9c36a9/volumes"
Jul 30 17:16:33.482874 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:33.482860    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89210773-9748-4652-8d4c-80f1c870dfc4" path="/var/lib/kubelet/pods/89210773-9748-4652-8d4c-80f1c870dfc4/volumes"
Jul 30 17:16:37.634460 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:37.634430    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-94aa4-predictor-86778f57d5-2hz7w"
Jul 30 17:16:46.385526 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:46.385481    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc"]
Jul 30 17:16:46.386061 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:46.385865    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00498391-6658-44f7-8284-19244f9c36a9" containerName="kube-rbac-proxy"
Jul 30 17:16:46.386061 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:46.385881    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="00498391-6658-44f7-8284-19244f9c36a9" containerName="kube-rbac-proxy"
Jul 30 17:16:46.386061 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:46.385901    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89210773-9748-4652-8d4c-80f1c870dfc4" containerName="storage-initializer"
Jul 30 17:16:46.386061 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:46.385910    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="89210773-9748-4652-8d4c-80f1c870dfc4" containerName="storage-initializer"
Jul 30 17:16:46.386061 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:46.385933    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00498391-6658-44f7-8284-19244f9c36a9" containerName="storage-initializer"
Jul 30 17:16:46.386061 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:46.385944    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="00498391-6658-44f7-8284-19244f9c36a9" containerName="storage-initializer"
Jul 30 17:16:46.386061 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:46.385953    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00498391-6658-44f7-8284-19244f9c36a9" containerName="kserve-container"
Jul 30 17:16:46.386061 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:46.385959    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="00498391-6658-44f7-8284-19244f9c36a9" containerName="kserve-container"
Jul 30 17:16:46.386061 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:46.385974    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89210773-9748-4652-8d4c-80f1c870dfc4" containerName="kube-rbac-proxy"
Jul 30 17:16:46.386061 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:46.385979    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="89210773-9748-4652-8d4c-80f1c870dfc4" containerName="kube-rbac-proxy"
Jul 30 17:16:46.386061 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:46.386007    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89210773-9748-4652-8d4c-80f1c870dfc4" containerName="kserve-container"
Jul 30 17:16:46.386061 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:46.386013    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="89210773-9748-4652-8d4c-80f1c870dfc4" containerName="kserve-container"
Jul 30 17:16:46.386496 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:46.386088    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="00498391-6658-44f7-8284-19244f9c36a9" containerName="kube-rbac-proxy"
Jul 30 17:16:46.386496 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:46.386099    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="00498391-6658-44f7-8284-19244f9c36a9" containerName="kserve-container"
Jul 30 17:16:46.386496 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:46.386109    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="89210773-9748-4652-8d4c-80f1c870dfc4" containerName="kserve-container"
Jul 30 17:16:46.386496 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:46.386117    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="89210773-9748-4652-8d4c-80f1c870dfc4" containerName="kube-rbac-proxy"
Jul 30 17:16:46.391101 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:46.391080    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc"
Jul 30 17:16:46.397803 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:46.394017    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-94aa4-predictor-serving-cert\""
Jul 30 17:16:46.397803 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:46.394238    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 30 17:16:46.397803 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:46.394369    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 30 17:16:46.397803 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:46.394521    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-94aa4-kube-rbac-proxy-sar-config\""
Jul 30 17:16:46.401724 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:46.401694    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc"]
Jul 30 17:16:46.466717 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:46.466660    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7d7d023d-4d93-4b21-8163-4bc6b39b7619-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc\" (UID: \"7d7d023d-4d93-4b21-8163-4bc6b39b7619\") " pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc"
Jul 30 17:16:46.466717 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:46.466714    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d7d023d-4d93-4b21-8163-4bc6b39b7619-cabundle-cert\") pod \"isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc\" (UID: \"7d7d023d-4d93-4b21-8163-4bc6b39b7619\") " pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc"
Jul 30 17:16:46.466946 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:46.466785    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpv5r\" (UniqueName: \"kubernetes.io/projected/7d7d023d-4d93-4b21-8163-4bc6b39b7619-kube-api-access-qpv5r\") pod \"isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc\" (UID: \"7d7d023d-4d93-4b21-8163-4bc6b39b7619\") " pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc"
Jul 30 17:16:46.466946 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:46.466847    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d7d023d-4d93-4b21-8163-4bc6b39b7619-kserve-provision-location\") pod \"isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc\" (UID: \"7d7d023d-4d93-4b21-8163-4bc6b39b7619\") " pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc"
Jul 30 17:16:46.466946 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:46.466893    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d7d023d-4d93-4b21-8163-4bc6b39b7619-proxy-tls\") pod \"isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc\" (UID: \"7d7d023d-4d93-4b21-8163-4bc6b39b7619\") " pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc"
Jul 30 17:16:46.567606 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:46.567561    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7d7d023d-4d93-4b21-8163-4bc6b39b7619-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc\" (UID: \"7d7d023d-4d93-4b21-8163-4bc6b39b7619\") " pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc"
Jul 30 17:16:46.567606 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:46.567606    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d7d023d-4d93-4b21-8163-4bc6b39b7619-cabundle-cert\") pod \"isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc\" (UID: \"7d7d023d-4d93-4b21-8163-4bc6b39b7619\") " pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc"
Jul 30 17:16:46.567892 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:46.567630    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpv5r\" (UniqueName: \"kubernetes.io/projected/7d7d023d-4d93-4b21-8163-4bc6b39b7619-kube-api-access-qpv5r\") pod \"isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc\" (UID: \"7d7d023d-4d93-4b21-8163-4bc6b39b7619\") " pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc"
Jul 30 17:16:46.567892 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:46.567657    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d7d023d-4d93-4b21-8163-4bc6b39b7619-kserve-provision-location\") pod \"isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc\" (UID: \"7d7d023d-4d93-4b21-8163-4bc6b39b7619\") " pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc"
Jul 30 17:16:46.567892 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:46.567702    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d7d023d-4d93-4b21-8163-4bc6b39b7619-proxy-tls\") pod \"isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc\" (UID: \"7d7d023d-4d93-4b21-8163-4bc6b39b7619\") " pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc"
Jul 30 17:16:46.568062 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:16:46.567912    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-serving-cert: secret "isvc-logger-raw-94aa4-predictor-serving-cert" not found
Jul 30 17:16:46.568062 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:16:46.567992    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d7d023d-4d93-4b21-8163-4bc6b39b7619-proxy-tls podName:7d7d023d-4d93-4b21-8163-4bc6b39b7619 nodeName:}" failed. No retries permitted until 2026-07-30 17:16:47.067970199 +0000 UTC m=+970.143270066 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7d7d023d-4d93-4b21-8163-4bc6b39b7619-proxy-tls") pod "isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc" (UID: "7d7d023d-4d93-4b21-8163-4bc6b39b7619") : secret "isvc-logger-raw-94aa4-predictor-serving-cert" not found
Jul 30 17:16:46.568210 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:46.568124    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d7d023d-4d93-4b21-8163-4bc6b39b7619-kserve-provision-location\") pod \"isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc\" (UID: \"7d7d023d-4d93-4b21-8163-4bc6b39b7619\") " pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc"
Jul 30 17:16:46.568428 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:46.568405    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d7d023d-4d93-4b21-8163-4bc6b39b7619-cabundle-cert\") pod \"isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc\" (UID: \"7d7d023d-4d93-4b21-8163-4bc6b39b7619\") " pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc"
Jul 30 17:16:46.568636 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:46.568617    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7d7d023d-4d93-4b21-8163-4bc6b39b7619-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc\" (UID: \"7d7d023d-4d93-4b21-8163-4bc6b39b7619\") " pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc"
Jul 30 17:16:46.579672 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:46.579638    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpv5r\" (UniqueName: \"kubernetes.io/projected/7d7d023d-4d93-4b21-8163-4bc6b39b7619-kube-api-access-qpv5r\") pod \"isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc\" (UID: \"7d7d023d-4d93-4b21-8163-4bc6b39b7619\") " pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc"
Jul 30 17:16:47.072547 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:47.072500    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d7d023d-4d93-4b21-8163-4bc6b39b7619-proxy-tls\") pod \"isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc\" (UID: \"7d7d023d-4d93-4b21-8163-4bc6b39b7619\") " pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc"
Jul 30 17:16:47.075149 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:47.075119    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d7d023d-4d93-4b21-8163-4bc6b39b7619-proxy-tls\") pod \"isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc\" (UID: \"7d7d023d-4d93-4b21-8163-4bc6b39b7619\") " pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc"
Jul 30 17:16:47.310668 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:47.310623    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc"
Jul 30 17:16:47.456488 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:47.456435    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc"]
Jul 30 17:16:47.460638 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:16:47.460601    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d7d023d_4d93_4b21_8163_4bc6b39b7619.slice/crio-296b7460658b0c18deb44d5c8104e1ce03a544277a71a5eba385a6039acc1de4 WatchSource:0}: Error finding container 296b7460658b0c18deb44d5c8104e1ce03a544277a71a5eba385a6039acc1de4: Status 404 returned error can't find the container with id 296b7460658b0c18deb44d5c8104e1ce03a544277a71a5eba385a6039acc1de4
Jul 30 17:16:47.696884 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:47.696783    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc" event={"ID":"7d7d023d-4d93-4b21-8163-4bc6b39b7619","Type":"ContainerStarted","Data":"9b7098a03b4007924298e16c7d48e8e5035053a2baea25d4bd5c4121ce525a59"}
Jul 30 17:16:47.696884 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:47.696858    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc" event={"ID":"7d7d023d-4d93-4b21-8163-4bc6b39b7619","Type":"ContainerStarted","Data":"296b7460658b0c18deb44d5c8104e1ce03a544277a71a5eba385a6039acc1de4"}
Jul 30 17:16:51.714289 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:51.714240    2580 generic.go:356] "Generic (PLEG): container finished" podID="7d7d023d-4d93-4b21-8163-4bc6b39b7619" containerID="9b7098a03b4007924298e16c7d48e8e5035053a2baea25d4bd5c4121ce525a59" exitCode=0
Jul 30 17:16:51.714716 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:51.714291    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc" event={"ID":"7d7d023d-4d93-4b21-8163-4bc6b39b7619","Type":"ContainerDied","Data":"9b7098a03b4007924298e16c7d48e8e5035053a2baea25d4bd5c4121ce525a59"}
Jul 30 17:16:52.720747 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:52.720705    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc" event={"ID":"7d7d023d-4d93-4b21-8163-4bc6b39b7619","Type":"ContainerStarted","Data":"79b29aecaf690ce02f43c9625621e03e65dc170d6e63da0ca7c4016392117a8e"}
Jul 30 17:16:52.721274 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:52.720780    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc" event={"ID":"7d7d023d-4d93-4b21-8163-4bc6b39b7619","Type":"ContainerStarted","Data":"525a46ec86289a65bc089da762b60f9f8d44049acfed1c6ff1abf29867f53863"}
Jul 30 17:16:52.721274 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:52.720806    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc" event={"ID":"7d7d023d-4d93-4b21-8163-4bc6b39b7619","Type":"ContainerStarted","Data":"547cc001beb0aa74ec62b305858b09b486bdcf9385236db9063d449f34214c05"}
Jul 30 17:16:52.721274 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:52.721119    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc"
Jul 30 17:16:52.721274 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:52.721199    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc"
Jul 30 17:16:52.721274 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:52.721238    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc"
Jul 30 17:16:52.723009 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:52.722969    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc" podUID="7d7d023d-4d93-4b21-8163-4bc6b39b7619" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 30 17:16:52.723806 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:52.723769    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc" podUID="7d7d023d-4d93-4b21-8163-4bc6b39b7619" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 17:16:52.745739 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:52.745677    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc" podStartSLOduration=3.167343878 podStartE2EDuration="6.745660129s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.578316251s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:16:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:16:52.743253842 +0000 UTC m=+975.818553743" watchObservedRunningTime="2026-07-30 17:16:52.745660129 +0000 UTC m=+975.820960018"
Jul 30 17:16:53.725649 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:53.725600    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc" podUID="7d7d023d-4d93-4b21-8163-4bc6b39b7619" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 30 17:16:53.726200 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:53.726146    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc" podUID="7d7d023d-4d93-4b21-8163-4bc6b39b7619" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 17:16:58.730587 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:58.730500    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc"
Jul 30 17:16:58.731248 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:58.731220    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc" podUID="7d7d023d-4d93-4b21-8163-4bc6b39b7619" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 30 17:16:58.731587 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:16:58.731560    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc" podUID="7d7d023d-4d93-4b21-8163-4bc6b39b7619" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 17:17:08.731247 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:17:08.731199    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc" podUID="7d7d023d-4d93-4b21-8163-4bc6b39b7619" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 30 17:17:08.731706 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:17:08.731631    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc" podUID="7d7d023d-4d93-4b21-8163-4bc6b39b7619" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 17:17:18.731463 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:17:18.731414    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc" podUID="7d7d023d-4d93-4b21-8163-4bc6b39b7619" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 30 17:17:18.732046 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:17:18.731920    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc" podUID="7d7d023d-4d93-4b21-8163-4bc6b39b7619" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 17:17:28.731716 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:17:28.731626    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc" podUID="7d7d023d-4d93-4b21-8163-4bc6b39b7619" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 30 17:17:28.732334 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:17:28.732153    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc" podUID="7d7d023d-4d93-4b21-8163-4bc6b39b7619" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 17:17:38.731666 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:17:38.731603    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc" podUID="7d7d023d-4d93-4b21-8163-4bc6b39b7619" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 30 17:17:38.732241 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:17:38.732016    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc" podUID="7d7d023d-4d93-4b21-8163-4bc6b39b7619" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 17:17:48.731613 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:17:48.731560    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc" podUID="7d7d023d-4d93-4b21-8163-4bc6b39b7619" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 30 17:17:48.732154 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:17:48.732055    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc" podUID="7d7d023d-4d93-4b21-8163-4bc6b39b7619" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 17:17:58.731929 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:17:58.731884    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc" podUID="7d7d023d-4d93-4b21-8163-4bc6b39b7619" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 30 17:17:58.732573 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:17:58.732470    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc" podUID="7d7d023d-4d93-4b21-8163-4bc6b39b7619" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 17:18:08.731113 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:08.731057    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc" podUID="7d7d023d-4d93-4b21-8163-4bc6b39b7619" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 30 17:18:08.731607 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:08.731572    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc" podUID="7d7d023d-4d93-4b21-8163-4bc6b39b7619" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 17:18:10.478246 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:10.478189    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc" podUID="7d7d023d-4d93-4b21-8163-4bc6b39b7619" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 30 17:18:10.478753 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:10.478691    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc" podUID="7d7d023d-4d93-4b21-8163-4bc6b39b7619" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 17:18:20.478392 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:20.478355    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc"
Jul 30 17:18:20.479007 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:20.478975    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc"
Jul 30 17:18:31.397328 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:31.397285    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-94aa4-predictor-86778f57d5-2hz7w_e864c7d3-57bd-4355-a494-2912f3f09f74/kserve-container/0.log"
Jul 30 17:18:31.625910 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:31.625865    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc"]
Jul 30 17:18:31.626226 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:31.626162    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc" podUID="7d7d023d-4d93-4b21-8163-4bc6b39b7619" containerName="kserve-container" containerID="cri-o://547cc001beb0aa74ec62b305858b09b486bdcf9385236db9063d449f34214c05" gracePeriod=30
Jul 30 17:18:31.626226 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:31.626189    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc" podUID="7d7d023d-4d93-4b21-8163-4bc6b39b7619" containerName="agent" containerID="cri-o://79b29aecaf690ce02f43c9625621e03e65dc170d6e63da0ca7c4016392117a8e" gracePeriod=30
Jul 30 17:18:31.626446 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:31.626237    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc" podUID="7d7d023d-4d93-4b21-8163-4bc6b39b7619" containerName="kube-rbac-proxy" containerID="cri-o://525a46ec86289a65bc089da762b60f9f8d44049acfed1c6ff1abf29867f53863" gracePeriod=30
Jul 30 17:18:31.701678 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:31.701581    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq"]
Jul 30 17:18:31.705360 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:31.705338    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq"
Jul 30 17:18:31.707840 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:31.707805    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-467c1-predictor-serving-cert\""
Jul 30 17:18:31.707840 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:31.707823    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-467c1-kube-rbac-proxy-sar-config\""
Jul 30 17:18:31.715286 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:31.715253    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq"]
Jul 30 17:18:31.750097 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:31.750053    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-94aa4-predictor-86778f57d5-2hz7w"]
Jul 30 17:18:31.750372 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:31.750345    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-94aa4-predictor-86778f57d5-2hz7w" podUID="e864c7d3-57bd-4355-a494-2912f3f09f74" containerName="kserve-container" containerID="cri-o://2a1b63b89ee1fced3c1d0e2573a0150107d6a676fa4621436d0999a8dd8a19e9" gracePeriod=30
Jul 30 17:18:31.750479 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:31.750435    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-94aa4-predictor-86778f57d5-2hz7w" podUID="e864c7d3-57bd-4355-a494-2912f3f09f74" containerName="kube-rbac-proxy" containerID="cri-o://4b6c228227898b0d9bfe4398bffc0cfb72f1cba7f4ddb7a8c702f4522e56670f" gracePeriod=30
Jul 30 17:18:31.797353 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:31.797311    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/902dff9d-901f-4056-bc40-367a61bb1477-cabundle-cert\") pod \"isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq\" (UID: \"902dff9d-901f-4056-bc40-367a61bb1477\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq"
Jul 30 17:18:31.797516 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:31.797377    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/902dff9d-901f-4056-bc40-367a61bb1477-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq\" (UID: \"902dff9d-901f-4056-bc40-367a61bb1477\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq"
Jul 30 17:18:31.797516 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:31.797449    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/902dff9d-901f-4056-bc40-367a61bb1477-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq\" (UID: \"902dff9d-901f-4056-bc40-367a61bb1477\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq"
Jul 30 17:18:31.797516 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:31.797480    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/902dff9d-901f-4056-bc40-367a61bb1477-proxy-tls\") pod \"isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq\" (UID: \"902dff9d-901f-4056-bc40-367a61bb1477\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq"
Jul 30 17:18:31.797516 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:31.797505    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpfwd\" (UniqueName: \"kubernetes.io/projected/902dff9d-901f-4056-bc40-367a61bb1477-kube-api-access-vpfwd\") pod \"isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq\" (UID: \"902dff9d-901f-4056-bc40-367a61bb1477\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq"
Jul 30 17:18:31.899216 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:31.898445    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpfwd\" (UniqueName: \"kubernetes.io/projected/902dff9d-901f-4056-bc40-367a61bb1477-kube-api-access-vpfwd\") pod \"isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq\" (UID: \"902dff9d-901f-4056-bc40-367a61bb1477\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq"
Jul 30 17:18:31.899216 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:31.898552    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/902dff9d-901f-4056-bc40-367a61bb1477-cabundle-cert\") pod \"isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq\" (UID: \"902dff9d-901f-4056-bc40-367a61bb1477\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq"
Jul 30 17:18:31.899216 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:31.898605    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/902dff9d-901f-4056-bc40-367a61bb1477-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq\" (UID: \"902dff9d-901f-4056-bc40-367a61bb1477\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq"
Jul 30 17:18:31.899216 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:31.898655    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/902dff9d-901f-4056-bc40-367a61bb1477-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq\" (UID: \"902dff9d-901f-4056-bc40-367a61bb1477\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq"
Jul 30 17:18:31.899216 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:31.898687    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/902dff9d-901f-4056-bc40-367a61bb1477-proxy-tls\") pod \"isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq\" (UID: \"902dff9d-901f-4056-bc40-367a61bb1477\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq"
Jul 30 17:18:31.899670 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:31.899645    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/902dff9d-901f-4056-bc40-367a61bb1477-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq\" (UID: \"902dff9d-901f-4056-bc40-367a61bb1477\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq"
Jul 30 17:18:31.899831 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:31.899810    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/902dff9d-901f-4056-bc40-367a61bb1477-cabundle-cert\") pod \"isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq\" (UID: \"902dff9d-901f-4056-bc40-367a61bb1477\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq"
Jul 30 17:18:31.899896 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:31.899852    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/902dff9d-901f-4056-bc40-367a61bb1477-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq\" (UID: \"902dff9d-901f-4056-bc40-367a61bb1477\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq"
Jul 30 17:18:31.901744 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:31.901718    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/902dff9d-901f-4056-bc40-367a61bb1477-proxy-tls\") pod \"isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq\" (UID: \"902dff9d-901f-4056-bc40-367a61bb1477\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq"
Jul 30 17:18:31.907591 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:31.907540    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpfwd\" (UniqueName: \"kubernetes.io/projected/902dff9d-901f-4056-bc40-367a61bb1477-kube-api-access-vpfwd\") pod \"isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq\" (UID: \"902dff9d-901f-4056-bc40-367a61bb1477\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq"
Jul 30 17:18:32.002975 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:32.002942    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-94aa4-predictor-86778f57d5-2hz7w"
Jul 30 17:18:32.017819 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:32.017781    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq"
Jul 30 17:18:32.099873 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:32.099839    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e864c7d3-57bd-4355-a494-2912f3f09f74-kube-rbac-proxy-sar-config\") pod \"e864c7d3-57bd-4355-a494-2912f3f09f74\" (UID: \"e864c7d3-57bd-4355-a494-2912f3f09f74\") "
Jul 30 17:18:32.100080 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:32.099886    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e864c7d3-57bd-4355-a494-2912f3f09f74-proxy-tls\") pod \"e864c7d3-57bd-4355-a494-2912f3f09f74\" (UID: \"e864c7d3-57bd-4355-a494-2912f3f09f74\") "
Jul 30 17:18:32.100080 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:32.099937    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xttws\" (UniqueName: \"kubernetes.io/projected/e864c7d3-57bd-4355-a494-2912f3f09f74-kube-api-access-xttws\") pod \"e864c7d3-57bd-4355-a494-2912f3f09f74\" (UID: \"e864c7d3-57bd-4355-a494-2912f3f09f74\") "
Jul 30 17:18:32.100336 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:32.100305    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e864c7d3-57bd-4355-a494-2912f3f09f74-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e864c7d3-57bd-4355-a494-2912f3f09f74" (UID: "e864c7d3-57bd-4355-a494-2912f3f09f74"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:18:32.102102 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:32.102067    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e864c7d3-57bd-4355-a494-2912f3f09f74-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e864c7d3-57bd-4355-a494-2912f3f09f74" (UID: "e864c7d3-57bd-4355-a494-2912f3f09f74"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:18:32.102224 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:32.102097    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e864c7d3-57bd-4355-a494-2912f3f09f74-kube-api-access-xttws" (OuterVolumeSpecName: "kube-api-access-xttws") pod "e864c7d3-57bd-4355-a494-2912f3f09f74" (UID: "e864c7d3-57bd-4355-a494-2912f3f09f74"). InnerVolumeSpecName "kube-api-access-xttws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:18:32.150909 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:32.150869    2580 generic.go:356] "Generic (PLEG): container finished" podID="e864c7d3-57bd-4355-a494-2912f3f09f74" containerID="4b6c228227898b0d9bfe4398bffc0cfb72f1cba7f4ddb7a8c702f4522e56670f" exitCode=2
Jul 30 17:18:32.150909 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:32.150906    2580 generic.go:356] "Generic (PLEG): container finished" podID="e864c7d3-57bd-4355-a494-2912f3f09f74" containerID="2a1b63b89ee1fced3c1d0e2573a0150107d6a676fa4621436d0999a8dd8a19e9" exitCode=2
Jul 30 17:18:32.151157 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:32.150927    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-94aa4-predictor-86778f57d5-2hz7w"
Jul 30 17:18:32.151157 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:32.150939    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-94aa4-predictor-86778f57d5-2hz7w" event={"ID":"e864c7d3-57bd-4355-a494-2912f3f09f74","Type":"ContainerDied","Data":"4b6c228227898b0d9bfe4398bffc0cfb72f1cba7f4ddb7a8c702f4522e56670f"}
Jul 30 17:18:32.151157 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:32.151013    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-94aa4-predictor-86778f57d5-2hz7w" event={"ID":"e864c7d3-57bd-4355-a494-2912f3f09f74","Type":"ContainerDied","Data":"2a1b63b89ee1fced3c1d0e2573a0150107d6a676fa4621436d0999a8dd8a19e9"}
Jul 30 17:18:32.151157 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:32.151042    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-94aa4-predictor-86778f57d5-2hz7w" event={"ID":"e864c7d3-57bd-4355-a494-2912f3f09f74","Type":"ContainerDied","Data":"67845f2a852feb206f43c7e2346efb890c669c37dcb71cf9053526b2aa14f41b"}
Jul 30 17:18:32.151157 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:32.151069    2580 scope.go:117] "RemoveContainer" containerID="4b6c228227898b0d9bfe4398bffc0cfb72f1cba7f4ddb7a8c702f4522e56670f"
Jul 30 17:18:32.154318 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:32.154288    2580 generic.go:356] "Generic (PLEG): container finished" podID="7d7d023d-4d93-4b21-8163-4bc6b39b7619" containerID="525a46ec86289a65bc089da762b60f9f8d44049acfed1c6ff1abf29867f53863" exitCode=2
Jul 30 17:18:32.154475 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:32.154355    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc" event={"ID":"7d7d023d-4d93-4b21-8163-4bc6b39b7619","Type":"ContainerDied","Data":"525a46ec86289a65bc089da762b60f9f8d44049acfed1c6ff1abf29867f53863"}
Jul 30 17:18:32.160439 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:32.160418    2580 scope.go:117] "RemoveContainer" containerID="2a1b63b89ee1fced3c1d0e2573a0150107d6a676fa4621436d0999a8dd8a19e9"
Jul 30 17:18:32.167265 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:32.167239    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq"]
Jul 30 17:18:32.170256 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:32.170226    2580 scope.go:117] "RemoveContainer" containerID="4b6c228227898b0d9bfe4398bffc0cfb72f1cba7f4ddb7a8c702f4522e56670f"
Jul 30 17:18:32.170585 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:18:32.170560    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b6c228227898b0d9bfe4398bffc0cfb72f1cba7f4ddb7a8c702f4522e56670f\": container with ID starting with 4b6c228227898b0d9bfe4398bffc0cfb72f1cba7f4ddb7a8c702f4522e56670f not found: ID does not exist" containerID="4b6c228227898b0d9bfe4398bffc0cfb72f1cba7f4ddb7a8c702f4522e56670f"
Jul 30 17:18:32.170671 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:32.170595    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b6c228227898b0d9bfe4398bffc0cfb72f1cba7f4ddb7a8c702f4522e56670f"} err="failed to get container status \"4b6c228227898b0d9bfe4398bffc0cfb72f1cba7f4ddb7a8c702f4522e56670f\": rpc error: code = NotFound desc = could not find container \"4b6c228227898b0d9bfe4398bffc0cfb72f1cba7f4ddb7a8c702f4522e56670f\": container with ID starting with 4b6c228227898b0d9bfe4398bffc0cfb72f1cba7f4ddb7a8c702f4522e56670f not found: ID does not exist"
Jul 30 17:18:32.170671 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:32.170614    2580 scope.go:117] "RemoveContainer" containerID="2a1b63b89ee1fced3c1d0e2573a0150107d6a676fa4621436d0999a8dd8a19e9"
Jul 30 17:18:32.170882 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:18:32.170863    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a1b63b89ee1fced3c1d0e2573a0150107d6a676fa4621436d0999a8dd8a19e9\": container with ID starting with 2a1b63b89ee1fced3c1d0e2573a0150107d6a676fa4621436d0999a8dd8a19e9 not found: ID does not exist" containerID="2a1b63b89ee1fced3c1d0e2573a0150107d6a676fa4621436d0999a8dd8a19e9"
Jul 30 17:18:32.170922 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:32.170889    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a1b63b89ee1fced3c1d0e2573a0150107d6a676fa4621436d0999a8dd8a19e9"} err="failed to get container status \"2a1b63b89ee1fced3c1d0e2573a0150107d6a676fa4621436d0999a8dd8a19e9\": rpc error: code = NotFound desc = could not find container \"2a1b63b89ee1fced3c1d0e2573a0150107d6a676fa4621436d0999a8dd8a19e9\": container with ID starting with 2a1b63b89ee1fced3c1d0e2573a0150107d6a676fa4621436d0999a8dd8a19e9 not found: ID does not exist"
Jul 30 17:18:32.170922 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:32.170907    2580 scope.go:117] "RemoveContainer" containerID="4b6c228227898b0d9bfe4398bffc0cfb72f1cba7f4ddb7a8c702f4522e56670f"
Jul 30 17:18:32.171164 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:32.171146    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b6c228227898b0d9bfe4398bffc0cfb72f1cba7f4ddb7a8c702f4522e56670f"} err="failed to get container status \"4b6c228227898b0d9bfe4398bffc0cfb72f1cba7f4ddb7a8c702f4522e56670f\": rpc error: code = NotFound desc = could not find container \"4b6c228227898b0d9bfe4398bffc0cfb72f1cba7f4ddb7a8c702f4522e56670f\": container with ID starting with 4b6c228227898b0d9bfe4398bffc0cfb72f1cba7f4ddb7a8c702f4522e56670f not found: ID does not exist"
Jul 30 17:18:32.171278 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:32.171190    2580 scope.go:117] "RemoveContainer" containerID="2a1b63b89ee1fced3c1d0e2573a0150107d6a676fa4621436d0999a8dd8a19e9"
Jul 30 17:18:32.171372 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:18:32.171347    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod902dff9d_901f_4056_bc40_367a61bb1477.slice/crio-e1668ae1b44e6ba9eba6a9ccb52f87643cc21262baaa7c685ea23c62a43f850d WatchSource:0}: Error finding container e1668ae1b44e6ba9eba6a9ccb52f87643cc21262baaa7c685ea23c62a43f850d: Status 404 returned error can't find the container with id e1668ae1b44e6ba9eba6a9ccb52f87643cc21262baaa7c685ea23c62a43f850d
Jul 30 17:18:32.171416 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:32.171399    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a1b63b89ee1fced3c1d0e2573a0150107d6a676fa4621436d0999a8dd8a19e9"} err="failed to get container status \"2a1b63b89ee1fced3c1d0e2573a0150107d6a676fa4621436d0999a8dd8a19e9\": rpc error: code = NotFound desc = could not find container \"2a1b63b89ee1fced3c1d0e2573a0150107d6a676fa4621436d0999a8dd8a19e9\": container with ID starting with 2a1b63b89ee1fced3c1d0e2573a0150107d6a676fa4621436d0999a8dd8a19e9 not found: ID does not exist"
Jul 30 17:18:32.177866 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:32.177837    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-94aa4-predictor-86778f57d5-2hz7w"]
Jul 30 17:18:32.183456 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:32.183412    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-94aa4-predictor-86778f57d5-2hz7w"]
Jul 30 17:18:32.200573 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:32.200517    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xttws\" (UniqueName: \"kubernetes.io/projected/e864c7d3-57bd-4355-a494-2912f3f09f74-kube-api-access-xttws\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:18:32.200573 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:32.200576    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e864c7d3-57bd-4355-a494-2912f3f09f74-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:18:32.200772 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:32.200588    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e864c7d3-57bd-4355-a494-2912f3f09f74-proxy-tls\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:18:33.161828 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:33.161787    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq" event={"ID":"902dff9d-901f-4056-bc40-367a61bb1477","Type":"ContainerStarted","Data":"2facb22aee08cb7483134dac1e9aa014d0622b1985864849c7b702eb17f4b837"}
Jul 30 17:18:33.162267 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:33.161891    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq" event={"ID":"902dff9d-901f-4056-bc40-367a61bb1477","Type":"ContainerStarted","Data":"e1668ae1b44e6ba9eba6a9ccb52f87643cc21262baaa7c685ea23c62a43f850d"}
Jul 30 17:18:33.482015 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:33.481926    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e864c7d3-57bd-4355-a494-2912f3f09f74" path="/var/lib/kubelet/pods/e864c7d3-57bd-4355-a494-2912f3f09f74/volumes"
Jul 30 17:18:33.726819 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:33.726762    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc" podUID="7d7d023d-4d93-4b21-8163-4bc6b39b7619" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 30 17:18:36.175690 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:36.175654    2580 generic.go:356] "Generic (PLEG): container finished" podID="902dff9d-901f-4056-bc40-367a61bb1477" containerID="2facb22aee08cb7483134dac1e9aa014d0622b1985864849c7b702eb17f4b837" exitCode=0
Jul 30 17:18:36.176092 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:36.175736    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq" event={"ID":"902dff9d-901f-4056-bc40-367a61bb1477","Type":"ContainerDied","Data":"2facb22aee08cb7483134dac1e9aa014d0622b1985864849c7b702eb17f4b837"}
Jul 30 17:18:37.183248 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:37.183209    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq" event={"ID":"902dff9d-901f-4056-bc40-367a61bb1477","Type":"ContainerStarted","Data":"54795049a620c579d4a74d33b2300cc45f2712e830111d201a2bc0e8b78ca1ee"}
Jul 30 17:18:37.183666 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:37.183281    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq" event={"ID":"902dff9d-901f-4056-bc40-367a61bb1477","Type":"ContainerStarted","Data":"997d869d5aea0186a80a2f14a51a5fbb969c71f60ef113766984ddd290d340e4"}
Jul 30 17:18:37.183666 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:37.183486    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq"
Jul 30 17:18:37.205181 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:37.205104    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq" podStartSLOduration=2.481506901 podStartE2EDuration="6.20508309s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.723576189s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:18:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:18:37.203190356 +0000 UTC m=+1080.278490240" watchObservedRunningTime="2026-07-30 17:18:37.20508309 +0000 UTC m=+1080.280383070"
Jul 30 17:18:38.190988 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:38.190949    2580 generic.go:356] "Generic (PLEG): container finished" podID="7d7d023d-4d93-4b21-8163-4bc6b39b7619" containerID="547cc001beb0aa74ec62b305858b09b486bdcf9385236db9063d449f34214c05" exitCode=0
Jul 30 17:18:38.191458 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:38.191018    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc" event={"ID":"7d7d023d-4d93-4b21-8163-4bc6b39b7619","Type":"ContainerDied","Data":"547cc001beb0aa74ec62b305858b09b486bdcf9385236db9063d449f34214c05"}
Jul 30 17:18:38.191458 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:38.191402    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq"
Jul 30 17:18:38.192386 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:38.192361    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq" podUID="902dff9d-901f-4056-bc40-367a61bb1477" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 30 17:18:38.726559 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:38.726509    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc" podUID="7d7d023d-4d93-4b21-8163-4bc6b39b7619" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 30 17:18:39.196549 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:39.196508    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq" podUID="902dff9d-901f-4056-bc40-367a61bb1477" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 30 17:18:40.478615 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:40.478568    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc" podUID="7d7d023d-4d93-4b21-8163-4bc6b39b7619" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 30 17:18:40.480317 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:40.480287    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc" podUID="7d7d023d-4d93-4b21-8163-4bc6b39b7619" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 17:18:43.726711 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:43.726654    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc" podUID="7d7d023d-4d93-4b21-8163-4bc6b39b7619" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 30 17:18:43.727184 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:43.726819    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc"
Jul 30 17:18:44.201477 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:44.201446    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq"
Jul 30 17:18:44.202124 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:44.202091    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq" podUID="902dff9d-901f-4056-bc40-367a61bb1477" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 30 17:18:48.726701 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:48.726642    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc" podUID="7d7d023d-4d93-4b21-8163-4bc6b39b7619" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 30 17:18:50.479138 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:50.479079    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc" podUID="7d7d023d-4d93-4b21-8163-4bc6b39b7619" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 30 17:18:50.479702 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:50.479676    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc" podUID="7d7d023d-4d93-4b21-8163-4bc6b39b7619" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 17:18:53.726717 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:53.726674    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc" podUID="7d7d023d-4d93-4b21-8163-4bc6b39b7619" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 30 17:18:54.203038 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:54.202995    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq" podUID="902dff9d-901f-4056-bc40-367a61bb1477" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 30 17:18:58.726555 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:18:58.726504    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc" podUID="7d7d023d-4d93-4b21-8163-4bc6b39b7619" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 30 17:19:00.479098 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:19:00.479047    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc" podUID="7d7d023d-4d93-4b21-8163-4bc6b39b7619" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 30 17:19:00.479644 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:19:00.479228    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc"
Jul 30 17:19:00.479886 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:19:00.479858    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc" podUID="7d7d023d-4d93-4b21-8163-4bc6b39b7619" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 17:19:00.480004 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:19:00.479988    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc"
Jul 30 17:19:01.789506 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:19:01.789475    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc"
Jul 30 17:19:01.864388 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:19:01.864347    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qpv5r\" (UniqueName: \"kubernetes.io/projected/7d7d023d-4d93-4b21-8163-4bc6b39b7619-kube-api-access-qpv5r\") pod \"7d7d023d-4d93-4b21-8163-4bc6b39b7619\" (UID: \"7d7d023d-4d93-4b21-8163-4bc6b39b7619\") "
Jul 30 17:19:01.864388 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:19:01.864390    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d7d023d-4d93-4b21-8163-4bc6b39b7619-cabundle-cert\") pod \"7d7d023d-4d93-4b21-8163-4bc6b39b7619\" (UID: \"7d7d023d-4d93-4b21-8163-4bc6b39b7619\") "
Jul 30 17:19:01.864656 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:19:01.864427    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d7d023d-4d93-4b21-8163-4bc6b39b7619-kserve-provision-location\") pod \"7d7d023d-4d93-4b21-8163-4bc6b39b7619\" (UID: \"7d7d023d-4d93-4b21-8163-4bc6b39b7619\") "
Jul 30 17:19:01.864656 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:19:01.864457    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7d7d023d-4d93-4b21-8163-4bc6b39b7619-kube-rbac-proxy-sar-config\") pod \"7d7d023d-4d93-4b21-8163-4bc6b39b7619\" (UID: \"7d7d023d-4d93-4b21-8163-4bc6b39b7619\") "
Jul 30 17:19:01.864656 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:19:01.864476    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d7d023d-4d93-4b21-8163-4bc6b39b7619-proxy-tls\") pod \"7d7d023d-4d93-4b21-8163-4bc6b39b7619\" (UID: \"7d7d023d-4d93-4b21-8163-4bc6b39b7619\") "
Jul 30 17:19:01.864831 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:19:01.864812    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d7d023d-4d93-4b21-8163-4bc6b39b7619-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7d7d023d-4d93-4b21-8163-4bc6b39b7619" (UID: "7d7d023d-4d93-4b21-8163-4bc6b39b7619"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 17:19:01.864893 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:19:01.864833    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d7d023d-4d93-4b21-8163-4bc6b39b7619-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7d7d023d-4d93-4b21-8163-4bc6b39b7619" (UID: "7d7d023d-4d93-4b21-8163-4bc6b39b7619"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:19:01.864893 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:19:01.864797    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d7d023d-4d93-4b21-8163-4bc6b39b7619-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7d7d023d-4d93-4b21-8163-4bc6b39b7619" (UID: "7d7d023d-4d93-4b21-8163-4bc6b39b7619"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:19:01.866709 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:19:01.866687    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d7d023d-4d93-4b21-8163-4bc6b39b7619-kube-api-access-qpv5r" (OuterVolumeSpecName: "kube-api-access-qpv5r") pod "7d7d023d-4d93-4b21-8163-4bc6b39b7619" (UID: "7d7d023d-4d93-4b21-8163-4bc6b39b7619"). InnerVolumeSpecName "kube-api-access-qpv5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:19:01.866790 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:19:01.866703    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d7d023d-4d93-4b21-8163-4bc6b39b7619-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7d7d023d-4d93-4b21-8163-4bc6b39b7619" (UID: "7d7d023d-4d93-4b21-8163-4bc6b39b7619"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:19:01.965993 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:19:01.965883    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d7d023d-4d93-4b21-8163-4bc6b39b7619-kserve-provision-location\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:19:01.965993 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:19:01.965919    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7d7d023d-4d93-4b21-8163-4bc6b39b7619-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:19:01.965993 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:19:01.965930    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d7d023d-4d93-4b21-8163-4bc6b39b7619-proxy-tls\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:19:01.965993 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:19:01.965941    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpv5r\" (UniqueName: \"kubernetes.io/projected/7d7d023d-4d93-4b21-8163-4bc6b39b7619-kube-api-access-qpv5r\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:19:01.965993 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:19:01.965950    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d7d023d-4d93-4b21-8163-4bc6b39b7619-cabundle-cert\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:19:02.311290 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:19:02.311249    2580 generic.go:356] "Generic (PLEG): container finished" podID="7d7d023d-4d93-4b21-8163-4bc6b39b7619" containerID="79b29aecaf690ce02f43c9625621e03e65dc170d6e63da0ca7c4016392117a8e" exitCode=0
Jul 30 17:19:02.311481 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:19:02.311304    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc" event={"ID":"7d7d023d-4d93-4b21-8163-4bc6b39b7619","Type":"ContainerDied","Data":"79b29aecaf690ce02f43c9625621e03e65dc170d6e63da0ca7c4016392117a8e"}
Jul 30 17:19:02.311481 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:19:02.311324    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc"
Jul 30 17:19:02.311481 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:19:02.311389    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc" event={"ID":"7d7d023d-4d93-4b21-8163-4bc6b39b7619","Type":"ContainerDied","Data":"296b7460658b0c18deb44d5c8104e1ce03a544277a71a5eba385a6039acc1de4"}
Jul 30 17:19:02.311481 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:19:02.311420    2580 scope.go:117] "RemoveContainer" containerID="79b29aecaf690ce02f43c9625621e03e65dc170d6e63da0ca7c4016392117a8e"
Jul 30 17:19:02.320641 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:19:02.320618    2580 scope.go:117] "RemoveContainer" containerID="525a46ec86289a65bc089da762b60f9f8d44049acfed1c6ff1abf29867f53863"
Jul 30 17:19:02.329771 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:19:02.329751    2580 scope.go:117] "RemoveContainer" containerID="547cc001beb0aa74ec62b305858b09b486bdcf9385236db9063d449f34214c05"
Jul 30 17:19:02.334444 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:19:02.334409    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc"]
Jul 30 17:19:02.338570 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:19:02.338542    2580 scope.go:117] "RemoveContainer" containerID="9b7098a03b4007924298e16c7d48e8e5035053a2baea25d4bd5c4121ce525a59"
Jul 30 17:19:02.338971 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:19:02.338947    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-94aa4-predictor-7d87468fcd-6vgdc"]
Jul 30 17:19:02.346519 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:19:02.346494    2580 scope.go:117] "RemoveContainer" containerID="79b29aecaf690ce02f43c9625621e03e65dc170d6e63da0ca7c4016392117a8e"
Jul 30 17:19:02.346882 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:19:02.346862    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79b29aecaf690ce02f43c9625621e03e65dc170d6e63da0ca7c4016392117a8e\": container with ID starting with 79b29aecaf690ce02f43c9625621e03e65dc170d6e63da0ca7c4016392117a8e not found: ID does not exist" containerID="79b29aecaf690ce02f43c9625621e03e65dc170d6e63da0ca7c4016392117a8e"
Jul 30 17:19:02.346945 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:19:02.346896    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79b29aecaf690ce02f43c9625621e03e65dc170d6e63da0ca7c4016392117a8e"} err="failed to get container status \"79b29aecaf690ce02f43c9625621e03e65dc170d6e63da0ca7c4016392117a8e\": rpc error: code = NotFound desc = could not find container \"79b29aecaf690ce02f43c9625621e03e65dc170d6e63da0ca7c4016392117a8e\": container with ID starting with 79b29aecaf690ce02f43c9625621e03e65dc170d6e63da0ca7c4016392117a8e not found: ID does not exist"
Jul 30 17:19:02.346945 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:19:02.346921    2580 scope.go:117] "RemoveContainer" containerID="525a46ec86289a65bc089da762b60f9f8d44049acfed1c6ff1abf29867f53863"
Jul 30 17:19:02.347216 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:19:02.347198    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"525a46ec86289a65bc089da762b60f9f8d44049acfed1c6ff1abf29867f53863\": container with ID starting with 525a46ec86289a65bc089da762b60f9f8d44049acfed1c6ff1abf29867f53863 not found: ID does not exist" containerID="525a46ec86289a65bc089da762b60f9f8d44049acfed1c6ff1abf29867f53863"
Jul 30 17:19:02.347272 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:19:02.347221    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"525a46ec86289a65bc089da762b60f9f8d44049acfed1c6ff1abf29867f53863"} err="failed to get container status \"525a46ec86289a65bc089da762b60f9f8d44049acfed1c6ff1abf29867f53863\": rpc error: code = NotFound desc = could not find container \"525a46ec86289a65bc089da762b60f9f8d44049acfed1c6ff1abf29867f53863\": container with ID starting with 525a46ec86289a65bc089da762b60f9f8d44049acfed1c6ff1abf29867f53863 not found: ID does not exist"
Jul 30 17:19:02.347272 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:19:02.347236    2580 scope.go:117] "RemoveContainer" containerID="547cc001beb0aa74ec62b305858b09b486bdcf9385236db9063d449f34214c05"
Jul 30 17:19:02.347477 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:19:02.347460    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"547cc001beb0aa74ec62b305858b09b486bdcf9385236db9063d449f34214c05\": container with ID starting with 547cc001beb0aa74ec62b305858b09b486bdcf9385236db9063d449f34214c05 not found: ID does not exist" containerID="547cc001beb0aa74ec62b305858b09b486bdcf9385236db9063d449f34214c05"
Jul 30 17:19:02.347522 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:19:02.347481    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"547cc001beb0aa74ec62b305858b09b486bdcf9385236db9063d449f34214c05"} err="failed to get container status \"547cc001beb0aa74ec62b305858b09b486bdcf9385236db9063d449f34214c05\": rpc error: code = NotFound desc = could not find container \"547cc001beb0aa74ec62b305858b09b486bdcf9385236db9063d449f34214c05\": container with ID starting with 547cc001beb0aa74ec62b305858b09b486bdcf9385236db9063d449f34214c05 not found: ID does not exist"
Jul 30 17:19:02.347522 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:19:02.347496    2580 scope.go:117] "RemoveContainer" containerID="9b7098a03b4007924298e16c7d48e8e5035053a2baea25d4bd5c4121ce525a59"
Jul 30 17:19:02.347725 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:19:02.347708    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b7098a03b4007924298e16c7d48e8e5035053a2baea25d4bd5c4121ce525a59\": container with ID starting with 9b7098a03b4007924298e16c7d48e8e5035053a2baea25d4bd5c4121ce525a59 not found: ID does not exist" containerID="9b7098a03b4007924298e16c7d48e8e5035053a2baea25d4bd5c4121ce525a59"
Jul 30 17:19:02.347771 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:19:02.347729    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b7098a03b4007924298e16c7d48e8e5035053a2baea25d4bd5c4121ce525a59"} err="failed to get container status \"9b7098a03b4007924298e16c7d48e8e5035053a2baea25d4bd5c4121ce525a59\": rpc error: code = NotFound desc = could not find container \"9b7098a03b4007924298e16c7d48e8e5035053a2baea25d4bd5c4121ce525a59\": container with ID starting with 9b7098a03b4007924298e16c7d48e8e5035053a2baea25d4bd5c4121ce525a59 not found: ID does not exist"
Jul 30 17:19:03.483447 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:19:03.483399    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d7d023d-4d93-4b21-8163-4bc6b39b7619" path="/var/lib/kubelet/pods/7d7d023d-4d93-4b21-8163-4bc6b39b7619/volumes"
Jul 30 17:19:04.202518 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:19:04.202476    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq" podUID="902dff9d-901f-4056-bc40-367a61bb1477" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 30 17:19:14.202826 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:19:14.202781    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq" podUID="902dff9d-901f-4056-bc40-367a61bb1477" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 30 17:19:24.202960 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:19:24.202911    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq" podUID="902dff9d-901f-4056-bc40-367a61bb1477" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 30 17:19:34.202704 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:19:34.202657    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq" podUID="902dff9d-901f-4056-bc40-367a61bb1477" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 30 17:19:44.202840 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:19:44.202794    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq" podUID="902dff9d-901f-4056-bc40-367a61bb1477" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 30 17:19:54.202343 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:19:54.202298    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq" podUID="902dff9d-901f-4056-bc40-367a61bb1477" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 30 17:20:03.478144 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:20:03.478042    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq" podUID="902dff9d-901f-4056-bc40-367a61bb1477" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 30 17:20:13.478214 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:20:13.478142    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq" podUID="902dff9d-901f-4056-bc40-367a61bb1477" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 30 17:20:23.478923 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:20:23.478874    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq" podUID="902dff9d-901f-4056-bc40-367a61bb1477" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 30 17:20:33.477994 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:20:33.477946    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq" podUID="902dff9d-901f-4056-bc40-367a61bb1477" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 30 17:20:37.375769 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:20:37.375737    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-wqszn_f231a916-1fda-4059-9130-d4112e738ee1/console-operator/1.log"
Jul 30 17:20:37.376824 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:20:37.376801    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-wqszn_f231a916-1fda-4059-9130-d4112e738ee1/console-operator/1.log"
Jul 30 17:20:43.478716 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:20:43.478663    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq" podUID="902dff9d-901f-4056-bc40-367a61bb1477" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 30 17:20:53.478420 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:20:53.478370    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq" podUID="902dff9d-901f-4056-bc40-367a61bb1477" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 30 17:21:03.478001 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:03.477948    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq" podUID="902dff9d-901f-4056-bc40-367a61bb1477" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 30 17:21:13.478250 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:13.478199    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq" podUID="902dff9d-901f-4056-bc40-367a61bb1477" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 30 17:21:23.482595 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:23.482564    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq"
Jul 30 17:21:31.862258 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:31.862210    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq"]
Jul 30 17:21:31.862778 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:31.862497    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq" podUID="902dff9d-901f-4056-bc40-367a61bb1477" containerName="kserve-container" containerID="cri-o://997d869d5aea0186a80a2f14a51a5fbb969c71f60ef113766984ddd290d340e4" gracePeriod=30
Jul 30 17:21:31.862778 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:31.862536    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq" podUID="902dff9d-901f-4056-bc40-367a61bb1477" containerName="kube-rbac-proxy" containerID="cri-o://54795049a620c579d4a74d33b2300cc45f2712e830111d201a2bc0e8b78ca1ee" gracePeriod=30
Jul 30 17:21:32.936555 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:32.936512    2580 generic.go:356] "Generic (PLEG): container finished" podID="902dff9d-901f-4056-bc40-367a61bb1477" containerID="54795049a620c579d4a74d33b2300cc45f2712e830111d201a2bc0e8b78ca1ee" exitCode=2
Jul 30 17:21:32.936983 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:32.936584    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq" event={"ID":"902dff9d-901f-4056-bc40-367a61bb1477","Type":"ContainerDied","Data":"54795049a620c579d4a74d33b2300cc45f2712e830111d201a2bc0e8b78ca1ee"}
Jul 30 17:21:33.478381 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:33.478327    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq" podUID="902dff9d-901f-4056-bc40-367a61bb1477" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 30 17:21:34.197584 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:34.197536    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq" podUID="902dff9d-901f-4056-bc40-367a61bb1477" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Jul 30 17:21:37.015374 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:37.015334    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-clswz"]
Jul 30 17:21:37.015783 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:37.015674    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d7d023d-4d93-4b21-8163-4bc6b39b7619" containerName="kserve-container"
Jul 30 17:21:37.015783 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:37.015685    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d7d023d-4d93-4b21-8163-4bc6b39b7619" containerName="kserve-container"
Jul 30 17:21:37.015783 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:37.015694    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d7d023d-4d93-4b21-8163-4bc6b39b7619" containerName="agent"
Jul 30 17:21:37.015783 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:37.015701    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d7d023d-4d93-4b21-8163-4bc6b39b7619" containerName="agent"
Jul 30 17:21:37.015783 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:37.015709    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e864c7d3-57bd-4355-a494-2912f3f09f74" containerName="kserve-container"
Jul 30 17:21:37.015783 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:37.015716    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e864c7d3-57bd-4355-a494-2912f3f09f74" containerName="kserve-container"
Jul 30 17:21:37.015783 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:37.015725    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d7d023d-4d93-4b21-8163-4bc6b39b7619" containerName="storage-initializer"
Jul 30 17:21:37.015783 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:37.015730    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d7d023d-4d93-4b21-8163-4bc6b39b7619" containerName="storage-initializer"
Jul 30 17:21:37.015783 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:37.015742    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d7d023d-4d93-4b21-8163-4bc6b39b7619" containerName="kube-rbac-proxy"
Jul 30 17:21:37.015783 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:37.015748    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d7d023d-4d93-4b21-8163-4bc6b39b7619" containerName="kube-rbac-proxy"
Jul 30 17:21:37.015783 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:37.015754    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e864c7d3-57bd-4355-a494-2912f3f09f74" containerName="kube-rbac-proxy"
Jul 30 17:21:37.015783 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:37.015759    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e864c7d3-57bd-4355-a494-2912f3f09f74" containerName="kube-rbac-proxy"
Jul 30 17:21:37.016138 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:37.015804    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e864c7d3-57bd-4355-a494-2912f3f09f74" containerName="kserve-container"
Jul 30 17:21:37.016138 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:37.015812    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e864c7d3-57bd-4355-a494-2912f3f09f74" containerName="kube-rbac-proxy"
Jul 30 17:21:37.016138 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:37.015818    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d7d023d-4d93-4b21-8163-4bc6b39b7619" containerName="kube-rbac-proxy"
Jul 30 17:21:37.016138 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:37.015825    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d7d023d-4d93-4b21-8163-4bc6b39b7619" containerName="kserve-container"
Jul 30 17:21:37.016138 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:37.015832    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d7d023d-4d93-4b21-8163-4bc6b39b7619" containerName="agent"
Jul 30 17:21:37.018426 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:37.018401    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-clswz"
Jul 30 17:21:37.018822 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:37.018795    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g"]
Jul 30 17:21:37.021219 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:37.021197    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Jul 30 17:21:37.021343 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:37.021232    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Jul 30 17:21:37.022057 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:37.022026    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g"
Jul 30 17:21:37.024679 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:37.024642    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Jul 30 17:21:37.032525 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:37.031624    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-clswz"]
Jul 30 17:21:37.034636 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:37.034605    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g"]
Jul 30 17:21:37.119286 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:37.119246    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c75219c-0563-4aa1-bc78-348c2eedf18f-kserve-provision-location\") pod \"isvc-canary-create-predictor-d9fd7b987-clswz\" (UID: \"9c75219c-0563-4aa1-bc78-348c2eedf18f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-clswz"
Jul 30 17:21:37.119489 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:37.119296    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6502fda1-f6ed-4140-86f0-c393aafd7aa0-proxy-tls\") pod \"isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g\" (UID: \"6502fda1-f6ed-4140-86f0-c393aafd7aa0\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g"
Jul 30 17:21:37.119489 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:37.119380    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6502fda1-f6ed-4140-86f0-c393aafd7aa0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g\" (UID: \"6502fda1-f6ed-4140-86f0-c393aafd7aa0\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g"
Jul 30 17:21:37.119489 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:37.119423    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c75219c-0563-4aa1-bc78-348c2eedf18f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-d9fd7b987-clswz\" (UID: \"9c75219c-0563-4aa1-bc78-348c2eedf18f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-clswz"
Jul 30 17:21:37.119489 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:37.119445    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6502fda1-f6ed-4140-86f0-c393aafd7aa0-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g\" (UID: \"6502fda1-f6ed-4140-86f0-c393aafd7aa0\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g"
Jul 30 17:21:37.119715 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:37.119571    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7xv5\" (UniqueName: \"kubernetes.io/projected/9c75219c-0563-4aa1-bc78-348c2eedf18f-kube-api-access-s7xv5\") pod \"isvc-canary-create-predictor-d9fd7b987-clswz\" (UID: \"9c75219c-0563-4aa1-bc78-348c2eedf18f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-clswz"
Jul 30 17:21:37.119715 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:37.119616    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c75219c-0563-4aa1-bc78-348c2eedf18f-cabundle-cert\") pod \"isvc-canary-create-predictor-d9fd7b987-clswz\" (UID: \"9c75219c-0563-4aa1-bc78-348c2eedf18f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-clswz"
Jul 30 17:21:37.119715 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:37.119688    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt2pw\" (UniqueName: \"kubernetes.io/projected/6502fda1-f6ed-4140-86f0-c393aafd7aa0-kube-api-access-jt2pw\") pod \"isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g\" (UID: \"6502fda1-f6ed-4140-86f0-c393aafd7aa0\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g"
Jul 30 17:21:37.119854 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:37.119724    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c75219c-0563-4aa1-bc78-348c2eedf18f-proxy-tls\") pod \"isvc-canary-create-predictor-d9fd7b987-clswz\" (UID: \"9c75219c-0563-4aa1-bc78-348c2eedf18f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-clswz"
Jul 30 17:21:37.119854 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:37.119750    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6502fda1-f6ed-4140-86f0-c393aafd7aa0-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g\" (UID: \"6502fda1-f6ed-4140-86f0-c393aafd7aa0\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g"
Jul 30 17:21:37.220600 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:37.220563    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7xv5\" (UniqueName: \"kubernetes.io/projected/9c75219c-0563-4aa1-bc78-348c2eedf18f-kube-api-access-s7xv5\") pod \"isvc-canary-create-predictor-d9fd7b987-clswz\" (UID: \"9c75219c-0563-4aa1-bc78-348c2eedf18f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-clswz"
Jul 30 17:21:37.220600 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:37.220607    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c75219c-0563-4aa1-bc78-348c2eedf18f-cabundle-cert\") pod \"isvc-canary-create-predictor-d9fd7b987-clswz\" (UID: \"9c75219c-0563-4aa1-bc78-348c2eedf18f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-clswz"
Jul 30 17:21:37.220865 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:37.220640    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt2pw\" (UniqueName: \"kubernetes.io/projected/6502fda1-f6ed-4140-86f0-c393aafd7aa0-kube-api-access-jt2pw\") pod \"isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g\" (UID: \"6502fda1-f6ed-4140-86f0-c393aafd7aa0\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g"
Jul 30 17:21:37.220865 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:37.220667    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c75219c-0563-4aa1-bc78-348c2eedf18f-proxy-tls\") pod \"isvc-canary-create-predictor-d9fd7b987-clswz\" (UID: \"9c75219c-0563-4aa1-bc78-348c2eedf18f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-clswz"
Jul 30 17:21:37.220865 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:37.220688    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6502fda1-f6ed-4140-86f0-c393aafd7aa0-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g\" (UID: \"6502fda1-f6ed-4140-86f0-c393aafd7aa0\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g"
Jul 30 17:21:37.220865 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:37.220706    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c75219c-0563-4aa1-bc78-348c2eedf18f-kserve-provision-location\") pod \"isvc-canary-create-predictor-d9fd7b987-clswz\" (UID: \"9c75219c-0563-4aa1-bc78-348c2eedf18f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-clswz"
Jul 30 17:21:37.220865 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:37.220730    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6502fda1-f6ed-4140-86f0-c393aafd7aa0-proxy-tls\") pod \"isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g\" (UID: \"6502fda1-f6ed-4140-86f0-c393aafd7aa0\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g"
Jul 30 17:21:37.221097 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:37.220951    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6502fda1-f6ed-4140-86f0-c393aafd7aa0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g\" (UID: \"6502fda1-f6ed-4140-86f0-c393aafd7aa0\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g"
Jul 30 17:21:37.221097 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:37.220995    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c75219c-0563-4aa1-bc78-348c2eedf18f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-d9fd7b987-clswz\" (UID: \"9c75219c-0563-4aa1-bc78-348c2eedf18f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-clswz"
Jul 30 17:21:37.221097 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:37.221022    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6502fda1-f6ed-4140-86f0-c393aafd7aa0-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g\" (UID: \"6502fda1-f6ed-4140-86f0-c393aafd7aa0\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g"
Jul 30 17:21:37.221460 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:37.221434    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6502fda1-f6ed-4140-86f0-c393aafd7aa0-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g\" (UID: \"6502fda1-f6ed-4140-86f0-c393aafd7aa0\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g"
Jul 30 17:21:37.221704 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:37.221659    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c75219c-0563-4aa1-bc78-348c2eedf18f-kserve-provision-location\") pod \"isvc-canary-create-predictor-d9fd7b987-clswz\" (UID: \"9c75219c-0563-4aa1-bc78-348c2eedf18f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-clswz"
Jul 30 17:21:37.221704 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:37.221670    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6502fda1-f6ed-4140-86f0-c393aafd7aa0-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g\" (UID: \"6502fda1-f6ed-4140-86f0-c393aafd7aa0\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g"
Jul 30 17:21:37.221704 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:37.221664    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6502fda1-f6ed-4140-86f0-c393aafd7aa0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g\" (UID: \"6502fda1-f6ed-4140-86f0-c393aafd7aa0\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g"
Jul 30 17:21:37.221882 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:37.221866    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c75219c-0563-4aa1-bc78-348c2eedf18f-cabundle-cert\") pod \"isvc-canary-create-predictor-d9fd7b987-clswz\" (UID: \"9c75219c-0563-4aa1-bc78-348c2eedf18f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-clswz"
Jul 30 17:21:37.222057 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:37.222037    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c75219c-0563-4aa1-bc78-348c2eedf18f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-d9fd7b987-clswz\" (UID: \"9c75219c-0563-4aa1-bc78-348c2eedf18f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-clswz"
Jul 30 17:21:37.223448 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:37.223428    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c75219c-0563-4aa1-bc78-348c2eedf18f-proxy-tls\") pod \"isvc-canary-create-predictor-d9fd7b987-clswz\" (UID: \"9c75219c-0563-4aa1-bc78-348c2eedf18f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-clswz"
Jul 30 17:21:37.223758 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:37.223737    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6502fda1-f6ed-4140-86f0-c393aafd7aa0-proxy-tls\") pod \"isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g\" (UID: \"6502fda1-f6ed-4140-86f0-c393aafd7aa0\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g"
Jul 30 17:21:37.228869 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:37.228843    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt2pw\" (UniqueName: \"kubernetes.io/projected/6502fda1-f6ed-4140-86f0-c393aafd7aa0-kube-api-access-jt2pw\") pod \"isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g\" (UID: \"6502fda1-f6ed-4140-86f0-c393aafd7aa0\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g"
Jul 30 17:21:37.229453 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:37.229431    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7xv5\" (UniqueName: \"kubernetes.io/projected/9c75219c-0563-4aa1-bc78-348c2eedf18f-kube-api-access-s7xv5\") pod \"isvc-canary-create-predictor-d9fd7b987-clswz\" (UID: \"9c75219c-0563-4aa1-bc78-348c2eedf18f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-clswz"
Jul 30 17:21:37.336906 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:37.336808    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-clswz"
Jul 30 17:21:37.343740 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:37.343719    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g"
Jul 30 17:21:37.489839 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:37.489803    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-clswz"]
Jul 30 17:21:37.493153 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:21:37.493123    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c75219c_0563_4aa1_bc78_348c2eedf18f.slice/crio-80b8e9b75a32acaba343cf9615b4f169ce02ff9f43038e264ccc0321e28c449a WatchSource:0}: Error finding container 80b8e9b75a32acaba343cf9615b4f169ce02ff9f43038e264ccc0321e28c449a: Status 404 returned error can't find the container with id 80b8e9b75a32acaba343cf9615b4f169ce02ff9f43038e264ccc0321e28c449a
Jul 30 17:21:37.515102 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:37.515076    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g"]
Jul 30 17:21:37.520407 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:21:37.520382    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6502fda1_f6ed_4140_86f0_c393aafd7aa0.slice/crio-df60b009209727da630391ff206f0a6df3648372493cfe7eabdbf4368f3c5c71 WatchSource:0}: Error finding container df60b009209727da630391ff206f0a6df3648372493cfe7eabdbf4368f3c5c71: Status 404 returned error can't find the container with id df60b009209727da630391ff206f0a6df3648372493cfe7eabdbf4368f3c5c71
Jul 30 17:21:37.959905 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:37.959858    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g" event={"ID":"6502fda1-f6ed-4140-86f0-c393aafd7aa0","Type":"ContainerStarted","Data":"710745b3d5c7c8ac5e65dc1315449c75c2a79e4248e2407c550ec7fd40ceaaf1"}
Jul 30 17:21:37.960089 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:37.959947    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g" event={"ID":"6502fda1-f6ed-4140-86f0-c393aafd7aa0","Type":"ContainerStarted","Data":"df60b009209727da630391ff206f0a6df3648372493cfe7eabdbf4368f3c5c71"}
Jul 30 17:21:37.961611 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:37.961582    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-clswz" event={"ID":"9c75219c-0563-4aa1-bc78-348c2eedf18f","Type":"ContainerStarted","Data":"aaaef553cae6ca1920613be4d197b3bd0fbc2422e2fb85d54c963ad135f6c42b"}
Jul 30 17:21:37.961727 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:37.961647    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-clswz" event={"ID":"9c75219c-0563-4aa1-bc78-348c2eedf18f","Type":"ContainerStarted","Data":"80b8e9b75a32acaba343cf9615b4f169ce02ff9f43038e264ccc0321e28c449a"}
Jul 30 17:21:39.197324 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:39.197280    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq" podUID="902dff9d-901f-4056-bc40-367a61bb1477" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Jul 30 17:21:41.978647 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:41.978553    2580 generic.go:356] "Generic (PLEG): container finished" podID="6502fda1-f6ed-4140-86f0-c393aafd7aa0" containerID="710745b3d5c7c8ac5e65dc1315449c75c2a79e4248e2407c550ec7fd40ceaaf1" exitCode=0
Jul 30 17:21:41.978647 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:41.978632    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g" event={"ID":"6502fda1-f6ed-4140-86f0-c393aafd7aa0","Type":"ContainerDied","Data":"710745b3d5c7c8ac5e65dc1315449c75c2a79e4248e2407c550ec7fd40ceaaf1"}
Jul 30 17:21:41.980377 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:41.980353    2580 generic.go:356] "Generic (PLEG): container finished" podID="9c75219c-0563-4aa1-bc78-348c2eedf18f" containerID="aaaef553cae6ca1920613be4d197b3bd0fbc2422e2fb85d54c963ad135f6c42b" exitCode=0
Jul 30 17:21:41.980479 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:41.980432    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-clswz" event={"ID":"9c75219c-0563-4aa1-bc78-348c2eedf18f","Type":"ContainerDied","Data":"aaaef553cae6ca1920613be4d197b3bd0fbc2422e2fb85d54c963ad135f6c42b"}
Jul 30 17:21:42.711526 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:42.711502    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq"
Jul 30 17:21:42.768546 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:42.768515    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/902dff9d-901f-4056-bc40-367a61bb1477-kube-rbac-proxy-sar-config\") pod \"902dff9d-901f-4056-bc40-367a61bb1477\" (UID: \"902dff9d-901f-4056-bc40-367a61bb1477\") "
Jul 30 17:21:42.768711 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:42.768587    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/902dff9d-901f-4056-bc40-367a61bb1477-cabundle-cert\") pod \"902dff9d-901f-4056-bc40-367a61bb1477\" (UID: \"902dff9d-901f-4056-bc40-367a61bb1477\") "
Jul 30 17:21:42.768711 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:42.768612    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vpfwd\" (UniqueName: \"kubernetes.io/projected/902dff9d-901f-4056-bc40-367a61bb1477-kube-api-access-vpfwd\") pod \"902dff9d-901f-4056-bc40-367a61bb1477\" (UID: \"902dff9d-901f-4056-bc40-367a61bb1477\") "
Jul 30 17:21:42.768711 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:42.768698    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/902dff9d-901f-4056-bc40-367a61bb1477-proxy-tls\") pod \"902dff9d-901f-4056-bc40-367a61bb1477\" (UID: \"902dff9d-901f-4056-bc40-367a61bb1477\") "
Jul 30 17:21:42.768874 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:42.768731    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/902dff9d-901f-4056-bc40-367a61bb1477-kserve-provision-location\") pod \"902dff9d-901f-4056-bc40-367a61bb1477\" (UID: \"902dff9d-901f-4056-bc40-367a61bb1477\") "
Jul 30 17:21:42.769057 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:42.769030    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/902dff9d-901f-4056-bc40-367a61bb1477-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "902dff9d-901f-4056-bc40-367a61bb1477" (UID: "902dff9d-901f-4056-bc40-367a61bb1477"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:21:42.769057 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:42.769048    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/902dff9d-901f-4056-bc40-367a61bb1477-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "902dff9d-901f-4056-bc40-367a61bb1477" (UID: "902dff9d-901f-4056-bc40-367a61bb1477"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:21:42.769225 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:42.769067    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/902dff9d-901f-4056-bc40-367a61bb1477-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "902dff9d-901f-4056-bc40-367a61bb1477" (UID: "902dff9d-901f-4056-bc40-367a61bb1477"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 17:21:42.770820 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:42.770800    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/902dff9d-901f-4056-bc40-367a61bb1477-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "902dff9d-901f-4056-bc40-367a61bb1477" (UID: "902dff9d-901f-4056-bc40-367a61bb1477"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:21:42.770820 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:42.770810    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/902dff9d-901f-4056-bc40-367a61bb1477-kube-api-access-vpfwd" (OuterVolumeSpecName: "kube-api-access-vpfwd") pod "902dff9d-901f-4056-bc40-367a61bb1477" (UID: "902dff9d-901f-4056-bc40-367a61bb1477"). InnerVolumeSpecName "kube-api-access-vpfwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:21:42.869596 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:42.869561    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/902dff9d-901f-4056-bc40-367a61bb1477-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:21:42.869596 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:42.869590    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/902dff9d-901f-4056-bc40-367a61bb1477-cabundle-cert\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:21:42.869596 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:42.869600    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpfwd\" (UniqueName: \"kubernetes.io/projected/902dff9d-901f-4056-bc40-367a61bb1477-kube-api-access-vpfwd\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:21:42.869823 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:42.869610    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/902dff9d-901f-4056-bc40-367a61bb1477-proxy-tls\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:21:42.869823 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:42.869620    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/902dff9d-901f-4056-bc40-367a61bb1477-kserve-provision-location\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:21:42.988925 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:42.988868    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g" event={"ID":"6502fda1-f6ed-4140-86f0-c393aafd7aa0","Type":"ContainerStarted","Data":"81534b8bd06dc8aa24d6546792b48911cff80580bf485ff2e163f11173ffa732"}
Jul 30 17:21:42.989556 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:42.988964    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g" event={"ID":"6502fda1-f6ed-4140-86f0-c393aafd7aa0","Type":"ContainerStarted","Data":"aa28f084cccc05caa5b7cc6ec3081809d77e2e97a8d42e2e3d126e5bad9baa9a"}
Jul 30 17:21:42.989556 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:42.989297    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g"
Jul 30 17:21:42.991248 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:42.991216    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-clswz" event={"ID":"9c75219c-0563-4aa1-bc78-348c2eedf18f","Type":"ContainerStarted","Data":"2932a34384ab5dc3c3dfddef55da64ec76babb56c819dbfa18de01de7da2289d"}
Jul 30 17:21:42.991384 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:42.991285    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-clswz" event={"ID":"9c75219c-0563-4aa1-bc78-348c2eedf18f","Type":"ContainerStarted","Data":"c4b4024de7a5b82a54c74e15fe0245a5594a53b359b575cf6b8d056c32864be8"}
Jul 30 17:21:42.991575 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:42.991542    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-clswz"
Jul 30 17:21:42.991714 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:42.991596    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-clswz"
Jul 30 17:21:42.993356 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:42.993324    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-clswz" podUID="9c75219c-0563-4aa1-bc78-348c2eedf18f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 30 17:21:42.993592 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:42.993565    2580 generic.go:356] "Generic (PLEG): container finished" podID="902dff9d-901f-4056-bc40-367a61bb1477" containerID="997d869d5aea0186a80a2f14a51a5fbb969c71f60ef113766984ddd290d340e4" exitCode=0
Jul 30 17:21:42.993686 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:42.993670    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq" event={"ID":"902dff9d-901f-4056-bc40-367a61bb1477","Type":"ContainerDied","Data":"997d869d5aea0186a80a2f14a51a5fbb969c71f60ef113766984ddd290d340e4"}
Jul 30 17:21:42.993738 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:42.993726    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq" event={"ID":"902dff9d-901f-4056-bc40-367a61bb1477","Type":"ContainerDied","Data":"e1668ae1b44e6ba9eba6a9ccb52f87643cc21262baaa7c685ea23c62a43f850d"}
Jul 30 17:21:42.993781 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:42.993673    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq"
Jul 30 17:21:42.993781 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:42.993765    2580 scope.go:117] "RemoveContainer" containerID="54795049a620c579d4a74d33b2300cc45f2712e830111d201a2bc0e8b78ca1ee"
Jul 30 17:21:43.003913 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:43.003890    2580 scope.go:117] "RemoveContainer" containerID="997d869d5aea0186a80a2f14a51a5fbb969c71f60ef113766984ddd290d340e4"
Jul 30 17:21:43.011773 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:43.011716    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g" podStartSLOduration=2.025867018 podStartE2EDuration="6.011697198s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.98583018s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:21:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:21:43.010400015 +0000 UTC m=+1266.085699927" watchObservedRunningTime="2026-07-30 17:21:43.011697198 +0000 UTC m=+1266.086997412"
Jul 30 17:21:43.013119 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:43.013092    2580 scope.go:117] "RemoveContainer" containerID="2facb22aee08cb7483134dac1e9aa014d0622b1985864849c7b702eb17f4b837"
Jul 30 17:21:43.021762 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:43.021739    2580 scope.go:117] "RemoveContainer" containerID="54795049a620c579d4a74d33b2300cc45f2712e830111d201a2bc0e8b78ca1ee"
Jul 30 17:21:43.022088 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:21:43.022070    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54795049a620c579d4a74d33b2300cc45f2712e830111d201a2bc0e8b78ca1ee\": container with ID starting with 54795049a620c579d4a74d33b2300cc45f2712e830111d201a2bc0e8b78ca1ee not found: ID does not exist" containerID="54795049a620c579d4a74d33b2300cc45f2712e830111d201a2bc0e8b78ca1ee"
Jul 30 17:21:43.022140 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:43.022098    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54795049a620c579d4a74d33b2300cc45f2712e830111d201a2bc0e8b78ca1ee"} err="failed to get container status \"54795049a620c579d4a74d33b2300cc45f2712e830111d201a2bc0e8b78ca1ee\": rpc error: code = NotFound desc = could not find container \"54795049a620c579d4a74d33b2300cc45f2712e830111d201a2bc0e8b78ca1ee\": container with ID starting with 54795049a620c579d4a74d33b2300cc45f2712e830111d201a2bc0e8b78ca1ee not found: ID does not exist"
Jul 30 17:21:43.022140 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:43.022118    2580 scope.go:117] "RemoveContainer" containerID="997d869d5aea0186a80a2f14a51a5fbb969c71f60ef113766984ddd290d340e4"
Jul 30 17:21:43.022412 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:21:43.022392    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"997d869d5aea0186a80a2f14a51a5fbb969c71f60ef113766984ddd290d340e4\": container with ID starting with 997d869d5aea0186a80a2f14a51a5fbb969c71f60ef113766984ddd290d340e4 not found: ID does not exist" containerID="997d869d5aea0186a80a2f14a51a5fbb969c71f60ef113766984ddd290d340e4"
Jul 30 17:21:43.022454 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:43.022419    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"997d869d5aea0186a80a2f14a51a5fbb969c71f60ef113766984ddd290d340e4"} err="failed to get container status \"997d869d5aea0186a80a2f14a51a5fbb969c71f60ef113766984ddd290d340e4\": rpc error: code = NotFound desc = could not find container \"997d869d5aea0186a80a2f14a51a5fbb969c71f60ef113766984ddd290d340e4\": container with ID starting with 997d869d5aea0186a80a2f14a51a5fbb969c71f60ef113766984ddd290d340e4 not found: ID does not exist"
Jul 30 17:21:43.022454 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:43.022439    2580 scope.go:117] "RemoveContainer" containerID="2facb22aee08cb7483134dac1e9aa014d0622b1985864849c7b702eb17f4b837"
Jul 30 17:21:43.022699 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:21:43.022681    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2facb22aee08cb7483134dac1e9aa014d0622b1985864849c7b702eb17f4b837\": container with ID starting with 2facb22aee08cb7483134dac1e9aa014d0622b1985864849c7b702eb17f4b837 not found: ID does not exist" containerID="2facb22aee08cb7483134dac1e9aa014d0622b1985864849c7b702eb17f4b837"
Jul 30 17:21:43.022740 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:43.022706    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2facb22aee08cb7483134dac1e9aa014d0622b1985864849c7b702eb17f4b837"} err="failed to get container status \"2facb22aee08cb7483134dac1e9aa014d0622b1985864849c7b702eb17f4b837\": rpc error: code = NotFound desc = could not find container \"2facb22aee08cb7483134dac1e9aa014d0622b1985864849c7b702eb17f4b837\": container with ID starting with 2facb22aee08cb7483134dac1e9aa014d0622b1985864849c7b702eb17f4b837 not found: ID does not exist"
Jul 30 17:21:43.030668 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:43.030518    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-clswz" podStartSLOduration=3.300632643 podStartE2EDuration="7.030495744s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.729863101s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:21:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:21:43.029821812 +0000 UTC m=+1266.105121726" watchObservedRunningTime="2026-07-30 17:21:43.030495744 +0000 UTC m=+1266.105795635"
Jul 30 17:21:43.043229 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:43.043182    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq"]
Jul 30 17:21:43.047314 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:43.047275    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-467c1-predictor-6d78f86d84-vctqq"]
Jul 30 17:21:43.482889 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:43.482811    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="902dff9d-901f-4056-bc40-367a61bb1477" path="/var/lib/kubelet/pods/902dff9d-901f-4056-bc40-367a61bb1477/volumes"
Jul 30 17:21:43.999667 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:43.999629    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g"
Jul 30 17:21:43.999667 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:43.999654    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-clswz" podUID="9c75219c-0563-4aa1-bc78-348c2eedf18f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 30 17:21:44.001284 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:44.001252    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g" podUID="6502fda1-f6ed-4140-86f0-c393aafd7aa0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 30 17:21:45.008054 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:45.008007    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g" podUID="6502fda1-f6ed-4140-86f0-c393aafd7aa0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 30 17:21:49.003401 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:49.003372    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-clswz"
Jul 30 17:21:49.003865 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:49.003830    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-clswz" podUID="9c75219c-0563-4aa1-bc78-348c2eedf18f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 30 17:21:50.010316 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:50.010288    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g"
Jul 30 17:21:50.010816 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:50.010786    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g" podUID="6502fda1-f6ed-4140-86f0-c393aafd7aa0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 30 17:21:59.004571 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:21:59.004523    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-clswz" podUID="9c75219c-0563-4aa1-bc78-348c2eedf18f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 30 17:22:00.011255 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:22:00.011212    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g" podUID="6502fda1-f6ed-4140-86f0-c393aafd7aa0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 30 17:22:09.004077 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:22:09.004036    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-clswz" podUID="9c75219c-0563-4aa1-bc78-348c2eedf18f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 30 17:22:10.011427 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:22:10.011385    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g" podUID="6502fda1-f6ed-4140-86f0-c393aafd7aa0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 30 17:22:19.004225 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:22:19.004184    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-clswz" podUID="9c75219c-0563-4aa1-bc78-348c2eedf18f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 30 17:22:20.011094 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:22:20.011050    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g" podUID="6502fda1-f6ed-4140-86f0-c393aafd7aa0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 30 17:22:29.003836 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:22:29.003790    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-clswz" podUID="9c75219c-0563-4aa1-bc78-348c2eedf18f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 30 17:22:30.011589 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:22:30.011547    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g" podUID="6502fda1-f6ed-4140-86f0-c393aafd7aa0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 30 17:22:39.004332 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:22:39.004279    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-clswz" podUID="9c75219c-0563-4aa1-bc78-348c2eedf18f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 30 17:22:40.011488 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:22:40.011445    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g" podUID="6502fda1-f6ed-4140-86f0-c393aafd7aa0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 30 17:22:49.003799 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:22:49.003761    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-clswz" podUID="9c75219c-0563-4aa1-bc78-348c2eedf18f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 30 17:22:50.011678 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:22:50.011635    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g" podUID="6502fda1-f6ed-4140-86f0-c393aafd7aa0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 30 17:22:55.482744 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:22:55.482708    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g"
Jul 30 17:22:59.005320 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:22:59.005234    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-clswz"
Jul 30 17:23:07.109536 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:07.109493    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-clswz"]
Jul 30 17:23:07.110445 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:07.110370    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-clswz" podUID="9c75219c-0563-4aa1-bc78-348c2eedf18f" containerName="kserve-container" containerID="cri-o://c4b4024de7a5b82a54c74e15fe0245a5594a53b359b575cf6b8d056c32864be8" gracePeriod=30
Jul 30 17:23:07.110800 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:07.110695    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-clswz" podUID="9c75219c-0563-4aa1-bc78-348c2eedf18f" containerName="kube-rbac-proxy" containerID="cri-o://2932a34384ab5dc3c3dfddef55da64ec76babb56c819dbfa18de01de7da2289d" gracePeriod=30
Jul 30 17:23:07.112558 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:07.112519    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g"]
Jul 30 17:23:07.112869 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:07.112815    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g" podUID="6502fda1-f6ed-4140-86f0-c393aafd7aa0" containerName="kserve-container" containerID="cri-o://aa28f084cccc05caa5b7cc6ec3081809d77e2e97a8d42e2e3d126e5bad9baa9a" gracePeriod=30
Jul 30 17:23:07.113007 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:07.112903    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g" podUID="6502fda1-f6ed-4140-86f0-c393aafd7aa0" containerName="kube-rbac-proxy" containerID="cri-o://81534b8bd06dc8aa24d6546792b48911cff80580bf485ff2e163f11173ffa732" gracePeriod=30
Jul 30 17:23:07.343817 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:07.343778    2580 generic.go:356] "Generic (PLEG): container finished" podID="6502fda1-f6ed-4140-86f0-c393aafd7aa0" containerID="81534b8bd06dc8aa24d6546792b48911cff80580bf485ff2e163f11173ffa732" exitCode=2
Jul 30 17:23:07.344019 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:07.343858    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g" event={"ID":"6502fda1-f6ed-4140-86f0-c393aafd7aa0","Type":"ContainerDied","Data":"81534b8bd06dc8aa24d6546792b48911cff80580bf485ff2e163f11173ffa732"}
Jul 30 17:23:07.345971 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:07.345946    2580 generic.go:356] "Generic (PLEG): container finished" podID="9c75219c-0563-4aa1-bc78-348c2eedf18f" containerID="2932a34384ab5dc3c3dfddef55da64ec76babb56c819dbfa18de01de7da2289d" exitCode=2
Jul 30 17:23:07.346119 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:07.346007    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-clswz" event={"ID":"9c75219c-0563-4aa1-bc78-348c2eedf18f","Type":"ContainerDied","Data":"2932a34384ab5dc3c3dfddef55da64ec76babb56c819dbfa18de01de7da2289d"}
Jul 30 17:23:09.000401 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:09.000353    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-clswz" podUID="9c75219c-0563-4aa1-bc78-348c2eedf18f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 30 17:23:09.004690 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:09.004656    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-clswz" podUID="9c75219c-0563-4aa1-bc78-348c2eedf18f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 30 17:23:10.005808 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:10.005756    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g" podUID="6502fda1-f6ed-4140-86f0-c393aafd7aa0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Jul 30 17:23:12.278296 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:12.278249    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-j5ss4"]
Jul 30 17:23:12.278726 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:12.278690    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="902dff9d-901f-4056-bc40-367a61bb1477" containerName="kube-rbac-proxy"
Jul 30 17:23:12.278726 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:12.278707    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="902dff9d-901f-4056-bc40-367a61bb1477" containerName="kube-rbac-proxy"
Jul 30 17:23:12.278726 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:12.278718    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="902dff9d-901f-4056-bc40-367a61bb1477" containerName="kserve-container"
Jul 30 17:23:12.278726 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:12.278724    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="902dff9d-901f-4056-bc40-367a61bb1477" containerName="kserve-container"
Jul 30 17:23:12.278885 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:12.278742    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="902dff9d-901f-4056-bc40-367a61bb1477" containerName="storage-initializer"
Jul 30 17:23:12.278885 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:12.278751    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="902dff9d-901f-4056-bc40-367a61bb1477" containerName="storage-initializer"
Jul 30 17:23:12.278885 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:12.278831    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="902dff9d-901f-4056-bc40-367a61bb1477" containerName="kube-rbac-proxy"
Jul 30 17:23:12.278885 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:12.278842    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="902dff9d-901f-4056-bc40-367a61bb1477" containerName="kserve-container"
Jul 30 17:23:12.282395 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:12.282355    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-j5ss4"
Jul 30 17:23:12.284572 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:12.284546    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5k5hb"]
Jul 30 17:23:12.284692 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:12.284660    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Jul 30 17:23:12.284859 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:12.284840    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Jul 30 17:23:12.288106 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:12.288085    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5k5hb"
Jul 30 17:23:12.291474 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:12.291434    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Jul 30 17:23:12.292515 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:12.292490    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-j5ss4"]
Jul 30 17:23:12.298297 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:12.298269    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5k5hb"]
Jul 30 17:23:12.325075 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:12.325032    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a9f9719-9ac1-4433-a65c-60e4d8b7d221-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-66f6768f84-5k5hb\" (UID: \"5a9f9719-9ac1-4433-a65c-60e4d8b7d221\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5k5hb"
Jul 30 17:23:12.325075 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:12.325079    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a9f9719-9ac1-4433-a65c-60e4d8b7d221-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-66f6768f84-5k5hb\" (UID: \"5a9f9719-9ac1-4433-a65c-60e4d8b7d221\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5k5hb"
Jul 30 17:23:12.325380 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:12.325121    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9vmt\" (UniqueName: \"kubernetes.io/projected/919c2def-aa85-45f3-8089-522a211eac3b-kube-api-access-q9vmt\") pod \"isvc-canary-promote-predictor-85f6464bd7-j5ss4\" (UID: \"919c2def-aa85-45f3-8089-522a211eac3b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-j5ss4"
Jul 30 17:23:12.325380 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:12.325208    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/919c2def-aa85-45f3-8089-522a211eac3b-cabundle-cert\") pod \"isvc-canary-promote-predictor-85f6464bd7-j5ss4\" (UID: \"919c2def-aa85-45f3-8089-522a211eac3b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-j5ss4"
Jul 30 17:23:12.325380 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:12.325230    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a9f9719-9ac1-4433-a65c-60e4d8b7d221-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-66f6768f84-5k5hb\" (UID: \"5a9f9719-9ac1-4433-a65c-60e4d8b7d221\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5k5hb"
Jul 30 17:23:12.325380 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:12.325251    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a9f9719-9ac1-4433-a65c-60e4d8b7d221-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-66f6768f84-5k5hb\" (UID: \"5a9f9719-9ac1-4433-a65c-60e4d8b7d221\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5k5hb"
Jul 30 17:23:12.325380 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:12.325311    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/919c2def-aa85-45f3-8089-522a211eac3b-kserve-provision-location\") pod \"isvc-canary-promote-predictor-85f6464bd7-j5ss4\" (UID: \"919c2def-aa85-45f3-8089-522a211eac3b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-j5ss4"
Jul 30 17:23:12.325568 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:12.325391    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/919c2def-aa85-45f3-8089-522a211eac3b-proxy-tls\") pod \"isvc-canary-promote-predictor-85f6464bd7-j5ss4\" (UID: \"919c2def-aa85-45f3-8089-522a211eac3b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-j5ss4"
Jul 30 17:23:12.325568 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:12.325448    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rgkh\" (UniqueName: \"kubernetes.io/projected/5a9f9719-9ac1-4433-a65c-60e4d8b7d221-kube-api-access-7rgkh\") pod \"isvc-canary-promote-v2-predictor-66f6768f84-5k5hb\" (UID: \"5a9f9719-9ac1-4433-a65c-60e4d8b7d221\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5k5hb"
Jul 30 17:23:12.325568 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:12.325502    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/919c2def-aa85-45f3-8089-522a211eac3b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-85f6464bd7-j5ss4\" (UID: \"919c2def-aa85-45f3-8089-522a211eac3b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-j5ss4"
Jul 30 17:23:12.425979 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:12.425933    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/919c2def-aa85-45f3-8089-522a211eac3b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-85f6464bd7-j5ss4\" (UID: \"919c2def-aa85-45f3-8089-522a211eac3b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-j5ss4"
Jul 30 17:23:12.426243 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:12.425993    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a9f9719-9ac1-4433-a65c-60e4d8b7d221-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-66f6768f84-5k5hb\" (UID: \"5a9f9719-9ac1-4433-a65c-60e4d8b7d221\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5k5hb"
Jul 30 17:23:12.426243 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:12.426024    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a9f9719-9ac1-4433-a65c-60e4d8b7d221-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-66f6768f84-5k5hb\" (UID: \"5a9f9719-9ac1-4433-a65c-60e4d8b7d221\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5k5hb"
Jul 30 17:23:12.426243 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:12.426050    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9vmt\" (UniqueName: \"kubernetes.io/projected/919c2def-aa85-45f3-8089-522a211eac3b-kube-api-access-q9vmt\") pod \"isvc-canary-promote-predictor-85f6464bd7-j5ss4\" (UID: \"919c2def-aa85-45f3-8089-522a211eac3b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-j5ss4"
Jul 30 17:23:12.426243 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:12.426088    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/919c2def-aa85-45f3-8089-522a211eac3b-cabundle-cert\") pod \"isvc-canary-promote-predictor-85f6464bd7-j5ss4\" (UID: \"919c2def-aa85-45f3-8089-522a211eac3b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-j5ss4"
Jul 30 17:23:12.426243 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:12.426111    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a9f9719-9ac1-4433-a65c-60e4d8b7d221-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-66f6768f84-5k5hb\" (UID: \"5a9f9719-9ac1-4433-a65c-60e4d8b7d221\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5k5hb"
Jul 30 17:23:12.426243 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:12.426137    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a9f9719-9ac1-4433-a65c-60e4d8b7d221-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-66f6768f84-5k5hb\" (UID: \"5a9f9719-9ac1-4433-a65c-60e4d8b7d221\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5k5hb"
Jul 30 17:23:12.426243 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:12.426190    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/919c2def-aa85-45f3-8089-522a211eac3b-kserve-provision-location\") pod \"isvc-canary-promote-predictor-85f6464bd7-j5ss4\" (UID: \"919c2def-aa85-45f3-8089-522a211eac3b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-j5ss4"
Jul 30 17:23:12.426243 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:12.426230    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/919c2def-aa85-45f3-8089-522a211eac3b-proxy-tls\") pod \"isvc-canary-promote-predictor-85f6464bd7-j5ss4\" (UID: \"919c2def-aa85-45f3-8089-522a211eac3b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-j5ss4"
Jul 30 17:23:12.426617 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:12.426272    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rgkh\" (UniqueName: \"kubernetes.io/projected/5a9f9719-9ac1-4433-a65c-60e4d8b7d221-kube-api-access-7rgkh\") pod \"isvc-canary-promote-v2-predictor-66f6768f84-5k5hb\" (UID: \"5a9f9719-9ac1-4433-a65c-60e4d8b7d221\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5k5hb"
Jul 30 17:23:12.427200 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:12.426766    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a9f9719-9ac1-4433-a65c-60e4d8b7d221-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-66f6768f84-5k5hb\" (UID: \"5a9f9719-9ac1-4433-a65c-60e4d8b7d221\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5k5hb"
Jul 30 17:23:12.427200 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:12.426867    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a9f9719-9ac1-4433-a65c-60e4d8b7d221-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-66f6768f84-5k5hb\" (UID: \"5a9f9719-9ac1-4433-a65c-60e4d8b7d221\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5k5hb"
Jul 30 17:23:12.427200 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:12.426871    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/919c2def-aa85-45f3-8089-522a211eac3b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-85f6464bd7-j5ss4\" (UID: \"919c2def-aa85-45f3-8089-522a211eac3b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-j5ss4"
Jul 30 17:23:12.427200 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:12.427047    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/919c2def-aa85-45f3-8089-522a211eac3b-kserve-provision-location\") pod \"isvc-canary-promote-predictor-85f6464bd7-j5ss4\" (UID: \"919c2def-aa85-45f3-8089-522a211eac3b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-j5ss4"
Jul 30 17:23:12.427200 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:12.427197    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/919c2def-aa85-45f3-8089-522a211eac3b-cabundle-cert\") pod \"isvc-canary-promote-predictor-85f6464bd7-j5ss4\" (UID: \"919c2def-aa85-45f3-8089-522a211eac3b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-j5ss4"
Jul 30 17:23:12.427546 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:12.427313    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a9f9719-9ac1-4433-a65c-60e4d8b7d221-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-66f6768f84-5k5hb\" (UID: \"5a9f9719-9ac1-4433-a65c-60e4d8b7d221\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5k5hb"
Jul 30 17:23:12.429121 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:12.429088    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a9f9719-9ac1-4433-a65c-60e4d8b7d221-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-66f6768f84-5k5hb\" (UID: \"5a9f9719-9ac1-4433-a65c-60e4d8b7d221\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5k5hb"
Jul 30 17:23:12.429647 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:12.429622    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/919c2def-aa85-45f3-8089-522a211eac3b-proxy-tls\") pod \"isvc-canary-promote-predictor-85f6464bd7-j5ss4\" (UID: \"919c2def-aa85-45f3-8089-522a211eac3b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-j5ss4"
Jul 30 17:23:12.434010 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:12.433976    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rgkh\" (UniqueName: \"kubernetes.io/projected/5a9f9719-9ac1-4433-a65c-60e4d8b7d221-kube-api-access-7rgkh\") pod \"isvc-canary-promote-v2-predictor-66f6768f84-5k5hb\" (UID: \"5a9f9719-9ac1-4433-a65c-60e4d8b7d221\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5k5hb"
Jul 30 17:23:12.434250 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:12.434213    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9vmt\" (UniqueName: \"kubernetes.io/projected/919c2def-aa85-45f3-8089-522a211eac3b-kube-api-access-q9vmt\") pod \"isvc-canary-promote-predictor-85f6464bd7-j5ss4\" (UID: \"919c2def-aa85-45f3-8089-522a211eac3b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-j5ss4"
Jul 30 17:23:12.594636 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:12.594523    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-j5ss4"
Jul 30 17:23:12.603059 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:12.603028    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5k5hb"
Jul 30 17:23:12.761485 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:12.761441    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-j5ss4"]
Jul 30 17:23:12.764853 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:23:12.764822    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod919c2def_aa85_45f3_8089_522a211eac3b.slice/crio-d01b946456b36d3c7d757de6ad40174d239a3d6007189615aeb4cf6344f18eba WatchSource:0}: Error finding container d01b946456b36d3c7d757de6ad40174d239a3d6007189615aeb4cf6344f18eba: Status 404 returned error can't find the container with id d01b946456b36d3c7d757de6ad40174d239a3d6007189615aeb4cf6344f18eba
Jul 30 17:23:12.780100 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:12.780071    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5k5hb"]
Jul 30 17:23:12.782808 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:23:12.782784    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a9f9719_9ac1_4433_a65c_60e4d8b7d221.slice/crio-6b7fbfa38a81364f9ebce7f589ca63eb4fa398a52a53561a5b7464222a87e876 WatchSource:0}: Error finding container 6b7fbfa38a81364f9ebce7f589ca63eb4fa398a52a53561a5b7464222a87e876: Status 404 returned error can't find the container with id 6b7fbfa38a81364f9ebce7f589ca63eb4fa398a52a53561a5b7464222a87e876
Jul 30 17:23:13.377203 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:13.377074    2580 generic.go:356] "Generic (PLEG): container finished" podID="9c75219c-0563-4aa1-bc78-348c2eedf18f" containerID="c4b4024de7a5b82a54c74e15fe0245a5594a53b359b575cf6b8d056c32864be8" exitCode=0
Jul 30 17:23:13.377203 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:13.377139    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-clswz" event={"ID":"9c75219c-0563-4aa1-bc78-348c2eedf18f","Type":"ContainerDied","Data":"c4b4024de7a5b82a54c74e15fe0245a5594a53b359b575cf6b8d056c32864be8"}
Jul 30 17:23:13.379303 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:13.379267    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5k5hb" event={"ID":"5a9f9719-9ac1-4433-a65c-60e4d8b7d221","Type":"ContainerStarted","Data":"916b1465204489e81c96492490d71389ca73cc0eba8f2afd2b50d5cdfed382e2"}
Jul 30 17:23:13.379497 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:13.379345    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5k5hb" event={"ID":"5a9f9719-9ac1-4433-a65c-60e4d8b7d221","Type":"ContainerStarted","Data":"6b7fbfa38a81364f9ebce7f589ca63eb4fa398a52a53561a5b7464222a87e876"}
Jul 30 17:23:13.381056 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:13.381026    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-j5ss4" event={"ID":"919c2def-aa85-45f3-8089-522a211eac3b","Type":"ContainerStarted","Data":"f3c8e6071c7110cf3c92e8852bc686399de85c90f6e7830d06f72e1c02f74c29"}
Jul 30 17:23:13.381219 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:13.381090    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-j5ss4" event={"ID":"919c2def-aa85-45f3-8089-522a211eac3b","Type":"ContainerStarted","Data":"d01b946456b36d3c7d757de6ad40174d239a3d6007189615aeb4cf6344f18eba"}
Jul 30 17:23:13.429018 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:13.428986    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-clswz"
Jul 30 17:23:13.537509 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:13.537457    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c75219c-0563-4aa1-bc78-348c2eedf18f-kserve-provision-location\") pod \"9c75219c-0563-4aa1-bc78-348c2eedf18f\" (UID: \"9c75219c-0563-4aa1-bc78-348c2eedf18f\") "
Jul 30 17:23:13.537734 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:13.537556    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c75219c-0563-4aa1-bc78-348c2eedf18f-proxy-tls\") pod \"9c75219c-0563-4aa1-bc78-348c2eedf18f\" (UID: \"9c75219c-0563-4aa1-bc78-348c2eedf18f\") "
Jul 30 17:23:13.537734 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:13.537625    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c75219c-0563-4aa1-bc78-348c2eedf18f-kube-rbac-proxy-sar-config\") pod \"9c75219c-0563-4aa1-bc78-348c2eedf18f\" (UID: \"9c75219c-0563-4aa1-bc78-348c2eedf18f\") "
Jul 30 17:23:13.537734 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:13.537703    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s7xv5\" (UniqueName: \"kubernetes.io/projected/9c75219c-0563-4aa1-bc78-348c2eedf18f-kube-api-access-s7xv5\") pod \"9c75219c-0563-4aa1-bc78-348c2eedf18f\" (UID: \"9c75219c-0563-4aa1-bc78-348c2eedf18f\") "
Jul 30 17:23:13.537900 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:13.537739    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c75219c-0563-4aa1-bc78-348c2eedf18f-cabundle-cert\") pod \"9c75219c-0563-4aa1-bc78-348c2eedf18f\" (UID: \"9c75219c-0563-4aa1-bc78-348c2eedf18f\") "
Jul 30 17:23:13.537900 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:13.537845    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c75219c-0563-4aa1-bc78-348c2eedf18f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9c75219c-0563-4aa1-bc78-348c2eedf18f" (UID: "9c75219c-0563-4aa1-bc78-348c2eedf18f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 17:23:13.538120 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:13.538019    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c75219c-0563-4aa1-bc78-348c2eedf18f-kserve-provision-location\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:23:13.538407 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:13.538228    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c75219c-0563-4aa1-bc78-348c2eedf18f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9c75219c-0563-4aa1-bc78-348c2eedf18f" (UID: "9c75219c-0563-4aa1-bc78-348c2eedf18f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:23:13.538507 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:13.538429    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c75219c-0563-4aa1-bc78-348c2eedf18f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9c75219c-0563-4aa1-bc78-348c2eedf18f" (UID: "9c75219c-0563-4aa1-bc78-348c2eedf18f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:23:13.540014 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:13.539984    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c75219c-0563-4aa1-bc78-348c2eedf18f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9c75219c-0563-4aa1-bc78-348c2eedf18f" (UID: "9c75219c-0563-4aa1-bc78-348c2eedf18f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:23:13.540305 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:13.540243    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c75219c-0563-4aa1-bc78-348c2eedf18f-kube-api-access-s7xv5" (OuterVolumeSpecName: "kube-api-access-s7xv5") pod "9c75219c-0563-4aa1-bc78-348c2eedf18f" (UID: "9c75219c-0563-4aa1-bc78-348c2eedf18f"). InnerVolumeSpecName "kube-api-access-s7xv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:23:13.584340 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:13.584311    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g"
Jul 30 17:23:13.639410 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:13.639301    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6502fda1-f6ed-4140-86f0-c393aafd7aa0-kube-rbac-proxy-sar-config\") pod \"6502fda1-f6ed-4140-86f0-c393aafd7aa0\" (UID: \"6502fda1-f6ed-4140-86f0-c393aafd7aa0\") "
Jul 30 17:23:13.639410 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:13.639358    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6502fda1-f6ed-4140-86f0-c393aafd7aa0-cabundle-cert\") pod \"6502fda1-f6ed-4140-86f0-c393aafd7aa0\" (UID: \"6502fda1-f6ed-4140-86f0-c393aafd7aa0\") "
Jul 30 17:23:13.639410 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:13.639382    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6502fda1-f6ed-4140-86f0-c393aafd7aa0-kserve-provision-location\") pod \"6502fda1-f6ed-4140-86f0-c393aafd7aa0\" (UID: \"6502fda1-f6ed-4140-86f0-c393aafd7aa0\") "
Jul 30 17:23:13.639713 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:13.639441    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6502fda1-f6ed-4140-86f0-c393aafd7aa0-proxy-tls\") pod \"6502fda1-f6ed-4140-86f0-c393aafd7aa0\" (UID: \"6502fda1-f6ed-4140-86f0-c393aafd7aa0\") "
Jul 30 17:23:13.639713 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:13.639468    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jt2pw\" (UniqueName: \"kubernetes.io/projected/6502fda1-f6ed-4140-86f0-c393aafd7aa0-kube-api-access-jt2pw\") pod \"6502fda1-f6ed-4140-86f0-c393aafd7aa0\" (UID: \"6502fda1-f6ed-4140-86f0-c393aafd7aa0\") "
Jul 30 17:23:13.639713 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:13.639689    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c75219c-0563-4aa1-bc78-348c2eedf18f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:23:13.639857 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:13.639710    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7xv5\" (UniqueName: \"kubernetes.io/projected/9c75219c-0563-4aa1-bc78-348c2eedf18f-kube-api-access-s7xv5\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:23:13.639857 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:13.639728    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c75219c-0563-4aa1-bc78-348c2eedf18f-cabundle-cert\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:23:13.639857 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:13.639744    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c75219c-0563-4aa1-bc78-348c2eedf18f-proxy-tls\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:23:13.639857 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:13.639773    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6502fda1-f6ed-4140-86f0-c393aafd7aa0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6502fda1-f6ed-4140-86f0-c393aafd7aa0" (UID: "6502fda1-f6ed-4140-86f0-c393aafd7aa0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:23:13.639857 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:13.639793    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6502fda1-f6ed-4140-86f0-c393aafd7aa0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6502fda1-f6ed-4140-86f0-c393aafd7aa0" (UID: "6502fda1-f6ed-4140-86f0-c393aafd7aa0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:23:13.639857 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:13.639778    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6502fda1-f6ed-4140-86f0-c393aafd7aa0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6502fda1-f6ed-4140-86f0-c393aafd7aa0" (UID: "6502fda1-f6ed-4140-86f0-c393aafd7aa0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 17:23:13.641730 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:13.641706    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6502fda1-f6ed-4140-86f0-c393aafd7aa0-kube-api-access-jt2pw" (OuterVolumeSpecName: "kube-api-access-jt2pw") pod "6502fda1-f6ed-4140-86f0-c393aafd7aa0" (UID: "6502fda1-f6ed-4140-86f0-c393aafd7aa0"). InnerVolumeSpecName "kube-api-access-jt2pw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:23:13.641856 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:13.641732    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6502fda1-f6ed-4140-86f0-c393aafd7aa0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6502fda1-f6ed-4140-86f0-c393aafd7aa0" (UID: "6502fda1-f6ed-4140-86f0-c393aafd7aa0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:23:13.740553 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:13.740505    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6502fda1-f6ed-4140-86f0-c393aafd7aa0-proxy-tls\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:23:13.740553 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:13.740542    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jt2pw\" (UniqueName: \"kubernetes.io/projected/6502fda1-f6ed-4140-86f0-c393aafd7aa0-kube-api-access-jt2pw\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:23:13.740553 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:13.740552    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6502fda1-f6ed-4140-86f0-c393aafd7aa0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:23:13.740553 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:13.740562    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6502fda1-f6ed-4140-86f0-c393aafd7aa0-cabundle-cert\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:23:13.740845 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:13.740571    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6502fda1-f6ed-4140-86f0-c393aafd7aa0-kserve-provision-location\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:23:14.387120 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:14.387081    2580 generic.go:356] "Generic (PLEG): container finished" podID="6502fda1-f6ed-4140-86f0-c393aafd7aa0" containerID="aa28f084cccc05caa5b7cc6ec3081809d77e2e97a8d42e2e3d126e5bad9baa9a" exitCode=0
Jul 30 17:23:14.387636 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:14.387140    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g"
Jul 30 17:23:14.387636 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:14.387188    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g" event={"ID":"6502fda1-f6ed-4140-86f0-c393aafd7aa0","Type":"ContainerDied","Data":"aa28f084cccc05caa5b7cc6ec3081809d77e2e97a8d42e2e3d126e5bad9baa9a"}
Jul 30 17:23:14.387636 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:14.387269    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g" event={"ID":"6502fda1-f6ed-4140-86f0-c393aafd7aa0","Type":"ContainerDied","Data":"df60b009209727da630391ff206f0a6df3648372493cfe7eabdbf4368f3c5c71"}
Jul 30 17:23:14.387636 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:14.387306    2580 scope.go:117] "RemoveContainer" containerID="81534b8bd06dc8aa24d6546792b48911cff80580bf485ff2e163f11173ffa732"
Jul 30 17:23:14.389327 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:14.389295    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-clswz" event={"ID":"9c75219c-0563-4aa1-bc78-348c2eedf18f","Type":"ContainerDied","Data":"80b8e9b75a32acaba343cf9615b4f169ce02ff9f43038e264ccc0321e28c449a"}
Jul 30 17:23:14.389327 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:14.389321    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-clswz"
Jul 30 17:23:14.399075 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:14.399038    2580 scope.go:117] "RemoveContainer" containerID="aa28f084cccc05caa5b7cc6ec3081809d77e2e97a8d42e2e3d126e5bad9baa9a"
Jul 30 17:23:14.411955 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:14.411919    2580 scope.go:117] "RemoveContainer" containerID="710745b3d5c7c8ac5e65dc1315449c75c2a79e4248e2407c550ec7fd40ceaaf1"
Jul 30 17:23:14.419803 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:14.419771    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-clswz"]
Jul 30 17:23:14.420611 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:14.420592    2580 scope.go:117] "RemoveContainer" containerID="81534b8bd06dc8aa24d6546792b48911cff80580bf485ff2e163f11173ffa732"
Jul 30 17:23:14.420952 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:23:14.420932    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81534b8bd06dc8aa24d6546792b48911cff80580bf485ff2e163f11173ffa732\": container with ID starting with 81534b8bd06dc8aa24d6546792b48911cff80580bf485ff2e163f11173ffa732 not found: ID does not exist" containerID="81534b8bd06dc8aa24d6546792b48911cff80580bf485ff2e163f11173ffa732"
Jul 30 17:23:14.421014 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:14.420962    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81534b8bd06dc8aa24d6546792b48911cff80580bf485ff2e163f11173ffa732"} err="failed to get container status \"81534b8bd06dc8aa24d6546792b48911cff80580bf485ff2e163f11173ffa732\": rpc error: code = NotFound desc = could not find container \"81534b8bd06dc8aa24d6546792b48911cff80580bf485ff2e163f11173ffa732\": container with ID starting with 81534b8bd06dc8aa24d6546792b48911cff80580bf485ff2e163f11173ffa732 not found: ID does not exist"
Jul 30 17:23:14.421014 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:14.420986    2580 scope.go:117] "RemoveContainer" containerID="aa28f084cccc05caa5b7cc6ec3081809d77e2e97a8d42e2e3d126e5bad9baa9a"
Jul 30 17:23:14.421257 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:23:14.421241    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa28f084cccc05caa5b7cc6ec3081809d77e2e97a8d42e2e3d126e5bad9baa9a\": container with ID starting with aa28f084cccc05caa5b7cc6ec3081809d77e2e97a8d42e2e3d126e5bad9baa9a not found: ID does not exist" containerID="aa28f084cccc05caa5b7cc6ec3081809d77e2e97a8d42e2e3d126e5bad9baa9a"
Jul 30 17:23:14.421300 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:14.421262    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa28f084cccc05caa5b7cc6ec3081809d77e2e97a8d42e2e3d126e5bad9baa9a"} err="failed to get container status \"aa28f084cccc05caa5b7cc6ec3081809d77e2e97a8d42e2e3d126e5bad9baa9a\": rpc error: code = NotFound desc = could not find container \"aa28f084cccc05caa5b7cc6ec3081809d77e2e97a8d42e2e3d126e5bad9baa9a\": container with ID starting with aa28f084cccc05caa5b7cc6ec3081809d77e2e97a8d42e2e3d126e5bad9baa9a not found: ID does not exist"
Jul 30 17:23:14.421300 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:14.421276    2580 scope.go:117] "RemoveContainer" containerID="710745b3d5c7c8ac5e65dc1315449c75c2a79e4248e2407c550ec7fd40ceaaf1"
Jul 30 17:23:14.421504 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:23:14.421478    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"710745b3d5c7c8ac5e65dc1315449c75c2a79e4248e2407c550ec7fd40ceaaf1\": container with ID starting with 710745b3d5c7c8ac5e65dc1315449c75c2a79e4248e2407c550ec7fd40ceaaf1 not found: ID does not exist" containerID="710745b3d5c7c8ac5e65dc1315449c75c2a79e4248e2407c550ec7fd40ceaaf1"
Jul 30 17:23:14.421580 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:14.421515    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"710745b3d5c7c8ac5e65dc1315449c75c2a79e4248e2407c550ec7fd40ceaaf1"} err="failed to get container status \"710745b3d5c7c8ac5e65dc1315449c75c2a79e4248e2407c550ec7fd40ceaaf1\": rpc error: code = NotFound desc = could not find container \"710745b3d5c7c8ac5e65dc1315449c75c2a79e4248e2407c550ec7fd40ceaaf1\": container with ID starting with 710745b3d5c7c8ac5e65dc1315449c75c2a79e4248e2407c550ec7fd40ceaaf1 not found: ID does not exist"
Jul 30 17:23:14.421580 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:14.421540    2580 scope.go:117] "RemoveContainer" containerID="2932a34384ab5dc3c3dfddef55da64ec76babb56c819dbfa18de01de7da2289d"
Jul 30 17:23:14.426466 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:14.426444    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-clswz"]
Jul 30 17:23:14.430017 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:14.429998    2580 scope.go:117] "RemoveContainer" containerID="c4b4024de7a5b82a54c74e15fe0245a5594a53b359b575cf6b8d056c32864be8"
Jul 30 17:23:14.437868 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:14.437840    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g"]
Jul 30 17:23:14.439963 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:14.439937    2580 scope.go:117] "RemoveContainer" containerID="aaaef553cae6ca1920613be4d197b3bd0fbc2422e2fb85d54c963ad135f6c42b"
Jul 30 17:23:14.444540 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:14.444509    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-mqj9g"]
Jul 30 17:23:15.482278 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:15.482239    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6502fda1-f6ed-4140-86f0-c393aafd7aa0" path="/var/lib/kubelet/pods/6502fda1-f6ed-4140-86f0-c393aafd7aa0/volumes"
Jul 30 17:23:15.482767 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:15.482753    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c75219c-0563-4aa1-bc78-348c2eedf18f" path="/var/lib/kubelet/pods/9c75219c-0563-4aa1-bc78-348c2eedf18f/volumes"
Jul 30 17:23:17.407412 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:17.407376    2580 generic.go:356] "Generic (PLEG): container finished" podID="5a9f9719-9ac1-4433-a65c-60e4d8b7d221" containerID="916b1465204489e81c96492490d71389ca73cc0eba8f2afd2b50d5cdfed382e2" exitCode=0
Jul 30 17:23:17.407873 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:17.407449    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5k5hb" event={"ID":"5a9f9719-9ac1-4433-a65c-60e4d8b7d221","Type":"ContainerDied","Data":"916b1465204489e81c96492490d71389ca73cc0eba8f2afd2b50d5cdfed382e2"}
Jul 30 17:23:17.409185 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:17.409146    2580 generic.go:356] "Generic (PLEG): container finished" podID="919c2def-aa85-45f3-8089-522a211eac3b" containerID="f3c8e6071c7110cf3c92e8852bc686399de85c90f6e7830d06f72e1c02f74c29" exitCode=0
Jul 30 17:23:17.409324 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:17.409227    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-j5ss4" event={"ID":"919c2def-aa85-45f3-8089-522a211eac3b","Type":"ContainerDied","Data":"f3c8e6071c7110cf3c92e8852bc686399de85c90f6e7830d06f72e1c02f74c29"}
Jul 30 17:23:18.416903 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:18.416858    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5k5hb" event={"ID":"5a9f9719-9ac1-4433-a65c-60e4d8b7d221","Type":"ContainerStarted","Data":"efe2fe9af915cebda3f5b4d48ec39bf519aa0ffa8aae76eadd7e3ba20611aba5"}
Jul 30 17:23:18.417411 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:18.416984    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5k5hb" event={"ID":"5a9f9719-9ac1-4433-a65c-60e4d8b7d221","Type":"ContainerStarted","Data":"56663149dae63d22e528eeb974c068af1d96ded6ba15ff2dcac2928992bdac7a"}
Jul 30 17:23:18.417411 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:18.417214    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5k5hb"
Jul 30 17:23:18.419054 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:18.419022    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-j5ss4" event={"ID":"919c2def-aa85-45f3-8089-522a211eac3b","Type":"ContainerStarted","Data":"960b19fbd3d78432e5ee2c46407471cd0c13b078212ae62b369648f9db800fb1"}
Jul 30 17:23:18.419203 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:18.419083    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-j5ss4" event={"ID":"919c2def-aa85-45f3-8089-522a211eac3b","Type":"ContainerStarted","Data":"838facefad39cfbaf216d35a88b12ecabbf6b7315bb51bf6a70915b54bf3deb4"}
Jul 30 17:23:18.419310 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:18.419292    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-j5ss4"
Jul 30 17:23:18.437029 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:18.436976    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5k5hb" podStartSLOduration=2.359603856 podStartE2EDuration="6.436959476s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.07735562s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:23:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:23:18.435660884 +0000 UTC m=+1361.510960783" watchObservedRunningTime="2026-07-30 17:23:18.436959476 +0000 UTC m=+1361.512259393"
Jul 30 17:23:18.454533 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:18.454469    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-j5ss4" podStartSLOduration=2.711388359 podStartE2EDuration="6.454447802s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.743059443s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:23:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:23:18.452388911 +0000 UTC m=+1361.527688802" watchObservedRunningTime="2026-07-30 17:23:18.454447802 +0000 UTC m=+1361.529747692"
Jul 30 17:23:19.425356 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:19.425316    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-j5ss4"
Jul 30 17:23:19.425861 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:19.425384    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5k5hb"
Jul 30 17:23:19.426321 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:19.426280    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5k5hb" podUID="5a9f9719-9ac1-4433-a65c-60e4d8b7d221" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 30 17:23:19.426452 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:19.426394    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-j5ss4" podUID="919c2def-aa85-45f3-8089-522a211eac3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 30 17:23:20.430883 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:20.430840    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5k5hb" podUID="5a9f9719-9ac1-4433-a65c-60e4d8b7d221" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 30 17:23:20.431392 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:20.430841    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-j5ss4" podUID="919c2def-aa85-45f3-8089-522a211eac3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 30 17:23:25.435831 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:25.435794    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-j5ss4"
Jul 30 17:23:25.436312 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:25.436044    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5k5hb"
Jul 30 17:23:25.436385 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:25.436338    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-j5ss4" podUID="919c2def-aa85-45f3-8089-522a211eac3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 30 17:23:25.436815 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:25.436791    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5k5hb" podUID="5a9f9719-9ac1-4433-a65c-60e4d8b7d221" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 30 17:23:35.436297 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:35.436256    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-j5ss4" podUID="919c2def-aa85-45f3-8089-522a211eac3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 30 17:23:35.436839 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:35.436808    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5k5hb" podUID="5a9f9719-9ac1-4433-a65c-60e4d8b7d221" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 30 17:23:45.436727 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:45.436669    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-j5ss4" podUID="919c2def-aa85-45f3-8089-522a211eac3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 30 17:23:45.437262 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:45.436831    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5k5hb" podUID="5a9f9719-9ac1-4433-a65c-60e4d8b7d221" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 30 17:23:55.436892 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:55.436849    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5k5hb" podUID="5a9f9719-9ac1-4433-a65c-60e4d8b7d221" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 30 17:23:55.437340 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:23:55.436848    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-j5ss4" podUID="919c2def-aa85-45f3-8089-522a211eac3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 30 17:24:05.437029 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:24:05.436989    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5k5hb" podUID="5a9f9719-9ac1-4433-a65c-60e4d8b7d221" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 30 17:24:05.437477 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:24:05.436991    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-j5ss4" podUID="919c2def-aa85-45f3-8089-522a211eac3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 30 17:24:15.436329 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:24:15.436282    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-j5ss4" podUID="919c2def-aa85-45f3-8089-522a211eac3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 30 17:24:15.436825 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:24:15.436752    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5k5hb" podUID="5a9f9719-9ac1-4433-a65c-60e4d8b7d221" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 30 17:24:25.436576 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:24:25.436534    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-j5ss4" podUID="919c2def-aa85-45f3-8089-522a211eac3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 30 17:24:25.437370 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:24:25.436673    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5k5hb" podUID="5a9f9719-9ac1-4433-a65c-60e4d8b7d221" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 30 17:24:35.436942 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:24:35.436847    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-j5ss4" podUID="919c2def-aa85-45f3-8089-522a211eac3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 30 17:24:35.437391 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:24:35.436847    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5k5hb" podUID="5a9f9719-9ac1-4433-a65c-60e4d8b7d221" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 30 17:24:42.478332 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:24:42.478295    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-j5ss4"
Jul 30 17:24:44.478326 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:24:44.478289    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5k5hb"
Jul 30 17:24:52.393600 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:24:52.393550    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-j5ss4"]
Jul 30 17:24:52.394083 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:24:52.393881    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-j5ss4" podUID="919c2def-aa85-45f3-8089-522a211eac3b" containerName="kube-rbac-proxy" containerID="cri-o://960b19fbd3d78432e5ee2c46407471cd0c13b078212ae62b369648f9db800fb1" gracePeriod=30
Jul 30 17:24:52.394083 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:24:52.393875    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-j5ss4" podUID="919c2def-aa85-45f3-8089-522a211eac3b" containerName="kserve-container" containerID="cri-o://838facefad39cfbaf216d35a88b12ecabbf6b7315bb51bf6a70915b54bf3deb4" gracePeriod=30
Jul 30 17:24:52.478084 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:24:52.478037    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-j5ss4" podUID="919c2def-aa85-45f3-8089-522a211eac3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 30 17:24:52.841162 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:24:52.841117    2580 generic.go:356] "Generic (PLEG): container finished" podID="919c2def-aa85-45f3-8089-522a211eac3b" containerID="960b19fbd3d78432e5ee2c46407471cd0c13b078212ae62b369648f9db800fb1" exitCode=2
Jul 30 17:24:52.841377 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:24:52.841209    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-j5ss4" event={"ID":"919c2def-aa85-45f3-8089-522a211eac3b","Type":"ContainerDied","Data":"960b19fbd3d78432e5ee2c46407471cd0c13b078212ae62b369648f9db800fb1"}
Jul 30 17:24:55.431856 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:24:55.431808    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-j5ss4" podUID="919c2def-aa85-45f3-8089-522a211eac3b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Jul 30 17:24:58.153135 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:24:58.153100    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-j5ss4"
Jul 30 17:24:58.296952 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:24:58.296910    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q9vmt\" (UniqueName: \"kubernetes.io/projected/919c2def-aa85-45f3-8089-522a211eac3b-kube-api-access-q9vmt\") pod \"919c2def-aa85-45f3-8089-522a211eac3b\" (UID: \"919c2def-aa85-45f3-8089-522a211eac3b\") "
Jul 30 17:24:58.297156 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:24:58.296963    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/919c2def-aa85-45f3-8089-522a211eac3b-kserve-provision-location\") pod \"919c2def-aa85-45f3-8089-522a211eac3b\" (UID: \"919c2def-aa85-45f3-8089-522a211eac3b\") "
Jul 30 17:24:58.297156 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:24:58.297011    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/919c2def-aa85-45f3-8089-522a211eac3b-kube-rbac-proxy-sar-config\") pod \"919c2def-aa85-45f3-8089-522a211eac3b\" (UID: \"919c2def-aa85-45f3-8089-522a211eac3b\") "
Jul 30 17:24:58.297156 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:24:58.297067    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/919c2def-aa85-45f3-8089-522a211eac3b-cabundle-cert\") pod \"919c2def-aa85-45f3-8089-522a211eac3b\" (UID: \"919c2def-aa85-45f3-8089-522a211eac3b\") "
Jul 30 17:24:58.297156 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:24:58.297095    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/919c2def-aa85-45f3-8089-522a211eac3b-proxy-tls\") pod \"919c2def-aa85-45f3-8089-522a211eac3b\" (UID: \"919c2def-aa85-45f3-8089-522a211eac3b\") "
Jul 30 17:24:58.297422 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:24:58.297397    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/919c2def-aa85-45f3-8089-522a211eac3b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "919c2def-aa85-45f3-8089-522a211eac3b" (UID: "919c2def-aa85-45f3-8089-522a211eac3b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 17:24:58.297520 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:24:58.297484    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/919c2def-aa85-45f3-8089-522a211eac3b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "919c2def-aa85-45f3-8089-522a211eac3b" (UID: "919c2def-aa85-45f3-8089-522a211eac3b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:24:58.297644 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:24:58.297488    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/919c2def-aa85-45f3-8089-522a211eac3b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "919c2def-aa85-45f3-8089-522a211eac3b" (UID: "919c2def-aa85-45f3-8089-522a211eac3b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:24:58.299324 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:24:58.299292    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/919c2def-aa85-45f3-8089-522a211eac3b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "919c2def-aa85-45f3-8089-522a211eac3b" (UID: "919c2def-aa85-45f3-8089-522a211eac3b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:24:58.299469 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:24:58.299442    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/919c2def-aa85-45f3-8089-522a211eac3b-kube-api-access-q9vmt" (OuterVolumeSpecName: "kube-api-access-q9vmt") pod "919c2def-aa85-45f3-8089-522a211eac3b" (UID: "919c2def-aa85-45f3-8089-522a211eac3b"). InnerVolumeSpecName "kube-api-access-q9vmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:24:58.398667 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:24:58.398624    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9vmt\" (UniqueName: \"kubernetes.io/projected/919c2def-aa85-45f3-8089-522a211eac3b-kube-api-access-q9vmt\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:24:58.398667 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:24:58.398660    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/919c2def-aa85-45f3-8089-522a211eac3b-kserve-provision-location\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:24:58.398667 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:24:58.398670    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/919c2def-aa85-45f3-8089-522a211eac3b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:24:58.398907 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:24:58.398680    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/919c2def-aa85-45f3-8089-522a211eac3b-cabundle-cert\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:24:58.398907 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:24:58.398688    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/919c2def-aa85-45f3-8089-522a211eac3b-proxy-tls\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:24:58.868534 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:24:58.868497    2580 generic.go:356] "Generic (PLEG): container finished" podID="919c2def-aa85-45f3-8089-522a211eac3b" containerID="838facefad39cfbaf216d35a88b12ecabbf6b7315bb51bf6a70915b54bf3deb4" exitCode=0
Jul 30 17:24:58.868748 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:24:58.868565    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-j5ss4"
Jul 30 17:24:58.868748 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:24:58.868586    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-j5ss4" event={"ID":"919c2def-aa85-45f3-8089-522a211eac3b","Type":"ContainerDied","Data":"838facefad39cfbaf216d35a88b12ecabbf6b7315bb51bf6a70915b54bf3deb4"}
Jul 30 17:24:58.868748 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:24:58.868646    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-j5ss4" event={"ID":"919c2def-aa85-45f3-8089-522a211eac3b","Type":"ContainerDied","Data":"d01b946456b36d3c7d757de6ad40174d239a3d6007189615aeb4cf6344f18eba"}
Jul 30 17:24:58.868748 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:24:58.868675    2580 scope.go:117] "RemoveContainer" containerID="960b19fbd3d78432e5ee2c46407471cd0c13b078212ae62b369648f9db800fb1"
Jul 30 17:24:58.877883 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:24:58.877860    2580 scope.go:117] "RemoveContainer" containerID="838facefad39cfbaf216d35a88b12ecabbf6b7315bb51bf6a70915b54bf3deb4"
Jul 30 17:24:58.885997 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:24:58.885973    2580 scope.go:117] "RemoveContainer" containerID="f3c8e6071c7110cf3c92e8852bc686399de85c90f6e7830d06f72e1c02f74c29"
Jul 30 17:24:58.892758 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:24:58.892727    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-j5ss4"]
Jul 30 17:24:58.895186 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:24:58.895146    2580 scope.go:117] "RemoveContainer" containerID="960b19fbd3d78432e5ee2c46407471cd0c13b078212ae62b369648f9db800fb1"
Jul 30 17:24:58.895552 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:24:58.895532    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"960b19fbd3d78432e5ee2c46407471cd0c13b078212ae62b369648f9db800fb1\": container with ID starting with 960b19fbd3d78432e5ee2c46407471cd0c13b078212ae62b369648f9db800fb1 not found: ID does not exist" containerID="960b19fbd3d78432e5ee2c46407471cd0c13b078212ae62b369648f9db800fb1"
Jul 30 17:24:58.895619 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:24:58.895561    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"960b19fbd3d78432e5ee2c46407471cd0c13b078212ae62b369648f9db800fb1"} err="failed to get container status \"960b19fbd3d78432e5ee2c46407471cd0c13b078212ae62b369648f9db800fb1\": rpc error: code = NotFound desc = could not find container \"960b19fbd3d78432e5ee2c46407471cd0c13b078212ae62b369648f9db800fb1\": container with ID starting with 960b19fbd3d78432e5ee2c46407471cd0c13b078212ae62b369648f9db800fb1 not found: ID does not exist"
Jul 30 17:24:58.895619 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:24:58.895585    2580 scope.go:117] "RemoveContainer" containerID="838facefad39cfbaf216d35a88b12ecabbf6b7315bb51bf6a70915b54bf3deb4"
Jul 30 17:24:58.895873 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:24:58.895854    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"838facefad39cfbaf216d35a88b12ecabbf6b7315bb51bf6a70915b54bf3deb4\": container with ID starting with 838facefad39cfbaf216d35a88b12ecabbf6b7315bb51bf6a70915b54bf3deb4 not found: ID does not exist" containerID="838facefad39cfbaf216d35a88b12ecabbf6b7315bb51bf6a70915b54bf3deb4"
Jul 30 17:24:58.895957 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:24:58.895882    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"838facefad39cfbaf216d35a88b12ecabbf6b7315bb51bf6a70915b54bf3deb4"} err="failed to get container status \"838facefad39cfbaf216d35a88b12ecabbf6b7315bb51bf6a70915b54bf3deb4\": rpc error: code = NotFound desc = could not find container \"838facefad39cfbaf216d35a88b12ecabbf6b7315bb51bf6a70915b54bf3deb4\": container with ID starting with 838facefad39cfbaf216d35a88b12ecabbf6b7315bb51bf6a70915b54bf3deb4 not found: ID does not exist"
Jul 30 17:24:58.895957 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:24:58.895904    2580 scope.go:117] "RemoveContainer" containerID="f3c8e6071c7110cf3c92e8852bc686399de85c90f6e7830d06f72e1c02f74c29"
Jul 30 17:24:58.896244 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:24:58.896214    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3c8e6071c7110cf3c92e8852bc686399de85c90f6e7830d06f72e1c02f74c29\": container with ID starting with f3c8e6071c7110cf3c92e8852bc686399de85c90f6e7830d06f72e1c02f74c29 not found: ID does not exist" containerID="f3c8e6071c7110cf3c92e8852bc686399de85c90f6e7830d06f72e1c02f74c29"
Jul 30 17:24:58.896365 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:24:58.896247    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3c8e6071c7110cf3c92e8852bc686399de85c90f6e7830d06f72e1c02f74c29"} err="failed to get container status \"f3c8e6071c7110cf3c92e8852bc686399de85c90f6e7830d06f72e1c02f74c29\": rpc error: code = NotFound desc = could not find container \"f3c8e6071c7110cf3c92e8852bc686399de85c90f6e7830d06f72e1c02f74c29\": container with ID starting with f3c8e6071c7110cf3c92e8852bc686399de85c90f6e7830d06f72e1c02f74c29 not found: ID does not exist"
Jul 30 17:24:58.899814 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:24:58.899786    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-j5ss4"]
Jul 30 17:24:59.483008 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:24:59.482965    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="919c2def-aa85-45f3-8089-522a211eac3b" path="/var/lib/kubelet/pods/919c2def-aa85-45f3-8089-522a211eac3b/volumes"
Jul 30 17:25:37.403670 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:25:37.403631    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-wqszn_f231a916-1fda-4059-9130-d4112e738ee1/console-operator/1.log"
Jul 30 17:25:37.405936 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:25:37.405903    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-wqszn_f231a916-1fda-4059-9130-d4112e738ee1/console-operator/1.log"
Jul 30 17:26:32.873132 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:32.873087    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5k5hb"]
Jul 30 17:26:32.875691 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:32.873421    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5k5hb" podUID="5a9f9719-9ac1-4433-a65c-60e4d8b7d221" containerName="kserve-container" containerID="cri-o://56663149dae63d22e528eeb974c068af1d96ded6ba15ff2dcac2928992bdac7a" gracePeriod=30
Jul 30 17:26:32.875691 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:32.873460    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5k5hb" podUID="5a9f9719-9ac1-4433-a65c-60e4d8b7d221" containerName="kube-rbac-proxy" containerID="cri-o://efe2fe9af915cebda3f5b4d48ec39bf519aa0ffa8aae76eadd7e3ba20611aba5" gracePeriod=30
Jul 30 17:26:33.263130 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:33.263087    2580 generic.go:356] "Generic (PLEG): container finished" podID="5a9f9719-9ac1-4433-a65c-60e4d8b7d221" containerID="efe2fe9af915cebda3f5b4d48ec39bf519aa0ffa8aae76eadd7e3ba20611aba5" exitCode=2
Jul 30 17:26:33.263372 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:33.263157    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5k5hb" event={"ID":"5a9f9719-9ac1-4433-a65c-60e4d8b7d221","Type":"ContainerDied","Data":"efe2fe9af915cebda3f5b4d48ec39bf519aa0ffa8aae76eadd7e3ba20611aba5"}
Jul 30 17:26:34.477844 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:34.477792    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5k5hb" podUID="5a9f9719-9ac1-4433-a65c-60e4d8b7d221" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 30 17:26:35.431404 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:35.431353    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5k5hb" podUID="5a9f9719-9ac1-4433-a65c-60e4d8b7d221" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Jul 30 17:26:37.959064 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:37.959020    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-m4xrx"]
Jul 30 17:26:37.959557 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:37.959438    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="919c2def-aa85-45f3-8089-522a211eac3b" containerName="kserve-container"
Jul 30 17:26:37.959557 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:37.959454    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="919c2def-aa85-45f3-8089-522a211eac3b" containerName="kserve-container"
Jul 30 17:26:37.959557 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:37.959465    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c75219c-0563-4aa1-bc78-348c2eedf18f" containerName="storage-initializer"
Jul 30 17:26:37.959557 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:37.959471    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c75219c-0563-4aa1-bc78-348c2eedf18f" containerName="storage-initializer"
Jul 30 17:26:37.959557 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:37.959478    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6502fda1-f6ed-4140-86f0-c393aafd7aa0" containerName="kube-rbac-proxy"
Jul 30 17:26:37.959557 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:37.959485    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6502fda1-f6ed-4140-86f0-c393aafd7aa0" containerName="kube-rbac-proxy"
Jul 30 17:26:37.959557 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:37.959493    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c75219c-0563-4aa1-bc78-348c2eedf18f" containerName="kserve-container"
Jul 30 17:26:37.959557 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:37.959499    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c75219c-0563-4aa1-bc78-348c2eedf18f" containerName="kserve-container"
Jul 30 17:26:37.959557 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:37.959507    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6502fda1-f6ed-4140-86f0-c393aafd7aa0" containerName="storage-initializer"
Jul 30 17:26:37.959557 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:37.959513    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6502fda1-f6ed-4140-86f0-c393aafd7aa0" containerName="storage-initializer"
Jul 30 17:26:37.959557 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:37.959519    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="919c2def-aa85-45f3-8089-522a211eac3b" containerName="storage-initializer"
Jul 30 17:26:37.959557 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:37.959524    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="919c2def-aa85-45f3-8089-522a211eac3b" containerName="storage-initializer"
Jul 30 17:26:37.959557 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:37.959535    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c75219c-0563-4aa1-bc78-348c2eedf18f" containerName="kube-rbac-proxy"
Jul 30 17:26:37.959557 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:37.959540    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c75219c-0563-4aa1-bc78-348c2eedf18f" containerName="kube-rbac-proxy"
Jul 30 17:26:37.959557 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:37.959550    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="919c2def-aa85-45f3-8089-522a211eac3b" containerName="kube-rbac-proxy"
Jul 30 17:26:37.959557 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:37.959556    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="919c2def-aa85-45f3-8089-522a211eac3b" containerName="kube-rbac-proxy"
Jul 30 17:26:37.959557 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:37.959561    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6502fda1-f6ed-4140-86f0-c393aafd7aa0" containerName="kserve-container"
Jul 30 17:26:37.959557 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:37.959567    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6502fda1-f6ed-4140-86f0-c393aafd7aa0" containerName="kserve-container"
Jul 30 17:26:37.960158 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:37.959636    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c75219c-0563-4aa1-bc78-348c2eedf18f" containerName="kube-rbac-proxy"
Jul 30 17:26:37.960158 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:37.959645    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c75219c-0563-4aa1-bc78-348c2eedf18f" containerName="kserve-container"
Jul 30 17:26:37.960158 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:37.959652    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6502fda1-f6ed-4140-86f0-c393aafd7aa0" containerName="kserve-container"
Jul 30 17:26:37.960158 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:37.959661    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="919c2def-aa85-45f3-8089-522a211eac3b" containerName="kserve-container"
Jul 30 17:26:37.960158 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:37.959667    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="919c2def-aa85-45f3-8089-522a211eac3b" containerName="kube-rbac-proxy"
Jul 30 17:26:37.960158 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:37.959673    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6502fda1-f6ed-4140-86f0-c393aafd7aa0" containerName="kube-rbac-proxy"
Jul 30 17:26:37.962824 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:37.962804    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-m4xrx"
Jul 30 17:26:37.965678 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:26:37.965645    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"isvc-canary-rollback-kube-rbac-proxy-sar-config\" is forbidden: User \"system:node:ip-10-0-136-68.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-136-68.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\"" type="*v1.ConfigMap"
Jul 30 17:26:37.965840 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:26:37.965755    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"isvc-canary-rollback-predictor-serving-cert\" is forbidden: User \"system:node:ip-10-0-136-68.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-136-68.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\"" type="*v1.Secret"
Jul 30 17:26:37.965840 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:26:37.965820    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"isvc-canary-rollback-predictor-7c9c87bcff-m4xrx\" is forbidden: User \"system:node:ip-10-0-136-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-136-68.ec2.internal' and this object" podUID="cf316334-a96d-48ab-b7cf-932b0f7d214a" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-m4xrx"
Jul 30 17:26:37.986760 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:37.986725    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-m4xrx"]
Jul 30 17:26:38.078297 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:38.078258    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhxd5\" (UniqueName: \"kubernetes.io/projected/cf316334-a96d-48ab-b7cf-932b0f7d214a-kube-api-access-fhxd5\") pod \"isvc-canary-rollback-predictor-7c9c87bcff-m4xrx\" (UID: \"cf316334-a96d-48ab-b7cf-932b0f7d214a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-m4xrx"
Jul 30 17:26:38.078502 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:38.078323    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf316334-a96d-48ab-b7cf-932b0f7d214a-proxy-tls\") pod \"isvc-canary-rollback-predictor-7c9c87bcff-m4xrx\" (UID: \"cf316334-a96d-48ab-b7cf-932b0f7d214a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-m4xrx"
Jul 30 17:26:38.078502 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:38.078368    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf316334-a96d-48ab-b7cf-932b0f7d214a-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7c9c87bcff-m4xrx\" (UID: \"cf316334-a96d-48ab-b7cf-932b0f7d214a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-m4xrx"
Jul 30 17:26:38.078502 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:38.078388    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf316334-a96d-48ab-b7cf-932b0f7d214a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7c9c87bcff-m4xrx\" (UID: \"cf316334-a96d-48ab-b7cf-932b0f7d214a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-m4xrx"
Jul 30 17:26:38.078502 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:38.078474    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf316334-a96d-48ab-b7cf-932b0f7d214a-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7c9c87bcff-m4xrx\" (UID: \"cf316334-a96d-48ab-b7cf-932b0f7d214a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-m4xrx"
Jul 30 17:26:38.179444 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:38.179406    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf316334-a96d-48ab-b7cf-932b0f7d214a-proxy-tls\") pod \"isvc-canary-rollback-predictor-7c9c87bcff-m4xrx\" (UID: \"cf316334-a96d-48ab-b7cf-932b0f7d214a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-m4xrx"
Jul 30 17:26:38.179444 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:38.179448    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf316334-a96d-48ab-b7cf-932b0f7d214a-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7c9c87bcff-m4xrx\" (UID: \"cf316334-a96d-48ab-b7cf-932b0f7d214a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-m4xrx"
Jul 30 17:26:38.179693 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:38.179479    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf316334-a96d-48ab-b7cf-932b0f7d214a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7c9c87bcff-m4xrx\" (UID: \"cf316334-a96d-48ab-b7cf-932b0f7d214a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-m4xrx"
Jul 30 17:26:38.179693 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:38.179532    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf316334-a96d-48ab-b7cf-932b0f7d214a-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7c9c87bcff-m4xrx\" (UID: \"cf316334-a96d-48ab-b7cf-932b0f7d214a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-m4xrx"
Jul 30 17:26:38.179693 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:38.179583    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhxd5\" (UniqueName: \"kubernetes.io/projected/cf316334-a96d-48ab-b7cf-932b0f7d214a-kube-api-access-fhxd5\") pod \"isvc-canary-rollback-predictor-7c9c87bcff-m4xrx\" (UID: \"cf316334-a96d-48ab-b7cf-932b0f7d214a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-m4xrx"
Jul 30 17:26:38.179948 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:38.179926    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf316334-a96d-48ab-b7cf-932b0f7d214a-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7c9c87bcff-m4xrx\" (UID: \"cf316334-a96d-48ab-b7cf-932b0f7d214a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-m4xrx"
Jul 30 17:26:38.180252 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:38.180230    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf316334-a96d-48ab-b7cf-932b0f7d214a-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7c9c87bcff-m4xrx\" (UID: \"cf316334-a96d-48ab-b7cf-932b0f7d214a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-m4xrx"
Jul 30 17:26:38.191777 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:38.191740    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhxd5\" (UniqueName: \"kubernetes.io/projected/cf316334-a96d-48ab-b7cf-932b0f7d214a-kube-api-access-fhxd5\") pod \"isvc-canary-rollback-predictor-7c9c87bcff-m4xrx\" (UID: \"cf316334-a96d-48ab-b7cf-932b0f7d214a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-m4xrx"
Jul 30 17:26:38.640430 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:38.640401    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5k5hb"
Jul 30 17:26:38.785391 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:38.785339    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a9f9719-9ac1-4433-a65c-60e4d8b7d221-cabundle-cert\") pod \"5a9f9719-9ac1-4433-a65c-60e4d8b7d221\" (UID: \"5a9f9719-9ac1-4433-a65c-60e4d8b7d221\") "
Jul 30 17:26:38.785583 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:38.785418    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a9f9719-9ac1-4433-a65c-60e4d8b7d221-kube-rbac-proxy-sar-config\") pod \"5a9f9719-9ac1-4433-a65c-60e4d8b7d221\" (UID: \"5a9f9719-9ac1-4433-a65c-60e4d8b7d221\") "
Jul 30 17:26:38.785583 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:38.785457    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a9f9719-9ac1-4433-a65c-60e4d8b7d221-proxy-tls\") pod \"5a9f9719-9ac1-4433-a65c-60e4d8b7d221\" (UID: \"5a9f9719-9ac1-4433-a65c-60e4d8b7d221\") "
Jul 30 17:26:38.785583 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:38.785520    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7rgkh\" (UniqueName: \"kubernetes.io/projected/5a9f9719-9ac1-4433-a65c-60e4d8b7d221-kube-api-access-7rgkh\") pod \"5a9f9719-9ac1-4433-a65c-60e4d8b7d221\" (UID: \"5a9f9719-9ac1-4433-a65c-60e4d8b7d221\") "
Jul 30 17:26:38.785583 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:38.785575    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a9f9719-9ac1-4433-a65c-60e4d8b7d221-kserve-provision-location\") pod \"5a9f9719-9ac1-4433-a65c-60e4d8b7d221\" (UID: \"5a9f9719-9ac1-4433-a65c-60e4d8b7d221\") "
Jul 30 17:26:38.785825 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:38.785793    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a9f9719-9ac1-4433-a65c-60e4d8b7d221-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5a9f9719-9ac1-4433-a65c-60e4d8b7d221" (UID: "5a9f9719-9ac1-4433-a65c-60e4d8b7d221"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:26:38.785944 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:38.785835    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a9f9719-9ac1-4433-a65c-60e4d8b7d221-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5a9f9719-9ac1-4433-a65c-60e4d8b7d221" (UID: "5a9f9719-9ac1-4433-a65c-60e4d8b7d221"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:26:38.785983 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:38.785948    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9f9719-9ac1-4433-a65c-60e4d8b7d221-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5a9f9719-9ac1-4433-a65c-60e4d8b7d221" (UID: "5a9f9719-9ac1-4433-a65c-60e4d8b7d221"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 17:26:38.787790 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:38.787772    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a9f9719-9ac1-4433-a65c-60e4d8b7d221-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5a9f9719-9ac1-4433-a65c-60e4d8b7d221" (UID: "5a9f9719-9ac1-4433-a65c-60e4d8b7d221"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:26:38.787867 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:38.787847    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a9f9719-9ac1-4433-a65c-60e4d8b7d221-kube-api-access-7rgkh" (OuterVolumeSpecName: "kube-api-access-7rgkh") pod "5a9f9719-9ac1-4433-a65c-60e4d8b7d221" (UID: "5a9f9719-9ac1-4433-a65c-60e4d8b7d221"). InnerVolumeSpecName "kube-api-access-7rgkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:26:38.886851 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:38.886801    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a9f9719-9ac1-4433-a65c-60e4d8b7d221-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:26:38.886851 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:38.886840    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a9f9719-9ac1-4433-a65c-60e4d8b7d221-proxy-tls\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:26:38.886851 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:38.886852    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rgkh\" (UniqueName: \"kubernetes.io/projected/5a9f9719-9ac1-4433-a65c-60e4d8b7d221-kube-api-access-7rgkh\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:26:38.886851 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:38.886860    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a9f9719-9ac1-4433-a65c-60e4d8b7d221-kserve-provision-location\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:26:38.887236 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:38.886870    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a9f9719-9ac1-4433-a65c-60e4d8b7d221-cabundle-cert\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:26:38.895507 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:38.895490    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Jul 30 17:26:38.900816 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:38.900790    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf316334-a96d-48ab-b7cf-932b0f7d214a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7c9c87bcff-m4xrx\" (UID: \"cf316334-a96d-48ab-b7cf-932b0f7d214a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-m4xrx"
Jul 30 17:26:39.106644 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:39.106556    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Jul 30 17:26:39.112422 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:39.112393    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf316334-a96d-48ab-b7cf-932b0f7d214a-proxy-tls\") pod \"isvc-canary-rollback-predictor-7c9c87bcff-m4xrx\" (UID: \"cf316334-a96d-48ab-b7cf-932b0f7d214a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-m4xrx"
Jul 30 17:26:39.173905 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:39.173858    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-m4xrx"
Jul 30 17:26:39.292119 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:39.292084    2580 generic.go:356] "Generic (PLEG): container finished" podID="5a9f9719-9ac1-4433-a65c-60e4d8b7d221" containerID="56663149dae63d22e528eeb974c068af1d96ded6ba15ff2dcac2928992bdac7a" exitCode=0
Jul 30 17:26:39.292320 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:39.292184    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5k5hb"
Jul 30 17:26:39.292320 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:39.292145    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5k5hb" event={"ID":"5a9f9719-9ac1-4433-a65c-60e4d8b7d221","Type":"ContainerDied","Data":"56663149dae63d22e528eeb974c068af1d96ded6ba15ff2dcac2928992bdac7a"}
Jul 30 17:26:39.292320 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:39.292311    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5k5hb" event={"ID":"5a9f9719-9ac1-4433-a65c-60e4d8b7d221","Type":"ContainerDied","Data":"6b7fbfa38a81364f9ebce7f589ca63eb4fa398a52a53561a5b7464222a87e876"}
Jul 30 17:26:39.292430 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:39.292340    2580 scope.go:117] "RemoveContainer" containerID="efe2fe9af915cebda3f5b4d48ec39bf519aa0ffa8aae76eadd7e3ba20611aba5"
Jul 30 17:26:39.301826 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:39.301804    2580 scope.go:117] "RemoveContainer" containerID="56663149dae63d22e528eeb974c068af1d96ded6ba15ff2dcac2928992bdac7a"
Jul 30 17:26:39.310370 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:39.310349    2580 scope.go:117] "RemoveContainer" containerID="916b1465204489e81c96492490d71389ca73cc0eba8f2afd2b50d5cdfed382e2"
Jul 30 17:26:39.319039 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:39.319013    2580 scope.go:117] "RemoveContainer" containerID="efe2fe9af915cebda3f5b4d48ec39bf519aa0ffa8aae76eadd7e3ba20611aba5"
Jul 30 17:26:39.319396 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:26:39.319376    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efe2fe9af915cebda3f5b4d48ec39bf519aa0ffa8aae76eadd7e3ba20611aba5\": container with ID starting with efe2fe9af915cebda3f5b4d48ec39bf519aa0ffa8aae76eadd7e3ba20611aba5 not found: ID does not exist" containerID="efe2fe9af915cebda3f5b4d48ec39bf519aa0ffa8aae76eadd7e3ba20611aba5"
Jul 30 17:26:39.319462 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:39.319406    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efe2fe9af915cebda3f5b4d48ec39bf519aa0ffa8aae76eadd7e3ba20611aba5"} err="failed to get container status \"efe2fe9af915cebda3f5b4d48ec39bf519aa0ffa8aae76eadd7e3ba20611aba5\": rpc error: code = NotFound desc = could not find container \"efe2fe9af915cebda3f5b4d48ec39bf519aa0ffa8aae76eadd7e3ba20611aba5\": container with ID starting with efe2fe9af915cebda3f5b4d48ec39bf519aa0ffa8aae76eadd7e3ba20611aba5 not found: ID does not exist"
Jul 30 17:26:39.319462 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:39.319428    2580 scope.go:117] "RemoveContainer" containerID="56663149dae63d22e528eeb974c068af1d96ded6ba15ff2dcac2928992bdac7a"
Jul 30 17:26:39.319651 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:26:39.319635    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56663149dae63d22e528eeb974c068af1d96ded6ba15ff2dcac2928992bdac7a\": container with ID starting with 56663149dae63d22e528eeb974c068af1d96ded6ba15ff2dcac2928992bdac7a not found: ID does not exist" containerID="56663149dae63d22e528eeb974c068af1d96ded6ba15ff2dcac2928992bdac7a"
Jul 30 17:26:39.319696 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:39.319654    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56663149dae63d22e528eeb974c068af1d96ded6ba15ff2dcac2928992bdac7a"} err="failed to get container status \"56663149dae63d22e528eeb974c068af1d96ded6ba15ff2dcac2928992bdac7a\": rpc error: code = NotFound desc = could not find container \"56663149dae63d22e528eeb974c068af1d96ded6ba15ff2dcac2928992bdac7a\": container with ID starting with 56663149dae63d22e528eeb974c068af1d96ded6ba15ff2dcac2928992bdac7a not found: ID does not exist"
Jul 30 17:26:39.319696 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:39.319667    2580 scope.go:117] "RemoveContainer" containerID="916b1465204489e81c96492490d71389ca73cc0eba8f2afd2b50d5cdfed382e2"
Jul 30 17:26:39.319885 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:26:39.319868    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"916b1465204489e81c96492490d71389ca73cc0eba8f2afd2b50d5cdfed382e2\": container with ID starting with 916b1465204489e81c96492490d71389ca73cc0eba8f2afd2b50d5cdfed382e2 not found: ID does not exist" containerID="916b1465204489e81c96492490d71389ca73cc0eba8f2afd2b50d5cdfed382e2"
Jul 30 17:26:39.319925 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:39.319891    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"916b1465204489e81c96492490d71389ca73cc0eba8f2afd2b50d5cdfed382e2"} err="failed to get container status \"916b1465204489e81c96492490d71389ca73cc0eba8f2afd2b50d5cdfed382e2\": rpc error: code = NotFound desc = could not find container \"916b1465204489e81c96492490d71389ca73cc0eba8f2afd2b50d5cdfed382e2\": container with ID starting with 916b1465204489e81c96492490d71389ca73cc0eba8f2afd2b50d5cdfed382e2 not found: ID does not exist"
Jul 30 17:26:39.338952 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:39.338914    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5k5hb"]
Jul 30 17:26:39.341991 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:39.341955    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5k5hb"]
Jul 30 17:26:39.359495 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:39.359410    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-m4xrx"]
Jul 30 17:26:39.363063 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:26:39.363034    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf316334_a96d_48ab_b7cf_932b0f7d214a.slice/crio-3b0d8e5c478b25dd6df5952c6e17a7369ea25cc564b6882257c54a9673382038 WatchSource:0}: Error finding container 3b0d8e5c478b25dd6df5952c6e17a7369ea25cc564b6882257c54a9673382038: Status 404 returned error can't find the container with id 3b0d8e5c478b25dd6df5952c6e17a7369ea25cc564b6882257c54a9673382038
Jul 30 17:26:39.483191 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:39.483132    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a9f9719-9ac1-4433-a65c-60e4d8b7d221" path="/var/lib/kubelet/pods/5a9f9719-9ac1-4433-a65c-60e4d8b7d221/volumes"
Jul 30 17:26:40.299360 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:40.299320    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-m4xrx" event={"ID":"cf316334-a96d-48ab-b7cf-932b0f7d214a","Type":"ContainerStarted","Data":"fa67c4d78a10ddc1959d71d2665eb56e5501c58d9e247d1661e7d50d4b9daae0"}
Jul 30 17:26:40.299774 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:40.299391    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-m4xrx" event={"ID":"cf316334-a96d-48ab-b7cf-932b0f7d214a","Type":"ContainerStarted","Data":"3b0d8e5c478b25dd6df5952c6e17a7369ea25cc564b6882257c54a9673382038"}
Jul 30 17:26:43.311744 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:43.311705    2580 generic.go:356] "Generic (PLEG): container finished" podID="cf316334-a96d-48ab-b7cf-932b0f7d214a" containerID="fa67c4d78a10ddc1959d71d2665eb56e5501c58d9e247d1661e7d50d4b9daae0" exitCode=0
Jul 30 17:26:43.312196 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:43.311783    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-m4xrx" event={"ID":"cf316334-a96d-48ab-b7cf-932b0f7d214a","Type":"ContainerDied","Data":"fa67c4d78a10ddc1959d71d2665eb56e5501c58d9e247d1661e7d50d4b9daae0"}
Jul 30 17:26:44.319060 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:44.319023    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-m4xrx" event={"ID":"cf316334-a96d-48ab-b7cf-932b0f7d214a","Type":"ContainerStarted","Data":"9153dea6782e9d401c1b316d88541514006e9d02a9da6e1a252d38bead2b10ea"}
Jul 30 17:26:44.319528 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:44.319091    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-m4xrx" event={"ID":"cf316334-a96d-48ab-b7cf-932b0f7d214a","Type":"ContainerStarted","Data":"91fa20ab31c06e1a747a21e6d5caaacb28c0355f6cc7ada428b2af600c3af1cc"}
Jul 30 17:26:44.319528 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:44.319391    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-m4xrx"
Jul 30 17:26:44.319528 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:44.319440    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-m4xrx"
Jul 30 17:26:44.320688 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:44.320657    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-m4xrx" podUID="cf316334-a96d-48ab-b7cf-932b0f7d214a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 30 17:26:44.348785 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:44.348720    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-m4xrx" podStartSLOduration=3.55385635 podStartE2EDuration="7.348697506s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.794841156s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:26:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:26:44.343157138 +0000 UTC m=+1567.418457048" watchObservedRunningTime="2026-07-30 17:26:44.348697506 +0000 UTC m=+1567.423997374"
Jul 30 17:26:45.323332 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:45.323288    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-m4xrx" podUID="cf316334-a96d-48ab-b7cf-932b0f7d214a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 30 17:26:50.327983 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:50.327945    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-m4xrx"
Jul 30 17:26:50.328623 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:26:50.328594    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-m4xrx" podUID="cf316334-a96d-48ab-b7cf-932b0f7d214a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 30 17:27:00.328929 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:27:00.328874    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-m4xrx" podUID="cf316334-a96d-48ab-b7cf-932b0f7d214a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 30 17:27:10.328897 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:27:10.328845    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-m4xrx" podUID="cf316334-a96d-48ab-b7cf-932b0f7d214a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 30 17:27:20.328545 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:27:20.328496    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-m4xrx" podUID="cf316334-a96d-48ab-b7cf-932b0f7d214a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 30 17:27:30.329345 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:27:30.329236    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-m4xrx" podUID="cf316334-a96d-48ab-b7cf-932b0f7d214a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 30 17:27:40.328962 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:27:40.328917    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-m4xrx" podUID="cf316334-a96d-48ab-b7cf-932b0f7d214a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 30 17:27:50.328720 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:27:50.328645    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-m4xrx" podUID="cf316334-a96d-48ab-b7cf-932b0f7d214a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 30 17:27:54.478015 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:27:54.477961    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-m4xrx" podUID="cf316334-a96d-48ab-b7cf-932b0f7d214a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 30 17:28:04.479461 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:28:04.479422    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-m4xrx"
Jul 30 17:29:38.258488 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:38.258439    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-m4xrx"]
Jul 30 17:29:38.259189 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:38.258785    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-m4xrx" podUID="cf316334-a96d-48ab-b7cf-932b0f7d214a" containerName="kserve-container" containerID="cri-o://91fa20ab31c06e1a747a21e6d5caaacb28c0355f6cc7ada428b2af600c3af1cc" gracePeriod=30
Jul 30 17:29:38.259189 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:38.258801    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-m4xrx" podUID="cf316334-a96d-48ab-b7cf-932b0f7d214a" containerName="kube-rbac-proxy" containerID="cri-o://9153dea6782e9d401c1b316d88541514006e9d02a9da6e1a252d38bead2b10ea" gracePeriod=30
Jul 30 17:29:39.038744 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:39.038708    2580 generic.go:356] "Generic (PLEG): container finished" podID="cf316334-a96d-48ab-b7cf-932b0f7d214a" containerID="9153dea6782e9d401c1b316d88541514006e9d02a9da6e1a252d38bead2b10ea" exitCode=2
Jul 30 17:29:39.038943 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:39.038792    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-m4xrx" event={"ID":"cf316334-a96d-48ab-b7cf-932b0f7d214a","Type":"ContainerDied","Data":"9153dea6782e9d401c1b316d88541514006e9d02a9da6e1a252d38bead2b10ea"}
Jul 30 17:29:40.323561 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:40.323515    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-m4xrx" podUID="cf316334-a96d-48ab-b7cf-932b0f7d214a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Jul 30 17:29:43.578613 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:43.578577    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-m76gp"]
Jul 30 17:29:43.579045 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:43.578916    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a9f9719-9ac1-4433-a65c-60e4d8b7d221" containerName="kube-rbac-proxy"
Jul 30 17:29:43.579045 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:43.578929    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a9f9719-9ac1-4433-a65c-60e4d8b7d221" containerName="kube-rbac-proxy"
Jul 30 17:29:43.579045 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:43.578940    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a9f9719-9ac1-4433-a65c-60e4d8b7d221" containerName="storage-initializer"
Jul 30 17:29:43.579045 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:43.578946    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a9f9719-9ac1-4433-a65c-60e4d8b7d221" containerName="storage-initializer"
Jul 30 17:29:43.579045 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:43.578956    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a9f9719-9ac1-4433-a65c-60e4d8b7d221" containerName="kserve-container"
Jul 30 17:29:43.579045 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:43.578961    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a9f9719-9ac1-4433-a65c-60e4d8b7d221" containerName="kserve-container"
Jul 30 17:29:43.579045 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:43.579027    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a9f9719-9ac1-4433-a65c-60e4d8b7d221" containerName="kube-rbac-proxy"
Jul 30 17:29:43.579045 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:43.579034    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a9f9719-9ac1-4433-a65c-60e4d8b7d221" containerName="kserve-container"
Jul 30 17:29:43.581809 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:43.581789    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-m76gp"
Jul 30 17:29:43.587358 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:43.587328    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Jul 30 17:29:43.587701 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:43.587682    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Jul 30 17:29:43.600235 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:43.600199    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-m76gp"]
Jul 30 17:29:43.622496 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:43.622457    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c07c8213-dd15-4e4c-95b2-aae55d328083-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-9b8c9778-m76gp\" (UID: \"c07c8213-dd15-4e4c-95b2-aae55d328083\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-m76gp"
Jul 30 17:29:43.622496 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:43.622502    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c07c8213-dd15-4e4c-95b2-aae55d328083-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-9b8c9778-m76gp\" (UID: \"c07c8213-dd15-4e4c-95b2-aae55d328083\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-m76gp"
Jul 30 17:29:43.622732 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:43.622562    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c07c8213-dd15-4e4c-95b2-aae55d328083-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-9b8c9778-m76gp\" (UID: \"c07c8213-dd15-4e4c-95b2-aae55d328083\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-m76gp"
Jul 30 17:29:43.622732 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:43.622581    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqh69\" (UniqueName: \"kubernetes.io/projected/c07c8213-dd15-4e4c-95b2-aae55d328083-kube-api-access-gqh69\") pod \"isvc-canary-stop-v2-predictor-9b8c9778-m76gp\" (UID: \"c07c8213-dd15-4e4c-95b2-aae55d328083\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-m76gp"
Jul 30 17:29:43.622732 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:43.622610    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c07c8213-dd15-4e4c-95b2-aae55d328083-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-9b8c9778-m76gp\" (UID: \"c07c8213-dd15-4e4c-95b2-aae55d328083\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-m76gp"
Jul 30 17:29:43.723333 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:43.723286    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c07c8213-dd15-4e4c-95b2-aae55d328083-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-9b8c9778-m76gp\" (UID: \"c07c8213-dd15-4e4c-95b2-aae55d328083\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-m76gp"
Jul 30 17:29:43.723570 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:43.723344    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c07c8213-dd15-4e4c-95b2-aae55d328083-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-9b8c9778-m76gp\" (UID: \"c07c8213-dd15-4e4c-95b2-aae55d328083\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-m76gp"
Jul 30 17:29:43.723570 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:43.723397    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c07c8213-dd15-4e4c-95b2-aae55d328083-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-9b8c9778-m76gp\" (UID: \"c07c8213-dd15-4e4c-95b2-aae55d328083\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-m76gp"
Jul 30 17:29:43.723570 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:43.723418    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqh69\" (UniqueName: \"kubernetes.io/projected/c07c8213-dd15-4e4c-95b2-aae55d328083-kube-api-access-gqh69\") pod \"isvc-canary-stop-v2-predictor-9b8c9778-m76gp\" (UID: \"c07c8213-dd15-4e4c-95b2-aae55d328083\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-m76gp"
Jul 30 17:29:43.723570 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:29:43.723451    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-serving-cert: secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Jul 30 17:29:43.723570 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:43.723550    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c07c8213-dd15-4e4c-95b2-aae55d328083-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-9b8c9778-m76gp\" (UID: \"c07c8213-dd15-4e4c-95b2-aae55d328083\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-m76gp"
Jul 30 17:29:43.723570 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:29:43.723561    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c07c8213-dd15-4e4c-95b2-aae55d328083-proxy-tls podName:c07c8213-dd15-4e4c-95b2-aae55d328083 nodeName:}" failed. No retries permitted until 2026-07-30 17:29:44.22353916 +0000 UTC m=+1747.298839037 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c07c8213-dd15-4e4c-95b2-aae55d328083-proxy-tls") pod "isvc-canary-stop-v2-predictor-9b8c9778-m76gp" (UID: "c07c8213-dd15-4e4c-95b2-aae55d328083") : secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Jul 30 17:29:43.723862 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:43.723831    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c07c8213-dd15-4e4c-95b2-aae55d328083-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-9b8c9778-m76gp\" (UID: \"c07c8213-dd15-4e4c-95b2-aae55d328083\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-m76gp"
Jul 30 17:29:43.724146 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:43.724124    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c07c8213-dd15-4e4c-95b2-aae55d328083-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-9b8c9778-m76gp\" (UID: \"c07c8213-dd15-4e4c-95b2-aae55d328083\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-m76gp"
Jul 30 17:29:43.724277 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:43.724151    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c07c8213-dd15-4e4c-95b2-aae55d328083-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-9b8c9778-m76gp\" (UID: \"c07c8213-dd15-4e4c-95b2-aae55d328083\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-m76gp"
Jul 30 17:29:43.741827 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:43.741790    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqh69\" (UniqueName: \"kubernetes.io/projected/c07c8213-dd15-4e4c-95b2-aae55d328083-kube-api-access-gqh69\") pod \"isvc-canary-stop-v2-predictor-9b8c9778-m76gp\" (UID: \"c07c8213-dd15-4e4c-95b2-aae55d328083\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-m76gp"
Jul 30 17:29:44.062480 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:44.062440    2580 generic.go:356] "Generic (PLEG): container finished" podID="cf316334-a96d-48ab-b7cf-932b0f7d214a" containerID="91fa20ab31c06e1a747a21e6d5caaacb28c0355f6cc7ada428b2af600c3af1cc" exitCode=0
Jul 30 17:29:44.062667 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:44.062508    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-m4xrx" event={"ID":"cf316334-a96d-48ab-b7cf-932b0f7d214a","Type":"ContainerDied","Data":"91fa20ab31c06e1a747a21e6d5caaacb28c0355f6cc7ada428b2af600c3af1cc"}
Jul 30 17:29:44.115647 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:44.115616    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-m4xrx"
Jul 30 17:29:44.126130 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:44.126102    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf316334-a96d-48ab-b7cf-932b0f7d214a-kube-rbac-proxy-sar-config\") pod \"cf316334-a96d-48ab-b7cf-932b0f7d214a\" (UID: \"cf316334-a96d-48ab-b7cf-932b0f7d214a\") "
Jul 30 17:29:44.126322 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:44.126137    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf316334-a96d-48ab-b7cf-932b0f7d214a-cabundle-cert\") pod \"cf316334-a96d-48ab-b7cf-932b0f7d214a\" (UID: \"cf316334-a96d-48ab-b7cf-932b0f7d214a\") "
Jul 30 17:29:44.126322 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:44.126182    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf316334-a96d-48ab-b7cf-932b0f7d214a-proxy-tls\") pod \"cf316334-a96d-48ab-b7cf-932b0f7d214a\" (UID: \"cf316334-a96d-48ab-b7cf-932b0f7d214a\") "
Jul 30 17:29:44.126322 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:44.126251    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fhxd5\" (UniqueName: \"kubernetes.io/projected/cf316334-a96d-48ab-b7cf-932b0f7d214a-kube-api-access-fhxd5\") pod \"cf316334-a96d-48ab-b7cf-932b0f7d214a\" (UID: \"cf316334-a96d-48ab-b7cf-932b0f7d214a\") "
Jul 30 17:29:44.126322 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:44.126274    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf316334-a96d-48ab-b7cf-932b0f7d214a-kserve-provision-location\") pod \"cf316334-a96d-48ab-b7cf-932b0f7d214a\" (UID: \"cf316334-a96d-48ab-b7cf-932b0f7d214a\") "
Jul 30 17:29:44.126659 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:44.126629    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf316334-a96d-48ab-b7cf-932b0f7d214a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cf316334-a96d-48ab-b7cf-932b0f7d214a" (UID: "cf316334-a96d-48ab-b7cf-932b0f7d214a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:29:44.126751 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:44.126629    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf316334-a96d-48ab-b7cf-932b0f7d214a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cf316334-a96d-48ab-b7cf-932b0f7d214a" (UID: "cf316334-a96d-48ab-b7cf-932b0f7d214a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 17:29:44.126751 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:44.126678    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf316334-a96d-48ab-b7cf-932b0f7d214a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cf316334-a96d-48ab-b7cf-932b0f7d214a" (UID: "cf316334-a96d-48ab-b7cf-932b0f7d214a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:29:44.128437 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:44.128412    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf316334-a96d-48ab-b7cf-932b0f7d214a-kube-api-access-fhxd5" (OuterVolumeSpecName: "kube-api-access-fhxd5") pod "cf316334-a96d-48ab-b7cf-932b0f7d214a" (UID: "cf316334-a96d-48ab-b7cf-932b0f7d214a"). InnerVolumeSpecName "kube-api-access-fhxd5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:29:44.128563 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:44.128471    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf316334-a96d-48ab-b7cf-932b0f7d214a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cf316334-a96d-48ab-b7cf-932b0f7d214a" (UID: "cf316334-a96d-48ab-b7cf-932b0f7d214a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:29:44.227248 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:44.227085    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c07c8213-dd15-4e4c-95b2-aae55d328083-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-9b8c9778-m76gp\" (UID: \"c07c8213-dd15-4e4c-95b2-aae55d328083\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-m76gp"
Jul 30 17:29:44.227436 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:44.227269    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhxd5\" (UniqueName: \"kubernetes.io/projected/cf316334-a96d-48ab-b7cf-932b0f7d214a-kube-api-access-fhxd5\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:29:44.227436 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:44.227289    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf316334-a96d-48ab-b7cf-932b0f7d214a-kserve-provision-location\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:29:44.227436 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:44.227303    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf316334-a96d-48ab-b7cf-932b0f7d214a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:29:44.227436 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:44.227316    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf316334-a96d-48ab-b7cf-932b0f7d214a-cabundle-cert\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:29:44.227436 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:44.227330    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf316334-a96d-48ab-b7cf-932b0f7d214a-proxy-tls\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:29:44.229781 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:44.229756    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c07c8213-dd15-4e4c-95b2-aae55d328083-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-9b8c9778-m76gp\" (UID: \"c07c8213-dd15-4e4c-95b2-aae55d328083\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-m76gp"
Jul 30 17:29:44.492411 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:44.492307    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-m76gp"
Jul 30 17:29:44.639458 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:44.639384    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-m76gp"]
Jul 30 17:29:44.642204 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:29:44.642152    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc07c8213_dd15_4e4c_95b2_aae55d328083.slice/crio-79603838eb179bb7c5a2f8307def555e8801ee54f328e37ff000d4db1ab124f7 WatchSource:0}: Error finding container 79603838eb179bb7c5a2f8307def555e8801ee54f328e37ff000d4db1ab124f7: Status 404 returned error can't find the container with id 79603838eb179bb7c5a2f8307def555e8801ee54f328e37ff000d4db1ab124f7
Jul 30 17:29:45.070541 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:45.070500    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-m4xrx"
Jul 30 17:29:45.070541 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:45.070523    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-m4xrx" event={"ID":"cf316334-a96d-48ab-b7cf-932b0f7d214a","Type":"ContainerDied","Data":"3b0d8e5c478b25dd6df5952c6e17a7369ea25cc564b6882257c54a9673382038"}
Jul 30 17:29:45.070825 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:45.070619    2580 scope.go:117] "RemoveContainer" containerID="9153dea6782e9d401c1b316d88541514006e9d02a9da6e1a252d38bead2b10ea"
Jul 30 17:29:45.072454 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:45.072416    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-m76gp" event={"ID":"c07c8213-dd15-4e4c-95b2-aae55d328083","Type":"ContainerStarted","Data":"4b12e65e06dd1c070249b4bc16b9d7e2d8ec59c506c764484fabaf5d5e0f8cde"}
Jul 30 17:29:45.072570 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:45.072476    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-m76gp" event={"ID":"c07c8213-dd15-4e4c-95b2-aae55d328083","Type":"ContainerStarted","Data":"79603838eb179bb7c5a2f8307def555e8801ee54f328e37ff000d4db1ab124f7"}
Jul 30 17:29:45.087226 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:45.087199    2580 scope.go:117] "RemoveContainer" containerID="91fa20ab31c06e1a747a21e6d5caaacb28c0355f6cc7ada428b2af600c3af1cc"
Jul 30 17:29:45.095596 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:45.095571    2580 scope.go:117] "RemoveContainer" containerID="fa67c4d78a10ddc1959d71d2665eb56e5501c58d9e247d1661e7d50d4b9daae0"
Jul 30 17:29:45.115764 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:45.115726    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-m4xrx"]
Jul 30 17:29:45.123522 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:45.123489    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-m4xrx"]
Jul 30 17:29:45.482112 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:45.482020    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf316334-a96d-48ab-b7cf-932b0f7d214a" path="/var/lib/kubelet/pods/cf316334-a96d-48ab-b7cf-932b0f7d214a/volumes"
Jul 30 17:29:49.091427 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:49.091389    2580 generic.go:356] "Generic (PLEG): container finished" podID="c07c8213-dd15-4e4c-95b2-aae55d328083" containerID="4b12e65e06dd1c070249b4bc16b9d7e2d8ec59c506c764484fabaf5d5e0f8cde" exitCode=0
Jul 30 17:29:49.091848 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:49.091444    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-m76gp" event={"ID":"c07c8213-dd15-4e4c-95b2-aae55d328083","Type":"ContainerDied","Data":"4b12e65e06dd1c070249b4bc16b9d7e2d8ec59c506c764484fabaf5d5e0f8cde"}
Jul 30 17:29:50.097666 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:50.097622    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-m76gp" event={"ID":"c07c8213-dd15-4e4c-95b2-aae55d328083","Type":"ContainerStarted","Data":"3aa9011eec5eac59412bd98f25da0105029822d1af4be62782e2b2cc7a870df9"}
Jul 30 17:29:50.098244 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:50.097734    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-m76gp" event={"ID":"c07c8213-dd15-4e4c-95b2-aae55d328083","Type":"ContainerStarted","Data":"d328e5d003af79f1eb3f3e02bb9fd884c283127245034a920a9e44380acc527c"}
Jul 30 17:29:50.098244 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:50.097976    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-m76gp"
Jul 30 17:29:50.119702 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:50.119633    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-m76gp" podStartSLOduration=3.325244653 podStartE2EDuration="7.119617135s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.794372482s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:29:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:29:50.118349016 +0000 UTC m=+1753.193648934" watchObservedRunningTime="2026-07-30 17:29:50.119617135 +0000 UTC m=+1753.194917025"
Jul 30 17:29:51.102711 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:51.102681    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-m76gp"
Jul 30 17:29:51.104299 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:51.104262    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-m76gp" podUID="c07c8213-dd15-4e4c-95b2-aae55d328083" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 30 17:29:52.107368 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:52.107316    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-m76gp" podUID="c07c8213-dd15-4e4c-95b2-aae55d328083" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 30 17:29:57.112239 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:57.112204    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-m76gp"
Jul 30 17:29:57.112895 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:29:57.112865    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-m76gp" podUID="c07c8213-dd15-4e4c-95b2-aae55d328083" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 30 17:30:07.113816 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:30:07.113765    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-m76gp" podUID="c07c8213-dd15-4e4c-95b2-aae55d328083" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 30 17:30:17.113471 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:30:17.113420    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-m76gp" podUID="c07c8213-dd15-4e4c-95b2-aae55d328083" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 30 17:30:27.113055 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:30:27.113006    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-m76gp" podUID="c07c8213-dd15-4e4c-95b2-aae55d328083" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 30 17:30:37.113399 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:30:37.113352    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-m76gp" podUID="c07c8213-dd15-4e4c-95b2-aae55d328083" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 30 17:30:37.429664 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:30:37.429568    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-wqszn_f231a916-1fda-4059-9130-d4112e738ee1/console-operator/1.log"
Jul 30 17:30:37.433347 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:30:37.433318    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-wqszn_f231a916-1fda-4059-9130-d4112e738ee1/console-operator/1.log"
Jul 30 17:30:47.113402 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:30:47.113355    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-m76gp" podUID="c07c8213-dd15-4e4c-95b2-aae55d328083" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 30 17:30:57.112949 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:30:57.112896    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-m76gp" podUID="c07c8213-dd15-4e4c-95b2-aae55d328083" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 30 17:31:07.113155 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:07.113104    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-m76gp" podUID="c07c8213-dd15-4e4c-95b2-aae55d328083" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 30 17:31:17.114395 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:17.114363    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-m76gp"
Jul 30 17:31:18.680112 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:18.680074    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-m76gp"]
Jul 30 17:31:18.680634 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:18.680344    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-m76gp" podUID="c07c8213-dd15-4e4c-95b2-aae55d328083" containerName="kserve-container" containerID="cri-o://d328e5d003af79f1eb3f3e02bb9fd884c283127245034a920a9e44380acc527c" gracePeriod=30
Jul 30 17:31:18.680634 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:18.680362    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-m76gp" podUID="c07c8213-dd15-4e4c-95b2-aae55d328083" containerName="kube-rbac-proxy" containerID="cri-o://3aa9011eec5eac59412bd98f25da0105029822d1af4be62782e2b2cc7a870df9" gracePeriod=30
Jul 30 17:31:19.468028 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:19.467990    2580 generic.go:356] "Generic (PLEG): container finished" podID="c07c8213-dd15-4e4c-95b2-aae55d328083" containerID="3aa9011eec5eac59412bd98f25da0105029822d1af4be62782e2b2cc7a870df9" exitCode=2
Jul 30 17:31:19.468251 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:19.468064    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-m76gp" event={"ID":"c07c8213-dd15-4e4c-95b2-aae55d328083","Type":"ContainerDied","Data":"3aa9011eec5eac59412bd98f25da0105029822d1af4be62782e2b2cc7a870df9"}
Jul 30 17:31:22.108075 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:22.108027    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-m76gp" podUID="c07c8213-dd15-4e4c-95b2-aae55d328083" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Jul 30 17:31:24.493800 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:24.493765    2580 generic.go:356] "Generic (PLEG): container finished" podID="c07c8213-dd15-4e4c-95b2-aae55d328083" containerID="d328e5d003af79f1eb3f3e02bb9fd884c283127245034a920a9e44380acc527c" exitCode=0
Jul 30 17:31:24.494275 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:24.493828    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-m76gp" event={"ID":"c07c8213-dd15-4e4c-95b2-aae55d328083","Type":"ContainerDied","Data":"d328e5d003af79f1eb3f3e02bb9fd884c283127245034a920a9e44380acc527c"}
Jul 30 17:31:24.528110 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:24.528081    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-m76gp"
Jul 30 17:31:24.711366 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:24.711259    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c07c8213-dd15-4e4c-95b2-aae55d328083-cabundle-cert\") pod \"c07c8213-dd15-4e4c-95b2-aae55d328083\" (UID: \"c07c8213-dd15-4e4c-95b2-aae55d328083\") "
Jul 30 17:31:24.711577 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:24.711368    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c07c8213-dd15-4e4c-95b2-aae55d328083-proxy-tls\") pod \"c07c8213-dd15-4e4c-95b2-aae55d328083\" (UID: \"c07c8213-dd15-4e4c-95b2-aae55d328083\") "
Jul 30 17:31:24.711577 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:24.711401    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c07c8213-dd15-4e4c-95b2-aae55d328083-kserve-provision-location\") pod \"c07c8213-dd15-4e4c-95b2-aae55d328083\" (UID: \"c07c8213-dd15-4e4c-95b2-aae55d328083\") "
Jul 30 17:31:24.711577 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:24.711457    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c07c8213-dd15-4e4c-95b2-aae55d328083-kube-rbac-proxy-sar-config\") pod \"c07c8213-dd15-4e4c-95b2-aae55d328083\" (UID: \"c07c8213-dd15-4e4c-95b2-aae55d328083\") "
Jul 30 17:31:24.711577 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:24.711506    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqh69\" (UniqueName: \"kubernetes.io/projected/c07c8213-dd15-4e4c-95b2-aae55d328083-kube-api-access-gqh69\") pod \"c07c8213-dd15-4e4c-95b2-aae55d328083\" (UID: \"c07c8213-dd15-4e4c-95b2-aae55d328083\") "
Jul 30 17:31:24.711781 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:24.711675    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c07c8213-dd15-4e4c-95b2-aae55d328083-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c07c8213-dd15-4e4c-95b2-aae55d328083" (UID: "c07c8213-dd15-4e4c-95b2-aae55d328083"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:31:24.711825 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:24.711791    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c07c8213-dd15-4e4c-95b2-aae55d328083-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c07c8213-dd15-4e4c-95b2-aae55d328083" (UID: "c07c8213-dd15-4e4c-95b2-aae55d328083"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 17:31:24.711887 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:24.711843    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c07c8213-dd15-4e4c-95b2-aae55d328083-cabundle-cert\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:31:24.711887 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:24.711859    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c07c8213-dd15-4e4c-95b2-aae55d328083-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c07c8213-dd15-4e4c-95b2-aae55d328083" (UID: "c07c8213-dd15-4e4c-95b2-aae55d328083"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:31:24.713697 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:24.713667    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c07c8213-dd15-4e4c-95b2-aae55d328083-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c07c8213-dd15-4e4c-95b2-aae55d328083" (UID: "c07c8213-dd15-4e4c-95b2-aae55d328083"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:31:24.713697 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:24.713671    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c07c8213-dd15-4e4c-95b2-aae55d328083-kube-api-access-gqh69" (OuterVolumeSpecName: "kube-api-access-gqh69") pod "c07c8213-dd15-4e4c-95b2-aae55d328083" (UID: "c07c8213-dd15-4e4c-95b2-aae55d328083"). InnerVolumeSpecName "kube-api-access-gqh69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:31:24.813034 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:24.812984    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c07c8213-dd15-4e4c-95b2-aae55d328083-proxy-tls\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:31:24.813034 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:24.813026    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c07c8213-dd15-4e4c-95b2-aae55d328083-kserve-provision-location\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:31:24.813034 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:24.813039    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c07c8213-dd15-4e4c-95b2-aae55d328083-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:31:24.813034 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:24.813049    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqh69\" (UniqueName: \"kubernetes.io/projected/c07c8213-dd15-4e4c-95b2-aae55d328083-kube-api-access-gqh69\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:31:25.500183 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:25.500142    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-m76gp"
Jul 30 17:31:25.500183 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:25.500152    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-m76gp" event={"ID":"c07c8213-dd15-4e4c-95b2-aae55d328083","Type":"ContainerDied","Data":"79603838eb179bb7c5a2f8307def555e8801ee54f328e37ff000d4db1ab124f7"}
Jul 30 17:31:25.500644 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:25.500248    2580 scope.go:117] "RemoveContainer" containerID="3aa9011eec5eac59412bd98f25da0105029822d1af4be62782e2b2cc7a870df9"
Jul 30 17:31:25.512294 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:25.512270    2580 scope.go:117] "RemoveContainer" containerID="d328e5d003af79f1eb3f3e02bb9fd884c283127245034a920a9e44380acc527c"
Jul 30 17:31:25.520769 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:25.520742    2580 scope.go:117] "RemoveContainer" containerID="4b12e65e06dd1c070249b4bc16b9d7e2d8ec59c506c764484fabaf5d5e0f8cde"
Jul 30 17:31:25.558945 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:25.558910    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-m76gp"]
Jul 30 17:31:25.560456 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:25.560426    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-m76gp"]
Jul 30 17:31:27.483374 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:27.483329    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c07c8213-dd15-4e4c-95b2-aae55d328083" path="/var/lib/kubelet/pods/c07c8213-dd15-4e4c-95b2-aae55d328083/volumes"
Jul 30 17:31:29.108705 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:29.108665    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-303984-predictor-8666b4654c-pz9g5"]
Jul 30 17:31:29.109137 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:29.108997    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf316334-a96d-48ab-b7cf-932b0f7d214a" containerName="storage-initializer"
Jul 30 17:31:29.109137 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:29.109008    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf316334-a96d-48ab-b7cf-932b0f7d214a" containerName="storage-initializer"
Jul 30 17:31:29.109137 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:29.109018    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c07c8213-dd15-4e4c-95b2-aae55d328083" containerName="kube-rbac-proxy"
Jul 30 17:31:29.109137 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:29.109024    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c07c8213-dd15-4e4c-95b2-aae55d328083" containerName="kube-rbac-proxy"
Jul 30 17:31:29.109137 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:29.109032    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c07c8213-dd15-4e4c-95b2-aae55d328083" containerName="storage-initializer"
Jul 30 17:31:29.109137 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:29.109038    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c07c8213-dd15-4e4c-95b2-aae55d328083" containerName="storage-initializer"
Jul 30 17:31:29.109137 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:29.109046    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf316334-a96d-48ab-b7cf-932b0f7d214a" containerName="kube-rbac-proxy"
Jul 30 17:31:29.109137 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:29.109052    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf316334-a96d-48ab-b7cf-932b0f7d214a" containerName="kube-rbac-proxy"
Jul 30 17:31:29.109137 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:29.109059    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf316334-a96d-48ab-b7cf-932b0f7d214a" containerName="kserve-container"
Jul 30 17:31:29.109137 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:29.109064    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf316334-a96d-48ab-b7cf-932b0f7d214a" containerName="kserve-container"
Jul 30 17:31:29.109137 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:29.109075    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c07c8213-dd15-4e4c-95b2-aae55d328083" containerName="kserve-container"
Jul 30 17:31:29.109137 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:29.109080    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c07c8213-dd15-4e4c-95b2-aae55d328083" containerName="kserve-container"
Jul 30 17:31:29.109137 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:29.109134    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf316334-a96d-48ab-b7cf-932b0f7d214a" containerName="kserve-container"
Jul 30 17:31:29.109137 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:29.109141    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c07c8213-dd15-4e4c-95b2-aae55d328083" containerName="kserve-container"
Jul 30 17:31:29.109592 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:29.109148    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c07c8213-dd15-4e4c-95b2-aae55d328083" containerName="kube-rbac-proxy"
Jul 30 17:31:29.109592 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:29.109154    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf316334-a96d-48ab-b7cf-932b0f7d214a" containerName="kube-rbac-proxy"
Jul 30 17:31:29.114113 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:29.114086    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-303984-predictor-8666b4654c-pz9g5"
Jul 30 17:31:29.120756 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:29.120720    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-303984-predictor-serving-cert\""
Jul 30 17:31:29.120756 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:29.120720    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 30 17:31:29.121041 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:29.120909    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 30 17:31:29.121885 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:29.121360    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 30 17:31:29.121885 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:29.121388    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 30 17:31:29.121885 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:29.121610    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-303984-kube-rbac-proxy-sar-config\""
Jul 30 17:31:29.121885 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:29.121610    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-qz9j4\""
Jul 30 17:31:29.147278 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:29.147235    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2649\" (UniqueName: \"kubernetes.io/projected/926f1662-23d6-47c4-81b1-e19078554bfa-kube-api-access-q2649\") pod \"isvc-primary-303984-predictor-8666b4654c-pz9g5\" (UID: \"926f1662-23d6-47c4-81b1-e19078554bfa\") " pod="kserve-ci-e2e-test/isvc-primary-303984-predictor-8666b4654c-pz9g5"
Jul 30 17:31:29.147278 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:29.147277    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/926f1662-23d6-47c4-81b1-e19078554bfa-cabundle-cert\") pod \"isvc-primary-303984-predictor-8666b4654c-pz9g5\" (UID: \"926f1662-23d6-47c4-81b1-e19078554bfa\") " pod="kserve-ci-e2e-test/isvc-primary-303984-predictor-8666b4654c-pz9g5"
Jul 30 17:31:29.147529 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:29.147305    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/926f1662-23d6-47c4-81b1-e19078554bfa-proxy-tls\") pod \"isvc-primary-303984-predictor-8666b4654c-pz9g5\" (UID: \"926f1662-23d6-47c4-81b1-e19078554bfa\") " pod="kserve-ci-e2e-test/isvc-primary-303984-predictor-8666b4654c-pz9g5"
Jul 30 17:31:29.147529 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:29.147346    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/926f1662-23d6-47c4-81b1-e19078554bfa-kube-rbac-proxy-sar-config\") pod \"isvc-primary-303984-predictor-8666b4654c-pz9g5\" (UID: \"926f1662-23d6-47c4-81b1-e19078554bfa\") " pod="kserve-ci-e2e-test/isvc-primary-303984-predictor-8666b4654c-pz9g5"
Jul 30 17:31:29.147529 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:29.147400    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/926f1662-23d6-47c4-81b1-e19078554bfa-kserve-provision-location\") pod \"isvc-primary-303984-predictor-8666b4654c-pz9g5\" (UID: \"926f1662-23d6-47c4-81b1-e19078554bfa\") " pod="kserve-ci-e2e-test/isvc-primary-303984-predictor-8666b4654c-pz9g5"
Jul 30 17:31:29.158985 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:29.158943    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-303984-predictor-8666b4654c-pz9g5"]
Jul 30 17:31:29.247978 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:29.247940    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2649\" (UniqueName: \"kubernetes.io/projected/926f1662-23d6-47c4-81b1-e19078554bfa-kube-api-access-q2649\") pod \"isvc-primary-303984-predictor-8666b4654c-pz9g5\" (UID: \"926f1662-23d6-47c4-81b1-e19078554bfa\") " pod="kserve-ci-e2e-test/isvc-primary-303984-predictor-8666b4654c-pz9g5"
Jul 30 17:31:29.247978 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:29.247979    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/926f1662-23d6-47c4-81b1-e19078554bfa-cabundle-cert\") pod \"isvc-primary-303984-predictor-8666b4654c-pz9g5\" (UID: \"926f1662-23d6-47c4-81b1-e19078554bfa\") " pod="kserve-ci-e2e-test/isvc-primary-303984-predictor-8666b4654c-pz9g5"
Jul 30 17:31:29.248331 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:29.248007    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/926f1662-23d6-47c4-81b1-e19078554bfa-proxy-tls\") pod \"isvc-primary-303984-predictor-8666b4654c-pz9g5\" (UID: \"926f1662-23d6-47c4-81b1-e19078554bfa\") " pod="kserve-ci-e2e-test/isvc-primary-303984-predictor-8666b4654c-pz9g5"
Jul 30 17:31:29.248331 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:29.248025    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/926f1662-23d6-47c4-81b1-e19078554bfa-kube-rbac-proxy-sar-config\") pod \"isvc-primary-303984-predictor-8666b4654c-pz9g5\" (UID: \"926f1662-23d6-47c4-81b1-e19078554bfa\") " pod="kserve-ci-e2e-test/isvc-primary-303984-predictor-8666b4654c-pz9g5"
Jul 30 17:31:29.248331 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:29.248061    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/926f1662-23d6-47c4-81b1-e19078554bfa-kserve-provision-location\") pod \"isvc-primary-303984-predictor-8666b4654c-pz9g5\" (UID: \"926f1662-23d6-47c4-81b1-e19078554bfa\") " pod="kserve-ci-e2e-test/isvc-primary-303984-predictor-8666b4654c-pz9g5"
Jul 30 17:31:29.248571 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:29.248551    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/926f1662-23d6-47c4-81b1-e19078554bfa-kserve-provision-location\") pod \"isvc-primary-303984-predictor-8666b4654c-pz9g5\" (UID: \"926f1662-23d6-47c4-81b1-e19078554bfa\") " pod="kserve-ci-e2e-test/isvc-primary-303984-predictor-8666b4654c-pz9g5"
Jul 30 17:31:29.248847 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:29.248825    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/926f1662-23d6-47c4-81b1-e19078554bfa-cabundle-cert\") pod \"isvc-primary-303984-predictor-8666b4654c-pz9g5\" (UID: \"926f1662-23d6-47c4-81b1-e19078554bfa\") " pod="kserve-ci-e2e-test/isvc-primary-303984-predictor-8666b4654c-pz9g5"
Jul 30 17:31:29.248942 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:29.248903    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/926f1662-23d6-47c4-81b1-e19078554bfa-kube-rbac-proxy-sar-config\") pod \"isvc-primary-303984-predictor-8666b4654c-pz9g5\" (UID: \"926f1662-23d6-47c4-81b1-e19078554bfa\") " pod="kserve-ci-e2e-test/isvc-primary-303984-predictor-8666b4654c-pz9g5"
Jul 30 17:31:29.250618 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:29.250597    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/926f1662-23d6-47c4-81b1-e19078554bfa-proxy-tls\") pod \"isvc-primary-303984-predictor-8666b4654c-pz9g5\" (UID: \"926f1662-23d6-47c4-81b1-e19078554bfa\") " pod="kserve-ci-e2e-test/isvc-primary-303984-predictor-8666b4654c-pz9g5"
Jul 30 17:31:29.266516 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:29.266480    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2649\" (UniqueName: \"kubernetes.io/projected/926f1662-23d6-47c4-81b1-e19078554bfa-kube-api-access-q2649\") pod \"isvc-primary-303984-predictor-8666b4654c-pz9g5\" (UID: \"926f1662-23d6-47c4-81b1-e19078554bfa\") " pod="kserve-ci-e2e-test/isvc-primary-303984-predictor-8666b4654c-pz9g5"
Jul 30 17:31:29.424938 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:29.424827    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-303984-predictor-8666b4654c-pz9g5"
Jul 30 17:31:29.577811 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:29.577773    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-303984-predictor-8666b4654c-pz9g5"]
Jul 30 17:31:29.582380 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:31:29.582338    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod926f1662_23d6_47c4_81b1_e19078554bfa.slice/crio-d329ebfff6ce1c83381fb0611160287b6afe32d167e49456cd4125e9dc09b329 WatchSource:0}: Error finding container d329ebfff6ce1c83381fb0611160287b6afe32d167e49456cd4125e9dc09b329: Status 404 returned error can't find the container with id d329ebfff6ce1c83381fb0611160287b6afe32d167e49456cd4125e9dc09b329
Jul 30 17:31:30.525204 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:30.525142    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-303984-predictor-8666b4654c-pz9g5" event={"ID":"926f1662-23d6-47c4-81b1-e19078554bfa","Type":"ContainerStarted","Data":"cf12c86e6352fbbfbe9651f9b41dff4624aef504d6f5028fac24038cf95b3ca9"}
Jul 30 17:31:30.525639 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:30.525244    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-303984-predictor-8666b4654c-pz9g5" event={"ID":"926f1662-23d6-47c4-81b1-e19078554bfa","Type":"ContainerStarted","Data":"d329ebfff6ce1c83381fb0611160287b6afe32d167e49456cd4125e9dc09b329"}
Jul 30 17:31:33.538878 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:33.538822    2580 generic.go:356] "Generic (PLEG): container finished" podID="926f1662-23d6-47c4-81b1-e19078554bfa" containerID="cf12c86e6352fbbfbe9651f9b41dff4624aef504d6f5028fac24038cf95b3ca9" exitCode=0
Jul 30 17:31:33.539461 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:33.538904    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-303984-predictor-8666b4654c-pz9g5" event={"ID":"926f1662-23d6-47c4-81b1-e19078554bfa","Type":"ContainerDied","Data":"cf12c86e6352fbbfbe9651f9b41dff4624aef504d6f5028fac24038cf95b3ca9"}
Jul 30 17:31:34.545313 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:34.545268    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-303984-predictor-8666b4654c-pz9g5" event={"ID":"926f1662-23d6-47c4-81b1-e19078554bfa","Type":"ContainerStarted","Data":"cd7b292503f88d1fc2608842d555bc4423a9f69acfd397da7cd189bfe742dc14"}
Jul 30 17:31:34.545850 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:34.545347    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-303984-predictor-8666b4654c-pz9g5" event={"ID":"926f1662-23d6-47c4-81b1-e19078554bfa","Type":"ContainerStarted","Data":"45a2be92d6d9c52714e2ff6a539e3a2733f6089fb812751ccb83ddf91b63a5b7"}
Jul 30 17:31:34.545850 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:34.545693    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-303984-predictor-8666b4654c-pz9g5"
Jul 30 17:31:34.545850 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:34.545741    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-303984-predictor-8666b4654c-pz9g5"
Jul 30 17:31:34.546972 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:34.546945    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-303984-predictor-8666b4654c-pz9g5" podUID="926f1662-23d6-47c4-81b1-e19078554bfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 30 17:31:34.584159 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:34.584093    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-303984-predictor-8666b4654c-pz9g5" podStartSLOduration=1.81645602 podStartE2EDuration="5.584076406s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.767620386s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:31:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:31:34.583113213 +0000 UTC m=+1857.658413103" watchObservedRunningTime="2026-07-30 17:31:34.584076406 +0000 UTC m=+1857.659376296"
Jul 30 17:31:35.549900 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:35.549857    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-303984-predictor-8666b4654c-pz9g5" podUID="926f1662-23d6-47c4-81b1-e19078554bfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 30 17:31:40.554899 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:40.554868    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-303984-predictor-8666b4654c-pz9g5"
Jul 30 17:31:40.555399 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:40.555372    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-303984-predictor-8666b4654c-pz9g5" podUID="926f1662-23d6-47c4-81b1-e19078554bfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 30 17:31:50.555902 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:31:50.555852    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-303984-predictor-8666b4654c-pz9g5" podUID="926f1662-23d6-47c4-81b1-e19078554bfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 30 17:32:00.555637 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:32:00.555538    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-303984-predictor-8666b4654c-pz9g5" podUID="926f1662-23d6-47c4-81b1-e19078554bfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 30 17:32:10.555952 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:32:10.555904    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-303984-predictor-8666b4654c-pz9g5" podUID="926f1662-23d6-47c4-81b1-e19078554bfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 30 17:32:20.555761 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:32:20.555711    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-303984-predictor-8666b4654c-pz9g5" podUID="926f1662-23d6-47c4-81b1-e19078554bfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 30 17:32:30.555433 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:32:30.555387    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-303984-predictor-8666b4654c-pz9g5" podUID="926f1662-23d6-47c4-81b1-e19078554bfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 30 17:32:40.555965 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:32:40.555923    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-303984-predictor-8666b4654c-pz9g5" podUID="926f1662-23d6-47c4-81b1-e19078554bfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 30 17:32:50.555817 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:32:50.555764    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-303984-predictor-8666b4654c-pz9g5" podUID="926f1662-23d6-47c4-81b1-e19078554bfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 30 17:32:53.482618 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:32:53.482585    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-303984-predictor-8666b4654c-pz9g5"
Jul 30 17:33:14.268464 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:14.268420    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-303984-predictor-6b948d9ddd-bw2sp"]
Jul 30 17:33:14.271935 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:14.271913    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-303984-predictor-6b948d9ddd-bw2sp"
Jul 30 17:33:14.276252 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:14.276221    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-303984\""
Jul 30 17:33:14.276551 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:14.276530    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-303984-kube-rbac-proxy-sar-config\""
Jul 30 17:33:14.276618 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:14.276577    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-303984-dockercfg-gsb2z\""
Jul 30 17:33:14.276667 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:14.276653    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-303984-predictor-serving-cert\""
Jul 30 17:33:14.290663 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:14.290626    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-303984-predictor-6b948d9ddd-bw2sp"]
Jul 30 17:33:14.443711 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:14.443672    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ce761c5-82b0-49f7-a36d-64aa30de9c15-proxy-tls\") pod \"isvc-secondary-303984-predictor-6b948d9ddd-bw2sp\" (UID: \"3ce761c5-82b0-49f7-a36d-64aa30de9c15\") " pod="kserve-ci-e2e-test/isvc-secondary-303984-predictor-6b948d9ddd-bw2sp"
Jul 30 17:33:14.443951 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:14.443735    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ce761c5-82b0-49f7-a36d-64aa30de9c15-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-303984-predictor-6b948d9ddd-bw2sp\" (UID: \"3ce761c5-82b0-49f7-a36d-64aa30de9c15\") " pod="kserve-ci-e2e-test/isvc-secondary-303984-predictor-6b948d9ddd-bw2sp"
Jul 30 17:33:14.443951 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:14.443767    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p5bd\" (UniqueName: \"kubernetes.io/projected/3ce761c5-82b0-49f7-a36d-64aa30de9c15-kube-api-access-8p5bd\") pod \"isvc-secondary-303984-predictor-6b948d9ddd-bw2sp\" (UID: \"3ce761c5-82b0-49f7-a36d-64aa30de9c15\") " pod="kserve-ci-e2e-test/isvc-secondary-303984-predictor-6b948d9ddd-bw2sp"
Jul 30 17:33:14.443951 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:14.443810    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ce761c5-82b0-49f7-a36d-64aa30de9c15-kserve-provision-location\") pod \"isvc-secondary-303984-predictor-6b948d9ddd-bw2sp\" (UID: \"3ce761c5-82b0-49f7-a36d-64aa30de9c15\") " pod="kserve-ci-e2e-test/isvc-secondary-303984-predictor-6b948d9ddd-bw2sp"
Jul 30 17:33:14.443951 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:14.443840    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3ce761c5-82b0-49f7-a36d-64aa30de9c15-cabundle-cert\") pod \"isvc-secondary-303984-predictor-6b948d9ddd-bw2sp\" (UID: \"3ce761c5-82b0-49f7-a36d-64aa30de9c15\") " pod="kserve-ci-e2e-test/isvc-secondary-303984-predictor-6b948d9ddd-bw2sp"
Jul 30 17:33:14.544633 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:14.544517    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ce761c5-82b0-49f7-a36d-64aa30de9c15-kserve-provision-location\") pod \"isvc-secondary-303984-predictor-6b948d9ddd-bw2sp\" (UID: \"3ce761c5-82b0-49f7-a36d-64aa30de9c15\") " pod="kserve-ci-e2e-test/isvc-secondary-303984-predictor-6b948d9ddd-bw2sp"
Jul 30 17:33:14.544633 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:14.544585    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3ce761c5-82b0-49f7-a36d-64aa30de9c15-cabundle-cert\") pod \"isvc-secondary-303984-predictor-6b948d9ddd-bw2sp\" (UID: \"3ce761c5-82b0-49f7-a36d-64aa30de9c15\") " pod="kserve-ci-e2e-test/isvc-secondary-303984-predictor-6b948d9ddd-bw2sp"
Jul 30 17:33:14.544633 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:14.544630    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ce761c5-82b0-49f7-a36d-64aa30de9c15-proxy-tls\") pod \"isvc-secondary-303984-predictor-6b948d9ddd-bw2sp\" (UID: \"3ce761c5-82b0-49f7-a36d-64aa30de9c15\") " pod="kserve-ci-e2e-test/isvc-secondary-303984-predictor-6b948d9ddd-bw2sp"
Jul 30 17:33:14.544952 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:14.544690    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ce761c5-82b0-49f7-a36d-64aa30de9c15-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-303984-predictor-6b948d9ddd-bw2sp\" (UID: \"3ce761c5-82b0-49f7-a36d-64aa30de9c15\") " pod="kserve-ci-e2e-test/isvc-secondary-303984-predictor-6b948d9ddd-bw2sp"
Jul 30 17:33:14.544952 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:14.544742    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p5bd\" (UniqueName: \"kubernetes.io/projected/3ce761c5-82b0-49f7-a36d-64aa30de9c15-kube-api-access-8p5bd\") pod \"isvc-secondary-303984-predictor-6b948d9ddd-bw2sp\" (UID: \"3ce761c5-82b0-49f7-a36d-64aa30de9c15\") " pod="kserve-ci-e2e-test/isvc-secondary-303984-predictor-6b948d9ddd-bw2sp"
Jul 30 17:33:14.545052 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:14.544991    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ce761c5-82b0-49f7-a36d-64aa30de9c15-kserve-provision-location\") pod \"isvc-secondary-303984-predictor-6b948d9ddd-bw2sp\" (UID: \"3ce761c5-82b0-49f7-a36d-64aa30de9c15\") " pod="kserve-ci-e2e-test/isvc-secondary-303984-predictor-6b948d9ddd-bw2sp"
Jul 30 17:33:14.545444 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:14.545421    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ce761c5-82b0-49f7-a36d-64aa30de9c15-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-303984-predictor-6b948d9ddd-bw2sp\" (UID: \"3ce761c5-82b0-49f7-a36d-64aa30de9c15\") " pod="kserve-ci-e2e-test/isvc-secondary-303984-predictor-6b948d9ddd-bw2sp"
Jul 30 17:33:14.545563 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:14.545429    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3ce761c5-82b0-49f7-a36d-64aa30de9c15-cabundle-cert\") pod \"isvc-secondary-303984-predictor-6b948d9ddd-bw2sp\" (UID: \"3ce761c5-82b0-49f7-a36d-64aa30de9c15\") " pod="kserve-ci-e2e-test/isvc-secondary-303984-predictor-6b948d9ddd-bw2sp"
Jul 30 17:33:14.547480 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:14.547456    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ce761c5-82b0-49f7-a36d-64aa30de9c15-proxy-tls\") pod \"isvc-secondary-303984-predictor-6b948d9ddd-bw2sp\" (UID: \"3ce761c5-82b0-49f7-a36d-64aa30de9c15\") " pod="kserve-ci-e2e-test/isvc-secondary-303984-predictor-6b948d9ddd-bw2sp"
Jul 30 17:33:14.557045 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:14.557006    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p5bd\" (UniqueName: \"kubernetes.io/projected/3ce761c5-82b0-49f7-a36d-64aa30de9c15-kube-api-access-8p5bd\") pod \"isvc-secondary-303984-predictor-6b948d9ddd-bw2sp\" (UID: \"3ce761c5-82b0-49f7-a36d-64aa30de9c15\") " pod="kserve-ci-e2e-test/isvc-secondary-303984-predictor-6b948d9ddd-bw2sp"
Jul 30 17:33:14.582998 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:14.582953    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-303984-predictor-6b948d9ddd-bw2sp"
Jul 30 17:33:14.747952 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:14.747914    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-303984-predictor-6b948d9ddd-bw2sp"]
Jul 30 17:33:14.751565 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:33:14.751508    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ce761c5_82b0_49f7_a36d_64aa30de9c15.slice/crio-500cd5f78e154700e559937dbf7804a7c5c679cd5d29741f8aa3f99dd6e694b5 WatchSource:0}: Error finding container 500cd5f78e154700e559937dbf7804a7c5c679cd5d29741f8aa3f99dd6e694b5: Status 404 returned error can't find the container with id 500cd5f78e154700e559937dbf7804a7c5c679cd5d29741f8aa3f99dd6e694b5
Jul 30 17:33:14.971039 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:14.970996    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-303984-predictor-6b948d9ddd-bw2sp" event={"ID":"3ce761c5-82b0-49f7-a36d-64aa30de9c15","Type":"ContainerStarted","Data":"feb90fa89d2e0d8f26896f15fbdfb83ddc7b3639ef0174b0a423022f2a940349"}
Jul 30 17:33:14.971254 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:14.971059    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-303984-predictor-6b948d9ddd-bw2sp" event={"ID":"3ce761c5-82b0-49f7-a36d-64aa30de9c15","Type":"ContainerStarted","Data":"500cd5f78e154700e559937dbf7804a7c5c679cd5d29741f8aa3f99dd6e694b5"}
Jul 30 17:33:20.998275 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:20.998243    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-303984-predictor-6b948d9ddd-bw2sp_3ce761c5-82b0-49f7-a36d-64aa30de9c15/storage-initializer/0.log"
Jul 30 17:33:20.998737 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:20.998285    2580 generic.go:356] "Generic (PLEG): container finished" podID="3ce761c5-82b0-49f7-a36d-64aa30de9c15" containerID="feb90fa89d2e0d8f26896f15fbdfb83ddc7b3639ef0174b0a423022f2a940349" exitCode=1
Jul 30 17:33:20.998737 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:20.998364    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-303984-predictor-6b948d9ddd-bw2sp" event={"ID":"3ce761c5-82b0-49f7-a36d-64aa30de9c15","Type":"ContainerDied","Data":"feb90fa89d2e0d8f26896f15fbdfb83ddc7b3639ef0174b0a423022f2a940349"}
Jul 30 17:33:22.010906 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:22.010875    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-303984-predictor-6b948d9ddd-bw2sp_3ce761c5-82b0-49f7-a36d-64aa30de9c15/storage-initializer/0.log"
Jul 30 17:33:22.011361 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:22.010949    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-303984-predictor-6b948d9ddd-bw2sp" event={"ID":"3ce761c5-82b0-49f7-a36d-64aa30de9c15","Type":"ContainerStarted","Data":"aeed5a65e7d6d08100f22901037bdad54a8c9d58324342df5e02745d9edeb904"}
Jul 30 17:33:25.025498 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:25.025466    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-303984-predictor-6b948d9ddd-bw2sp_3ce761c5-82b0-49f7-a36d-64aa30de9c15/storage-initializer/1.log"
Jul 30 17:33:25.025947 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:25.025858    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-303984-predictor-6b948d9ddd-bw2sp_3ce761c5-82b0-49f7-a36d-64aa30de9c15/storage-initializer/0.log"
Jul 30 17:33:25.025947 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:25.025899    2580 generic.go:356] "Generic (PLEG): container finished" podID="3ce761c5-82b0-49f7-a36d-64aa30de9c15" containerID="aeed5a65e7d6d08100f22901037bdad54a8c9d58324342df5e02745d9edeb904" exitCode=1
Jul 30 17:33:25.026028 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:25.025984    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-303984-predictor-6b948d9ddd-bw2sp" event={"ID":"3ce761c5-82b0-49f7-a36d-64aa30de9c15","Type":"ContainerDied","Data":"aeed5a65e7d6d08100f22901037bdad54a8c9d58324342df5e02745d9edeb904"}
Jul 30 17:33:25.026077 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:25.026065    2580 scope.go:117] "RemoveContainer" containerID="feb90fa89d2e0d8f26896f15fbdfb83ddc7b3639ef0174b0a423022f2a940349"
Jul 30 17:33:25.026531 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:25.026509    2580 scope.go:117] "RemoveContainer" containerID="feb90fa89d2e0d8f26896f15fbdfb83ddc7b3639ef0174b0a423022f2a940349"
Jul 30 17:33:25.037808 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:33:25.037777    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-303984-predictor-6b948d9ddd-bw2sp_kserve-ci-e2e-test(3ce761c5-82b0-49f7-a36d-64aa30de9c15)\"" pod="kserve-ci-e2e-test/isvc-secondary-303984-predictor-6b948d9ddd-bw2sp" podUID="3ce761c5-82b0-49f7-a36d-64aa30de9c15"
Jul 30 17:33:25.037888 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:33:25.037840    2580 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-303984-predictor-6b948d9ddd-bw2sp_kserve-ci-e2e-test_3ce761c5-82b0-49f7-a36d-64aa30de9c15_0 in pod sandbox 500cd5f78e154700e559937dbf7804a7c5c679cd5d29741f8aa3f99dd6e694b5 from index: no such id: 'feb90fa89d2e0d8f26896f15fbdfb83ddc7b3639ef0174b0a423022f2a940349'" containerID="feb90fa89d2e0d8f26896f15fbdfb83ddc7b3639ef0174b0a423022f2a940349"
Jul 30 17:33:25.037888 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:25.037872    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"feb90fa89d2e0d8f26896f15fbdfb83ddc7b3639ef0174b0a423022f2a940349"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-303984-predictor-6b948d9ddd-bw2sp_kserve-ci-e2e-test_3ce761c5-82b0-49f7-a36d-64aa30de9c15_0 in pod sandbox 500cd5f78e154700e559937dbf7804a7c5c679cd5d29741f8aa3f99dd6e694b5 from index: no such id: 'feb90fa89d2e0d8f26896f15fbdfb83ddc7b3639ef0174b0a423022f2a940349'"
Jul 30 17:33:26.032554 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:26.032523    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-303984-predictor-6b948d9ddd-bw2sp_3ce761c5-82b0-49f7-a36d-64aa30de9c15/storage-initializer/1.log"
Jul 30 17:33:32.303540 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:32.303437    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-303984-predictor-6b948d9ddd-bw2sp"]
Jul 30 17:33:32.394219 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:32.394181    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-303984-predictor-8666b4654c-pz9g5"]
Jul 30 17:33:32.394538 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:32.394504    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-303984-predictor-8666b4654c-pz9g5" podUID="926f1662-23d6-47c4-81b1-e19078554bfa" containerName="kserve-container" containerID="cri-o://45a2be92d6d9c52714e2ff6a539e3a2733f6089fb812751ccb83ddf91b63a5b7" gracePeriod=30
Jul 30 17:33:32.394769 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:32.394568    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-303984-predictor-8666b4654c-pz9g5" podUID="926f1662-23d6-47c4-81b1-e19078554bfa" containerName="kube-rbac-proxy" containerID="cri-o://cd7b292503f88d1fc2608842d555bc4423a9f69acfd397da7cd189bfe742dc14" gracePeriod=30
Jul 30 17:33:32.515607 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:32.515582    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-303984-predictor-6b948d9ddd-bw2sp_3ce761c5-82b0-49f7-a36d-64aa30de9c15/storage-initializer/1.log"
Jul 30 17:33:32.517837 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:32.517817    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-303984-predictor-6b948d9ddd-bw2sp"
Jul 30 17:33:32.590052 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:32.589944    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3ce761c5-82b0-49f7-a36d-64aa30de9c15-cabundle-cert\") pod \"3ce761c5-82b0-49f7-a36d-64aa30de9c15\" (UID: \"3ce761c5-82b0-49f7-a36d-64aa30de9c15\") "
Jul 30 17:33:32.590052 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:32.589996    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ce761c5-82b0-49f7-a36d-64aa30de9c15-kserve-provision-location\") pod \"3ce761c5-82b0-49f7-a36d-64aa30de9c15\" (UID: \"3ce761c5-82b0-49f7-a36d-64aa30de9c15\") "
Jul 30 17:33:32.590052 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:32.590020    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ce761c5-82b0-49f7-a36d-64aa30de9c15-kube-rbac-proxy-sar-config\") pod \"3ce761c5-82b0-49f7-a36d-64aa30de9c15\" (UID: \"3ce761c5-82b0-49f7-a36d-64aa30de9c15\") "
Jul 30 17:33:32.590327 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:32.590064    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8p5bd\" (UniqueName: \"kubernetes.io/projected/3ce761c5-82b0-49f7-a36d-64aa30de9c15-kube-api-access-8p5bd\") pod \"3ce761c5-82b0-49f7-a36d-64aa30de9c15\" (UID: \"3ce761c5-82b0-49f7-a36d-64aa30de9c15\") "
Jul 30 17:33:32.590327 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:32.590118    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ce761c5-82b0-49f7-a36d-64aa30de9c15-proxy-tls\") pod \"3ce761c5-82b0-49f7-a36d-64aa30de9c15\" (UID: \"3ce761c5-82b0-49f7-a36d-64aa30de9c15\") "
Jul 30 17:33:32.590416 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:32.590381    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ce761c5-82b0-49f7-a36d-64aa30de9c15-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3ce761c5-82b0-49f7-a36d-64aa30de9c15" (UID: "3ce761c5-82b0-49f7-a36d-64aa30de9c15"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 17:33:32.590469 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:32.590452    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ce761c5-82b0-49f7-a36d-64aa30de9c15-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3ce761c5-82b0-49f7-a36d-64aa30de9c15" (UID: "3ce761c5-82b0-49f7-a36d-64aa30de9c15"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:33:32.590509 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:32.590487    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ce761c5-82b0-49f7-a36d-64aa30de9c15-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3ce761c5-82b0-49f7-a36d-64aa30de9c15" (UID: "3ce761c5-82b0-49f7-a36d-64aa30de9c15"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:33:32.592415 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:32.592374    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ce761c5-82b0-49f7-a36d-64aa30de9c15-kube-api-access-8p5bd" (OuterVolumeSpecName: "kube-api-access-8p5bd") pod "3ce761c5-82b0-49f7-a36d-64aa30de9c15" (UID: "3ce761c5-82b0-49f7-a36d-64aa30de9c15"). InnerVolumeSpecName "kube-api-access-8p5bd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:33:32.592562 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:32.592448    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ce761c5-82b0-49f7-a36d-64aa30de9c15-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3ce761c5-82b0-49f7-a36d-64aa30de9c15" (UID: "3ce761c5-82b0-49f7-a36d-64aa30de9c15"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:33:32.691340 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:32.691291    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ce761c5-82b0-49f7-a36d-64aa30de9c15-proxy-tls\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:33:32.691340 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:32.691327    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3ce761c5-82b0-49f7-a36d-64aa30de9c15-cabundle-cert\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:33:32.691340 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:32.691337    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ce761c5-82b0-49f7-a36d-64aa30de9c15-kserve-provision-location\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:33:32.691340 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:32.691347    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ce761c5-82b0-49f7-a36d-64aa30de9c15-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:33:32.691340 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:32.691355    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p5bd\" (UniqueName: \"kubernetes.io/projected/3ce761c5-82b0-49f7-a36d-64aa30de9c15-kube-api-access-8p5bd\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:33:33.064696 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:33.064663    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-303984-predictor-6b948d9ddd-bw2sp_3ce761c5-82b0-49f7-a36d-64aa30de9c15/storage-initializer/1.log"
Jul 30 17:33:33.064929 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:33.064798    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-303984-predictor-6b948d9ddd-bw2sp"
Jul 30 17:33:33.064929 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:33.064803    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-303984-predictor-6b948d9ddd-bw2sp" event={"ID":"3ce761c5-82b0-49f7-a36d-64aa30de9c15","Type":"ContainerDied","Data":"500cd5f78e154700e559937dbf7804a7c5c679cd5d29741f8aa3f99dd6e694b5"}
Jul 30 17:33:33.064929 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:33.064915    2580 scope.go:117] "RemoveContainer" containerID="aeed5a65e7d6d08100f22901037bdad54a8c9d58324342df5e02745d9edeb904"
Jul 30 17:33:33.067309 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:33.067273    2580 generic.go:356] "Generic (PLEG): container finished" podID="926f1662-23d6-47c4-81b1-e19078554bfa" containerID="cd7b292503f88d1fc2608842d555bc4423a9f69acfd397da7cd189bfe742dc14" exitCode=2
Jul 30 17:33:33.067447 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:33.067343    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-303984-predictor-8666b4654c-pz9g5" event={"ID":"926f1662-23d6-47c4-81b1-e19078554bfa","Type":"ContainerDied","Data":"cd7b292503f88d1fc2608842d555bc4423a9f69acfd397da7cd189bfe742dc14"}
Jul 30 17:33:33.104026 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:33.103981    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-303984-predictor-6b948d9ddd-bw2sp"]
Jul 30 17:33:33.108476 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:33.108440    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-303984-predictor-6b948d9ddd-bw2sp"]
Jul 30 17:33:33.478073 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:33.477953    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-303984-predictor-8666b4654c-pz9g5" podUID="926f1662-23d6-47c4-81b1-e19078554bfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 30 17:33:33.483054 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:33.483021    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ce761c5-82b0-49f7-a36d-64aa30de9c15" path="/var/lib/kubelet/pods/3ce761c5-82b0-49f7-a36d-64aa30de9c15/volumes"
Jul 30 17:33:35.550294 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:35.550243    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-303984-predictor-8666b4654c-pz9g5" podUID="926f1662-23d6-47c4-81b1-e19078554bfa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Jul 30 17:33:37.482803 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:37.482773    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-61f75d-predictor-fc548b88d-njzqx"]
Jul 30 17:33:37.483216 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:37.483118    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ce761c5-82b0-49f7-a36d-64aa30de9c15" containerName="storage-initializer"
Jul 30 17:33:37.483216 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:37.483131    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ce761c5-82b0-49f7-a36d-64aa30de9c15" containerName="storage-initializer"
Jul 30 17:33:37.483216 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:37.483147    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ce761c5-82b0-49f7-a36d-64aa30de9c15" containerName="storage-initializer"
Jul 30 17:33:37.483216 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:37.483153    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ce761c5-82b0-49f7-a36d-64aa30de9c15" containerName="storage-initializer"
Jul 30 17:33:37.483360 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:37.483232    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ce761c5-82b0-49f7-a36d-64aa30de9c15" containerName="storage-initializer"
Jul 30 17:33:37.483360 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:37.483243    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ce761c5-82b0-49f7-a36d-64aa30de9c15" containerName="storage-initializer"
Jul 30 17:33:37.487933 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:37.487908    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-61f75d-predictor-fc548b88d-njzqx"
Jul 30 17:33:37.490980 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:37.490948    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-61f75d-predictor-serving-cert\""
Jul 30 17:33:37.490980 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:37.490956    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-61f75d\""
Jul 30 17:33:37.491252 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:37.491018    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-61f75d-dockercfg-6ptst\""
Jul 30 17:33:37.491315 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:37.491272    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-61f75d-kube-rbac-proxy-sar-config\""
Jul 30 17:33:37.500695 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:37.500667    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-61f75d-predictor-fc548b88d-njzqx"]
Jul 30 17:33:37.532514 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:37.532464    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfngm\" (UniqueName: \"kubernetes.io/projected/987cad56-29db-4deb-a983-4f2aa7803f7a-kube-api-access-hfngm\") pod \"isvc-init-fail-61f75d-predictor-fc548b88d-njzqx\" (UID: \"987cad56-29db-4deb-a983-4f2aa7803f7a\") " pod="kserve-ci-e2e-test/isvc-init-fail-61f75d-predictor-fc548b88d-njzqx"
Jul 30 17:33:37.532773 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:37.532527    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/987cad56-29db-4deb-a983-4f2aa7803f7a-cabundle-cert\") pod \"isvc-init-fail-61f75d-predictor-fc548b88d-njzqx\" (UID: \"987cad56-29db-4deb-a983-4f2aa7803f7a\") " pod="kserve-ci-e2e-test/isvc-init-fail-61f75d-predictor-fc548b88d-njzqx"
Jul 30 17:33:37.532773 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:37.532588    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/987cad56-29db-4deb-a983-4f2aa7803f7a-kserve-provision-location\") pod \"isvc-init-fail-61f75d-predictor-fc548b88d-njzqx\" (UID: \"987cad56-29db-4deb-a983-4f2aa7803f7a\") " pod="kserve-ci-e2e-test/isvc-init-fail-61f75d-predictor-fc548b88d-njzqx"
Jul 30 17:33:37.532773 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:37.532630    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/987cad56-29db-4deb-a983-4f2aa7803f7a-proxy-tls\") pod \"isvc-init-fail-61f75d-predictor-fc548b88d-njzqx\" (UID: \"987cad56-29db-4deb-a983-4f2aa7803f7a\") " pod="kserve-ci-e2e-test/isvc-init-fail-61f75d-predictor-fc548b88d-njzqx"
Jul 30 17:33:37.532773 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:37.532679    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/987cad56-29db-4deb-a983-4f2aa7803f7a-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-61f75d-predictor-fc548b88d-njzqx\" (UID: \"987cad56-29db-4deb-a983-4f2aa7803f7a\") " pod="kserve-ci-e2e-test/isvc-init-fail-61f75d-predictor-fc548b88d-njzqx"
Jul 30 17:33:37.633786 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:37.633737    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/987cad56-29db-4deb-a983-4f2aa7803f7a-cabundle-cert\") pod \"isvc-init-fail-61f75d-predictor-fc548b88d-njzqx\" (UID: \"987cad56-29db-4deb-a983-4f2aa7803f7a\") " pod="kserve-ci-e2e-test/isvc-init-fail-61f75d-predictor-fc548b88d-njzqx"
Jul 30 17:33:37.634024 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:37.633799    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/987cad56-29db-4deb-a983-4f2aa7803f7a-kserve-provision-location\") pod \"isvc-init-fail-61f75d-predictor-fc548b88d-njzqx\" (UID: \"987cad56-29db-4deb-a983-4f2aa7803f7a\") " pod="kserve-ci-e2e-test/isvc-init-fail-61f75d-predictor-fc548b88d-njzqx"
Jul 30 17:33:37.634024 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:37.633832    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/987cad56-29db-4deb-a983-4f2aa7803f7a-proxy-tls\") pod \"isvc-init-fail-61f75d-predictor-fc548b88d-njzqx\" (UID: \"987cad56-29db-4deb-a983-4f2aa7803f7a\") " pod="kserve-ci-e2e-test/isvc-init-fail-61f75d-predictor-fc548b88d-njzqx"
Jul 30 17:33:37.634024 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:37.633877    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/987cad56-29db-4deb-a983-4f2aa7803f7a-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-61f75d-predictor-fc548b88d-njzqx\" (UID: \"987cad56-29db-4deb-a983-4f2aa7803f7a\") " pod="kserve-ci-e2e-test/isvc-init-fail-61f75d-predictor-fc548b88d-njzqx"
Jul 30 17:33:37.634024 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:37.633917    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfngm\" (UniqueName: \"kubernetes.io/projected/987cad56-29db-4deb-a983-4f2aa7803f7a-kube-api-access-hfngm\") pod \"isvc-init-fail-61f75d-predictor-fc548b88d-njzqx\" (UID: \"987cad56-29db-4deb-a983-4f2aa7803f7a\") " pod="kserve-ci-e2e-test/isvc-init-fail-61f75d-predictor-fc548b88d-njzqx"
Jul 30 17:33:37.634336 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:33:37.634022    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-61f75d-predictor-serving-cert: secret "isvc-init-fail-61f75d-predictor-serving-cert" not found
Jul 30 17:33:37.634336 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:33:37.634114    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/987cad56-29db-4deb-a983-4f2aa7803f7a-proxy-tls podName:987cad56-29db-4deb-a983-4f2aa7803f7a nodeName:}" failed. No retries permitted until 2026-07-30 17:33:38.134088288 +0000 UTC m=+1981.209388173 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/987cad56-29db-4deb-a983-4f2aa7803f7a-proxy-tls") pod "isvc-init-fail-61f75d-predictor-fc548b88d-njzqx" (UID: "987cad56-29db-4deb-a983-4f2aa7803f7a") : secret "isvc-init-fail-61f75d-predictor-serving-cert" not found
Jul 30 17:33:37.634470 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:37.634428    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/987cad56-29db-4deb-a983-4f2aa7803f7a-kserve-provision-location\") pod \"isvc-init-fail-61f75d-predictor-fc548b88d-njzqx\" (UID: \"987cad56-29db-4deb-a983-4f2aa7803f7a\") " pod="kserve-ci-e2e-test/isvc-init-fail-61f75d-predictor-fc548b88d-njzqx"
Jul 30 17:33:37.634660 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:37.634638    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/987cad56-29db-4deb-a983-4f2aa7803f7a-cabundle-cert\") pod \"isvc-init-fail-61f75d-predictor-fc548b88d-njzqx\" (UID: \"987cad56-29db-4deb-a983-4f2aa7803f7a\") " pod="kserve-ci-e2e-test/isvc-init-fail-61f75d-predictor-fc548b88d-njzqx"
Jul 30 17:33:37.634748 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:37.634729    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/987cad56-29db-4deb-a983-4f2aa7803f7a-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-61f75d-predictor-fc548b88d-njzqx\" (UID: \"987cad56-29db-4deb-a983-4f2aa7803f7a\") " pod="kserve-ci-e2e-test/isvc-init-fail-61f75d-predictor-fc548b88d-njzqx"
Jul 30 17:33:37.650160 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:37.650127    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfngm\" (UniqueName: \"kubernetes.io/projected/987cad56-29db-4deb-a983-4f2aa7803f7a-kube-api-access-hfngm\") pod \"isvc-init-fail-61f75d-predictor-fc548b88d-njzqx\" (UID: \"987cad56-29db-4deb-a983-4f2aa7803f7a\") " pod="kserve-ci-e2e-test/isvc-init-fail-61f75d-predictor-fc548b88d-njzqx"
Jul 30 17:33:38.095268 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:38.095232    2580 generic.go:356] "Generic (PLEG): container finished" podID="926f1662-23d6-47c4-81b1-e19078554bfa" containerID="45a2be92d6d9c52714e2ff6a539e3a2733f6089fb812751ccb83ddf91b63a5b7" exitCode=0
Jul 30 17:33:38.095440 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:38.095304    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-303984-predictor-8666b4654c-pz9g5" event={"ID":"926f1662-23d6-47c4-81b1-e19078554bfa","Type":"ContainerDied","Data":"45a2be92d6d9c52714e2ff6a539e3a2733f6089fb812751ccb83ddf91b63a5b7"}
Jul 30 17:33:38.137848 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:38.137820    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/987cad56-29db-4deb-a983-4f2aa7803f7a-proxy-tls\") pod \"isvc-init-fail-61f75d-predictor-fc548b88d-njzqx\" (UID: \"987cad56-29db-4deb-a983-4f2aa7803f7a\") " pod="kserve-ci-e2e-test/isvc-init-fail-61f75d-predictor-fc548b88d-njzqx"
Jul 30 17:33:38.140465 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:38.140437    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/987cad56-29db-4deb-a983-4f2aa7803f7a-proxy-tls\") pod \"isvc-init-fail-61f75d-predictor-fc548b88d-njzqx\" (UID: \"987cad56-29db-4deb-a983-4f2aa7803f7a\") " pod="kserve-ci-e2e-test/isvc-init-fail-61f75d-predictor-fc548b88d-njzqx"
Jul 30 17:33:38.155728 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:38.155688    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-303984-predictor-8666b4654c-pz9g5"
Jul 30 17:33:38.238908 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:38.238797    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q2649\" (UniqueName: \"kubernetes.io/projected/926f1662-23d6-47c4-81b1-e19078554bfa-kube-api-access-q2649\") pod \"926f1662-23d6-47c4-81b1-e19078554bfa\" (UID: \"926f1662-23d6-47c4-81b1-e19078554bfa\") "
Jul 30 17:33:38.238908 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:38.238846    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/926f1662-23d6-47c4-81b1-e19078554bfa-proxy-tls\") pod \"926f1662-23d6-47c4-81b1-e19078554bfa\" (UID: \"926f1662-23d6-47c4-81b1-e19078554bfa\") "
Jul 30 17:33:38.238908 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:38.238890    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/926f1662-23d6-47c4-81b1-e19078554bfa-kserve-provision-location\") pod \"926f1662-23d6-47c4-81b1-e19078554bfa\" (UID: \"926f1662-23d6-47c4-81b1-e19078554bfa\") "
Jul 30 17:33:38.239256 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:38.238916    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/926f1662-23d6-47c4-81b1-e19078554bfa-cabundle-cert\") pod \"926f1662-23d6-47c4-81b1-e19078554bfa\" (UID: \"926f1662-23d6-47c4-81b1-e19078554bfa\") "
Jul 30 17:33:38.239256 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:38.238943    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/926f1662-23d6-47c4-81b1-e19078554bfa-kube-rbac-proxy-sar-config\") pod \"926f1662-23d6-47c4-81b1-e19078554bfa\" (UID: \"926f1662-23d6-47c4-81b1-e19078554bfa\") "
Jul 30 17:33:38.239380 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:38.239307    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/926f1662-23d6-47c4-81b1-e19078554bfa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "926f1662-23d6-47c4-81b1-e19078554bfa" (UID: "926f1662-23d6-47c4-81b1-e19078554bfa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 17:33:38.239380 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:38.239362    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/926f1662-23d6-47c4-81b1-e19078554bfa-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "926f1662-23d6-47c4-81b1-e19078554bfa" (UID: "926f1662-23d6-47c4-81b1-e19078554bfa"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:33:38.239458 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:38.239394    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/926f1662-23d6-47c4-81b1-e19078554bfa-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "926f1662-23d6-47c4-81b1-e19078554bfa" (UID: "926f1662-23d6-47c4-81b1-e19078554bfa"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:33:38.241125 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:38.241097    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/926f1662-23d6-47c4-81b1-e19078554bfa-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "926f1662-23d6-47c4-81b1-e19078554bfa" (UID: "926f1662-23d6-47c4-81b1-e19078554bfa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:33:38.241125 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:38.241102    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/926f1662-23d6-47c4-81b1-e19078554bfa-kube-api-access-q2649" (OuterVolumeSpecName: "kube-api-access-q2649") pod "926f1662-23d6-47c4-81b1-e19078554bfa" (UID: "926f1662-23d6-47c4-81b1-e19078554bfa"). InnerVolumeSpecName "kube-api-access-q2649". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:33:38.339857 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:38.339818    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/926f1662-23d6-47c4-81b1-e19078554bfa-kserve-provision-location\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:33:38.339857 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:38.339853    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/926f1662-23d6-47c4-81b1-e19078554bfa-cabundle-cert\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:33:38.339857 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:38.339863    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/926f1662-23d6-47c4-81b1-e19078554bfa-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:33:38.340117 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:38.339873    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2649\" (UniqueName: \"kubernetes.io/projected/926f1662-23d6-47c4-81b1-e19078554bfa-kube-api-access-q2649\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:33:38.340117 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:38.339883    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/926f1662-23d6-47c4-81b1-e19078554bfa-proxy-tls\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:33:38.399422 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:38.399382    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-61f75d-predictor-fc548b88d-njzqx"
Jul 30 17:33:38.544195 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:38.544140    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-61f75d-predictor-fc548b88d-njzqx"]
Jul 30 17:33:38.547276 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:33:38.547242    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod987cad56_29db_4deb_a983_4f2aa7803f7a.slice/crio-10543f4da1b626bced40015869c62ad004ddcb1d6904e9aa360c27f259fbba0a WatchSource:0}: Error finding container 10543f4da1b626bced40015869c62ad004ddcb1d6904e9aa360c27f259fbba0a: Status 404 returned error can't find the container with id 10543f4da1b626bced40015869c62ad004ddcb1d6904e9aa360c27f259fbba0a
Jul 30 17:33:39.102248 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:39.102206    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-303984-predictor-8666b4654c-pz9g5" event={"ID":"926f1662-23d6-47c4-81b1-e19078554bfa","Type":"ContainerDied","Data":"d329ebfff6ce1c83381fb0611160287b6afe32d167e49456cd4125e9dc09b329"}
Jul 30 17:33:39.102248 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:39.102242    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-303984-predictor-8666b4654c-pz9g5"
Jul 30 17:33:39.102534 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:39.102291    2580 scope.go:117] "RemoveContainer" containerID="cd7b292503f88d1fc2608842d555bc4423a9f69acfd397da7cd189bfe742dc14"
Jul 30 17:33:39.104248 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:39.104199    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-61f75d-predictor-fc548b88d-njzqx" event={"ID":"987cad56-29db-4deb-a983-4f2aa7803f7a","Type":"ContainerStarted","Data":"cf71d58fd2c0a26ba14a04bc3c6777bd9a0f1d0c5dc40156be2c6f842d281573"}
Jul 30 17:33:39.104413 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:39.104282    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-61f75d-predictor-fc548b88d-njzqx" event={"ID":"987cad56-29db-4deb-a983-4f2aa7803f7a","Type":"ContainerStarted","Data":"10543f4da1b626bced40015869c62ad004ddcb1d6904e9aa360c27f259fbba0a"}
Jul 30 17:33:39.114090 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:39.114065    2580 scope.go:117] "RemoveContainer" containerID="45a2be92d6d9c52714e2ff6a539e3a2733f6089fb812751ccb83ddf91b63a5b7"
Jul 30 17:33:39.123852 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:39.123833    2580 scope.go:117] "RemoveContainer" containerID="cf12c86e6352fbbfbe9651f9b41dff4624aef504d6f5028fac24038cf95b3ca9"
Jul 30 17:33:39.147074 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:39.147039    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-303984-predictor-8666b4654c-pz9g5"]
Jul 30 17:33:39.152055 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:39.152017    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-303984-predictor-8666b4654c-pz9g5"]
Jul 30 17:33:39.482810 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:39.482718    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="926f1662-23d6-47c4-81b1-e19078554bfa" path="/var/lib/kubelet/pods/926f1662-23d6-47c4-81b1-e19078554bfa/volumes"
Jul 30 17:33:42.120233 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:42.120204    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-61f75d-predictor-fc548b88d-njzqx_987cad56-29db-4deb-a983-4f2aa7803f7a/storage-initializer/0.log"
Jul 30 17:33:42.120676 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:42.120246    2580 generic.go:356] "Generic (PLEG): container finished" podID="987cad56-29db-4deb-a983-4f2aa7803f7a" containerID="cf71d58fd2c0a26ba14a04bc3c6777bd9a0f1d0c5dc40156be2c6f842d281573" exitCode=1
Jul 30 17:33:42.120676 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:42.120318    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-61f75d-predictor-fc548b88d-njzqx" event={"ID":"987cad56-29db-4deb-a983-4f2aa7803f7a","Type":"ContainerDied","Data":"cf71d58fd2c0a26ba14a04bc3c6777bd9a0f1d0c5dc40156be2c6f842d281573"}
Jul 30 17:33:42.517594 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:42.517558    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-61f75d-predictor-fc548b88d-njzqx"]
Jul 30 17:33:43.126226 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:43.126195    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-61f75d-predictor-fc548b88d-njzqx_987cad56-29db-4deb-a983-4f2aa7803f7a/storage-initializer/0.log"
Jul 30 17:33:43.126664 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:43.126333    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-61f75d-predictor-fc548b88d-njzqx" event={"ID":"987cad56-29db-4deb-a983-4f2aa7803f7a","Type":"ContainerStarted","Data":"a9ef3e274c1239b8abfbcc7b6eeb5fb2198b30af7e3b845e01d74c424c9cd2b6"}
Jul 30 17:33:43.126664 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:43.126382    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-61f75d-predictor-fc548b88d-njzqx" podUID="987cad56-29db-4deb-a983-4f2aa7803f7a" containerName="storage-initializer" containerID="cri-o://a9ef3e274c1239b8abfbcc7b6eeb5fb2198b30af7e3b845e01d74c424c9cd2b6" gracePeriod=30
Jul 30 17:33:47.985071 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:47.985042    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-61f75d-predictor-fc548b88d-njzqx_987cad56-29db-4deb-a983-4f2aa7803f7a/storage-initializer/1.log"
Jul 30 17:33:47.985488 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:47.985461    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-61f75d-predictor-fc548b88d-njzqx_987cad56-29db-4deb-a983-4f2aa7803f7a/storage-initializer/0.log"
Jul 30 17:33:47.987822 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:47.987806    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-61f75d-predictor-fc548b88d-njzqx"
Jul 30 17:33:48.023828 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:48.023733    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/987cad56-29db-4deb-a983-4f2aa7803f7a-proxy-tls\") pod \"987cad56-29db-4deb-a983-4f2aa7803f7a\" (UID: \"987cad56-29db-4deb-a983-4f2aa7803f7a\") "
Jul 30 17:33:48.023828 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:48.023793    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/987cad56-29db-4deb-a983-4f2aa7803f7a-kserve-provision-location\") pod \"987cad56-29db-4deb-a983-4f2aa7803f7a\" (UID: \"987cad56-29db-4deb-a983-4f2aa7803f7a\") "
Jul 30 17:33:48.024042 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:48.023866    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/987cad56-29db-4deb-a983-4f2aa7803f7a-kube-rbac-proxy-sar-config\") pod \"987cad56-29db-4deb-a983-4f2aa7803f7a\" (UID: \"987cad56-29db-4deb-a983-4f2aa7803f7a\") "
Jul 30 17:33:48.024042 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:48.023905    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/987cad56-29db-4deb-a983-4f2aa7803f7a-cabundle-cert\") pod \"987cad56-29db-4deb-a983-4f2aa7803f7a\" (UID: \"987cad56-29db-4deb-a983-4f2aa7803f7a\") "
Jul 30 17:33:48.024042 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:48.023948    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hfngm\" (UniqueName: \"kubernetes.io/projected/987cad56-29db-4deb-a983-4f2aa7803f7a-kube-api-access-hfngm\") pod \"987cad56-29db-4deb-a983-4f2aa7803f7a\" (UID: \"987cad56-29db-4deb-a983-4f2aa7803f7a\") "
Jul 30 17:33:48.024252 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:48.024078    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/987cad56-29db-4deb-a983-4f2aa7803f7a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "987cad56-29db-4deb-a983-4f2aa7803f7a" (UID: "987cad56-29db-4deb-a983-4f2aa7803f7a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 17:33:48.024252 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:48.024241    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/987cad56-29db-4deb-a983-4f2aa7803f7a-kserve-provision-location\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:33:48.024356 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:48.024312    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/987cad56-29db-4deb-a983-4f2aa7803f7a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "987cad56-29db-4deb-a983-4f2aa7803f7a" (UID: "987cad56-29db-4deb-a983-4f2aa7803f7a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:33:48.024356 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:48.024326    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/987cad56-29db-4deb-a983-4f2aa7803f7a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "987cad56-29db-4deb-a983-4f2aa7803f7a" (UID: "987cad56-29db-4deb-a983-4f2aa7803f7a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:33:48.025992 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:48.025972    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/987cad56-29db-4deb-a983-4f2aa7803f7a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "987cad56-29db-4deb-a983-4f2aa7803f7a" (UID: "987cad56-29db-4deb-a983-4f2aa7803f7a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:33:48.026060 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:48.026012    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/987cad56-29db-4deb-a983-4f2aa7803f7a-kube-api-access-hfngm" (OuterVolumeSpecName: "kube-api-access-hfngm") pod "987cad56-29db-4deb-a983-4f2aa7803f7a" (UID: "987cad56-29db-4deb-a983-4f2aa7803f7a"). InnerVolumeSpecName "kube-api-access-hfngm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:33:48.125512 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:48.125471    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/987cad56-29db-4deb-a983-4f2aa7803f7a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:33:48.125512 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:48.125505    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/987cad56-29db-4deb-a983-4f2aa7803f7a-cabundle-cert\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:33:48.125512 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:48.125516    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfngm\" (UniqueName: \"kubernetes.io/projected/987cad56-29db-4deb-a983-4f2aa7803f7a-kube-api-access-hfngm\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:33:48.125767 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:48.125525    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/987cad56-29db-4deb-a983-4f2aa7803f7a-proxy-tls\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:33:48.148655 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:48.148624    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-61f75d-predictor-fc548b88d-njzqx_987cad56-29db-4deb-a983-4f2aa7803f7a/storage-initializer/1.log"
Jul 30 17:33:48.149056 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:48.149037    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-61f75d-predictor-fc548b88d-njzqx_987cad56-29db-4deb-a983-4f2aa7803f7a/storage-initializer/0.log"
Jul 30 17:33:48.149135 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:48.149079    2580 generic.go:356] "Generic (PLEG): container finished" podID="987cad56-29db-4deb-a983-4f2aa7803f7a" containerID="a9ef3e274c1239b8abfbcc7b6eeb5fb2198b30af7e3b845e01d74c424c9cd2b6" exitCode=1
Jul 30 17:33:48.149216 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:48.149164    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-61f75d-predictor-fc548b88d-njzqx"
Jul 30 17:33:48.149216 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:48.149183    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-61f75d-predictor-fc548b88d-njzqx" event={"ID":"987cad56-29db-4deb-a983-4f2aa7803f7a","Type":"ContainerDied","Data":"a9ef3e274c1239b8abfbcc7b6eeb5fb2198b30af7e3b845e01d74c424c9cd2b6"}
Jul 30 17:33:48.149335 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:48.149258    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-61f75d-predictor-fc548b88d-njzqx" event={"ID":"987cad56-29db-4deb-a983-4f2aa7803f7a","Type":"ContainerDied","Data":"10543f4da1b626bced40015869c62ad004ddcb1d6904e9aa360c27f259fbba0a"}
Jul 30 17:33:48.149335 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:48.149290    2580 scope.go:117] "RemoveContainer" containerID="a9ef3e274c1239b8abfbcc7b6eeb5fb2198b30af7e3b845e01d74c424c9cd2b6"
Jul 30 17:33:48.158637 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:48.158608    2580 scope.go:117] "RemoveContainer" containerID="cf71d58fd2c0a26ba14a04bc3c6777bd9a0f1d0c5dc40156be2c6f842d281573"
Jul 30 17:33:48.166865 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:48.166842    2580 scope.go:117] "RemoveContainer" containerID="a9ef3e274c1239b8abfbcc7b6eeb5fb2198b30af7e3b845e01d74c424c9cd2b6"
Jul 30 17:33:48.167191 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:33:48.167147    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9ef3e274c1239b8abfbcc7b6eeb5fb2198b30af7e3b845e01d74c424c9cd2b6\": container with ID starting with a9ef3e274c1239b8abfbcc7b6eeb5fb2198b30af7e3b845e01d74c424c9cd2b6 not found: ID does not exist" containerID="a9ef3e274c1239b8abfbcc7b6eeb5fb2198b30af7e3b845e01d74c424c9cd2b6"
Jul 30 17:33:48.167248 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:48.167194    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9ef3e274c1239b8abfbcc7b6eeb5fb2198b30af7e3b845e01d74c424c9cd2b6"} err="failed to get container status \"a9ef3e274c1239b8abfbcc7b6eeb5fb2198b30af7e3b845e01d74c424c9cd2b6\": rpc error: code = NotFound desc = could not find container \"a9ef3e274c1239b8abfbcc7b6eeb5fb2198b30af7e3b845e01d74c424c9cd2b6\": container with ID starting with a9ef3e274c1239b8abfbcc7b6eeb5fb2198b30af7e3b845e01d74c424c9cd2b6 not found: ID does not exist"
Jul 30 17:33:48.167248 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:48.167216    2580 scope.go:117] "RemoveContainer" containerID="cf71d58fd2c0a26ba14a04bc3c6777bd9a0f1d0c5dc40156be2c6f842d281573"
Jul 30 17:33:48.167480 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:33:48.167460    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf71d58fd2c0a26ba14a04bc3c6777bd9a0f1d0c5dc40156be2c6f842d281573\": container with ID starting with cf71d58fd2c0a26ba14a04bc3c6777bd9a0f1d0c5dc40156be2c6f842d281573 not found: ID does not exist" containerID="cf71d58fd2c0a26ba14a04bc3c6777bd9a0f1d0c5dc40156be2c6f842d281573"
Jul 30 17:33:48.167535 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:48.167488    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf71d58fd2c0a26ba14a04bc3c6777bd9a0f1d0c5dc40156be2c6f842d281573"} err="failed to get container status \"cf71d58fd2c0a26ba14a04bc3c6777bd9a0f1d0c5dc40156be2c6f842d281573\": rpc error: code = NotFound desc = could not find container \"cf71d58fd2c0a26ba14a04bc3c6777bd9a0f1d0c5dc40156be2c6f842d281573\": container with ID starting with cf71d58fd2c0a26ba14a04bc3c6777bd9a0f1d0c5dc40156be2c6f842d281573 not found: ID does not exist"
Jul 30 17:33:48.197031 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:48.196987    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-61f75d-predictor-fc548b88d-njzqx"]
Jul 30 17:33:48.201236 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:48.201180    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-61f75d-predictor-fc548b88d-njzqx"]
Jul 30 17:33:49.482834 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:33:49.482795    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="987cad56-29db-4deb-a983-4f2aa7803f7a" path="/var/lib/kubelet/pods/987cad56-29db-4deb-a983-4f2aa7803f7a/volumes"
Jul 30 17:35:23.024297 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:35:23.024266    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr"]
Jul 30 17:35:23.024807 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:35:23.024558    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="926f1662-23d6-47c4-81b1-e19078554bfa" containerName="kserve-container"
Jul 30 17:35:23.024807 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:35:23.024572    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="926f1662-23d6-47c4-81b1-e19078554bfa" containerName="kserve-container"
Jul 30 17:35:23.024807 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:35:23.024592    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="926f1662-23d6-47c4-81b1-e19078554bfa" containerName="storage-initializer"
Jul 30 17:35:23.024807 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:35:23.024600    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="926f1662-23d6-47c4-81b1-e19078554bfa" containerName="storage-initializer"
Jul 30 17:35:23.024807 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:35:23.024606    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="987cad56-29db-4deb-a983-4f2aa7803f7a" containerName="storage-initializer"
Jul 30 17:35:23.024807 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:35:23.024612    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="987cad56-29db-4deb-a983-4f2aa7803f7a" containerName="storage-initializer"
Jul 30 17:35:23.024807 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:35:23.024624    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="926f1662-23d6-47c4-81b1-e19078554bfa" containerName="kube-rbac-proxy"
Jul 30 17:35:23.024807 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:35:23.024629    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="926f1662-23d6-47c4-81b1-e19078554bfa" containerName="kube-rbac-proxy"
Jul 30 17:35:23.024807 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:35:23.024696    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="926f1662-23d6-47c4-81b1-e19078554bfa" containerName="kserve-container"
Jul 30 17:35:23.024807 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:35:23.024707    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="987cad56-29db-4deb-a983-4f2aa7803f7a" containerName="storage-initializer"
Jul 30 17:35:23.024807 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:35:23.024713    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="987cad56-29db-4deb-a983-4f2aa7803f7a" containerName="storage-initializer"
Jul 30 17:35:23.024807 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:35:23.024722    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="926f1662-23d6-47c4-81b1-e19078554bfa" containerName="kube-rbac-proxy"
Jul 30 17:35:23.024807 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:35:23.024783    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="987cad56-29db-4deb-a983-4f2aa7803f7a" containerName="storage-initializer"
Jul 30 17:35:23.024807 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:35:23.024790    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="987cad56-29db-4deb-a983-4f2aa7803f7a" containerName="storage-initializer"
Jul 30 17:35:23.031976 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:35:23.031952    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr"
Jul 30 17:35:23.034739 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:35:23.034708    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 30 17:35:23.034739 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:35:23.034718    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-4cfc1-kube-rbac-proxy-sar-config\""
Jul 30 17:35:23.034923 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:35:23.034738    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-qz9j4\""
Jul 30 17:35:23.035602 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:35:23.035584    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 30 17:35:23.035674 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:35:23.035584    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 30 17:35:23.035674 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:35:23.035658    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 30 17:35:23.035674 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:35:23.035670    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-4cfc1-predictor-serving-cert\""
Jul 30 17:35:23.042573 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:35:23.042551    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr"]
Jul 30 17:35:23.056489 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:35:23.056464    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42da4e5f-e5a5-4875-ac8e-349b27e158c4-kserve-provision-location\") pod \"raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr\" (UID: \"42da4e5f-e5a5-4875-ac8e-349b27e158c4\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr"
Jul 30 17:35:23.056568 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:35:23.056498    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42da4e5f-e5a5-4875-ac8e-349b27e158c4-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr\" (UID: \"42da4e5f-e5a5-4875-ac8e-349b27e158c4\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr"
Jul 30 17:35:23.056568 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:35:23.056523    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7xsd\" (UniqueName: \"kubernetes.io/projected/42da4e5f-e5a5-4875-ac8e-349b27e158c4-kube-api-access-c7xsd\") pod \"raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr\" (UID: \"42da4e5f-e5a5-4875-ac8e-349b27e158c4\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr"
Jul 30 17:35:23.056659 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:35:23.056582    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/42da4e5f-e5a5-4875-ac8e-349b27e158c4-cabundle-cert\") pod \"raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr\" (UID: \"42da4e5f-e5a5-4875-ac8e-349b27e158c4\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr"
Jul 30 17:35:23.056659 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:35:23.056619    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42da4e5f-e5a5-4875-ac8e-349b27e158c4-proxy-tls\") pod \"raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr\" (UID: \"42da4e5f-e5a5-4875-ac8e-349b27e158c4\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr"
Jul 30 17:35:23.157304 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:35:23.157280    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42da4e5f-e5a5-4875-ac8e-349b27e158c4-proxy-tls\") pod \"raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr\" (UID: \"42da4e5f-e5a5-4875-ac8e-349b27e158c4\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr"
Jul 30 17:35:23.157411 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:35:23.157351    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42da4e5f-e5a5-4875-ac8e-349b27e158c4-kserve-provision-location\") pod \"raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr\" (UID: \"42da4e5f-e5a5-4875-ac8e-349b27e158c4\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr"
Jul 30 17:35:23.157411 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:35:23.157385    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42da4e5f-e5a5-4875-ac8e-349b27e158c4-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr\" (UID: \"42da4e5f-e5a5-4875-ac8e-349b27e158c4\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr"
Jul 30 17:35:23.157532 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:35:23.157418    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7xsd\" (UniqueName: \"kubernetes.io/projected/42da4e5f-e5a5-4875-ac8e-349b27e158c4-kube-api-access-c7xsd\") pod \"raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr\" (UID: \"42da4e5f-e5a5-4875-ac8e-349b27e158c4\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr"
Jul 30 17:35:23.157532 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:35:23.157455    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/42da4e5f-e5a5-4875-ac8e-349b27e158c4-cabundle-cert\") pod \"raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr\" (UID: \"42da4e5f-e5a5-4875-ac8e-349b27e158c4\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr"
Jul 30 17:35:23.157789 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:35:23.157769    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42da4e5f-e5a5-4875-ac8e-349b27e158c4-kserve-provision-location\") pod \"raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr\" (UID: \"42da4e5f-e5a5-4875-ac8e-349b27e158c4\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr"
Jul 30 17:35:23.158106 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:35:23.158086    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42da4e5f-e5a5-4875-ac8e-349b27e158c4-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr\" (UID: \"42da4e5f-e5a5-4875-ac8e-349b27e158c4\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr"
Jul 30 17:35:23.158185 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:35:23.158083    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/42da4e5f-e5a5-4875-ac8e-349b27e158c4-cabundle-cert\") pod \"raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr\" (UID: \"42da4e5f-e5a5-4875-ac8e-349b27e158c4\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr"
Jul 30 17:35:23.159692 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:35:23.159675    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42da4e5f-e5a5-4875-ac8e-349b27e158c4-proxy-tls\") pod \"raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr\" (UID: \"42da4e5f-e5a5-4875-ac8e-349b27e158c4\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr"
Jul 30 17:35:23.166476 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:35:23.166458    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7xsd\" (UniqueName: \"kubernetes.io/projected/42da4e5f-e5a5-4875-ac8e-349b27e158c4-kube-api-access-c7xsd\") pod \"raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr\" (UID: \"42da4e5f-e5a5-4875-ac8e-349b27e158c4\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr"
Jul 30 17:35:23.343524 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:35:23.343462    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr"
Jul 30 17:35:23.473313 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:35:23.473288    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr"]
Jul 30 17:35:23.475858 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:35:23.475822    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42da4e5f_e5a5_4875_ac8e_349b27e158c4.slice/crio-281c51de9d6d459ab04c762fb43d0f4e0c5812fd71030b0116a3741862fd0b89 WatchSource:0}: Error finding container 281c51de9d6d459ab04c762fb43d0f4e0c5812fd71030b0116a3741862fd0b89: Status 404 returned error can't find the container with id 281c51de9d6d459ab04c762fb43d0f4e0c5812fd71030b0116a3741862fd0b89
Jul 30 17:35:23.532694 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:35:23.532669    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr" event={"ID":"42da4e5f-e5a5-4875-ac8e-349b27e158c4","Type":"ContainerStarted","Data":"281c51de9d6d459ab04c762fb43d0f4e0c5812fd71030b0116a3741862fd0b89"}
Jul 30 17:35:24.538241 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:35:24.538201    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr" event={"ID":"42da4e5f-e5a5-4875-ac8e-349b27e158c4","Type":"ContainerStarted","Data":"956266b2b58cfe951fe14be374eb059c861774cda1077f33bf1cc13889a16873"}
Jul 30 17:35:27.550118 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:35:27.550086    2580 generic.go:356] "Generic (PLEG): container finished" podID="42da4e5f-e5a5-4875-ac8e-349b27e158c4" containerID="956266b2b58cfe951fe14be374eb059c861774cda1077f33bf1cc13889a16873" exitCode=0
Jul 30 17:35:27.550531 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:35:27.550139    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr" event={"ID":"42da4e5f-e5a5-4875-ac8e-349b27e158c4","Type":"ContainerDied","Data":"956266b2b58cfe951fe14be374eb059c861774cda1077f33bf1cc13889a16873"}
Jul 30 17:35:28.556695 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:35:28.556656    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr" event={"ID":"42da4e5f-e5a5-4875-ac8e-349b27e158c4","Type":"ContainerStarted","Data":"c116630cd9dffedff802b61625bb556ef7dd28270158e88e6a6f3119c657bb97"}
Jul 30 17:35:28.557085 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:35:28.556743    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr" event={"ID":"42da4e5f-e5a5-4875-ac8e-349b27e158c4","Type":"ContainerStarted","Data":"9b510521dea6494f233e707fdefcd59fdd4c4c5ab997e3eaa9a7c164dd38f914"}
Jul 30 17:35:28.557085 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:35:28.557021    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr"
Jul 30 17:35:28.557218 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:35:28.557198    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr"
Jul 30 17:35:28.558458 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:35:28.558435    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr" podUID="42da4e5f-e5a5-4875-ac8e-349b27e158c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 30 17:35:28.577813 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:35:28.577765    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr" podStartSLOduration=1.685133263 podStartE2EDuration="5.577752525s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.892619262s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:35:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:35:28.576132757 +0000 UTC m=+2091.651432648" watchObservedRunningTime="2026-07-30 17:35:28.577752525 +0000 UTC m=+2091.653052416"
Jul 30 17:35:29.560925 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:35:29.560884    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr" podUID="42da4e5f-e5a5-4875-ac8e-349b27e158c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 30 17:35:34.565271 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:35:34.565236    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr"
Jul 30 17:35:34.565724 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:35:34.565694    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr" podUID="42da4e5f-e5a5-4875-ac8e-349b27e158c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 30 17:35:37.454480 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:35:37.454450    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-wqszn_f231a916-1fda-4059-9130-d4112e738ee1/console-operator/1.log"
Jul 30 17:35:37.459301 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:35:37.459279    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-wqszn_f231a916-1fda-4059-9130-d4112e738ee1/console-operator/1.log"
Jul 30 17:35:44.565656 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:35:44.565612    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr" podUID="42da4e5f-e5a5-4875-ac8e-349b27e158c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 30 17:35:54.565720 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:35:54.565681    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr" podUID="42da4e5f-e5a5-4875-ac8e-349b27e158c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 30 17:36:04.565641 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:04.565600    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr" podUID="42da4e5f-e5a5-4875-ac8e-349b27e158c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 30 17:36:14.565917 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:14.565873    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr" podUID="42da4e5f-e5a5-4875-ac8e-349b27e158c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 30 17:36:24.566231 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:24.566161    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr" podUID="42da4e5f-e5a5-4875-ac8e-349b27e158c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 30 17:36:34.566040 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:34.565999    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr" podUID="42da4e5f-e5a5-4875-ac8e-349b27e158c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 30 17:36:44.566741 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:44.566709    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr"
Jul 30 17:36:53.092738 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:53.092698    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr"]
Jul 30 17:36:53.093256 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:53.092981    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr" podUID="42da4e5f-e5a5-4875-ac8e-349b27e158c4" containerName="kserve-container" containerID="cri-o://9b510521dea6494f233e707fdefcd59fdd4c4c5ab997e3eaa9a7c164dd38f914" gracePeriod=30
Jul 30 17:36:53.093256 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:53.092985    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr" podUID="42da4e5f-e5a5-4875-ac8e-349b27e158c4" containerName="kube-rbac-proxy" containerID="cri-o://c116630cd9dffedff802b61625bb556ef7dd28270158e88e6a6f3119c657bb97" gracePeriod=30
Jul 30 17:36:53.887492 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:53.887462    2580 generic.go:356] "Generic (PLEG): container finished" podID="42da4e5f-e5a5-4875-ac8e-349b27e158c4" containerID="c116630cd9dffedff802b61625bb556ef7dd28270158e88e6a6f3119c657bb97" exitCode=2
Jul 30 17:36:53.887633 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:53.887533    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr" event={"ID":"42da4e5f-e5a5-4875-ac8e-349b27e158c4","Type":"ContainerDied","Data":"c116630cd9dffedff802b61625bb556ef7dd28270158e88e6a6f3119c657bb97"}
Jul 30 17:36:54.561362 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:54.561316    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr" podUID="42da4e5f-e5a5-4875-ac8e-349b27e158c4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Jul 30 17:36:54.566687 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:54.566656    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr" podUID="42da4e5f-e5a5-4875-ac8e-349b27e158c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 30 17:36:57.030985 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:57.030964    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr"
Jul 30 17:36:57.122415 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:57.122362    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42da4e5f-e5a5-4875-ac8e-349b27e158c4-proxy-tls\") pod \"42da4e5f-e5a5-4875-ac8e-349b27e158c4\" (UID: \"42da4e5f-e5a5-4875-ac8e-349b27e158c4\") "
Jul 30 17:36:57.122415 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:57.122407    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42da4e5f-e5a5-4875-ac8e-349b27e158c4-kube-rbac-proxy-sar-config\") pod \"42da4e5f-e5a5-4875-ac8e-349b27e158c4\" (UID: \"42da4e5f-e5a5-4875-ac8e-349b27e158c4\") "
Jul 30 17:36:57.122554 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:57.122428    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c7xsd\" (UniqueName: \"kubernetes.io/projected/42da4e5f-e5a5-4875-ac8e-349b27e158c4-kube-api-access-c7xsd\") pod \"42da4e5f-e5a5-4875-ac8e-349b27e158c4\" (UID: \"42da4e5f-e5a5-4875-ac8e-349b27e158c4\") "
Jul 30 17:36:57.122554 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:57.122461    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42da4e5f-e5a5-4875-ac8e-349b27e158c4-kserve-provision-location\") pod \"42da4e5f-e5a5-4875-ac8e-349b27e158c4\" (UID: \"42da4e5f-e5a5-4875-ac8e-349b27e158c4\") "
Jul 30 17:36:57.122554 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:57.122478    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/42da4e5f-e5a5-4875-ac8e-349b27e158c4-cabundle-cert\") pod \"42da4e5f-e5a5-4875-ac8e-349b27e158c4\" (UID: \"42da4e5f-e5a5-4875-ac8e-349b27e158c4\") "
Jul 30 17:36:57.122809 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:57.122781    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42da4e5f-e5a5-4875-ac8e-349b27e158c4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "42da4e5f-e5a5-4875-ac8e-349b27e158c4" (UID: "42da4e5f-e5a5-4875-ac8e-349b27e158c4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:36:57.122886 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:57.122810    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42da4e5f-e5a5-4875-ac8e-349b27e158c4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "42da4e5f-e5a5-4875-ac8e-349b27e158c4" (UID: "42da4e5f-e5a5-4875-ac8e-349b27e158c4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 17:36:57.122942 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:57.122902    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42da4e5f-e5a5-4875-ac8e-349b27e158c4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "42da4e5f-e5a5-4875-ac8e-349b27e158c4" (UID: "42da4e5f-e5a5-4875-ac8e-349b27e158c4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:36:57.124344 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:57.124324    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42da4e5f-e5a5-4875-ac8e-349b27e158c4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "42da4e5f-e5a5-4875-ac8e-349b27e158c4" (UID: "42da4e5f-e5a5-4875-ac8e-349b27e158c4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:36:57.124467 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:57.124447    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42da4e5f-e5a5-4875-ac8e-349b27e158c4-kube-api-access-c7xsd" (OuterVolumeSpecName: "kube-api-access-c7xsd") pod "42da4e5f-e5a5-4875-ac8e-349b27e158c4" (UID: "42da4e5f-e5a5-4875-ac8e-349b27e158c4"). InnerVolumeSpecName "kube-api-access-c7xsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:36:57.223207 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:57.223163    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42da4e5f-e5a5-4875-ac8e-349b27e158c4-kserve-provision-location\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:36:57.223207 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:57.223205    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/42da4e5f-e5a5-4875-ac8e-349b27e158c4-cabundle-cert\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:36:57.223335 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:57.223214    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42da4e5f-e5a5-4875-ac8e-349b27e158c4-proxy-tls\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:36:57.223335 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:57.223225    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42da4e5f-e5a5-4875-ac8e-349b27e158c4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:36:57.223335 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:57.223233    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7xsd\" (UniqueName: \"kubernetes.io/projected/42da4e5f-e5a5-4875-ac8e-349b27e158c4-kube-api-access-c7xsd\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:36:57.905493 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:57.905461    2580 generic.go:356] "Generic (PLEG): container finished" podID="42da4e5f-e5a5-4875-ac8e-349b27e158c4" containerID="9b510521dea6494f233e707fdefcd59fdd4c4c5ab997e3eaa9a7c164dd38f914" exitCode=0
Jul 30 17:36:57.905653 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:57.905503    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr" event={"ID":"42da4e5f-e5a5-4875-ac8e-349b27e158c4","Type":"ContainerDied","Data":"9b510521dea6494f233e707fdefcd59fdd4c4c5ab997e3eaa9a7c164dd38f914"}
Jul 30 17:36:57.905653 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:57.905535    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr"
Jul 30 17:36:57.905653 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:57.905574    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr" event={"ID":"42da4e5f-e5a5-4875-ac8e-349b27e158c4","Type":"ContainerDied","Data":"281c51de9d6d459ab04c762fb43d0f4e0c5812fd71030b0116a3741862fd0b89"}
Jul 30 17:36:57.905653 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:57.905604    2580 scope.go:117] "RemoveContainer" containerID="c116630cd9dffedff802b61625bb556ef7dd28270158e88e6a6f3119c657bb97"
Jul 30 17:36:57.914298 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:57.914091    2580 scope.go:117] "RemoveContainer" containerID="9b510521dea6494f233e707fdefcd59fdd4c4c5ab997e3eaa9a7c164dd38f914"
Jul 30 17:36:57.921552 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:57.921528    2580 scope.go:117] "RemoveContainer" containerID="956266b2b58cfe951fe14be374eb059c861774cda1077f33bf1cc13889a16873"
Jul 30 17:36:57.925586 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:57.925557    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr"]
Jul 30 17:36:57.929889 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:57.929859    2580 scope.go:117] "RemoveContainer" containerID="c116630cd9dffedff802b61625bb556ef7dd28270158e88e6a6f3119c657bb97"
Jul 30 17:36:57.930214 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:36:57.930191    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c116630cd9dffedff802b61625bb556ef7dd28270158e88e6a6f3119c657bb97\": container with ID starting with c116630cd9dffedff802b61625bb556ef7dd28270158e88e6a6f3119c657bb97 not found: ID does not exist" containerID="c116630cd9dffedff802b61625bb556ef7dd28270158e88e6a6f3119c657bb97"
Jul 30 17:36:57.930286 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:57.930222    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c116630cd9dffedff802b61625bb556ef7dd28270158e88e6a6f3119c657bb97"} err="failed to get container status \"c116630cd9dffedff802b61625bb556ef7dd28270158e88e6a6f3119c657bb97\": rpc error: code = NotFound desc = could not find container \"c116630cd9dffedff802b61625bb556ef7dd28270158e88e6a6f3119c657bb97\": container with ID starting with c116630cd9dffedff802b61625bb556ef7dd28270158e88e6a6f3119c657bb97 not found: ID does not exist"
Jul 30 17:36:57.930286 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:57.930241    2580 scope.go:117] "RemoveContainer" containerID="9b510521dea6494f233e707fdefcd59fdd4c4c5ab997e3eaa9a7c164dd38f914"
Jul 30 17:36:57.930539 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:36:57.930520    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b510521dea6494f233e707fdefcd59fdd4c4c5ab997e3eaa9a7c164dd38f914\": container with ID starting with 9b510521dea6494f233e707fdefcd59fdd4c4c5ab997e3eaa9a7c164dd38f914 not found: ID does not exist" containerID="9b510521dea6494f233e707fdefcd59fdd4c4c5ab997e3eaa9a7c164dd38f914"
Jul 30 17:36:57.930590 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:57.930546    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b510521dea6494f233e707fdefcd59fdd4c4c5ab997e3eaa9a7c164dd38f914"} err="failed to get container status \"9b510521dea6494f233e707fdefcd59fdd4c4c5ab997e3eaa9a7c164dd38f914\": rpc error: code = NotFound desc = could not find container \"9b510521dea6494f233e707fdefcd59fdd4c4c5ab997e3eaa9a7c164dd38f914\": container with ID starting with 9b510521dea6494f233e707fdefcd59fdd4c4c5ab997e3eaa9a7c164dd38f914 not found: ID does not exist"
Jul 30 17:36:57.930590 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:57.930565    2580 scope.go:117] "RemoveContainer" containerID="956266b2b58cfe951fe14be374eb059c861774cda1077f33bf1cc13889a16873"
Jul 30 17:36:57.930789 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:57.930767    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-4cfc1-predictor-85c4786b7d-8rfdr"]
Jul 30 17:36:57.930825 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:36:57.930808    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"956266b2b58cfe951fe14be374eb059c861774cda1077f33bf1cc13889a16873\": container with ID starting with 956266b2b58cfe951fe14be374eb059c861774cda1077f33bf1cc13889a16873 not found: ID does not exist" containerID="956266b2b58cfe951fe14be374eb059c861774cda1077f33bf1cc13889a16873"
Jul 30 17:36:57.930863 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:57.930834    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"956266b2b58cfe951fe14be374eb059c861774cda1077f33bf1cc13889a16873"} err="failed to get container status \"956266b2b58cfe951fe14be374eb059c861774cda1077f33bf1cc13889a16873\": rpc error: code = NotFound desc = could not find container \"956266b2b58cfe951fe14be374eb059c861774cda1077f33bf1cc13889a16873\": container with ID starting with 956266b2b58cfe951fe14be374eb059c861774cda1077f33bf1cc13889a16873 not found: ID does not exist"
Jul 30 17:36:59.009740 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:59.009711    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-gsxmp/must-gather-d2ppr"]
Jul 30 17:36:59.010307 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:59.010143    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42da4e5f-e5a5-4875-ac8e-349b27e158c4" containerName="storage-initializer"
Jul 30 17:36:59.010307 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:59.010164    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="42da4e5f-e5a5-4875-ac8e-349b27e158c4" containerName="storage-initializer"
Jul 30 17:36:59.010307 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:59.010198    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42da4e5f-e5a5-4875-ac8e-349b27e158c4" containerName="kserve-container"
Jul 30 17:36:59.010307 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:59.010207    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="42da4e5f-e5a5-4875-ac8e-349b27e158c4" containerName="kserve-container"
Jul 30 17:36:59.010307 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:59.010221    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42da4e5f-e5a5-4875-ac8e-349b27e158c4" containerName="kube-rbac-proxy"
Jul 30 17:36:59.010307 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:59.010230    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="42da4e5f-e5a5-4875-ac8e-349b27e158c4" containerName="kube-rbac-proxy"
Jul 30 17:36:59.010620 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:59.010333    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="42da4e5f-e5a5-4875-ac8e-349b27e158c4" containerName="kserve-container"
Jul 30 17:36:59.010620 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:59.010346    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="42da4e5f-e5a5-4875-ac8e-349b27e158c4" containerName="kube-rbac-proxy"
Jul 30 17:36:59.014422 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:59.014401    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gsxmp/must-gather-d2ppr"
Jul 30 17:36:59.017767 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:59.017748    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gsxmp\"/\"openshift-service-ca.crt\""
Jul 30 17:36:59.019017 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:59.019003    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gsxmp\"/\"kube-root-ca.crt\""
Jul 30 17:36:59.033531 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:59.033511    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcdw7\" (UniqueName: \"kubernetes.io/projected/580a2c98-621d-4e4a-bedb-4925e441acfb-kube-api-access-zcdw7\") pod \"must-gather-d2ppr\" (UID: \"580a2c98-621d-4e4a-bedb-4925e441acfb\") " pod="openshift-must-gather-gsxmp/must-gather-d2ppr"
Jul 30 17:36:59.033641 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:59.033569    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/580a2c98-621d-4e4a-bedb-4925e441acfb-must-gather-output\") pod \"must-gather-d2ppr\" (UID: \"580a2c98-621d-4e4a-bedb-4925e441acfb\") " pod="openshift-must-gather-gsxmp/must-gather-d2ppr"
Jul 30 17:36:59.077934 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:59.077907    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gsxmp/must-gather-d2ppr"]
Jul 30 17:36:59.134476 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:59.134445    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/580a2c98-621d-4e4a-bedb-4925e441acfb-must-gather-output\") pod \"must-gather-d2ppr\" (UID: \"580a2c98-621d-4e4a-bedb-4925e441acfb\") " pod="openshift-must-gather-gsxmp/must-gather-d2ppr"
Jul 30 17:36:59.134599 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:59.134485    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcdw7\" (UniqueName: \"kubernetes.io/projected/580a2c98-621d-4e4a-bedb-4925e441acfb-kube-api-access-zcdw7\") pod \"must-gather-d2ppr\" (UID: \"580a2c98-621d-4e4a-bedb-4925e441acfb\") " pod="openshift-must-gather-gsxmp/must-gather-d2ppr"
Jul 30 17:36:59.134773 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:59.134755    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/580a2c98-621d-4e4a-bedb-4925e441acfb-must-gather-output\") pod \"must-gather-d2ppr\" (UID: \"580a2c98-621d-4e4a-bedb-4925e441acfb\") " pod="openshift-must-gather-gsxmp/must-gather-d2ppr"
Jul 30 17:36:59.147265 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:59.147243    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcdw7\" (UniqueName: \"kubernetes.io/projected/580a2c98-621d-4e4a-bedb-4925e441acfb-kube-api-access-zcdw7\") pod \"must-gather-d2ppr\" (UID: \"580a2c98-621d-4e4a-bedb-4925e441acfb\") " pod="openshift-must-gather-gsxmp/must-gather-d2ppr"
Jul 30 17:36:59.323255 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:59.323196    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gsxmp/must-gather-d2ppr"
Jul 30 17:36:59.451133 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:59.451104    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gsxmp/must-gather-d2ppr"]
Jul 30 17:36:59.454798 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:36:59.454774    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod580a2c98_621d_4e4a_bedb_4925e441acfb.slice/crio-70337be76d5c9a6788c43201d538fe37bf2d4244535521a778b6b862325d981d WatchSource:0}: Error finding container 70337be76d5c9a6788c43201d538fe37bf2d4244535521a778b6b862325d981d: Status 404 returned error can't find the container with id 70337be76d5c9a6788c43201d538fe37bf2d4244535521a778b6b862325d981d
Jul 30 17:36:59.456432 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:59.456416    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 17:36:59.484927 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:59.484900    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42da4e5f-e5a5-4875-ac8e-349b27e158c4" path="/var/lib/kubelet/pods/42da4e5f-e5a5-4875-ac8e-349b27e158c4/volumes"
Jul 30 17:36:59.916567 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:36:59.916534    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gsxmp/must-gather-d2ppr" event={"ID":"580a2c98-621d-4e4a-bedb-4925e441acfb","Type":"ContainerStarted","Data":"70337be76d5c9a6788c43201d538fe37bf2d4244535521a778b6b862325d981d"}
Jul 30 17:37:05.947326 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:05.947288    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gsxmp/must-gather-d2ppr" event={"ID":"580a2c98-621d-4e4a-bedb-4925e441acfb","Type":"ContainerStarted","Data":"15673f84e0ffc36c05d426abf38a7ba76732effb6bb98294af3f3da2e6b1a944"}
Jul 30 17:37:05.947326 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:05.947333    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gsxmp/must-gather-d2ppr" event={"ID":"580a2c98-621d-4e4a-bedb-4925e441acfb","Type":"ContainerStarted","Data":"0395a077bdd2c206f91a539fa6624a609c9a466f88e24701c19750867518ff8c"}
Jul 30 17:37:05.964125 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:05.964065    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-gsxmp/must-gather-d2ppr" podStartSLOduration=2.139711886 podStartE2EDuration="7.964050478s" totalImagesPullingTime="5.824338592s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:36:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:37:05.962201766 +0000 UTC m=+2189.037501655" watchObservedRunningTime="2026-07-30 17:37:05.964050478 +0000 UTC m=+2189.039350366"
Jul 30 17:37:28.045182 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:28.045140    2580 generic.go:356] "Generic (PLEG): container finished" podID="580a2c98-621d-4e4a-bedb-4925e441acfb" containerID="0395a077bdd2c206f91a539fa6624a609c9a466f88e24701c19750867518ff8c" exitCode=0
Jul 30 17:37:28.045557 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:28.045212    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gsxmp/must-gather-d2ppr" event={"ID":"580a2c98-621d-4e4a-bedb-4925e441acfb","Type":"ContainerDied","Data":"0395a077bdd2c206f91a539fa6624a609c9a466f88e24701c19750867518ff8c"}
Jul 30 17:37:28.045557 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:28.045486    2580 scope.go:117] "RemoveContainer" containerID="0395a077bdd2c206f91a539fa6624a609c9a466f88e24701c19750867518ff8c"
Jul 30 17:37:28.085160 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:28.085127    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-gsxmp_must-gather-d2ppr_580a2c98-621d-4e4a-bedb-4925e441acfb/gather/0.log"
Jul 30 17:37:31.358278 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:31.358238    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-k7lnc_ad2d9b30-7d70-4236-9398-93ff51843c30/global-pull-secret-syncer/0.log"
Jul 30 17:37:31.452110 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:31.452083    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-d8nwh_440bb960-b449-499c-b4b8-dabc4cc9d3cf/konnectivity-agent/0.log"
Jul 30 17:37:31.573535 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:31.573508    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-68.ec2.internal_74460565773fc50cdf3aee810c639568/haproxy/0.log"
Jul 30 17:37:33.489102 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:33.489063    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-gsxmp/must-gather-d2ppr"]
Jul 30 17:37:33.489560 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:33.489259    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-gsxmp/must-gather-d2ppr" podUID="580a2c98-621d-4e4a-bedb-4925e441acfb" containerName="copy" containerID="cri-o://15673f84e0ffc36c05d426abf38a7ba76732effb6bb98294af3f3da2e6b1a944" gracePeriod=2
Jul 30 17:37:33.495009 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:33.494964    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-gsxmp/must-gather-d2ppr"]
Jul 30 17:37:33.715278 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:33.715257    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-gsxmp_must-gather-d2ppr_580a2c98-621d-4e4a-bedb-4925e441acfb/copy/0.log"
Jul 30 17:37:33.716907 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:33.716891    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gsxmp/must-gather-d2ppr"
Jul 30 17:37:33.719447 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:37:33.719422    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-d2ppr\" is forbidden: User \"system:node:ip-10-0-136-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-gsxmp\": no relationship found between node 'ip-10-0-136-68.ec2.internal' and this object" podUID="580a2c98-621d-4e4a-bedb-4925e441acfb" pod="openshift-must-gather-gsxmp/must-gather-d2ppr"
Jul 30 17:37:33.809808 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:33.809789    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zcdw7\" (UniqueName: \"kubernetes.io/projected/580a2c98-621d-4e4a-bedb-4925e441acfb-kube-api-access-zcdw7\") pod \"580a2c98-621d-4e4a-bedb-4925e441acfb\" (UID: \"580a2c98-621d-4e4a-bedb-4925e441acfb\") "
Jul 30 17:37:33.809905 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:33.809863    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/580a2c98-621d-4e4a-bedb-4925e441acfb-must-gather-output\") pod \"580a2c98-621d-4e4a-bedb-4925e441acfb\" (UID: \"580a2c98-621d-4e4a-bedb-4925e441acfb\") "
Jul 30 17:37:33.811224 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:33.811198    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/580a2c98-621d-4e4a-bedb-4925e441acfb-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "580a2c98-621d-4e4a-bedb-4925e441acfb" (UID: "580a2c98-621d-4e4a-bedb-4925e441acfb"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 17:37:33.811880 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:33.811853    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/580a2c98-621d-4e4a-bedb-4925e441acfb-kube-api-access-zcdw7" (OuterVolumeSpecName: "kube-api-access-zcdw7") pod "580a2c98-621d-4e4a-bedb-4925e441acfb" (UID: "580a2c98-621d-4e4a-bedb-4925e441acfb"). InnerVolumeSpecName "kube-api-access-zcdw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:37:33.910794 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:33.910770    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcdw7\" (UniqueName: \"kubernetes.io/projected/580a2c98-621d-4e4a-bedb-4925e441acfb-kube-api-access-zcdw7\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:37:33.910794 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:33.910790    2580 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/580a2c98-621d-4e4a-bedb-4925e441acfb-must-gather-output\") on node \"ip-10-0-136-68.ec2.internal\" DevicePath \"\""
Jul 30 17:37:34.070131 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:34.070072    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-gsxmp_must-gather-d2ppr_580a2c98-621d-4e4a-bedb-4925e441acfb/copy/0.log"
Jul 30 17:37:34.070527 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:34.070499    2580 generic.go:356] "Generic (PLEG): container finished" podID="580a2c98-621d-4e4a-bedb-4925e441acfb" containerID="15673f84e0ffc36c05d426abf38a7ba76732effb6bb98294af3f3da2e6b1a944" exitCode=143
Jul 30 17:37:34.070640 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:34.070551    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gsxmp/must-gather-d2ppr"
Jul 30 17:37:34.070640 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:34.070621    2580 scope.go:117] "RemoveContainer" containerID="15673f84e0ffc36c05d426abf38a7ba76732effb6bb98294af3f3da2e6b1a944"
Jul 30 17:37:34.073366 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:37:34.073332    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-d2ppr\" is forbidden: User \"system:node:ip-10-0-136-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-gsxmp\": no relationship found between node 'ip-10-0-136-68.ec2.internal' and this object" podUID="580a2c98-621d-4e4a-bedb-4925e441acfb" pod="openshift-must-gather-gsxmp/must-gather-d2ppr"
Jul 30 17:37:34.080277 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:34.080258    2580 scope.go:117] "RemoveContainer" containerID="0395a077bdd2c206f91a539fa6624a609c9a466f88e24701c19750867518ff8c"
Jul 30 17:37:34.082367 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:37:34.082338    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-d2ppr\" is forbidden: User \"system:node:ip-10-0-136-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-gsxmp\": no relationship found between node 'ip-10-0-136-68.ec2.internal' and this object" podUID="580a2c98-621d-4e4a-bedb-4925e441acfb" pod="openshift-must-gather-gsxmp/must-gather-d2ppr"
Jul 30 17:37:34.092281 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:34.092258    2580 scope.go:117] "RemoveContainer" containerID="15673f84e0ffc36c05d426abf38a7ba76732effb6bb98294af3f3da2e6b1a944"
Jul 30 17:37:34.092562 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:37:34.092542    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15673f84e0ffc36c05d426abf38a7ba76732effb6bb98294af3f3da2e6b1a944\": container with ID starting with 15673f84e0ffc36c05d426abf38a7ba76732effb6bb98294af3f3da2e6b1a944 not found: ID does not exist" containerID="15673f84e0ffc36c05d426abf38a7ba76732effb6bb98294af3f3da2e6b1a944"
Jul 30 17:37:34.092612 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:34.092571    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15673f84e0ffc36c05d426abf38a7ba76732effb6bb98294af3f3da2e6b1a944"} err="failed to get container status \"15673f84e0ffc36c05d426abf38a7ba76732effb6bb98294af3f3da2e6b1a944\": rpc error: code = NotFound desc = could not find container \"15673f84e0ffc36c05d426abf38a7ba76732effb6bb98294af3f3da2e6b1a944\": container with ID starting with 15673f84e0ffc36c05d426abf38a7ba76732effb6bb98294af3f3da2e6b1a944 not found: ID does not exist"
Jul 30 17:37:34.092612 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:34.092590    2580 scope.go:117] "RemoveContainer" containerID="0395a077bdd2c206f91a539fa6624a609c9a466f88e24701c19750867518ff8c"
Jul 30 17:37:34.092861 ip-10-0-136-68 kubenswrapper[2580]: E0730 17:37:34.092840    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0395a077bdd2c206f91a539fa6624a609c9a466f88e24701c19750867518ff8c\": container with ID starting with 0395a077bdd2c206f91a539fa6624a609c9a466f88e24701c19750867518ff8c not found: ID does not exist" containerID="0395a077bdd2c206f91a539fa6624a609c9a466f88e24701c19750867518ff8c"
Jul 30 17:37:34.092902 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:34.092866    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0395a077bdd2c206f91a539fa6624a609c9a466f88e24701c19750867518ff8c"} err="failed to get container status \"0395a077bdd2c206f91a539fa6624a609c9a466f88e24701c19750867518ff8c\": rpc error: code = NotFound desc = could not find container \"0395a077bdd2c206f91a539fa6624a609c9a466f88e24701c19750867518ff8c\": container with ID starting with 0395a077bdd2c206f91a539fa6624a609c9a466f88e24701c19750867518ff8c not found: ID does not exist"
Jul 30 17:37:35.481929 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:35.481893    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="580a2c98-621d-4e4a-bedb-4925e441acfb" path="/var/lib/kubelet/pods/580a2c98-621d-4e4a-bedb-4925e441acfb/volumes"
Jul 30 17:37:35.866543 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:35.866510    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-6xrrb_badf66a2-ec5d-4590-b910-2f6cddef1caf/cluster-monitoring-operator/0.log"
Jul 30 17:37:35.923031 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:35.923005    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-99w2g_f775eb31-353d-410e-add2-e2aa791f31c8/kube-state-metrics/0.log"
Jul 30 17:37:35.959917 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:35.959893    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-99w2g_f775eb31-353d-410e-add2-e2aa791f31c8/kube-rbac-proxy-main/0.log"
Jul 30 17:37:35.999368 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:35.999338    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-99w2g_f775eb31-353d-410e-add2-e2aa791f31c8/kube-rbac-proxy-self/0.log"
Jul 30 17:37:36.052429 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:36.052391    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-545cd44b95-ndcrt_5f6b791b-3b27-4b3a-8cc0-f2cf0f3163c5/metrics-server/0.log"
Jul 30 17:37:36.279714 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:36.279682    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l7cvg_9b737c7c-d8a8-4567-86cd-bbbbe37b1a36/node-exporter/0.log"
Jul 30 17:37:36.316459 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:36.316420    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l7cvg_9b737c7c-d8a8-4567-86cd-bbbbe37b1a36/kube-rbac-proxy/0.log"
Jul 30 17:37:36.354194 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:36.354161    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l7cvg_9b737c7c-d8a8-4567-86cd-bbbbe37b1a36/init-textfile/0.log"
Jul 30 17:37:37.002338 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:37.002308    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-69625_5e497af4-651c-4078-a0b5-bf1b7697e869/prometheus-operator/0.log"
Jul 30 17:37:37.039704 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:37.039683    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-69625_5e497af4-651c-4078-a0b5-bf1b7697e869/kube-rbac-proxy/0.log"
Jul 30 17:37:37.086354 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:37.086323    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-qgll2_abad2709-49a5-41f9-bc4d-25ff3db9e554/prometheus-operator-admission-webhook/0.log"
Jul 30 17:37:37.388307 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:37.388242    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5989fc4c8b-gdxvq_8e8869c6-d815-4f74-b761-b5309510e276/thanos-query/0.log"
Jul 30 17:37:37.428574 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:37.428550    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5989fc4c8b-gdxvq_8e8869c6-d815-4f74-b761-b5309510e276/kube-rbac-proxy-web/0.log"
Jul 30 17:37:37.473403 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:37.473385    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5989fc4c8b-gdxvq_8e8869c6-d815-4f74-b761-b5309510e276/kube-rbac-proxy/0.log"
Jul 30 17:37:37.516307 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:37.516286    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5989fc4c8b-gdxvq_8e8869c6-d815-4f74-b761-b5309510e276/prom-label-proxy/0.log"
Jul 30 17:37:37.563942 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:37.563918    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5989fc4c8b-gdxvq_8e8869c6-d815-4f74-b761-b5309510e276/kube-rbac-proxy-rules/0.log"
Jul 30 17:37:37.604346 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:37.604330    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5989fc4c8b-gdxvq_8e8869c6-d815-4f74-b761-b5309510e276/kube-rbac-proxy-metrics/0.log"
Jul 30 17:37:39.234960 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:39.234924    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-wqszn_f231a916-1fda-4059-9130-d4112e738ee1/console-operator/1.log"
Jul 30 17:37:39.246405 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:39.246372    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-wqszn_f231a916-1fda-4059-9130-d4112e738ee1/console-operator/2.log"
Jul 30 17:37:39.321749 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:39.321721    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5wwbm/perf-node-gather-daemonset-tgq6z"]
Jul 30 17:37:39.322040 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:39.322027    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="580a2c98-621d-4e4a-bedb-4925e441acfb" containerName="gather"
Jul 30 17:37:39.322092 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:39.322043    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="580a2c98-621d-4e4a-bedb-4925e441acfb" containerName="gather"
Jul 30 17:37:39.322092 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:39.322050    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="580a2c98-621d-4e4a-bedb-4925e441acfb" containerName="copy"
Jul 30 17:37:39.322092 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:39.322055    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="580a2c98-621d-4e4a-bedb-4925e441acfb" containerName="copy"
Jul 30 17:37:39.322212 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:39.322111    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="580a2c98-621d-4e4a-bedb-4925e441acfb" containerName="gather"
Jul 30 17:37:39.322212 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:39.322122    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="580a2c98-621d-4e4a-bedb-4925e441acfb" containerName="copy"
Jul 30 17:37:39.326078 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:39.326060    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5wwbm/perf-node-gather-daemonset-tgq6z"
Jul 30 17:37:39.329312 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:39.329292    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5wwbm\"/\"default-dockercfg-tg7fp\""
Jul 30 17:37:39.329852 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:39.329833    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5wwbm\"/\"openshift-service-ca.crt\""
Jul 30 17:37:39.330656 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:39.330639    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5wwbm\"/\"kube-root-ca.crt\""
Jul 30 17:37:39.339049 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:39.339029    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5wwbm/perf-node-gather-daemonset-tgq6z"]
Jul 30 17:37:39.451879 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:39.451860    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b008d452-1faf-4604-a3c3-cb06a6b05358-proc\") pod \"perf-node-gather-daemonset-tgq6z\" (UID: \"b008d452-1faf-4604-a3c3-cb06a6b05358\") " pod="openshift-must-gather-5wwbm/perf-node-gather-daemonset-tgq6z"
Jul 30 17:37:39.451968 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:39.451888    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g8p6\" (UniqueName: \"kubernetes.io/projected/b008d452-1faf-4604-a3c3-cb06a6b05358-kube-api-access-4g8p6\") pod \"perf-node-gather-daemonset-tgq6z\" (UID: \"b008d452-1faf-4604-a3c3-cb06a6b05358\") " pod="openshift-must-gather-5wwbm/perf-node-gather-daemonset-tgq6z"
Jul 30 17:37:39.451968 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:39.451914    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b008d452-1faf-4604-a3c3-cb06a6b05358-lib-modules\") pod \"perf-node-gather-daemonset-tgq6z\" (UID: \"b008d452-1faf-4604-a3c3-cb06a6b05358\") " pod="openshift-must-gather-5wwbm/perf-node-gather-daemonset-tgq6z"
Jul 30 17:37:39.451968 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:39.451933    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b008d452-1faf-4604-a3c3-cb06a6b05358-podres\") pod \"perf-node-gather-daemonset-tgq6z\" (UID: \"b008d452-1faf-4604-a3c3-cb06a6b05358\") " pod="openshift-must-gather-5wwbm/perf-node-gather-daemonset-tgq6z"
Jul 30 17:37:39.452085 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:39.451982    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b008d452-1faf-4604-a3c3-cb06a6b05358-sys\") pod \"perf-node-gather-daemonset-tgq6z\" (UID: \"b008d452-1faf-4604-a3c3-cb06a6b05358\") " pod="openshift-must-gather-5wwbm/perf-node-gather-daemonset-tgq6z"
Jul 30 17:37:39.552787 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:39.552760    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b008d452-1faf-4604-a3c3-cb06a6b05358-proc\") pod \"perf-node-gather-daemonset-tgq6z\" (UID: \"b008d452-1faf-4604-a3c3-cb06a6b05358\") " pod="openshift-must-gather-5wwbm/perf-node-gather-daemonset-tgq6z"
Jul 30 17:37:39.552885 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:39.552796    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g8p6\" (UniqueName: \"kubernetes.io/projected/b008d452-1faf-4604-a3c3-cb06a6b05358-kube-api-access-4g8p6\") pod \"perf-node-gather-daemonset-tgq6z\" (UID: \"b008d452-1faf-4604-a3c3-cb06a6b05358\") " pod="openshift-must-gather-5wwbm/perf-node-gather-daemonset-tgq6z"
Jul 30 17:37:39.552885 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:39.552829    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b008d452-1faf-4604-a3c3-cb06a6b05358-lib-modules\") pod \"perf-node-gather-daemonset-tgq6z\" (UID: \"b008d452-1faf-4604-a3c3-cb06a6b05358\") " pod="openshift-must-gather-5wwbm/perf-node-gather-daemonset-tgq6z"
Jul 30 17:37:39.552885 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:39.552854    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b008d452-1faf-4604-a3c3-cb06a6b05358-podres\") pod \"perf-node-gather-daemonset-tgq6z\" (UID: \"b008d452-1faf-4604-a3c3-cb06a6b05358\") " pod="openshift-must-gather-5wwbm/perf-node-gather-daemonset-tgq6z"
Jul 30 17:37:39.552885 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:39.552882    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b008d452-1faf-4604-a3c3-cb06a6b05358-sys\") pod \"perf-node-gather-daemonset-tgq6z\" (UID: \"b008d452-1faf-4604-a3c3-cb06a6b05358\") " pod="openshift-must-gather-5wwbm/perf-node-gather-daemonset-tgq6z"
Jul 30 17:37:39.553039 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:39.552886    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b008d452-1faf-4604-a3c3-cb06a6b05358-proc\") pod \"perf-node-gather-daemonset-tgq6z\" (UID: \"b008d452-1faf-4604-a3c3-cb06a6b05358\") " pod="openshift-must-gather-5wwbm/perf-node-gather-daemonset-tgq6z"
Jul 30 17:37:39.553039 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:39.552937    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b008d452-1faf-4604-a3c3-cb06a6b05358-sys\") pod \"perf-node-gather-daemonset-tgq6z\" (UID: \"b008d452-1faf-4604-a3c3-cb06a6b05358\") " pod="openshift-must-gather-5wwbm/perf-node-gather-daemonset-tgq6z"
Jul 30 17:37:39.553039 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:39.552989    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b008d452-1faf-4604-a3c3-cb06a6b05358-lib-modules\") pod \"perf-node-gather-daemonset-tgq6z\" (UID: \"b008d452-1faf-4604-a3c3-cb06a6b05358\") " pod="openshift-must-gather-5wwbm/perf-node-gather-daemonset-tgq6z"
Jul 30 17:37:39.553039 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:39.553011    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b008d452-1faf-4604-a3c3-cb06a6b05358-podres\") pod \"perf-node-gather-daemonset-tgq6z\" (UID: \"b008d452-1faf-4604-a3c3-cb06a6b05358\") " pod="openshift-must-gather-5wwbm/perf-node-gather-daemonset-tgq6z"
Jul 30 17:37:39.561965 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:39.561930    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g8p6\" (UniqueName: \"kubernetes.io/projected/b008d452-1faf-4604-a3c3-cb06a6b05358-kube-api-access-4g8p6\") pod \"perf-node-gather-daemonset-tgq6z\" (UID: \"b008d452-1faf-4604-a3c3-cb06a6b05358\") " pod="openshift-must-gather-5wwbm/perf-node-gather-daemonset-tgq6z"
Jul 30 17:37:39.636146 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:39.636120    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5wwbm/perf-node-gather-daemonset-tgq6z"
Jul 30 17:37:39.728518 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:39.728464    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-58c4fb9c59-zbljv_74b18cf4-f2b0-400e-b7df-87d89f76fe0b/console/0.log"
Jul 30 17:37:39.772865 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:39.772839    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5wwbm/perf-node-gather-daemonset-tgq6z"]
Jul 30 17:37:39.776702 ip-10-0-136-68 kubenswrapper[2580]: W0730 17:37:39.776673    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb008d452_1faf_4604_a3c3_cb06a6b05358.slice/crio-c9c762102970d707b9709de782ebec091b458d7894702eba4a026b8db4b3cdca WatchSource:0}: Error finding container c9c762102970d707b9709de782ebec091b458d7894702eba4a026b8db4b3cdca: Status 404 returned error can't find the container with id c9c762102970d707b9709de782ebec091b458d7894702eba4a026b8db4b3cdca
Jul 30 17:37:40.097288 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:40.097212    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5wwbm/perf-node-gather-daemonset-tgq6z" event={"ID":"b008d452-1faf-4604-a3c3-cb06a6b05358","Type":"ContainerStarted","Data":"c759c76ac820659e8cb9ed635cb10004d4c495c326072d111f0d4bf62d14d3da"}
Jul 30 17:37:40.097416 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:40.097301    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5wwbm/perf-node-gather-daemonset-tgq6z"
Jul 30 17:37:40.097416 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:40.097329    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5wwbm/perf-node-gather-daemonset-tgq6z" event={"ID":"b008d452-1faf-4604-a3c3-cb06a6b05358","Type":"ContainerStarted","Data":"c9c762102970d707b9709de782ebec091b458d7894702eba4a026b8db4b3cdca"}
Jul 30 17:37:41.338415 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:41.338384    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8l786_d7e1b15f-2f31-42d4-b8e6-56bee84ad252/dns/0.log"
Jul 30 17:37:41.362620 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:41.362589    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8l786_d7e1b15f-2f31-42d4-b8e6-56bee84ad252/kube-rbac-proxy/0.log"
Jul 30 17:37:41.416415 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:41.416393    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-c77l8_e4e1630f-f153-4be6-883a-1fbeca875a69/dns-node-resolver/0.log"
Jul 30 17:37:41.935500 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:41.935455    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tdkrg_adf2f8d3-a0f4-4514-b6e3-b1507fa0c0bb/node-ca/0.log"
Jul 30 17:37:43.086379 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:43.086353    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-d2cnh_439955b8-2c26-4450-8a8c-ab8b9f7f0b3f/serve-healthcheck-canary/0.log"
Jul 30 17:37:43.785090 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:43.785066    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bjfhv_3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61/kube-rbac-proxy/0.log"
Jul 30 17:37:43.822593 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:43.822570    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bjfhv_3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61/exporter/0.log"
Jul 30 17:37:43.851449 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:43.851426    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bjfhv_3263a9ff-a9b4-4eb3-9af6-e03df8dc4c61/extractor/0.log"
Jul 30 17:37:46.113601 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:46.113568    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5wwbm/perf-node-gather-daemonset-tgq6z"
Jul 30 17:37:46.133225 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:46.133150    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5wwbm/perf-node-gather-daemonset-tgq6z" podStartSLOduration=7.133136043 podStartE2EDuration="7.133136043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 17:37:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:37:40.116560375 +0000 UTC m=+2223.191860265" watchObservedRunningTime="2026-07-30 17:37:46.133136043 +0000 UTC m=+2229.208435933"
Jul 30 17:37:46.533790 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:46.533756    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-654c557856-69jnn_e74064b8-5845-4d74-af33-e9c18de02def/manager/0.log"
Jul 30 17:37:51.188161 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:51.188092    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-b65z5_25af7388-ce5d-49e0-8ecb-c4aa3c1f91be/migrator/0.log"
Jul 30 17:37:51.210872 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:51.210843    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-b65z5_25af7388-ce5d-49e0-8ecb-c4aa3c1f91be/graceful-termination/0.log"
Jul 30 17:37:52.486062 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:52.486032    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-2f2qc_752fa9f3-f89c-4b80-ac71-c32c0e4ae5b1/kube-multus/0.log"
Jul 30 17:37:52.907571 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:52.907546    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mz6bl_f366886f-2162-4153-a077-55dcd0fef1e8/kube-multus-additional-cni-plugins/0.log"
Jul 30 17:37:52.936400 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:52.936379    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mz6bl_f366886f-2162-4153-a077-55dcd0fef1e8/egress-router-binary-copy/0.log"
Jul 30 17:37:52.962668 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:52.962644    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mz6bl_f366886f-2162-4153-a077-55dcd0fef1e8/cni-plugins/0.log"
Jul 30 17:37:52.987886 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:52.987865    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mz6bl_f366886f-2162-4153-a077-55dcd0fef1e8/bond-cni-plugin/0.log"
Jul 30 17:37:53.011077 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:53.011062    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mz6bl_f366886f-2162-4153-a077-55dcd0fef1e8/routeoverride-cni/0.log"
Jul 30 17:37:53.037245 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:53.037222    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mz6bl_f366886f-2162-4153-a077-55dcd0fef1e8/whereabouts-cni-bincopy/0.log"
Jul 30 17:37:53.066346 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:53.066328    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mz6bl_f366886f-2162-4153-a077-55dcd0fef1e8/whereabouts-cni/0.log"
Jul 30 17:37:53.235873 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:53.235807    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-kbsf2_4f8b90e9-e705-4b9c-89f7-047151a2b098/network-metrics-daemon/0.log"
Jul 30 17:37:53.269993 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:53.269973    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-kbsf2_4f8b90e9-e705-4b9c-89f7-047151a2b098/kube-rbac-proxy/0.log"
Jul 30 17:37:54.538726 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:54.538704    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tblxl_19e7774c-aa26-4558-a50c-0c5ff05d9767/ovn-controller/0.log"
Jul 30 17:37:54.556764 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:54.556736    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tblxl_19e7774c-aa26-4558-a50c-0c5ff05d9767/ovn-acl-logging/0.log"
Jul 30 17:37:54.584019 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:54.583995    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tblxl_19e7774c-aa26-4558-a50c-0c5ff05d9767/kube-rbac-proxy-node/0.log"
Jul 30 17:37:54.606820 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:54.606801    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tblxl_19e7774c-aa26-4558-a50c-0c5ff05d9767/kube-rbac-proxy-ovn-metrics/0.log"
Jul 30 17:37:54.629755 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:54.629735    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tblxl_19e7774c-aa26-4558-a50c-0c5ff05d9767/northd/0.log"
Jul 30 17:37:54.656245 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:54.656223    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tblxl_19e7774c-aa26-4558-a50c-0c5ff05d9767/nbdb/0.log"
Jul 30 17:37:54.701098 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:54.701070    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tblxl_19e7774c-aa26-4558-a50c-0c5ff05d9767/sbdb/0.log"
Jul 30 17:37:54.769019 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:54.768995    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tblxl_19e7774c-aa26-4558-a50c-0c5ff05d9767/ovnkube-controller/0.log"
Jul 30 17:37:55.821406 ip-10-0-136-68 kubenswrapper[2580]: I0730 17:37:55.821376    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-ltj2t_33ffc7a5-96d7-4a2a-b6c2-ec922b4de57e/network-check-target-container/0.log"