Jul 07 16:45:30.385597 ip-10-0-136-217 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 07 16:45:30.385606 ip-10-0-136-217 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 07 16:45:30.385613 ip-10-0-136-217 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 07 16:45:30.385873 ip-10-0-136-217 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 07 16:45:40.421498 ip-10-0-136-217 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 07 16:45:40.421518 ip-10-0-136-217 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 65f7864e5147436e9730989cebab3ea3 --
Jul 07 16:47:43.112044 ip-10-0-136-217 systemd[1]: Starting Kubernetes Kubelet...
Jul 07 16:47:43.482959 ip-10-0-136-217 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 16:47:43.482959 ip-10-0-136-217 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 07 16:47:43.482959 ip-10-0-136-217 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 16:47:43.482959 ip-10-0-136-217 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 07 16:47:43.482959 ip-10-0-136-217 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 16:47:43.483857 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.483773    2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 07 16:47:43.488272 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488256    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 16:47:43.488272 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488272    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 16:47:43.488346 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488275    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 16:47:43.488346 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488279    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 16:47:43.488346 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488284    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 16:47:43.488346 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488288    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 16:47:43.488346 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488291    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 16:47:43.488346 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488294    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 16:47:43.488346 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488297    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 16:47:43.488346 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488300    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 16:47:43.488346 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488302    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 16:47:43.488346 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488305    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 16:47:43.488346 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488307    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 16:47:43.488346 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488310    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 16:47:43.488346 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488313    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 16:47:43.488346 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488316    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 16:47:43.488346 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488318    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 16:47:43.488346 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488321    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 16:47:43.488346 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488324    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 16:47:43.488346 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488326    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 16:47:43.488346 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488329    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 16:47:43.488829 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488331    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 16:47:43.488829 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488334    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 16:47:43.488829 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488337    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 16:47:43.488829 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488339    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 16:47:43.488829 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488341    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 16:47:43.488829 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488344    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 16:47:43.488829 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488346    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 16:47:43.488829 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488349    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 16:47:43.488829 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488352    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 16:47:43.488829 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488355    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 16:47:43.488829 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488358    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 16:47:43.488829 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488360    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 16:47:43.488829 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488364    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 16:47:43.488829 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488366    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 16:47:43.488829 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488369    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 16:47:43.488829 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488371    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 16:47:43.488829 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488374    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 16:47:43.488829 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488377    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 16:47:43.488829 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488379    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 16:47:43.488829 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488381    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 16:47:43.489320 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488384    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 16:47:43.489320 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488387    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 16:47:43.489320 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488389    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 16:47:43.489320 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488391    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 16:47:43.489320 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488394    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 16:47:43.489320 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488396    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 16:47:43.489320 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488398    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 16:47:43.489320 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488401    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 16:47:43.489320 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488403    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 16:47:43.489320 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488405    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 16:47:43.489320 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488408    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 16:47:43.489320 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488411    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 16:47:43.489320 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488414    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 16:47:43.489320 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488416    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 16:47:43.489320 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488419    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 16:47:43.489320 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488422    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 16:47:43.489320 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488424    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 16:47:43.489320 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488426    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 16:47:43.489320 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488429    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 16:47:43.489320 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488432    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 16:47:43.489807 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488436    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 16:47:43.489807 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488439    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 16:47:43.489807 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488442    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 16:47:43.489807 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488445    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 16:47:43.489807 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488448    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 16:47:43.489807 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488450    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 16:47:43.489807 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488453    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 16:47:43.489807 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488455    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 16:47:43.489807 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488459    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 16:47:43.489807 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488462    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 16:47:43.489807 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488465    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 16:47:43.489807 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488468    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 16:47:43.489807 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488470    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 16:47:43.489807 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488473    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 16:47:43.489807 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488476    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 16:47:43.489807 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488478    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 16:47:43.489807 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488481    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 16:47:43.489807 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488483    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 16:47:43.489807 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488485    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 16:47:43.490270 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488488    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 16:47:43.490270 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488490    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 16:47:43.490270 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488494    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 16:47:43.490270 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488497    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 16:47:43.490270 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488501    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 16:47:43.490270 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488504    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 16:47:43.490270 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488508    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 16:47:43.490270 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488511    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 16:47:43.490270 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488513    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 16:47:43.490270 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488516    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 16:47:43.490270 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488519    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 16:47:43.490270 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488521    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 16:47:43.490270 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488525    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 16:47:43.490270 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488528    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 16:47:43.490270 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488531    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 16:47:43.490270 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488533    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 16:47:43.490270 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488536    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 16:47:43.490270 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488539    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 16:47:43.490270 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.488542    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 16:47:43.490781 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489256    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 16:47:43.490781 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489263    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 16:47:43.490781 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489267    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 16:47:43.490781 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489270    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 16:47:43.490781 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489274    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 16:47:43.490781 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489277    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 16:47:43.490781 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489280    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 16:47:43.490781 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489284    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 16:47:43.490781 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489287    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 16:47:43.490781 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489289    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 16:47:43.490781 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489292    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 16:47:43.490781 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489294    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 16:47:43.490781 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489297    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 16:47:43.490781 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489299    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 16:47:43.490781 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489302    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 16:47:43.490781 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489304    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 16:47:43.490781 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489307    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 16:47:43.490781 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489310    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 16:47:43.490781 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489312    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 16:47:43.491273 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489316    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 16:47:43.491273 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489318    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 16:47:43.491273 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489321    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 16:47:43.491273 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489324    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 16:47:43.491273 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489327    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 16:47:43.491273 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489329    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 16:47:43.491273 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489332    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 16:47:43.491273 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489335    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 16:47:43.491273 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489337    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 16:47:43.491273 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489340    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 16:47:43.491273 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489342    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 16:47:43.491273 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489345    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 16:47:43.491273 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489347    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 16:47:43.491273 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489350    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 16:47:43.491273 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489353    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 16:47:43.491273 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489355    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 16:47:43.491273 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489358    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 16:47:43.491273 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489361    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 16:47:43.491273 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489363    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 16:47:43.491273 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489367    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 16:47:43.491766 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489369    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 16:47:43.491766 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489372    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 16:47:43.491766 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489375    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 16:47:43.491766 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489377    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 16:47:43.491766 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489381    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 16:47:43.491766 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489386    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 16:47:43.491766 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489389    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 16:47:43.491766 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489393    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 16:47:43.491766 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489396    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 16:47:43.491766 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489399    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 16:47:43.491766 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489402    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 16:47:43.491766 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489405    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 16:47:43.491766 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489408    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 16:47:43.491766 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489410    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 16:47:43.491766 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489413    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 16:47:43.491766 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489415    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 16:47:43.491766 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489418    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 16:47:43.491766 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489420    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 16:47:43.491766 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489423    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 16:47:43.492226 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489425    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 16:47:43.492226 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489428    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 16:47:43.492226 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489430    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 16:47:43.492226 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489433    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 16:47:43.492226 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489436    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 16:47:43.492226 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489438    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 16:47:43.492226 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489441    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 16:47:43.492226 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489445    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 16:47:43.492226 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489448    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 16:47:43.492226 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489451    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 16:47:43.492226 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489454    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 16:47:43.492226 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489456    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 16:47:43.492226 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489459    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 16:47:43.492226 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489464    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 16:47:43.492226 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489467    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 16:47:43.492226 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489469    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 16:47:43.492226 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489472    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 16:47:43.492226 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489474    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 16:47:43.492226 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489478    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 16:47:43.492226 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489480    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 16:47:43.492717 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489483    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 16:47:43.492717 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489486    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 16:47:43.492717 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489488    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 16:47:43.492717 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489491    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 16:47:43.492717 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489493    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 16:47:43.492717 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489496    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 16:47:43.492717 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489498    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 16:47:43.492717 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489501    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 16:47:43.492717 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489503    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 16:47:43.492717 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489505    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 16:47:43.492717 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489508    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 16:47:43.492717 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489511    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 16:47:43.492717 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489513    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 16:47:43.492717 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489516    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 16:47:43.492717 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489519    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 16:47:43.492717 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489522    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 16:47:43.492717 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489524    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 16:47:43.492717 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489526    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 16:47:43.492717 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489529    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 16:47:43.492717 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489531    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 16:47:43.493217 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489534    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 16:47:43.493217 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489913    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 16:47:43.493217 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489919    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 16:47:43.493217 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489922    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 16:47:43.493217 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489925    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 16:47:43.493217 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489929    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 16:47:43.493217 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489932    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 16:47:43.493217 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489935    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 16:47:43.493217 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489938    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 16:47:43.493217 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489941    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 16:47:43.493217 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489944    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 16:47:43.493217 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489947    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 16:47:43.493217 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489949    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 16:47:43.493217 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489953    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 16:47:43.493217 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489955    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 16:47:43.493217 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489958    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 16:47:43.493217 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489962    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 16:47:43.493217 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489965    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 16:47:43.493217 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489968    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 16:47:43.493217 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489971    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 16:47:43.493697 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489974    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 16:47:43.493697 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489976    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 16:47:43.493697 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489979    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 16:47:43.493697 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489981    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 16:47:43.493697 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489984    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 16:47:43.493697 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489986    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 16:47:43.493697 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489989    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 16:47:43.493697 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489993    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 16:47:43.493697 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489995    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 16:47:43.493697 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.489998    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 16:47:43.493697 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490001    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 16:47:43.493697 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490003    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 16:47:43.493697 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490007    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 16:47:43.493697 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490010    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 16:47:43.493697 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490013    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 16:47:43.493697 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490016    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 16:47:43.493697 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490019    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 16:47:43.493697 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490021    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 16:47:43.493697 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490028    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 16:47:43.494162 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490031    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 16:47:43.494162 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490034    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 16:47:43.494162 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490037    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 16:47:43.494162 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490039    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 16:47:43.494162 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490042    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 16:47:43.494162 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490044    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 16:47:43.494162 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490047    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 16:47:43.494162 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490050    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 16:47:43.494162 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490053    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 16:47:43.494162 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490055    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 16:47:43.494162 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490058    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 16:47:43.494162 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490060    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 16:47:43.494162 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490063    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 16:47:43.494162 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490065    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 16:47:43.494162 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490068    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 16:47:43.494162 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490070    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 16:47:43.494162 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490073    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 16:47:43.494162 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490076    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 16:47:43.494162 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490078    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 16:47:43.494162 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490080    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 16:47:43.494633 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490083    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 16:47:43.494633 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490086    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 16:47:43.494633 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490089    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 16:47:43.494633 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490091    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 16:47:43.494633 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490094    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 16:47:43.494633 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490096    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 16:47:43.494633 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490099    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 16:47:43.494633 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490102    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 16:47:43.494633 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490104    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 16:47:43.494633 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490107    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 16:47:43.494633 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490109    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 16:47:43.494633 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490112    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 16:47:43.494633 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490115    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 16:47:43.494633 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490118    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 16:47:43.494633 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490120    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 16:47:43.494633 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490123    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 16:47:43.494633 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490125    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 16:47:43.494633 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490128    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 16:47:43.494633 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490131    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 16:47:43.494633 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490133    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 16:47:43.495116 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490136    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 16:47:43.495116 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490138    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 16:47:43.495116 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490141    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 16:47:43.495116 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490143    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 16:47:43.495116 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490146    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 16:47:43.495116 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490148    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 16:47:43.495116 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490152    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 16:47:43.495116 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490154    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 16:47:43.495116 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490157    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 16:47:43.495116 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490159    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 16:47:43.495116 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490162    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 16:47:43.495116 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490164    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 16:47:43.495116 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490167    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 16:47:43.495116 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490170    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 16:47:43.495116 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490172    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 16:47:43.495116 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490175    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 16:47:43.495116 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490177    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 16:47:43.495116 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490180    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 16:47:43.495116 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490184    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 16:47:43.495558 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490187    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 16:47:43.495558 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490190    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 16:47:43.495558 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490263    2578 flags.go:64] FLAG: --address="0.0.0.0"
Jul 07 16:47:43.495558 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490270    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 07 16:47:43.495558 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490276    2578 flags.go:64] FLAG: --anonymous-auth="true"
Jul 07 16:47:43.495558 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490280    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 07 16:47:43.495558 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490284    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 07 16:47:43.495558 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490287    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 07 16:47:43.495558 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490299    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 07 16:47:43.495558 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490304    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 07 16:47:43.495558 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490307    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 07 16:47:43.495558 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490310    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 07 16:47:43.495558 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490314    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 07 16:47:43.495558 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490317    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 07 16:47:43.495558 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490320    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 07 16:47:43.495558 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490323    2578 flags.go:64] FLAG: --cgroup-root=""
Jul 07 16:47:43.495558 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490325    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 07 16:47:43.495558 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490328    2578 flags.go:64] FLAG: --client-ca-file=""
Jul 07 16:47:43.495558 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490331    2578 flags.go:64] FLAG: --cloud-provider="external"
Jul 07 16:47:43.495558 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490334    2578 flags.go:64] FLAG: --cluster-dns="[]"
Jul 07 16:47:43.495558 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490338    2578 flags.go:64] FLAG: --cluster-domain=""
Jul 07 16:47:43.495558 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490341    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 07 16:47:43.495558 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490344    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 07 16:47:43.495558 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490347    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 07 16:47:43.496148 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490351    2578 flags.go:64] FLAG: --container-log-max-files="5"
Jul 07 16:47:43.496148 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490355    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 07 16:47:43.496148 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490358    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 07 16:47:43.496148 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490362    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 07 16:47:43.496148 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490365    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 07 16:47:43.496148 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490368    2578 flags.go:64] FLAG: --contention-profiling="false"
Jul 07 16:47:43.496148 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490371    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 07 16:47:43.496148 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490373    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 07 16:47:43.496148 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490376    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 07 16:47:43.496148 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490379    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 07 16:47:43.496148 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490383    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 07 16:47:43.496148 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490386    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 07 16:47:43.496148 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490389    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 07 16:47:43.496148 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490391    2578 flags.go:64] FLAG: --enable-load-reader="false"
Jul 07 16:47:43.496148 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490394    2578 flags.go:64] FLAG: --enable-server="true"
Jul 07 16:47:43.496148 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490398    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 07 16:47:43.496148 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490402    2578 flags.go:64] FLAG: --event-burst="100"
Jul 07 16:47:43.496148 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490405    2578 flags.go:64] FLAG: --event-qps="50"
Jul 07 16:47:43.496148 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490407    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 07 16:47:43.496148 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490414    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 07 16:47:43.496148 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490416    2578 flags.go:64] FLAG: --eviction-hard=""
Jul 07 16:47:43.496148 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490420    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 07 16:47:43.496148 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490423    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 07 16:47:43.496148 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490426    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 07 16:47:43.496148 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490429    2578 flags.go:64] FLAG: --eviction-soft=""
Jul 07 16:47:43.496879 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490432    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 07 16:47:43.496879 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490434    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 07 16:47:43.496879 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490437    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 07 16:47:43.496879 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490440    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 07 16:47:43.496879 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490443    2578 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 07 16:47:43.496879 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490446    2578 flags.go:64] FLAG: --fail-swap-on="true"
Jul 07 16:47:43.496879 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490448    2578 flags.go:64] FLAG: --feature-gates=""
Jul 07 16:47:43.496879 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490452    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 07 16:47:43.496879 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490457    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 07 16:47:43.496879 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490460    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 07 16:47:43.496879 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490463    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 07 16:47:43.496879 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490466    2578 flags.go:64] FLAG: --healthz-port="10248"
Jul 07 16:47:43.496879 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490469    2578 flags.go:64] FLAG: --help="false"
Jul 07 16:47:43.496879 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490471    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-136-217.ec2.internal"
Jul 07 16:47:43.496879 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490474    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 07 16:47:43.496879 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490477    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 07 16:47:43.496879 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490480    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 07 16:47:43.496879 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490483    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 07 16:47:43.496879 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490487    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 07 16:47:43.496879 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490489    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 07 16:47:43.496879 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490492    2578 flags.go:64] FLAG: --image-service-endpoint=""
Jul 07 16:47:43.496879 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490495    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 07 16:47:43.496879 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490497    2578 flags.go:64] FLAG: --kube-api-burst="100"
Jul 07 16:47:43.497472 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490501    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 07 16:47:43.497472 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490504    2578 flags.go:64] FLAG: --kube-api-qps="50"
Jul 07 16:47:43.497472 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490507    2578 flags.go:64] FLAG: --kube-reserved=""
Jul 07 16:47:43.497472 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490511    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 07 16:47:43.497472 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490514    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 07 16:47:43.497472 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490517    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 07 16:47:43.497472 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490520    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 07 16:47:43.497472 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490523    2578 flags.go:64] FLAG: --lock-file=""
Jul 07 16:47:43.497472 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490526    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 07 16:47:43.497472 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490528    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 07 16:47:43.497472 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490531    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 07 16:47:43.497472 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490536    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 07 16:47:43.497472 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490539    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 07 16:47:43.497472 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490542    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 07 16:47:43.497472 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490545    2578 flags.go:64] FLAG: --logging-format="text"
Jul 07 16:47:43.497472 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490547    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 07 16:47:43.497472 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490551    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 07 16:47:43.497472 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490555    2578 flags.go:64] FLAG: --manifest-url=""
Jul 07 16:47:43.497472 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490558    2578 flags.go:64] FLAG: --manifest-url-header=""
Jul 07 16:47:43.497472 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490562    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 07 16:47:43.497472 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490565    2578 flags.go:64] FLAG: --max-open-files="1000000"
Jul 07 16:47:43.497472 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490569    2578 flags.go:64] FLAG: --max-pods="110"
Jul 07 16:47:43.497472 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490571    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 07 16:47:43.497472 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490574    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 07 16:47:43.497472 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490577    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 07 16:47:43.498098 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490580    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 07 16:47:43.498098 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490583    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 07 16:47:43.498098 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490586    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 07 16:47:43.498098 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490589    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 07 16:47:43.498098 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490595    2578 flags.go:64] FLAG: --node-status-max-images="50"
Jul 07 16:47:43.498098 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490598    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 07 16:47:43.498098 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490602    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 07 16:47:43.498098 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490605    2578 flags.go:64] FLAG: --pod-cidr=""
Jul 07 16:47:43.498098 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490608    2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5"
Jul 07 16:47:43.498098 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490629    2578 flags.go:64] FLAG: --pod-manifest-path=""
Jul 07 16:47:43.498098 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490632    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 07 16:47:43.498098 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490637    2578 flags.go:64] FLAG: --pods-per-core="0"
Jul 07 16:47:43.498098 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490640    2578 flags.go:64] FLAG: --port="10250"
Jul 07 16:47:43.498098 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490643    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 07 16:47:43.498098 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490646    2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03b6ae8924d79cd9c"
Jul 07 16:47:43.498098 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490650    2578 flags.go:64] FLAG: --qos-reserved=""
Jul 07 16:47:43.498098 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490652    2578 flags.go:64] FLAG: --read-only-port="10255"
Jul 07 16:47:43.498098 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490655    2578 flags.go:64] FLAG: --register-node="true"
Jul 07 16:47:43.498098 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490658    2578 flags.go:64] FLAG: --register-with-taints=""
Jul 07 16:47:43.498098 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490662    2578 flags.go:64] FLAG: --registry-burst="10"
Jul 07 16:47:43.498098 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490664    2578 flags.go:64] FLAG: --registry-qps="5"
Jul 07 16:47:43.498098 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490667    2578 flags.go:64] FLAG: --reserved-cpus=""
Jul 07 16:47:43.498098 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490670    2578 flags.go:64] FLAG: --reserved-memory=""
Jul 07 16:47:43.498098 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490673    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 07 16:47:43.498098 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490677    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 07 16:47:43.498711 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490680    2578 flags.go:64] FLAG: --rotate-certificates="false"
Jul 07 16:47:43.498711 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490683    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 07 16:47:43.498711 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490686    2578 flags.go:64] FLAG: --runonce="false"
Jul 07 16:47:43.498711 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490688    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 07 16:47:43.498711 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490691    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 07 16:47:43.498711 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490694    2578 flags.go:64] FLAG: --seccomp-default="false"
Jul 07 16:47:43.498711 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490697    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 07 16:47:43.498711 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490699    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 07 16:47:43.498711 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490702    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 07 16:47:43.498711 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490705    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 07 16:47:43.498711 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490708    2578 flags.go:64] FLAG: --storage-driver-password="root"
Jul 07 16:47:43.498711 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490711    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 07 16:47:43.498711 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490714    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 07 16:47:43.498711 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490716    2578 flags.go:64] FLAG: --storage-driver-user="root"
Jul 07 16:47:43.498711 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490719    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 07 16:47:43.498711 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490722    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 07 16:47:43.498711 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490726    2578 flags.go:64] FLAG: --system-cgroups=""
Jul 07 16:47:43.498711 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490741    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 07 16:47:43.498711 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490748    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 07 16:47:43.498711 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490751    2578 flags.go:64] FLAG: --tls-cert-file=""
Jul 07 16:47:43.498711 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490753    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 07 16:47:43.498711 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490758    2578 flags.go:64] FLAG: --tls-min-version=""
Jul 07 16:47:43.498711 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490761    2578 flags.go:64] FLAG: --tls-private-key-file=""
Jul 07 16:47:43.498711 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490763    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 07 16:47:43.498711 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490766    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 07 16:47:43.499328 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490769    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 07 16:47:43.499328 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490772    2578 flags.go:64] FLAG: --v="2"
Jul 07 16:47:43.499328 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490776    2578 flags.go:64] FLAG: --version="false"
Jul 07 16:47:43.499328 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490780    2578 flags.go:64] FLAG: --vmodule=""
Jul 07 16:47:43.499328 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490783    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 07 16:47:43.499328 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.490787    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 07 16:47:43.499328 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490899    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 16:47:43.499328 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490903    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 16:47:43.499328 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490906    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 16:47:43.499328 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490910    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 16:47:43.499328 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490914    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 16:47:43.499328 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490917    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 16:47:43.499328 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490920    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 16:47:43.499328 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490923    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 16:47:43.499328 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490925    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 16:47:43.499328 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490928    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 16:47:43.499328 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490930    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 16:47:43.499328 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490933    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 16:47:43.499328 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490936    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 16:47:43.499328 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490938    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 16:47:43.499328 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490941    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 16:47:43.499867 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490943    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 16:47:43.499867 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490946    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 16:47:43.499867 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490948    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 16:47:43.499867 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490953    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 16:47:43.499867 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490958    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 16:47:43.499867 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490961    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 16:47:43.499867 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490964    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 16:47:43.499867 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490967    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 16:47:43.499867 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490970    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 16:47:43.499867 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490972    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 16:47:43.499867 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490975    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 16:47:43.499867 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490978    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 16:47:43.499867 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490981    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 16:47:43.499867 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490983    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 16:47:43.499867 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490986    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 16:47:43.499867 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490988    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 16:47:43.499867 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490991    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 16:47:43.499867 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490995    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 16:47:43.499867 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.490998    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 16:47:43.500359 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491001    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 16:47:43.500359 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491003    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 16:47:43.500359 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491006    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 16:47:43.500359 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491008    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 16:47:43.500359 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491011    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 16:47:43.500359 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491013    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 16:47:43.500359 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491016    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 16:47:43.500359 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491018    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 16:47:43.500359 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491020    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 16:47:43.500359 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491023    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 16:47:43.500359 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491025    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 16:47:43.500359 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491028    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 16:47:43.500359 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491030    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 16:47:43.500359 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491033    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 16:47:43.500359 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491035    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 16:47:43.500359 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491038    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 16:47:43.500359 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491040    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 16:47:43.500359 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491045    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 16:47:43.500359 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491048    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 16:47:43.500359 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491050    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 16:47:43.500913 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491053    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 16:47:43.500913 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491055    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 16:47:43.500913 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491058    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 16:47:43.500913 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491061    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 16:47:43.500913 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491063    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 16:47:43.500913 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491066    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 16:47:43.500913 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491069    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 16:47:43.500913 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491071    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 16:47:43.500913 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491074    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 16:47:43.500913 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491077    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 16:47:43.500913 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491080    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 16:47:43.500913 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491083    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 16:47:43.500913 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491086    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 16:47:43.500913 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491088    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 16:47:43.500913 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491092    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 16:47:43.500913 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491095    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 16:47:43.500913 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491098    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 16:47:43.500913 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491101    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 16:47:43.500913 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491104    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 16:47:43.501364 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491106    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 16:47:43.501364 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491109    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 16:47:43.501364 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491112    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 16:47:43.501364 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491114    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 16:47:43.501364 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491117    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 16:47:43.501364 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491119    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 16:47:43.501364 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491122    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 16:47:43.501364 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491124    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 16:47:43.501364 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491127    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 16:47:43.501364 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491129    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 16:47:43.501364 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491134    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 16:47:43.501364 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491141    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 16:47:43.501364 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491144    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 16:47:43.501364 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491146    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 16:47:43.501364 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491149    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 16:47:43.501364 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491151    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 16:47:43.501364 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491154    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 16:47:43.501364 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491157    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 16:47:43.501364 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491159    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 16:47:43.501364 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491162    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 16:47:43.501904 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491165    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 16:47:43.501904 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491167    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 16:47:43.501904 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491170    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 16:47:43.501904 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491175    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 16:47:43.501904 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491177    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 16:47:43.501904 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.491180    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 16:47:43.501904 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.491701    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 16:47:43.501904 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.498291    2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 07 16:47:43.501904 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.498306    2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 07 16:47:43.501904 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.498326    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 07 16:47:43.501904 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.498332    2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 07 16:47:43.501904 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498386    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 16:47:43.501904 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498391    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 16:47:43.501904 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498395    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 16:47:43.501904 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498399    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 16:47:43.502291 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498402    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 16:47:43.502291 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498406    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 16:47:43.502291 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498410    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 16:47:43.502291 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498413    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 16:47:43.502291 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498416    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 16:47:43.502291 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498419    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 16:47:43.502291 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498422    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 16:47:43.502291 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498425    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 16:47:43.502291 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498427    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 16:47:43.502291 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498430    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 16:47:43.502291 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498433    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 16:47:43.502291 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498436    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 16:47:43.502291 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498438    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 16:47:43.502291 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498441    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 16:47:43.502291 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498444    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 16:47:43.502291 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498446    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 16:47:43.502291 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498449    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 16:47:43.502291 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498451    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 16:47:43.502291 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498454    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 16:47:43.502758 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498457    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 16:47:43.502758 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498459    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 16:47:43.502758 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498462    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 16:47:43.502758 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498466    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 16:47:43.502758 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498469    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 16:47:43.502758 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498472    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 16:47:43.502758 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498477    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 16:47:43.502758 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498480    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 16:47:43.502758 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498483    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 16:47:43.502758 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498485    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 16:47:43.502758 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498488    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 16:47:43.502758 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498490    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 16:47:43.502758 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498493    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 16:47:43.502758 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498496    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 16:47:43.502758 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498498    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 16:47:43.502758 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498500    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 16:47:43.502758 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498503    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 16:47:43.502758 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498505    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 16:47:43.502758 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498508    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 16:47:43.502758 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498510    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 16:47:43.503253 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498514    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 16:47:43.503253 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498518    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 16:47:43.503253 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498520    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 16:47:43.503253 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498523    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 16:47:43.503253 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498525    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 16:47:43.503253 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498528    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 16:47:43.503253 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498530    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 16:47:43.503253 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498533    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 16:47:43.503253 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498536    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 16:47:43.503253 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498538    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 16:47:43.503253 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498540    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 16:47:43.503253 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498543    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 16:47:43.503253 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498545    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 16:47:43.503253 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498548    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 16:47:43.503253 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498550    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 16:47:43.503253 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498553    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 16:47:43.503253 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498555    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 16:47:43.503253 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498558    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 16:47:43.503253 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498561    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 16:47:43.503701 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498564    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 16:47:43.503701 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498567    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 16:47:43.503701 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498569    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 16:47:43.503701 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498572    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 16:47:43.503701 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498574    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 16:47:43.503701 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498577    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 16:47:43.503701 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498579    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 16:47:43.503701 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498582    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 16:47:43.503701 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498584    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 16:47:43.503701 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498587    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 16:47:43.503701 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498589    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 16:47:43.503701 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498592    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 16:47:43.503701 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498594    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 16:47:43.503701 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498597    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 16:47:43.503701 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498599    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 16:47:43.503701 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498602    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 16:47:43.503701 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498604    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 16:47:43.503701 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498607    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 16:47:43.503701 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498609    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 16:47:43.503701 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498612    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 16:47:43.504216 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498614    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 16:47:43.504216 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498617    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 16:47:43.504216 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498619    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 16:47:43.504216 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498622    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 16:47:43.504216 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498624    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 16:47:43.504216 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498627    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 16:47:43.504216 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498630    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 16:47:43.504216 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498632    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 16:47:43.504216 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498635    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 16:47:43.504216 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498638    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 16:47:43.504216 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498640    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 16:47:43.504216 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498643    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 16:47:43.504216 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498646    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 16:47:43.504216 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498649    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 16:47:43.504216 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498651    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 16:47:43.504216 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498654    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 16:47:43.504216 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498657    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 16:47:43.504622 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.498662    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 16:47:43.504622 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498793    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 16:47:43.504622 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498799    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 16:47:43.504622 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498802    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 16:47:43.504622 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498805    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 16:47:43.504622 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498808    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 16:47:43.504622 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498810    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 16:47:43.504622 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498813    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 16:47:43.504622 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498816    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 16:47:43.504622 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498818    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 16:47:43.504622 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498821    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 16:47:43.504622 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498823    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 16:47:43.504622 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498826    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 16:47:43.504622 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498829    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 16:47:43.504622 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498831    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 16:47:43.505029 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498834    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 16:47:43.505029 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498837    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 16:47:43.505029 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498839    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 16:47:43.505029 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498842    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 16:47:43.505029 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498844    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 16:47:43.505029 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498846    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 16:47:43.505029 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498849    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 16:47:43.505029 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498851    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 16:47:43.505029 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498854    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 16:47:43.505029 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498856    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 16:47:43.505029 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498859    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 16:47:43.505029 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498862    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 16:47:43.505029 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498864    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 16:47:43.505029 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498867    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 16:47:43.505029 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498870    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 16:47:43.505029 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498872    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 16:47:43.505029 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498875    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 16:47:43.505029 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498877    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 16:47:43.505029 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498880    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 16:47:43.505029 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498882    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 16:47:43.505707 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498885    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 16:47:43.505707 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498887    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 16:47:43.505707 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498890    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 16:47:43.505707 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498892    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 16:47:43.505707 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498894    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 16:47:43.505707 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498897    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 16:47:43.505707 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498900    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 16:47:43.505707 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498902    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 16:47:43.505707 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498905    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 16:47:43.505707 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498908    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 16:47:43.505707 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498911    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 16:47:43.505707 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498915    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 16:47:43.505707 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498918    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 16:47:43.505707 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498921    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 16:47:43.505707 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498923    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 16:47:43.505707 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498926    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 16:47:43.505707 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498928    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 16:47:43.505707 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498931    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 16:47:43.505707 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498933    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 16:47:43.506478 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498936    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 16:47:43.506478 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498938    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 16:47:43.506478 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498941    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 16:47:43.506478 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498943    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 16:47:43.506478 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498946    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 16:47:43.506478 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498948    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 16:47:43.506478 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498951    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 16:47:43.506478 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498954    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 16:47:43.506478 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498957    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 16:47:43.506478 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498959    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 16:47:43.506478 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498962    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 16:47:43.506478 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498964    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 16:47:43.506478 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498967    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 16:47:43.506478 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498969    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 16:47:43.506478 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498971    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 16:47:43.506478 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498974    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 16:47:43.506478 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498976    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 16:47:43.506478 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498979    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 16:47:43.506478 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498981    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 16:47:43.506478 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498984    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 16:47:43.507026 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498986    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 16:47:43.507026 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498989    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 16:47:43.507026 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498991    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 16:47:43.507026 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498994    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 16:47:43.507026 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498996    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 16:47:43.507026 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.498999    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 16:47:43.507026 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.499002    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 16:47:43.507026 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.499004    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 16:47:43.507026 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.499006    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 16:47:43.507026 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.499009    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 16:47:43.507026 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.499011    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 16:47:43.507026 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.499014    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 16:47:43.507026 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.499016    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 16:47:43.507026 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.499019    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 16:47:43.507026 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.499021    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 16:47:43.507026 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.499024    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 16:47:43.507026 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.499028    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 16:47:43.507026 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.499031    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 16:47:43.507026 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.499034    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 16:47:43.507486 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.499037    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 16:47:43.507486 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.499039    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 16:47:43.507486 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.499044    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 16:47:43.507486 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.499047    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 16:47:43.507486 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.499050    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 16:47:43.507486 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.499053    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 16:47:43.507486 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:43.499055    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 16:47:43.507486 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.499060    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 16:47:43.507486 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.499181    2578 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 07 16:47:43.507486 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.501390    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 07 16:47:43.507486 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.502785    2578 server.go:1019] "Starting client certificate rotation"
Jul 07 16:47:43.507486 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.502889    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 07 16:47:43.507486 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.502955    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 07 16:47:43.507834 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.504920    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 07 16:47:43.509619 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.509595    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 07 16:47:43.521343 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.521319    2578 log.go:25] "Validated CRI v1 runtime API"
Jul 07 16:47:43.523837 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.523824    2578 log.go:25] "Validated CRI v1 image API"
Jul 07 16:47:43.524586 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.524572    2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 07 16:47:43.527036 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.527017    2578 fs.go:135] Filesystem UUIDs: map[49849c4f-6945-46b7-96e5-4410438bdef0:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 ceca9b67-b710-4ca0-b20d-ffeb063995dd:/dev/nvme0n1p3]
Jul 07 16:47:43.527082 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.527036    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 07 16:47:43.532708 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.532599    2578 manager.go:216] Machine: {Timestamp:2026-07-07 16:47:43.531319119 +0000 UTC m=+0.307633336 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3109106 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec212a4101159e5a3b24814c730ff04a SystemUUID:ec212a41-0115-9e5a-3b24-814c730ff04a BootID:65f7864e-5147-436e-9730-989cebab3ea3 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:e2:a6:d5:65:f3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:e2:a6:d5:65:f3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:12:2b:0b:7c:aa:f6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 07 16:47:43.532708 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.532706    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 07 16:47:43.532818 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.532804    2578 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 07 16:47:43.535726 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.535699    2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 07 16:47:43.535872 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.535729    2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-217.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 07 16:47:43.536517 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.536507    2578 topology_manager.go:138] "Creating topology manager with none policy"
Jul 07 16:47:43.536546 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.536520    2578 container_manager_linux.go:309] "Creating device plugin manager"
Jul 07 16:47:43.536546 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.536527    2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 07 16:47:43.537129 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.537120    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 07 16:47:43.539085 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.539075    2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 07 16:47:43.542396 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.542382    2578 state_mem.go:36] "Initialized new in-memory state store"
Jul 07 16:47:43.542501 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.542492    2578 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 07 16:47:43.542542 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.542521    2578 kubelet.go:486] "Attempting to sync node with API server"
Jul 07 16:47:43.542542 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.542530    2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 07 16:47:43.542605 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.542544    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 07 16:47:43.542605 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.542553    2578 kubelet.go:398] "Adding apiserver pod source"
Jul 07 16:47:43.542605 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.542561    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 07 16:47:43.543077 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.543054    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 07 16:47:43.543077 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.543073    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 07 16:47:43.544494 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.544477    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 07 16:47:43.545084 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.545070    2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 07 16:47:43.547369 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.547355    2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 07 16:47:43.547459 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.547393    2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 07 16:47:43.548453 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.548437    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 07 16:47:43.548526 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.548459    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 07 16:47:43.548526 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.548471    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 07 16:47:43.548526 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.548482    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 07 16:47:43.548526 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.548498    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 07 16:47:43.548526 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.548510    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 07 16:47:43.548526 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.548525    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 07 16:47:43.548682 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.548557    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 07 16:47:43.548682 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.548572    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 07 16:47:43.548682 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.548583    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 07 16:47:43.548682 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.548625    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 07 16:47:43.548682 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.548636    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 07 16:47:43.549571 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.549549    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 07 16:47:43.549660 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.549595    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 07 16:47:43.553637 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.553622    2578 server.go:1268] "Started kubelet"
Jul 07 16:47:43.553786 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.553718    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 07 16:47:43.553786 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.553768    2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 07 16:47:43.553892 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.553810    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 07 16:47:43.554603 ip-10-0-136-217 systemd[1]: Started Kubernetes Kubelet.
Jul 07 16:47:43.555322 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.555305    2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 07 16:47:43.555999 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.555937    2578 server.go:310] "Adding debug handlers to kubelet server"
Jul 07 16:47:43.558437 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:43.558413    2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 07 16:47:43.562021 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.562004    2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 07 16:47:43.562021 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.562001    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 07 16:47:43.562455 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:43.562423    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-217.ec2.internal\" not found"
Jul 07 16:47:43.562836 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.562807    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 07 16:47:43.562836 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.562833    2578 factory.go:55] Registering systemd factory
Jul 07 16:47:43.563039 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.562842    2578 factory.go:223] Registration of the systemd container factory successfully
Jul 07 16:47:43.563039 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.563024    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 07 16:47:43.563133 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.563048    2578 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 07 16:47:43.563133 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.563084    2578 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 07 16:47:43.563239 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.563142    2578 factory.go:153] Registering CRI-O factory
Jul 07 16:47:43.563239 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.563158    2578 factory.go:223] Registration of the crio container factory successfully
Jul 07 16:47:43.563239 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.563181    2578 factory.go:103] Registering Raw factory
Jul 07 16:47:43.563239 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.563196    2578 manager.go:1196] Started watching for new ooms in manager
Jul 07 16:47:43.563239 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.563219    2578 reconstruct.go:97] "Volume reconstruction finished"
Jul 07 16:47:43.563239 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.563229    2578 reconciler.go:29] "Reconciler: start to sync state"
Jul 07 16:47:43.563644 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.563613    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-217.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 07 16:47:43.564082 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.563986    2578 manager.go:349] Starting recovery of all containers
Jul 07 16:47:43.564941 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:43.563432    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-217.ec2.internal.18c00fd5d750d556  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-217.ec2.internal,UID:ip-10-0-136-217.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-217.ec2.internal,},FirstTimestamp:2026-07-07 16:47:43.553566038 +0000 UTC m=+0.329880234,LastTimestamp:2026-07-07 16:47:43.553566038 +0000 UTC m=+0.329880234,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-217.ec2.internal,}"
Jul 07 16:47:43.565882 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:43.565856    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 07 16:47:43.568200 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:43.568161    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-217.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 07 16:47:43.574227 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.574041    2578 manager.go:354] Recovery completed
Jul 07 16:47:43.574382 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:43.574354    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 07 16:47:43.575277 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:43.575253    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-217.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 07 16:47:43.580119 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.580105    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 16:47:43.581955 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.581929    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-217.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 16:47:43.582011 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.581968    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-217.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 16:47:43.582011 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.581978    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-217.ec2.internal" event="NodeHasSufficientPID"
Jul 07 16:47:43.582447 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.582426    2578 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 07 16:47:43.582447 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.582432    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 16:47:43.582447 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.582440    2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 07 16:47:43.582590 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.582459    2578 state_mem.go:36] "Initialized new in-memory state store"
Jul 07 16:47:43.583079 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.583066    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-217.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 16:47:43.583126 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.583089    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-217.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 16:47:43.583126 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.583099    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-217.ec2.internal" event="NodeHasSufficientPID"
Jul 07 16:47:43.583984 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.583974    2578 policy_none.go:49] "None policy: Start"
Jul 07 16:47:43.584018 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.583989    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 07 16:47:43.584018 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.583998    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 07 16:47:43.584591 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:43.584525    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-217.ec2.internal.18c00fd5d9020630  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-217.ec2.internal,UID:ip-10-0-136-217.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-217.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-217.ec2.internal,},FirstTimestamp:2026-07-07 16:47:43.581955632 +0000 UTC m=+0.358269830,LastTimestamp:2026-07-07 16:47:43.581955632 +0000 UTC m=+0.358269830,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-217.ec2.internal,}"
Jul 07 16:47:43.585045 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.585034    2578 policy_none.go:47] "Start"
Jul 07 16:47:43.592816 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:43.592723    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-217.ec2.internal.18c00fd5d90249e8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-217.ec2.internal,UID:ip-10-0-136-217.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-136-217.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-136-217.ec2.internal,},FirstTimestamp:2026-07-07 16:47:43.581972968 +0000 UTC m=+0.358287164,LastTimestamp:2026-07-07 16:47:43.581972968 +0000 UTC m=+0.358287164,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-217.ec2.internal,}"
Jul 07 16:47:43.595031 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:43.594973    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-217.ec2.internal.18c00fd5d9026c5c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-217.ec2.internal,UID:ip-10-0-136-217.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-136-217.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-136-217.ec2.internal,},FirstTimestamp:2026-07-07 16:47:43.581981788 +0000 UTC m=+0.358295985,LastTimestamp:2026-07-07 16:47:43.581981788 +0000 UTC m=+0.358295985,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-217.ec2.internal,}"
Jul 07 16:47:43.604180 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:43.604115    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-136-217.ec2.internal.18c00fd5d9020630\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-217.ec2.internal.18c00fd5d9020630  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-217.ec2.internal,UID:ip-10-0-136-217.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-217.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-217.ec2.internal,},FirstTimestamp:2026-07-07 16:47:43.581955632 +0000 UTC m=+0.358269830,LastTimestamp:2026-07-07 16:47:43.583078938 +0000 UTC m=+0.359393136,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-217.ec2.internal,}"
Jul 07 16:47:43.612590 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:43.612536    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-136-217.ec2.internal.18c00fd5d90249e8\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-217.ec2.internal.18c00fd5d90249e8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-217.ec2.internal,UID:ip-10-0-136-217.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-136-217.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-136-217.ec2.internal,},FirstTimestamp:2026-07-07 16:47:43.581972968 +0000 UTC m=+0.358287164,LastTimestamp:2026-07-07 16:47:43.583093786 +0000 UTC m=+0.359407983,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-217.ec2.internal,}"
Jul 07 16:47:43.615709 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:43.615653    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-136-217.ec2.internal.18c00fd5d9026c5c\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-217.ec2.internal.18c00fd5d9026c5c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-217.ec2.internal,UID:ip-10-0-136-217.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-136-217.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-136-217.ec2.internal,},FirstTimestamp:2026-07-07 16:47:43.581981788 +0000 UTC m=+0.358295985,LastTimestamp:2026-07-07 16:47:43.583103021 +0000 UTC m=+0.359417217,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-217.ec2.internal,}"
Jul 07 16:47:43.623183 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.623164    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gw56n"
Jul 07 16:47:43.625331 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.625319    2578 manager.go:337] "Starting Device Plugin manager"
Jul 07 16:47:43.625374 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:43.625348    2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 07 16:47:43.625374 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.625358    2578 server.go:87] "Starting device plugin registration server"
Jul 07 16:47:43.625597 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.625586    2578 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 07 16:47:43.625647 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.625601    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 07 16:47:43.625715 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.625695    2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 07 16:47:43.625796 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.625784    2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 07 16:47:43.625796 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.625794    2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 07 16:47:43.626255 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:43.626235    2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 07 16:47:43.626334 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:43.626291    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-217.ec2.internal\" not found"
Jul 07 16:47:43.630902 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:43.630848    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-217.ec2.internal.18c00fd5dbb81e39  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-217.ec2.internal,UID:ip-10-0-136-217.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-136-217.ec2.internal,},FirstTimestamp:2026-07-07 16:47:43.627443769 +0000 UTC m=+0.403757953,LastTimestamp:2026-07-07 16:47:43.627443769 +0000 UTC m=+0.403757953,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-217.ec2.internal,}"
Jul 07 16:47:43.640834 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.640811    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gw56n"
Jul 07 16:47:43.718330 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.718289    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 07 16:47:43.719534 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.719514    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 07 16:47:43.719586 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.719561    2578 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 07 16:47:43.719586 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.719585    2578 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 07 16:47:43.719659 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:43.719619    2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 07 16:47:43.726299 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.726278    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 16:47:43.726998 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.726984    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-217.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 16:47:43.727066 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.727012    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-217.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 16:47:43.727066 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.727022    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-217.ec2.internal" event="NodeHasSufficientPID"
Jul 07 16:47:43.727066 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.727044    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-217.ec2.internal"
Jul 07 16:47:43.735528 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.735480    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 16:47:43.743147 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.743134    2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-217.ec2.internal"
Jul 07 16:47:43.743217 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:43.743157    2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-217.ec2.internal\": node \"ip-10-0-136-217.ec2.internal\" not found"
Jul 07 16:47:43.760219 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:43.760201    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-217.ec2.internal\" not found"
Jul 07 16:47:43.820441 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.820419    2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-217.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-217.ec2.internal"]
Jul 07 16:47:43.820505 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.820493    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 16:47:43.821973 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.821957    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-217.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 16:47:43.822025 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.821987    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-217.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 16:47:43.822025 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.821997    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-217.ec2.internal" event="NodeHasSufficientPID"
Jul 07 16:47:43.823140 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.823127    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 16:47:43.823263 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.823249    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-217.ec2.internal"
Jul 07 16:47:43.823324 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.823285    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 16:47:43.823821 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.823799    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-217.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 16:47:43.823899 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.823828    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-217.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 16:47:43.823899 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.823842    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-217.ec2.internal" event="NodeHasSufficientPID"
Jul 07 16:47:43.824435 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.824420    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-217.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 16:47:43.824499 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.824446    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-217.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 16:47:43.824499 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.824457    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-217.ec2.internal" event="NodeHasSufficientPID"
Jul 07 16:47:43.825599 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.825583    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-217.ec2.internal"
Jul 07 16:47:43.825699 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.825610    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 16:47:43.826278 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.826259    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-217.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 16:47:43.826382 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.826288    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-217.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 16:47:43.826382 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.826302    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-217.ec2.internal" event="NodeHasSufficientPID"
Jul 07 16:47:43.839273 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:43.839255    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-217.ec2.internal\" not found" node="ip-10-0-136-217.ec2.internal"
Jul 07 16:47:43.843557 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:43.843543    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-217.ec2.internal\" not found" node="ip-10-0-136-217.ec2.internal"
Jul 07 16:47:43.860458 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:43.860443    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-217.ec2.internal\" not found"
Jul 07 16:47:43.864758 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.864744    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d3cfd0e1ebd8169b57d9e712a094a397-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-217.ec2.internal\" (UID: \"d3cfd0e1ebd8169b57d9e712a094a397\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-217.ec2.internal"
Jul 07 16:47:43.864820 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.864770    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3cfd0e1ebd8169b57d9e712a094a397-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-217.ec2.internal\" (UID: \"d3cfd0e1ebd8169b57d9e712a094a397\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-217.ec2.internal"
Jul 07 16:47:43.864820 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.864789    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e0242a47c8f42188a150225d6e13e270-config\") pod \"kube-apiserver-proxy-ip-10-0-136-217.ec2.internal\" (UID: \"e0242a47c8f42188a150225d6e13e270\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-217.ec2.internal"
Jul 07 16:47:43.961067 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:43.961044    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-217.ec2.internal\" not found"
Jul 07 16:47:43.965331 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.965315    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d3cfd0e1ebd8169b57d9e712a094a397-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-217.ec2.internal\" (UID: \"d3cfd0e1ebd8169b57d9e712a094a397\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-217.ec2.internal"
Jul 07 16:47:43.965399 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.965369    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d3cfd0e1ebd8169b57d9e712a094a397-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-217.ec2.internal\" (UID: \"d3cfd0e1ebd8169b57d9e712a094a397\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-217.ec2.internal"
Jul 07 16:47:43.965431 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.965397    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3cfd0e1ebd8169b57d9e712a094a397-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-217.ec2.internal\" (UID: \"d3cfd0e1ebd8169b57d9e712a094a397\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-217.ec2.internal"
Jul 07 16:47:43.965431 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.965424    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e0242a47c8f42188a150225d6e13e270-config\") pod \"kube-apiserver-proxy-ip-10-0-136-217.ec2.internal\" (UID: \"e0242a47c8f42188a150225d6e13e270\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-217.ec2.internal"
Jul 07 16:47:43.965485 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.965455    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e0242a47c8f42188a150225d6e13e270-config\") pod \"kube-apiserver-proxy-ip-10-0-136-217.ec2.internal\" (UID: \"e0242a47c8f42188a150225d6e13e270\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-217.ec2.internal"
Jul 07 16:47:43.965516 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:43.965486    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3cfd0e1ebd8169b57d9e712a094a397-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-217.ec2.internal\" (UID: \"d3cfd0e1ebd8169b57d9e712a094a397\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-217.ec2.internal"
Jul 07 16:47:44.061797 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:44.061702    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-217.ec2.internal\" not found"
Jul 07 16:47:44.143094 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:44.143069    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-217.ec2.internal"
Jul 07 16:47:44.147459 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:44.147384    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-217.ec2.internal"
Jul 07 16:47:44.162795 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:44.162774    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-217.ec2.internal\" not found"
Jul 07 16:47:44.263192 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:44.263155    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-217.ec2.internal\" not found"
Jul 07 16:47:44.363571 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:44.363501    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-217.ec2.internal\" not found"
Jul 07 16:47:44.464028 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:44.464001    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-217.ec2.internal\" not found"
Jul 07 16:47:44.502293 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:44.502268    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 07 16:47:44.502728 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:44.502435    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 07 16:47:44.564085 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:44.564061    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-217.ec2.internal\" not found"
Jul 07 16:47:44.570102 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:44.570086    2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 16:47:44.625422 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:44.625399    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3cfd0e1ebd8169b57d9e712a094a397.slice/crio-f804a8577349ae7ca1823ec3cc15971b48f46dfd2c585b4c32fd9010718a8ab0 WatchSource:0}: Error finding container f804a8577349ae7ca1823ec3cc15971b48f46dfd2c585b4c32fd9010718a8ab0: Status 404 returned error can't find the container with id f804a8577349ae7ca1823ec3cc15971b48f46dfd2c585b4c32fd9010718a8ab0
Jul 07 16:47:44.625616 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:44.625596    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0242a47c8f42188a150225d6e13e270.slice/crio-18db7977196a368181606e1b3b81a378d592464397ae2e94b424623e4536b071 WatchSource:0}: Error finding container 18db7977196a368181606e1b3b81a378d592464397ae2e94b424623e4536b071: Status 404 returned error can't find the container with id 18db7977196a368181606e1b3b81a378d592464397ae2e94b424623e4536b071
Jul 07 16:47:44.629844 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:44.629832    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 16:47:44.645103 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:44.645071    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-06 16:42:43 +0000 UTC" deadline="2028-04-12 05:28:46.77751811 +0000 UTC"
Jul 07 16:47:44.645103 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:44.645102    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15468h41m2.132419068s"
Jul 07 16:47:44.663202 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:44.663183    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-217.ec2.internal"
Jul 07 16:47:44.677268 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:44.677252    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 07 16:47:44.678708 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:44.678697    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-217.ec2.internal"
Jul 07 16:47:44.687737 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:44.687719    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 07 16:47:44.725538 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:44.725496    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-217.ec2.internal" event={"ID":"e0242a47c8f42188a150225d6e13e270","Type":"ContainerStarted","Data":"18db7977196a368181606e1b3b81a378d592464397ae2e94b424623e4536b071"}
Jul 07 16:47:44.726395 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:44.726374    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-217.ec2.internal" event={"ID":"d3cfd0e1ebd8169b57d9e712a094a397","Type":"ContainerStarted","Data":"f804a8577349ae7ca1823ec3cc15971b48f46dfd2c585b4c32fd9010718a8ab0"}
Jul 07 16:47:44.973451 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:44.973378    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 16:47:45.068688 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.068659    2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 16:47:45.544000 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.543948    2578 apiserver.go:52] "Watching apiserver"
Jul 07 16:47:45.551828 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.551801    2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 07 16:47:45.553071 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.553046    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-86jhq","openshift-multus/multus-p92bt","openshift-network-diagnostics/network-check-target-wwb84","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pqq66","openshift-dns/node-resolver-f9r9j","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-217.ec2.internal","openshift-multus/network-metrics-daemon-kdcss","openshift-network-operator/iptables-alerter-llvkg","openshift-ovn-kubernetes/ovnkube-node-bjtgg","kube-system/global-pull-secret-syncer-h5f7f","kube-system/konnectivity-agent-snjp4","kube-system/kube-apiserver-proxy-ip-10-0-136-217.ec2.internal","openshift-cluster-node-tuning-operator/tuned-bvj2w","openshift-image-registry/node-ca-dbsvg"]
Jul 07 16:47:45.554851 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.554826    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdcss"
Jul 07 16:47:45.554954 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:45.554910    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdcss" podUID="91e42c9a-31b0-42bf-9055-28fd68ab4ba9"
Jul 07 16:47:45.556017 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.555999    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bvj2w"
Jul 07 16:47:45.557383 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.557290    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wwb84"
Jul 07 16:47:45.557383 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:45.557347    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wwb84" podUID="57008745-859f-49ec-9269-7671fa8164a2"
Jul 07 16:47:45.558391 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.558372    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 07 16:47:45.558474 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.558393    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-5gdtl\""
Jul 07 16:47:45.558474 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.558380    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 07 16:47:45.558898 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.558879    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pqq66"
Jul 07 16:47:45.560280 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.560202    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-f9r9j"
Jul 07 16:47:45.561109 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.561013    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 07 16:47:45.561109 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.561086    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 07 16:47:45.561252 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.561215    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 07 16:47:45.561582 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.561354    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dbsvg"
Jul 07 16:47:45.561582 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.561445    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jmj9k\""
Jul 07 16:47:45.562447 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.562431    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5khqq\""
Jul 07 16:47:45.562720 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.562678    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 07 16:47:45.562720 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.562713    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 07 16:47:45.562859 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.562762    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:47:45.562859 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.562840    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 07 16:47:45.564880 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.564824    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 07 16:47:45.564972 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.564900    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-86jhq"
Jul 07 16:47:45.565032 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.564982    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 07 16:47:45.565176 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.564982    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 07 16:47:45.565539 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.565436    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pktdb\""
Jul 07 16:47:45.565816 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.565799    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 07 16:47:45.565912 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.565853    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 07 16:47:45.570341 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.569312    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 07 16:47:45.570341 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.569719    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 07 16:47:45.570341 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.569758    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 07 16:47:45.570341 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.569803    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-p92bt"
Jul 07 16:47:45.570341 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.569819    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 07 16:47:45.570341 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.570004    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7lhcz\""
Jul 07 16:47:45.570341 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.570032    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 07 16:47:45.570341 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.570115    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 07 16:47:45.570341 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.569722    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 07 16:47:45.570341 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.570226    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 07 16:47:45.570341 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.570342    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 07 16:47:45.570871 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.570407    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bpnr5\""
Jul 07 16:47:45.570871 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.570475    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h5f7f"
Jul 07 16:47:45.570871 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:45.570547    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h5f7f" podUID="816e0df7-c9cf-4303-a920-a90a16ee015c"
Jul 07 16:47:45.572035 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.571497    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/26ecad8c-d52d-4d74-b5c1-0fdba353728e-etc-openvswitch\") pod \"ovnkube-node-bjtgg\" (UID: \"26ecad8c-d52d-4d74-b5c1-0fdba353728e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:47:45.572035 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.571535    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/26ecad8c-d52d-4d74-b5c1-0fdba353728e-log-socket\") pod \"ovnkube-node-bjtgg\" (UID: \"26ecad8c-d52d-4d74-b5c1-0fdba353728e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:47:45.572035 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.571584    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/26ecad8c-d52d-4d74-b5c1-0fdba353728e-host-cni-bin\") pod \"ovnkube-node-bjtgg\" (UID: \"26ecad8c-d52d-4d74-b5c1-0fdba353728e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:47:45.572035 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.571615    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca-system-cni-dir\") pod \"multus-additional-cni-plugins-86jhq\" (UID: \"40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca\") " pod="openshift-multus/multus-additional-cni-plugins-86jhq"
Jul 07 16:47:45.572035 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.571645    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/60d7869e-bef8-4020-b685-5f597d1031b9-serviceca\") pod \"node-ca-dbsvg\" (UID: \"60d7869e-bef8-4020-b685-5f597d1031b9\") " pod="openshift-image-registry/node-ca-dbsvg"
Jul 07 16:47:45.572035 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.571679    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhj26\" (UniqueName: \"kubernetes.io/projected/91e42c9a-31b0-42bf-9055-28fd68ab4ba9-kube-api-access-mhj26\") pod \"network-metrics-daemon-kdcss\" (UID: \"91e42c9a-31b0-42bf-9055-28fd68ab4ba9\") " pod="openshift-multus/network-metrics-daemon-kdcss"
Jul 07 16:47:45.572035 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.571712    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/26ecad8c-d52d-4d74-b5c1-0fdba353728e-ovnkube-script-lib\") pod \"ovnkube-node-bjtgg\" (UID: \"26ecad8c-d52d-4d74-b5c1-0fdba353728e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:47:45.572035 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.571756    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/82568637-39b9-4e53-a44f-6c7148d449a0-run\") pod \"tuned-bvj2w\" (UID: \"82568637-39b9-4e53-a44f-6c7148d449a0\") " pod="openshift-cluster-node-tuning-operator/tuned-bvj2w"
Jul 07 16:47:45.572035 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.571785    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1cc86643-1ed6-4849-ad30-923acbf45ab7-hosts-file\") pod \"node-resolver-f9r9j\" (UID: \"1cc86643-1ed6-4849-ad30-923acbf45ab7\") " pod="openshift-dns/node-resolver-f9r9j"
Jul 07 16:47:45.572035 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.571815    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8f86\" (UniqueName: \"kubernetes.io/projected/60d7869e-bef8-4020-b685-5f597d1031b9-kube-api-access-f8f86\") pod \"node-ca-dbsvg\" (UID: \"60d7869e-bef8-4020-b685-5f597d1031b9\") " pod="openshift-image-registry/node-ca-dbsvg"
Jul 07 16:47:45.572035 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.571860    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/26ecad8c-d52d-4d74-b5c1-0fdba353728e-host-slash\") pod \"ovnkube-node-bjtgg\" (UID: \"26ecad8c-d52d-4d74-b5c1-0fdba353728e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:47:45.572035 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.571909    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/26ecad8c-d52d-4d74-b5c1-0fdba353728e-var-lib-openvswitch\") pod \"ovnkube-node-bjtgg\" (UID: \"26ecad8c-d52d-4d74-b5c1-0fdba353728e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:47:45.572035 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.571972    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/82568637-39b9-4e53-a44f-6c7148d449a0-etc-systemd\") pod \"tuned-bvj2w\" (UID: \"82568637-39b9-4e53-a44f-6c7148d449a0\") " pod="openshift-cluster-node-tuning-operator/tuned-bvj2w"
Jul 07 16:47:45.572035 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.572007    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/82568637-39b9-4e53-a44f-6c7148d449a0-var-lib-kubelet\") pod \"tuned-bvj2w\" (UID: \"82568637-39b9-4e53-a44f-6c7148d449a0\") " pod="openshift-cluster-node-tuning-operator/tuned-bvj2w"
Jul 07 16:47:45.572748 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.572181    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 07 16:47:45.572748 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.572351    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-snjp4"
Jul 07 16:47:45.572849 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.572800    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-795k2\""
Jul 07 16:47:45.573788 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.573389    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/82568637-39b9-4e53-a44f-6c7148d449a0-etc-kubernetes\") pod \"tuned-bvj2w\" (UID: \"82568637-39b9-4e53-a44f-6c7148d449a0\") " pod="openshift-cluster-node-tuning-operator/tuned-bvj2w"
Jul 07 16:47:45.573788 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.573425    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v9tq\" (UniqueName: \"kubernetes.io/projected/40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca-kube-api-access-7v9tq\") pod \"multus-additional-cni-plugins-86jhq\" (UID: \"40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca\") " pod="openshift-multus/multus-additional-cni-plugins-86jhq"
Jul 07 16:47:45.573788 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.573470    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ca3370c2-512b-41bb-a09e-b26cac03572d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pqq66\" (UID: \"ca3370c2-512b-41bb-a09e-b26cac03572d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pqq66"
Jul 07 16:47:45.573788 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.573511    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ca3370c2-512b-41bb-a09e-b26cac03572d-device-dir\") pod \"aws-ebs-csi-driver-node-pqq66\" (UID: \"ca3370c2-512b-41bb-a09e-b26cac03572d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pqq66"
Jul 07 16:47:45.573788 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.573539    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/82568637-39b9-4e53-a44f-6c7148d449a0-etc-sysctl-d\") pod \"tuned-bvj2w\" (UID: \"82568637-39b9-4e53-a44f-6c7148d449a0\") " pod="openshift-cluster-node-tuning-operator/tuned-bvj2w"
Jul 07 16:47:45.573788 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.573568    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/26ecad8c-d52d-4d74-b5c1-0fdba353728e-run-ovn\") pod \"ovnkube-node-bjtgg\" (UID: \"26ecad8c-d52d-4d74-b5c1-0fdba353728e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:47:45.573788 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.573598    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26ecad8c-d52d-4d74-b5c1-0fdba353728e-host-run-ovn-kubernetes\") pod \"ovnkube-node-bjtgg\" (UID: \"26ecad8c-d52d-4d74-b5c1-0fdba353728e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:47:45.573788 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.573626    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/82568637-39b9-4e53-a44f-6c7148d449a0-etc-modprobe-d\") pod \"tuned-bvj2w\" (UID: \"82568637-39b9-4e53-a44f-6c7148d449a0\") " pod="openshift-cluster-node-tuning-operator/tuned-bvj2w"
Jul 07 16:47:45.573788 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.573650    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/82568637-39b9-4e53-a44f-6c7148d449a0-etc-sysconfig\") pod \"tuned-bvj2w\" (UID: \"82568637-39b9-4e53-a44f-6c7148d449a0\") " pod="openshift-cluster-node-tuning-operator/tuned-bvj2w"
Jul 07 16:47:45.573788 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.573684    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/82568637-39b9-4e53-a44f-6c7148d449a0-etc-tuned\") pod \"tuned-bvj2w\" (UID: \"82568637-39b9-4e53-a44f-6c7148d449a0\") " pod="openshift-cluster-node-tuning-operator/tuned-bvj2w"
Jul 07 16:47:45.573788 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.573783    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/26ecad8c-d52d-4d74-b5c1-0fdba353728e-systemd-units\") pod \"ovnkube-node-bjtgg\" (UID: \"26ecad8c-d52d-4d74-b5c1-0fdba353728e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:47:45.574349 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.573809    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/26ecad8c-d52d-4d74-b5c1-0fdba353728e-host-run-netns\") pod \"ovnkube-node-bjtgg\" (UID: \"26ecad8c-d52d-4d74-b5c1-0fdba353728e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:47:45.574349 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.573831    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnd5w\" (UniqueName: \"kubernetes.io/projected/26ecad8c-d52d-4d74-b5c1-0fdba353728e-kube-api-access-rnd5w\") pod \"ovnkube-node-bjtgg\" (UID: \"26ecad8c-d52d-4d74-b5c1-0fdba353728e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:47:45.574349 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.573852    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/82568637-39b9-4e53-a44f-6c7148d449a0-host\") pod \"tuned-bvj2w\" (UID: \"82568637-39b9-4e53-a44f-6c7148d449a0\") " pod="openshift-cluster-node-tuning-operator/tuned-bvj2w"
Jul 07 16:47:45.574349 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.573873    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-86jhq\" (UID: \"40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca\") " pod="openshift-multus/multus-additional-cni-plugins-86jhq"
Jul 07 16:47:45.574349 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.573895    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/26ecad8c-d52d-4d74-b5c1-0fdba353728e-ovn-node-metrics-cert\") pod \"ovnkube-node-bjtgg\" (UID: \"26ecad8c-d52d-4d74-b5c1-0fdba353728e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:47:45.574349 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.573915    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/82568637-39b9-4e53-a44f-6c7148d449a0-etc-sysctl-conf\") pod \"tuned-bvj2w\" (UID: \"82568637-39b9-4e53-a44f-6c7148d449a0\") " pod="openshift-cluster-node-tuning-operator/tuned-bvj2w"
Jul 07 16:47:45.574349 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.573936    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26ecad8c-d52d-4d74-b5c1-0fdba353728e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bjtgg\" (UID: \"26ecad8c-d52d-4d74-b5c1-0fdba353728e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:47:45.574349 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.573956    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82568637-39b9-4e53-a44f-6c7148d449a0-tmp\") pod \"tuned-bvj2w\" (UID: \"82568637-39b9-4e53-a44f-6c7148d449a0\") " pod="openshift-cluster-node-tuning-operator/tuned-bvj2w"
Jul 07 16:47:45.574349 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.573978    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca-tuning-conf-dir\") pod \"multus-additional-cni-plugins-86jhq\" (UID: \"40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca\") " pod="openshift-multus/multus-additional-cni-plugins-86jhq"
Jul 07 16:47:45.574349 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.574005    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91e42c9a-31b0-42bf-9055-28fd68ab4ba9-metrics-certs\") pod \"network-metrics-daemon-kdcss\" (UID: \"91e42c9a-31b0-42bf-9055-28fd68ab4ba9\") " pod="openshift-multus/network-metrics-daemon-kdcss"
Jul 07 16:47:45.574349 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.574025    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ca3370c2-512b-41bb-a09e-b26cac03572d-registration-dir\") pod \"aws-ebs-csi-driver-node-pqq66\" (UID: \"ca3370c2-512b-41bb-a09e-b26cac03572d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pqq66"
Jul 07 16:47:45.574349 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.574045    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/26ecad8c-d52d-4d74-b5c1-0fdba353728e-host-kubelet\") pod \"ovnkube-node-bjtgg\" (UID: \"26ecad8c-d52d-4d74-b5c1-0fdba353728e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:47:45.574349 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.574066    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/26ecad8c-d52d-4d74-b5c1-0fdba353728e-node-log\") pod \"ovnkube-node-bjtgg\" (UID: \"26ecad8c-d52d-4d74-b5c1-0fdba353728e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:47:45.574349 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.574086    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxlds\" (UniqueName: \"kubernetes.io/projected/1cc86643-1ed6-4849-ad30-923acbf45ab7-kube-api-access-mxlds\") pod \"node-resolver-f9r9j\" (UID: \"1cc86643-1ed6-4849-ad30-923acbf45ab7\") " pod="openshift-dns/node-resolver-f9r9j"
Jul 07 16:47:45.574349 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.574109    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86hrs\" (UniqueName: \"kubernetes.io/projected/ca3370c2-512b-41bb-a09e-b26cac03572d-kube-api-access-86hrs\") pod \"aws-ebs-csi-driver-node-pqq66\" (UID: \"ca3370c2-512b-41bb-a09e-b26cac03572d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pqq66"
Jul 07 16:47:45.574349 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.574129    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/26ecad8c-d52d-4d74-b5c1-0fdba353728e-run-openvswitch\") pod \"ovnkube-node-bjtgg\" (UID: \"26ecad8c-d52d-4d74-b5c1-0fdba353728e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:47:45.575151 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.574149    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/82568637-39b9-4e53-a44f-6c7148d449a0-sys\") pod \"tuned-bvj2w\" (UID: \"82568637-39b9-4e53-a44f-6c7148d449a0\") " pod="openshift-cluster-node-tuning-operator/tuned-bvj2w"
Jul 07 16:47:45.575151 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.574173    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcwrg\" (UniqueName: \"kubernetes.io/projected/82568637-39b9-4e53-a44f-6c7148d449a0-kube-api-access-jcwrg\") pod \"tuned-bvj2w\" (UID: \"82568637-39b9-4e53-a44f-6c7148d449a0\") " pod="openshift-cluster-node-tuning-operator/tuned-bvj2w"
Jul 07 16:47:45.575151 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.574212    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-86jhq\" (UID: \"40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca\") " pod="openshift-multus/multus-additional-cni-plugins-86jhq"
Jul 07 16:47:45.575151 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.574235    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/60d7869e-bef8-4020-b685-5f597d1031b9-host\") pod \"node-ca-dbsvg\" (UID: \"60d7869e-bef8-4020-b685-5f597d1031b9\") " pod="openshift-image-registry/node-ca-dbsvg"
Jul 07 16:47:45.575151 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.574256    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/26ecad8c-d52d-4d74-b5c1-0fdba353728e-ovnkube-config\") pod \"ovnkube-node-bjtgg\" (UID: \"26ecad8c-d52d-4d74-b5c1-0fdba353728e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:47:45.575151 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.574277    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpnrf\" (UniqueName: \"kubernetes.io/projected/57008745-859f-49ec-9269-7671fa8164a2-kube-api-access-zpnrf\") pod \"network-check-target-wwb84\" (UID: \"57008745-859f-49ec-9269-7671fa8164a2\") " pod="openshift-network-diagnostics/network-check-target-wwb84"
Jul 07 16:47:45.575151 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.574317    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ca3370c2-512b-41bb-a09e-b26cac03572d-etc-selinux\") pod \"aws-ebs-csi-driver-node-pqq66\" (UID: \"ca3370c2-512b-41bb-a09e-b26cac03572d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pqq66"
Jul 07 16:47:45.575151 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.574338    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ca3370c2-512b-41bb-a09e-b26cac03572d-sys-fs\") pod \"aws-ebs-csi-driver-node-pqq66\" (UID: \"ca3370c2-512b-41bb-a09e-b26cac03572d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pqq66"
Jul 07 16:47:45.575151 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.574358    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/26ecad8c-d52d-4d74-b5c1-0fdba353728e-run-systemd\") pod \"ovnkube-node-bjtgg\" (UID: \"26ecad8c-d52d-4d74-b5c1-0fdba353728e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:47:45.575151 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.574379    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/26ecad8c-d52d-4d74-b5c1-0fdba353728e-host-cni-netd\") pod \"ovnkube-node-bjtgg\" (UID: \"26ecad8c-d52d-4d74-b5c1-0fdba353728e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:47:45.575151 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.574404    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/26ecad8c-d52d-4d74-b5c1-0fdba353728e-env-overrides\") pod \"ovnkube-node-bjtgg\" (UID: \"26ecad8c-d52d-4d74-b5c1-0fdba353728e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:47:45.575151 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.574424    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ca3370c2-512b-41bb-a09e-b26cac03572d-socket-dir\") pod \"aws-ebs-csi-driver-node-pqq66\" (UID: \"ca3370c2-512b-41bb-a09e-b26cac03572d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pqq66"
Jul 07 16:47:45.575151 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.574443    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cc86643-1ed6-4849-ad30-923acbf45ab7-tmp-dir\") pod \"node-resolver-f9r9j\" (UID: \"1cc86643-1ed6-4849-ad30-923acbf45ab7\") " pod="openshift-dns/node-resolver-f9r9j"
Jul 07 16:47:45.575151 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.574464    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca-cnibin\") pod \"multus-additional-cni-plugins-86jhq\" (UID: \"40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca\") " pod="openshift-multus/multus-additional-cni-plugins-86jhq"
Jul 07 16:47:45.575151 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.574502    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca-os-release\") pod \"multus-additional-cni-plugins-86jhq\" (UID: \"40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca\") " pod="openshift-multus/multus-additional-cni-plugins-86jhq"
Jul 07 16:47:45.575151 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.574538    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/82568637-39b9-4e53-a44f-6c7148d449a0-lib-modules\") pod \"tuned-bvj2w\" (UID: \"82568637-39b9-4e53-a44f-6c7148d449a0\") " pod="openshift-cluster-node-tuning-operator/tuned-bvj2w"
Jul 07 16:47:45.575914 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.574565    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca-cni-binary-copy\") pod \"multus-additional-cni-plugins-86jhq\" (UID: \"40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca\") " pod="openshift-multus/multus-additional-cni-plugins-86jhq"
Jul 07 16:47:45.576489 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.576048    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-m48fk\""
Jul 07 16:47:45.576489 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.576120    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 07 16:47:45.576489 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.576140    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 07 16:47:45.577641 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.577420    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-llvkg"
Jul 07 16:47:45.578419 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.578399    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 07 16:47:45.579825 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.579626    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 07 16:47:45.579825 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.579695    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 07 16:47:45.579958 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.579894    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 07 16:47:45.580078 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.580029    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ql8q6\""
Jul 07 16:47:45.595080 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.595065    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2gngl"
Jul 07 16:47:45.602850 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.602834    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2gngl"
Jul 07 16:47:45.664604 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.664580    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 07 16:47:45.675131 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.675104    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/82568637-39b9-4e53-a44f-6c7148d449a0-etc-systemd\") pod \"tuned-bvj2w\" (UID: \"82568637-39b9-4e53-a44f-6c7148d449a0\") " pod="openshift-cluster-node-tuning-operator/tuned-bvj2w"
Jul 07 16:47:45.675250 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.675166    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/82568637-39b9-4e53-a44f-6c7148d449a0-var-lib-kubelet\") pod \"tuned-bvj2w\" (UID: \"82568637-39b9-4e53-a44f-6c7148d449a0\") " pod="openshift-cluster-node-tuning-operator/tuned-bvj2w"
Jul 07 16:47:45.675250 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.675186    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/82568637-39b9-4e53-a44f-6c7148d449a0-etc-kubernetes\") pod \"tuned-bvj2w\" (UID: \"82568637-39b9-4e53-a44f-6c7148d449a0\") " pod="openshift-cluster-node-tuning-operator/tuned-bvj2w"
Jul 07 16:47:45.675250 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.675207    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v9tq\" (UniqueName: \"kubernetes.io/projected/40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca-kube-api-access-7v9tq\") pod \"multus-additional-cni-plugins-86jhq\" (UID: \"40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca\") " pod="openshift-multus/multus-additional-cni-plugins-86jhq"
Jul 07 16:47:45.675250 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.675210    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/82568637-39b9-4e53-a44f-6c7148d449a0-etc-systemd\") pod \"tuned-bvj2w\" (UID: \"82568637-39b9-4e53-a44f-6c7148d449a0\") " pod="openshift-cluster-node-tuning-operator/tuned-bvj2w"
Jul 07 16:47:45.675250 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.675235    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0bb06482-d151-4a1f-8e49-9101ad901e7a-iptables-alerter-script\") pod \"iptables-alerter-llvkg\" (UID: \"0bb06482-d151-4a1f-8e49-9101ad901e7a\") " pod="openshift-network-operator/iptables-alerter-llvkg"
Jul 07 16:47:45.675475 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.675266    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/82568637-39b9-4e53-a44f-6c7148d449a0-var-lib-kubelet\") pod \"tuned-bvj2w\" (UID: \"82568637-39b9-4e53-a44f-6c7148d449a0\") " pod="openshift-cluster-node-tuning-operator/tuned-bvj2w"
Jul 07 16:47:45.675475 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.675284    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/82568637-39b9-4e53-a44f-6c7148d449a0-etc-kubernetes\") pod \"tuned-bvj2w\" (UID: \"82568637-39b9-4e53-a44f-6c7148d449a0\") " pod="openshift-cluster-node-tuning-operator/tuned-bvj2w"
Jul 07 16:47:45.675475 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.675296    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ca3370c2-512b-41bb-a09e-b26cac03572d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pqq66\" (UID: \"ca3370c2-512b-41bb-a09e-b26cac03572d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pqq66"
Jul 07 16:47:45.675475 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.675326    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ca3370c2-512b-41bb-a09e-b26cac03572d-device-dir\") pod \"aws-ebs-csi-driver-node-pqq66\" (UID: \"ca3370c2-512b-41bb-a09e-b26cac03572d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pqq66"
Jul 07 16:47:45.675475 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.675348    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/82568637-39b9-4e53-a44f-6c7148d449a0-etc-sysctl-d\") pod \"tuned-bvj2w\" (UID: \"82568637-39b9-4e53-a44f-6c7148d449a0\") " pod="openshift-cluster-node-tuning-operator/tuned-bvj2w"
Jul 07 16:47:45.675475 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.675373    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f7e6b596-4e4f-47d2-80e6-900c70180ef5-multus-cni-dir\") pod \"multus-p92bt\" (UID: \"f7e6b596-4e4f-47d2-80e6-900c70180ef5\") " pod="openshift-multus/multus-p92bt"
Jul 07 16:47:45.675475 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.675385    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ca3370c2-512b-41bb-a09e-b26cac03572d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pqq66\" (UID: \"ca3370c2-512b-41bb-a09e-b26cac03572d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pqq66"
Jul 07 16:47:45.675475 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.675394    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/26ecad8c-d52d-4d74-b5c1-0fdba353728e-run-ovn\") pod \"ovnkube-node-bjtgg\" (UID: \"26ecad8c-d52d-4d74-b5c1-0fdba353728e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:47:45.675475 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.675404    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ca3370c2-512b-41bb-a09e-b26cac03572d-device-dir\") pod \"aws-ebs-csi-driver-node-pqq66\" (UID: \"ca3370c2-512b-41bb-a09e-b26cac03572d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pqq66"
Jul 07 16:47:45.675475 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.675429    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26ecad8c-d52d-4d74-b5c1-0fdba353728e-host-run-ovn-kubernetes\") pod \"ovnkube-node-bjtgg\" (UID: \"26ecad8c-d52d-4d74-b5c1-0fdba353728e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:47:45.675475 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.675442    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/26ecad8c-d52d-4d74-b5c1-0fdba353728e-run-ovn\") pod \"ovnkube-node-bjtgg\" (UID: \"26ecad8c-d52d-4d74-b5c1-0fdba353728e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:47:45.675475 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.675452    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/82568637-39b9-4e53-a44f-6c7148d449a0-etc-modprobe-d\") pod \"tuned-bvj2w\" (UID: \"82568637-39b9-4e53-a44f-6c7148d449a0\") " pod="openshift-cluster-node-tuning-operator/tuned-bvj2w"
Jul 07 16:47:45.675475 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.675471    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26ecad8c-d52d-4d74-b5c1-0fdba353728e-host-run-ovn-kubernetes\") pod \"ovnkube-node-bjtgg\" (UID: \"26ecad8c-d52d-4d74-b5c1-0fdba353728e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:47:45.675978 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.675495    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/82568637-39b9-4e53-a44f-6c7148d449a0-etc-sysconfig\") pod \"tuned-bvj2w\" (UID: \"82568637-39b9-4e53-a44f-6c7148d449a0\") " pod="openshift-cluster-node-tuning-operator/tuned-bvj2w"
Jul 07 16:47:45.675978 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.675518    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/82568637-39b9-4e53-a44f-6c7148d449a0-etc-tuned\") pod \"tuned-bvj2w\" (UID: \"82568637-39b9-4e53-a44f-6c7148d449a0\") " pod="openshift-cluster-node-tuning-operator/tuned-bvj2w"
Jul 07 16:47:45.675978 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.675515    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/82568637-39b9-4e53-a44f-6c7148d449a0-etc-sysctl-d\") pod \"tuned-bvj2w\" (UID: \"82568637-39b9-4e53-a44f-6c7148d449a0\") " pod="openshift-cluster-node-tuning-operator/tuned-bvj2w"
Jul 07 16:47:45.675978 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.675542    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7e6b596-4e4f-47d2-80e6-900c70180ef5-etc-kubernetes\") pod \"multus-p92bt\" (UID: \"f7e6b596-4e4f-47d2-80e6-900c70180ef5\") " pod="openshift-multus/multus-p92bt"
Jul 07 16:47:45.675978 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.675565    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/82568637-39b9-4e53-a44f-6c7148d449a0-etc-sysconfig\") pod \"tuned-bvj2w\" (UID: \"82568637-39b9-4e53-a44f-6c7148d449a0\") " pod="openshift-cluster-node-tuning-operator/tuned-bvj2w"
Jul 07 16:47:45.675978 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.675571    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/26ecad8c-d52d-4d74-b5c1-0fdba353728e-systemd-units\") pod \"ovnkube-node-bjtgg\" (UID: \"26ecad8c-d52d-4d74-b5c1-0fdba353728e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:47:45.675978 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.675602    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/26ecad8c-d52d-4d74-b5c1-0fdba353728e-host-run-netns\") pod \"ovnkube-node-bjtgg\" (UID: \"26ecad8c-d52d-4d74-b5c1-0fdba353728e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:47:45.675978 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.675602    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/26ecad8c-d52d-4d74-b5c1-0fdba353728e-systemd-units\") pod \"ovnkube-node-bjtgg\" (UID: \"26ecad8c-d52d-4d74-b5c1-0fdba353728e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:47:45.675978 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.675629    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnd5w\" (UniqueName: \"kubernetes.io/projected/26ecad8c-d52d-4d74-b5c1-0fdba353728e-kube-api-access-rnd5w\") pod \"ovnkube-node-bjtgg\" (UID: \"26ecad8c-d52d-4d74-b5c1-0fdba353728e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:47:45.675978 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.675651    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/82568637-39b9-4e53-a44f-6c7148d449a0-host\") pod \"tuned-bvj2w\" (UID: \"82568637-39b9-4e53-a44f-6c7148d449a0\") " pod="openshift-cluster-node-tuning-operator/tuned-bvj2w"
Jul 07 16:47:45.675978 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.675668    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-86jhq\" (UID: \"40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca\") " pod="openshift-multus/multus-additional-cni-plugins-86jhq"
Jul 07 16:47:45.675978 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.675689    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7e6b596-4e4f-47d2-80e6-900c70180ef5-host-var-lib-cni-bin\") pod \"multus-p92bt\" (UID: \"f7e6b596-4e4f-47d2-80e6-900c70180ef5\") " pod="openshift-multus/multus-p92bt"
Jul 07 16:47:45.675978 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.675720    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/26ecad8c-d52d-4d74-b5c1-0fdba353728e-ovn-node-metrics-cert\") pod \"ovnkube-node-bjtgg\" (UID: \"26ecad8c-d52d-4d74-b5c1-0fdba353728e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:47:45.675978 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.675743    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/82568637-39b9-4e53-a44f-6c7148d449a0-host\") pod \"tuned-bvj2w\" (UID: \"82568637-39b9-4e53-a44f-6c7148d449a0\") " pod="openshift-cluster-node-tuning-operator/tuned-bvj2w"
Jul 07 16:47:45.675978 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.675632    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/26ecad8c-d52d-4d74-b5c1-0fdba353728e-host-run-netns\") pod \"ovnkube-node-bjtgg\" (UID: \"26ecad8c-d52d-4d74-b5c1-0fdba353728e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:47:45.675978 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.675758    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/82568637-39b9-4e53-a44f-6c7148d449a0-etc-sysctl-conf\") pod \"tuned-bvj2w\" (UID: \"82568637-39b9-4e53-a44f-6c7148d449a0\") " pod="openshift-cluster-node-tuning-operator/tuned-bvj2w"
Jul 07 16:47:45.675978 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.675781    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b5aa7217-bfcc-4c09-8d6f-51bb83cce667-konnectivity-ca\") pod \"konnectivity-agent-snjp4\" (UID: \"b5aa7217-bfcc-4c09-8d6f-51bb83cce667\") " pod="kube-system/konnectivity-agent-snjp4"
Jul 07 16:47:45.676714 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.675803    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f7e6b596-4e4f-47d2-80e6-900c70180ef5-host-var-lib-cni-multus\") pod \"multus-p92bt\" (UID: \"f7e6b596-4e4f-47d2-80e6-900c70180ef5\") " pod="openshift-multus/multus-p92bt"
Jul 07 16:47:45.676714 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.675827    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26ecad8c-d52d-4d74-b5c1-0fdba353728e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bjtgg\" (UID: \"26ecad8c-d52d-4d74-b5c1-0fdba353728e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:47:45.676714 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.675839    2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 07 16:47:45.676714 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.675868    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82568637-39b9-4e53-a44f-6c7148d449a0-tmp\") pod \"tuned-bvj2w\" (UID: \"82568637-39b9-4e53-a44f-6c7148d449a0\") " pod="openshift-cluster-node-tuning-operator/tuned-bvj2w"
Jul 07 16:47:45.676714 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.675897    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca-tuning-conf-dir\") pod \"multus-additional-cni-plugins-86jhq\" (UID: \"40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca\") " pod="openshift-multus/multus-additional-cni-plugins-86jhq"
Jul 07 16:47:45.676714 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.675921    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/816e0df7-c9cf-4303-a920-a90a16ee015c-dbus\") pod \"global-pull-secret-syncer-h5f7f\" (UID: \"816e0df7-c9cf-4303-a920-a90a16ee015c\") " pod="kube-system/global-pull-secret-syncer-h5f7f"
Jul 07 16:47:45.676714 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.675944    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmdwc\" (UniqueName: \"kubernetes.io/projected/0bb06482-d151-4a1f-8e49-9101ad901e7a-kube-api-access-qmdwc\") pod \"iptables-alerter-llvkg\" (UID: \"0bb06482-d151-4a1f-8e49-9101ad901e7a\") " pod="openshift-network-operator/iptables-alerter-llvkg"
Jul 07 16:47:45.676714 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.675950    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/82568637-39b9-4e53-a44f-6c7148d449a0-etc-sysctl-conf\") pod \"tuned-bvj2w\" (UID: \"82568637-39b9-4e53-a44f-6c7148d449a0\") " pod="openshift-cluster-node-tuning-operator/tuned-bvj2w"
Jul 07 16:47:45.676714 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.676013    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/82568637-39b9-4e53-a44f-6c7148d449a0-etc-modprobe-d\") pod \"tuned-bvj2w\" (UID: \"82568637-39b9-4e53-a44f-6c7148d449a0\") " pod="openshift-cluster-node-tuning-operator/tuned-bvj2w"
Jul 07 16:47:45.676714 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.676082    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca-tuning-conf-dir\") pod \"multus-additional-cni-plugins-86jhq\" (UID: \"40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca\") " pod="openshift-multus/multus-additional-cni-plugins-86jhq"
Jul 07 16:47:45.676714 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.676290    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-86jhq\" (UID: \"40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca\") " pod="openshift-multus/multus-additional-cni-plugins-86jhq"
Jul 07 16:47:45.676714 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.675973    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f7e6b596-4e4f-47d2-80e6-900c70180ef5-multus-socket-dir-parent\") pod \"multus-p92bt\" (UID: \"f7e6b596-4e4f-47d2-80e6-900c70180ef5\") " pod="openshift-multus/multus-p92bt"
Jul 07 16:47:45.676714 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.676438    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91e42c9a-31b0-42bf-9055-28fd68ab4ba9-metrics-certs\") pod \"network-metrics-daemon-kdcss\" (UID: \"91e42c9a-31b0-42bf-9055-28fd68ab4ba9\") " pod="openshift-multus/network-metrics-daemon-kdcss"
Jul 07 16:47:45.676714 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.676462    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ca3370c2-512b-41bb-a09e-b26cac03572d-registration-dir\") pod \"aws-ebs-csi-driver-node-pqq66\" (UID: \"ca3370c2-512b-41bb-a09e-b26cac03572d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pqq66"
Jul 07 16:47:45.676714 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.676461    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26ecad8c-d52d-4d74-b5c1-0fdba353728e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bjtgg\" (UID: \"26ecad8c-d52d-4d74-b5c1-0fdba353728e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:47:45.676714 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.676476    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/26ecad8c-d52d-4d74-b5c1-0fdba353728e-host-kubelet\") pod \"ovnkube-node-bjtgg\" (UID: \"26ecad8c-d52d-4d74-b5c1-0fdba353728e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:47:45.676714 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.676509    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/26ecad8c-d52d-4d74-b5c1-0fdba353728e-host-kubelet\") pod \"ovnkube-node-bjtgg\" (UID: \"26ecad8c-d52d-4d74-b5c1-0fdba353728e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:47:45.677444 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.676511    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/26ecad8c-d52d-4d74-b5c1-0fdba353728e-node-log\") pod \"ovnkube-node-bjtgg\" (UID: \"26ecad8c-d52d-4d74-b5c1-0fdba353728e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:47:45.677444 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.676545    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxlds\" (UniqueName: \"kubernetes.io/projected/1cc86643-1ed6-4849-ad30-923acbf45ab7-kube-api-access-mxlds\") pod \"node-resolver-f9r9j\" (UID: \"1cc86643-1ed6-4849-ad30-923acbf45ab7\") " pod="openshift-dns/node-resolver-f9r9j"
Jul 07 16:47:45.677444 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.676549    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/26ecad8c-d52d-4d74-b5c1-0fdba353728e-node-log\") pod \"ovnkube-node-bjtgg\" (UID: \"26ecad8c-d52d-4d74-b5c1-0fdba353728e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:47:45.677444 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.676580    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ca3370c2-512b-41bb-a09e-b26cac03572d-registration-dir\") pod \"aws-ebs-csi-driver-node-pqq66\" (UID: \"ca3370c2-512b-41bb-a09e-b26cac03572d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pqq66"
Jul 07 16:47:45.677444 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.676585    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7e6b596-4e4f-47d2-80e6-900c70180ef5-host-run-netns\") pod \"multus-p92bt\" (UID: \"f7e6b596-4e4f-47d2-80e6-900c70180ef5\") " pod="openshift-multus/multus-p92bt"
Jul 07 16:47:45.677444 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:45.676603    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:47:45.677444 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.676611    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f7e6b596-4e4f-47d2-80e6-900c70180ef5-multus-conf-dir\") pod \"multus-p92bt\" (UID: \"f7e6b596-4e4f-47d2-80e6-900c70180ef5\") " pod="openshift-multus/multus-p92bt"
Jul 07 16:47:45.677444 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.676640    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86hrs\" (UniqueName: \"kubernetes.io/projected/ca3370c2-512b-41bb-a09e-b26cac03572d-kube-api-access-86hrs\") pod \"aws-ebs-csi-driver-node-pqq66\" (UID: \"ca3370c2-512b-41bb-a09e-b26cac03572d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pqq66"
Jul 07 16:47:45.677444 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:45.676661    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91e42c9a-31b0-42bf-9055-28fd68ab4ba9-metrics-certs podName:91e42c9a-31b0-42bf-9055-28fd68ab4ba9 nodeName:}" failed. No retries permitted until 2026-07-07 16:47:46.176643019 +0000 UTC m=+2.952957216 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/91e42c9a-31b0-42bf-9055-28fd68ab4ba9-metrics-certs") pod "network-metrics-daemon-kdcss" (UID: "91e42c9a-31b0-42bf-9055-28fd68ab4ba9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:47:45.677444 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.676765    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/26ecad8c-d52d-4d74-b5c1-0fdba353728e-run-openvswitch\") pod \"ovnkube-node-bjtgg\" (UID: \"26ecad8c-d52d-4d74-b5c1-0fdba353728e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:47:45.677444 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.676799    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/82568637-39b9-4e53-a44f-6c7148d449a0-sys\") pod \"tuned-bvj2w\" (UID: \"82568637-39b9-4e53-a44f-6c7148d449a0\") " pod="openshift-cluster-node-tuning-operator/tuned-bvj2w"
Jul 07 16:47:45.677444 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.676840    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcwrg\" (UniqueName: \"kubernetes.io/projected/82568637-39b9-4e53-a44f-6c7148d449a0-kube-api-access-jcwrg\") pod \"tuned-bvj2w\" (UID: \"82568637-39b9-4e53-a44f-6c7148d449a0\") " pod="openshift-cluster-node-tuning-operator/tuned-bvj2w"
Jul 07 16:47:45.677444 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.676844    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/82568637-39b9-4e53-a44f-6c7148d449a0-sys\") pod \"tuned-bvj2w\" (UID: \"82568637-39b9-4e53-a44f-6c7148d449a0\") " pod="openshift-cluster-node-tuning-operator/tuned-bvj2w"
Jul 07 16:47:45.677444 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.676839    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/26ecad8c-d52d-4d74-b5c1-0fdba353728e-run-openvswitch\") pod \"ovnkube-node-bjtgg\" (UID: \"26ecad8c-d52d-4d74-b5c1-0fdba353728e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:47:45.677444 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.676873    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-86jhq\" (UID: \"40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca\") " pod="openshift-multus/multus-additional-cni-plugins-86jhq"
Jul 07 16:47:45.677444 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.676902    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b5aa7217-bfcc-4c09-8d6f-51bb83cce667-agent-certs\") pod \"konnectivity-agent-snjp4\" (UID: \"b5aa7217-bfcc-4c09-8d6f-51bb83cce667\") " pod="kube-system/konnectivity-agent-snjp4"
Jul 07 16:47:45.677444 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.676925    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f7e6b596-4e4f-47d2-80e6-900c70180ef5-cnibin\") pod \"multus-p92bt\" (UID: \"f7e6b596-4e4f-47d2-80e6-900c70180ef5\") " pod="openshift-multus/multus-p92bt"
Jul 07 16:47:45.678184 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.676944    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f7e6b596-4e4f-47d2-80e6-900c70180ef5-os-release\") pod \"multus-p92bt\" (UID: \"f7e6b596-4e4f-47d2-80e6-900c70180ef5\") " pod="openshift-multus/multus-p92bt"
Jul 07 16:47:45.678184 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.676966    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/60d7869e-bef8-4020-b685-5f597d1031b9-host\") pod \"node-ca-dbsvg\" (UID: \"60d7869e-bef8-4020-b685-5f597d1031b9\") " pod="openshift-image-registry/node-ca-dbsvg"
Jul 07 16:47:45.678184 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.676995    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/26ecad8c-d52d-4d74-b5c1-0fdba353728e-ovnkube-config\") pod \"ovnkube-node-bjtgg\" (UID: \"26ecad8c-d52d-4d74-b5c1-0fdba353728e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:47:45.678184 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.677002    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/60d7869e-bef8-4020-b685-5f597d1031b9-host\") pod \"node-ca-dbsvg\" (UID: \"60d7869e-bef8-4020-b685-5f597d1031b9\") " pod="openshift-image-registry/node-ca-dbsvg"
Jul 07 16:47:45.678184 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.677051    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpnrf\" (UniqueName: \"kubernetes.io/projected/57008745-859f-49ec-9269-7671fa8164a2-kube-api-access-zpnrf\") pod \"network-check-target-wwb84\" (UID: \"57008745-859f-49ec-9269-7671fa8164a2\") " pod="openshift-network-diagnostics/network-check-target-wwb84"
Jul 07 16:47:45.678184 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.677120    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ca3370c2-512b-41bb-a09e-b26cac03572d-etc-selinux\") pod \"aws-ebs-csi-driver-node-pqq66\" (UID: \"ca3370c2-512b-41bb-a09e-b26cac03572d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pqq66"
Jul 07 16:47:45.678184 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.677221    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ca3370c2-512b-41bb-a09e-b26cac03572d-sys-fs\") pod \"aws-ebs-csi-driver-node-pqq66\" (UID: \"ca3370c2-512b-41bb-a09e-b26cac03572d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pqq66"
Jul 07 16:47:45.678184 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.677269    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ca3370c2-512b-41bb-a09e-b26cac03572d-etc-selinux\") pod \"aws-ebs-csi-driver-node-pqq66\" (UID: \"ca3370c2-512b-41bb-a09e-b26cac03572d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pqq66"
Jul 07 16:47:45.678184 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.677285    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/26ecad8c-d52d-4d74-b5c1-0fdba353728e-run-systemd\") pod \"ovnkube-node-bjtgg\" (UID: \"26ecad8c-d52d-4d74-b5c1-0fdba353728e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:47:45.678184 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.677297    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-86jhq\" (UID: \"40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca\") " pod="openshift-multus/multus-additional-cni-plugins-86jhq"
Jul 07 16:47:45.678184 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.677311    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/26ecad8c-d52d-4d74-b5c1-0fdba353728e-host-cni-netd\") pod \"ovnkube-node-bjtgg\" (UID: \"26ecad8c-d52d-4d74-b5c1-0fdba353728e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:47:45.678184 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.677344    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/26ecad8c-d52d-4d74-b5c1-0fdba353728e-env-overrides\") pod \"ovnkube-node-bjtgg\" (UID: \"26ecad8c-d52d-4d74-b5c1-0fdba353728e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:47:45.678184 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.677349    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/26ecad8c-d52d-4d74-b5c1-0fdba353728e-host-cni-netd\") pod \"ovnkube-node-bjtgg\" (UID: \"26ecad8c-d52d-4d74-b5c1-0fdba353728e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:47:45.678184 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.677373    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f7e6b596-4e4f-47d2-80e6-900c70180ef5-host-run-multus-certs\") pod \"multus-p92bt\" (UID: \"f7e6b596-4e4f-47d2-80e6-900c70180ef5\") " pod="openshift-multus/multus-p92bt"
Jul 07 16:47:45.678184 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.677390    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/26ecad8c-d52d-4d74-b5c1-0fdba353728e-run-systemd\") pod \"ovnkube-node-bjtgg\" (UID: \"26ecad8c-d52d-4d74-b5c1-0fdba353728e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:47:45.678184 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.677272    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ca3370c2-512b-41bb-a09e-b26cac03572d-sys-fs\") pod \"aws-ebs-csi-driver-node-pqq66\" (UID: \"ca3370c2-512b-41bb-a09e-b26cac03572d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pqq66"
Jul 07 16:47:45.678184 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.677399    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ca3370c2-512b-41bb-a09e-b26cac03572d-socket-dir\") pod \"aws-ebs-csi-driver-node-pqq66\" (UID: \"ca3370c2-512b-41bb-a09e-b26cac03572d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pqq66"
Jul 07 16:47:45.678800 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.677435    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cc86643-1ed6-4849-ad30-923acbf45ab7-tmp-dir\") pod \"node-resolver-f9r9j\" (UID: \"1cc86643-1ed6-4849-ad30-923acbf45ab7\") " pod="openshift-dns/node-resolver-f9r9j"
Jul 07 16:47:45.678800 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.677458    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca-cnibin\") pod \"multus-additional-cni-plugins-86jhq\" (UID: \"40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca\") " pod="openshift-multus/multus-additional-cni-plugins-86jhq"
Jul 07 16:47:45.678800 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.677483    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca-os-release\") pod \"multus-additional-cni-plugins-86jhq\" (UID: \"40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca\") " pod="openshift-multus/multus-additional-cni-plugins-86jhq"
Jul 07 16:47:45.678800 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.677487    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ca3370c2-512b-41bb-a09e-b26cac03572d-socket-dir\") pod \"aws-ebs-csi-driver-node-pqq66\" (UID: \"ca3370c2-512b-41bb-a09e-b26cac03572d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pqq66"
Jul 07 16:47:45.678800 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.677483    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/26ecad8c-d52d-4d74-b5c1-0fdba353728e-ovnkube-config\") pod \"ovnkube-node-bjtgg\" (UID: \"26ecad8c-d52d-4d74-b5c1-0fdba353728e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:47:45.678800 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.677518    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca-cnibin\") pod \"multus-additional-cni-plugins-86jhq\" (UID: \"40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca\") " pod="openshift-multus/multus-additional-cni-plugins-86jhq"
Jul 07 16:47:45.678800 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.677545    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f7e6b596-4e4f-47d2-80e6-900c70180ef5-host-run-k8s-cni-cncf-io\") pod \"multus-p92bt\" (UID: \"f7e6b596-4e4f-47d2-80e6-900c70180ef5\") " pod="openshift-multus/multus-p92bt"
Jul 07 16:47:45.678800 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.677545    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca-os-release\") pod \"multus-additional-cni-plugins-86jhq\" (UID: \"40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca\") " pod="openshift-multus/multus-additional-cni-plugins-86jhq"
Jul 07 16:47:45.678800 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.677569    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f7e6b596-4e4f-47d2-80e6-900c70180ef5-multus-daemon-config\") pod \"multus-p92bt\" (UID: \"f7e6b596-4e4f-47d2-80e6-900c70180ef5\") " pod="openshift-multus/multus-p92bt"
Jul 07 16:47:45.678800 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.677640    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/82568637-39b9-4e53-a44f-6c7148d449a0-lib-modules\") pod \"tuned-bvj2w\" (UID: \"82568637-39b9-4e53-a44f-6c7148d449a0\") " pod="openshift-cluster-node-tuning-operator/tuned-bvj2w"
Jul 07 16:47:45.678800 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.677664    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca-cni-binary-copy\") pod \"multus-additional-cni-plugins-86jhq\" (UID: \"40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca\") " pod="openshift-multus/multus-additional-cni-plugins-86jhq"
Jul 07 16:47:45.678800 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.677685    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/26ecad8c-d52d-4d74-b5c1-0fdba353728e-etc-openvswitch\") pod \"ovnkube-node-bjtgg\" (UID: \"26ecad8c-d52d-4d74-b5c1-0fdba353728e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:47:45.678800 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.677709    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/816e0df7-c9cf-4303-a920-a90a16ee015c-kubelet-config\") pod \"global-pull-secret-syncer-h5f7f\" (UID: \"816e0df7-c9cf-4303-a920-a90a16ee015c\") " pod="kube-system/global-pull-secret-syncer-h5f7f"
Jul 07 16:47:45.678800 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.677756    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0bb06482-d151-4a1f-8e49-9101ad901e7a-host-slash\") pod \"iptables-alerter-llvkg\" (UID: \"0bb06482-d151-4a1f-8e49-9101ad901e7a\") " pod="openshift-network-operator/iptables-alerter-llvkg"
Jul 07 16:47:45.678800 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.677717    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cc86643-1ed6-4849-ad30-923acbf45ab7-tmp-dir\") pod \"node-resolver-f9r9j\" (UID: \"1cc86643-1ed6-4849-ad30-923acbf45ab7\") " pod="openshift-dns/node-resolver-f9r9j"
Jul 07 16:47:45.678800 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.677765    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/26ecad8c-d52d-4d74-b5c1-0fdba353728e-etc-openvswitch\") pod \"ovnkube-node-bjtgg\" (UID: \"26ecad8c-d52d-4d74-b5c1-0fdba353728e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:47:45.678800 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.677767    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/82568637-39b9-4e53-a44f-6c7148d449a0-lib-modules\") pod \"tuned-bvj2w\" (UID: \"82568637-39b9-4e53-a44f-6c7148d449a0\") " pod="openshift-cluster-node-tuning-operator/tuned-bvj2w"
Jul 07 16:47:45.680028 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.677782    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f7e6b596-4e4f-47d2-80e6-900c70180ef5-cni-binary-copy\") pod \"multus-p92bt\" (UID: \"f7e6b596-4e4f-47d2-80e6-900c70180ef5\") " pod="openshift-multus/multus-p92bt"
Jul 07 16:47:45.680028 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.677756    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/26ecad8c-d52d-4d74-b5c1-0fdba353728e-env-overrides\") pod \"ovnkube-node-bjtgg\" (UID: \"26ecad8c-d52d-4d74-b5c1-0fdba353728e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:47:45.680028 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.677812    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb2l8\" (UniqueName: \"kubernetes.io/projected/f7e6b596-4e4f-47d2-80e6-900c70180ef5-kube-api-access-rb2l8\") pod \"multus-p92bt\" (UID: \"f7e6b596-4e4f-47d2-80e6-900c70180ef5\") " pod="openshift-multus/multus-p92bt"
Jul 07 16:47:45.680028 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.677837    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/26ecad8c-d52d-4d74-b5c1-0fdba353728e-log-socket\") pod \"ovnkube-node-bjtgg\" (UID: \"26ecad8c-d52d-4d74-b5c1-0fdba353728e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:47:45.680028 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.677867    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/26ecad8c-d52d-4d74-b5c1-0fdba353728e-host-cni-bin\") pod \"ovnkube-node-bjtgg\" (UID: \"26ecad8c-d52d-4d74-b5c1-0fdba353728e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:47:45.680028 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.677893    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/26ecad8c-d52d-4d74-b5c1-0fdba353728e-log-socket\") pod \"ovnkube-node-bjtgg\" (UID: \"26ecad8c-d52d-4d74-b5c1-0fdba353728e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:47:45.680028 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.677895    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca-system-cni-dir\") pod \"multus-additional-cni-plugins-86jhq\" (UID: \"40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca\") " pod="openshift-multus/multus-additional-cni-plugins-86jhq"
Jul 07 16:47:45.680028 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.677941    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca-system-cni-dir\") pod \"multus-additional-cni-plugins-86jhq\" (UID: \"40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca\") " pod="openshift-multus/multus-additional-cni-plugins-86jhq"
Jul 07 16:47:45.680028 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.677958    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/816e0df7-c9cf-4303-a920-a90a16ee015c-original-pull-secret\") pod \"global-pull-secret-syncer-h5f7f\" (UID: \"816e0df7-c9cf-4303-a920-a90a16ee015c\") " pod="kube-system/global-pull-secret-syncer-h5f7f"
Jul 07 16:47:45.680028 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.677978    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/26ecad8c-d52d-4d74-b5c1-0fdba353728e-host-cni-bin\") pod \"ovnkube-node-bjtgg\" (UID: \"26ecad8c-d52d-4d74-b5c1-0fdba353728e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:47:45.680028 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.677995    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f7e6b596-4e4f-47d2-80e6-900c70180ef5-system-cni-dir\") pod \"multus-p92bt\" (UID: \"f7e6b596-4e4f-47d2-80e6-900c70180ef5\") " pod="openshift-multus/multus-p92bt"
Jul 07 16:47:45.680028 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.678018    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/60d7869e-bef8-4020-b685-5f597d1031b9-serviceca\") pod \"node-ca-dbsvg\" (UID: \"60d7869e-bef8-4020-b685-5f597d1031b9\") " pod="openshift-image-registry/node-ca-dbsvg"
Jul 07 16:47:45.680028 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.678058    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhj26\" (UniqueName: \"kubernetes.io/projected/91e42c9a-31b0-42bf-9055-28fd68ab4ba9-kube-api-access-mhj26\") pod \"network-metrics-daemon-kdcss\" (UID: \"91e42c9a-31b0-42bf-9055-28fd68ab4ba9\") " pod="openshift-multus/network-metrics-daemon-kdcss"
Jul 07 16:47:45.680028 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.678097    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca-cni-binary-copy\") pod \"multus-additional-cni-plugins-86jhq\" (UID: \"40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca\") " pod="openshift-multus/multus-additional-cni-plugins-86jhq"
Jul 07 16:47:45.680028 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.678141    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/26ecad8c-d52d-4d74-b5c1-0fdba353728e-ovnkube-script-lib\") pod \"ovnkube-node-bjtgg\" (UID: \"26ecad8c-d52d-4d74-b5c1-0fdba353728e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:47:45.680028 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.678247    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/82568637-39b9-4e53-a44f-6c7148d449a0-run\") pod \"tuned-bvj2w\" (UID: \"82568637-39b9-4e53-a44f-6c7148d449a0\") " pod="openshift-cluster-node-tuning-operator/tuned-bvj2w"
Jul 07 16:47:45.680028 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.678272    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1cc86643-1ed6-4849-ad30-923acbf45ab7-hosts-file\") pod \"node-resolver-f9r9j\" (UID: \"1cc86643-1ed6-4849-ad30-923acbf45ab7\") " pod="openshift-dns/node-resolver-f9r9j"
Jul 07 16:47:45.680538 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.678296    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7e6b596-4e4f-47d2-80e6-900c70180ef5-host-var-lib-kubelet\") pod \"multus-p92bt\" (UID: \"f7e6b596-4e4f-47d2-80e6-900c70180ef5\") " pod="openshift-multus/multus-p92bt"
Jul 07 16:47:45.680538 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.678316    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f7e6b596-4e4f-47d2-80e6-900c70180ef5-hostroot\") pod \"multus-p92bt\" (UID: \"f7e6b596-4e4f-47d2-80e6-900c70180ef5\") " pod="openshift-multus/multus-p92bt"
Jul 07 16:47:45.680538 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.678339    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8f86\" (UniqueName: \"kubernetes.io/projected/60d7869e-bef8-4020-b685-5f597d1031b9-kube-api-access-f8f86\") pod \"node-ca-dbsvg\" (UID: \"60d7869e-bef8-4020-b685-5f597d1031b9\") " pod="openshift-image-registry/node-ca-dbsvg"
Jul 07 16:47:45.680538 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.678363    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/26ecad8c-d52d-4d74-b5c1-0fdba353728e-host-slash\") pod \"ovnkube-node-bjtgg\" (UID: \"26ecad8c-d52d-4d74-b5c1-0fdba353728e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:47:45.680538 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.678384    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/26ecad8c-d52d-4d74-b5c1-0fdba353728e-var-lib-openvswitch\") pod \"ovnkube-node-bjtgg\" (UID: \"26ecad8c-d52d-4d74-b5c1-0fdba353728e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:47:45.680538 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.678387    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/60d7869e-bef8-4020-b685-5f597d1031b9-serviceca\") pod \"node-ca-dbsvg\" (UID: \"60d7869e-bef8-4020-b685-5f597d1031b9\") " pod="openshift-image-registry/node-ca-dbsvg"
Jul 07 16:47:45.680538 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.678467    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/82568637-39b9-4e53-a44f-6c7148d449a0-run\") pod \"tuned-bvj2w\" (UID: \"82568637-39b9-4e53-a44f-6c7148d449a0\") " pod="openshift-cluster-node-tuning-operator/tuned-bvj2w"
Jul 07 16:47:45.680538 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.678545    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1cc86643-1ed6-4849-ad30-923acbf45ab7-hosts-file\") pod \"node-resolver-f9r9j\" (UID: \"1cc86643-1ed6-4849-ad30-923acbf45ab7\") " pod="openshift-dns/node-resolver-f9r9j"
Jul 07 16:47:45.680538 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.678568    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/26ecad8c-d52d-4d74-b5c1-0fdba353728e-ovnkube-script-lib\") pod \"ovnkube-node-bjtgg\" (UID: \"26ecad8c-d52d-4d74-b5c1-0fdba353728e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:47:45.680538 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.678628    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/26ecad8c-d52d-4d74-b5c1-0fdba353728e-host-slash\") pod \"ovnkube-node-bjtgg\" (UID: \"26ecad8c-d52d-4d74-b5c1-0fdba353728e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:47:45.680538 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.678699    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/26ecad8c-d52d-4d74-b5c1-0fdba353728e-var-lib-openvswitch\") pod \"ovnkube-node-bjtgg\" (UID: \"26ecad8c-d52d-4d74-b5c1-0fdba353728e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:47:45.680538 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.679157    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82568637-39b9-4e53-a44f-6c7148d449a0-tmp\") pod \"tuned-bvj2w\" (UID: \"82568637-39b9-4e53-a44f-6c7148d449a0\") " pod="openshift-cluster-node-tuning-operator/tuned-bvj2w"
Jul 07 16:47:45.680538 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.679290    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/82568637-39b9-4e53-a44f-6c7148d449a0-etc-tuned\") pod \"tuned-bvj2w\" (UID: \"82568637-39b9-4e53-a44f-6c7148d449a0\") " pod="openshift-cluster-node-tuning-operator/tuned-bvj2w"
Jul 07 16:47:45.680538 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.679333    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/26ecad8c-d52d-4d74-b5c1-0fdba353728e-ovn-node-metrics-cert\") pod \"ovnkube-node-bjtgg\" (UID: \"26ecad8c-d52d-4d74-b5c1-0fdba353728e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:47:45.688291 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.686146    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v9tq\" (UniqueName: \"kubernetes.io/projected/40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca-kube-api-access-7v9tq\") pod \"multus-additional-cni-plugins-86jhq\" (UID: \"40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca\") " pod="openshift-multus/multus-additional-cni-plugins-86jhq"
Jul 07 16:47:45.688869 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:45.688849    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 16:47:45.688968 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:45.688874    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 16:47:45.688968 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:45.688885    2578 projected.go:196] Error preparing data for projected volume kube-api-access-zpnrf for pod openshift-network-diagnostics/network-check-target-wwb84: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:47:45.688968 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:45.688933    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57008745-859f-49ec-9269-7671fa8164a2-kube-api-access-zpnrf podName:57008745-859f-49ec-9269-7671fa8164a2 nodeName:}" failed. No retries permitted until 2026-07-07 16:47:46.188918981 +0000 UTC m=+2.965233180 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-zpnrf" (UniqueName: "kubernetes.io/projected/57008745-859f-49ec-9269-7671fa8164a2-kube-api-access-zpnrf") pod "network-check-target-wwb84" (UID: "57008745-859f-49ec-9269-7671fa8164a2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:47:45.691558 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.691540    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnd5w\" (UniqueName: \"kubernetes.io/projected/26ecad8c-d52d-4d74-b5c1-0fdba353728e-kube-api-access-rnd5w\") pod \"ovnkube-node-bjtgg\" (UID: \"26ecad8c-d52d-4d74-b5c1-0fdba353728e\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:47:45.691748 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.691701    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhj26\" (UniqueName: \"kubernetes.io/projected/91e42c9a-31b0-42bf-9055-28fd68ab4ba9-kube-api-access-mhj26\") pod \"network-metrics-daemon-kdcss\" (UID: \"91e42c9a-31b0-42bf-9055-28fd68ab4ba9\") " pod="openshift-multus/network-metrics-daemon-kdcss"
Jul 07 16:47:45.692147 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.692113    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxlds\" (UniqueName: \"kubernetes.io/projected/1cc86643-1ed6-4849-ad30-923acbf45ab7-kube-api-access-mxlds\") pod \"node-resolver-f9r9j\" (UID: \"1cc86643-1ed6-4849-ad30-923acbf45ab7\") " pod="openshift-dns/node-resolver-f9r9j"
Jul 07 16:47:45.692147 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.692136    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8f86\" (UniqueName: \"kubernetes.io/projected/60d7869e-bef8-4020-b685-5f597d1031b9-kube-api-access-f8f86\") pod \"node-ca-dbsvg\" (UID: \"60d7869e-bef8-4020-b685-5f597d1031b9\") " pod="openshift-image-registry/node-ca-dbsvg"
Jul 07 16:47:45.692267 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.692116    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcwrg\" (UniqueName: \"kubernetes.io/projected/82568637-39b9-4e53-a44f-6c7148d449a0-kube-api-access-jcwrg\") pod \"tuned-bvj2w\" (UID: \"82568637-39b9-4e53-a44f-6c7148d449a0\") " pod="openshift-cluster-node-tuning-operator/tuned-bvj2w"
Jul 07 16:47:45.692267 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.692241    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86hrs\" (UniqueName: \"kubernetes.io/projected/ca3370c2-512b-41bb-a09e-b26cac03572d-kube-api-access-86hrs\") pod \"aws-ebs-csi-driver-node-pqq66\" (UID: \"ca3370c2-512b-41bb-a09e-b26cac03572d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pqq66"
Jul 07 16:47:45.778747 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.778705    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/816e0df7-c9cf-4303-a920-a90a16ee015c-dbus\") pod \"global-pull-secret-syncer-h5f7f\" (UID: \"816e0df7-c9cf-4303-a920-a90a16ee015c\") " pod="kube-system/global-pull-secret-syncer-h5f7f"
Jul 07 16:47:45.778880 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.778815    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmdwc\" (UniqueName: \"kubernetes.io/projected/0bb06482-d151-4a1f-8e49-9101ad901e7a-kube-api-access-qmdwc\") pod \"iptables-alerter-llvkg\" (UID: \"0bb06482-d151-4a1f-8e49-9101ad901e7a\") " pod="openshift-network-operator/iptables-alerter-llvkg"
Jul 07 16:47:45.778880 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.778830    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f7e6b596-4e4f-47d2-80e6-900c70180ef5-multus-socket-dir-parent\") pod \"multus-p92bt\" (UID: \"f7e6b596-4e4f-47d2-80e6-900c70180ef5\") " pod="openshift-multus/multus-p92bt"
Jul 07 16:47:45.778880 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.778853    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7e6b596-4e4f-47d2-80e6-900c70180ef5-host-run-netns\") pod \"multus-p92bt\" (UID: \"f7e6b596-4e4f-47d2-80e6-900c70180ef5\") " pod="openshift-multus/multus-p92bt"
Jul 07 16:47:45.778880 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.778866    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f7e6b596-4e4f-47d2-80e6-900c70180ef5-multus-conf-dir\") pod \"multus-p92bt\" (UID: \"f7e6b596-4e4f-47d2-80e6-900c70180ef5\") " pod="openshift-multus/multus-p92bt"
Jul 07 16:47:45.779042 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.778893    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/816e0df7-c9cf-4303-a920-a90a16ee015c-dbus\") pod \"global-pull-secret-syncer-h5f7f\" (UID: \"816e0df7-c9cf-4303-a920-a90a16ee015c\") " pod="kube-system/global-pull-secret-syncer-h5f7f"
Jul 07 16:47:45.779042 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.778915    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b5aa7217-bfcc-4c09-8d6f-51bb83cce667-agent-certs\") pod \"konnectivity-agent-snjp4\" (UID: \"b5aa7217-bfcc-4c09-8d6f-51bb83cce667\") " pod="kube-system/konnectivity-agent-snjp4"
Jul 07 16:47:45.779042 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.778921    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f7e6b596-4e4f-47d2-80e6-900c70180ef5-multus-socket-dir-parent\") pod \"multus-p92bt\" (UID: \"f7e6b596-4e4f-47d2-80e6-900c70180ef5\") " pod="openshift-multus/multus-p92bt"
Jul 07 16:47:45.779042 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.778904    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7e6b596-4e4f-47d2-80e6-900c70180ef5-host-run-netns\") pod \"multus-p92bt\" (UID: \"f7e6b596-4e4f-47d2-80e6-900c70180ef5\") " pod="openshift-multus/multus-p92bt"
Jul 07 16:47:45.779042 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.778968    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f7e6b596-4e4f-47d2-80e6-900c70180ef5-multus-conf-dir\") pod \"multus-p92bt\" (UID: \"f7e6b596-4e4f-47d2-80e6-900c70180ef5\") " pod="openshift-multus/multus-p92bt"
Jul 07 16:47:45.779042 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.778993    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f7e6b596-4e4f-47d2-80e6-900c70180ef5-cnibin\") pod \"multus-p92bt\" (UID: \"f7e6b596-4e4f-47d2-80e6-900c70180ef5\") " pod="openshift-multus/multus-p92bt"
Jul 07 16:47:45.779042 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.779041    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f7e6b596-4e4f-47d2-80e6-900c70180ef5-os-release\") pod \"multus-p92bt\" (UID: \"f7e6b596-4e4f-47d2-80e6-900c70180ef5\") " pod="openshift-multus/multus-p92bt"
Jul 07 16:47:45.779401 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.779088    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f7e6b596-4e4f-47d2-80e6-900c70180ef5-host-run-multus-certs\") pod \"multus-p92bt\" (UID: \"f7e6b596-4e4f-47d2-80e6-900c70180ef5\") " pod="openshift-multus/multus-p92bt"
Jul 07 16:47:45.779401 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.779039    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f7e6b596-4e4f-47d2-80e6-900c70180ef5-cnibin\") pod \"multus-p92bt\" (UID: \"f7e6b596-4e4f-47d2-80e6-900c70180ef5\") " pod="openshift-multus/multus-p92bt"
Jul 07 16:47:45.779401 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.779124    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f7e6b596-4e4f-47d2-80e6-900c70180ef5-host-run-k8s-cni-cncf-io\") pod \"multus-p92bt\" (UID: \"f7e6b596-4e4f-47d2-80e6-900c70180ef5\") " pod="openshift-multus/multus-p92bt"
Jul 07 16:47:45.779401 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.779141    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f7e6b596-4e4f-47d2-80e6-900c70180ef5-host-run-multus-certs\") pod \"multus-p92bt\" (UID: \"f7e6b596-4e4f-47d2-80e6-900c70180ef5\") " pod="openshift-multus/multus-p92bt"
Jul 07 16:47:45.779401 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.779147    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f7e6b596-4e4f-47d2-80e6-900c70180ef5-multus-daemon-config\") pod \"multus-p92bt\" (UID: \"f7e6b596-4e4f-47d2-80e6-900c70180ef5\") " pod="openshift-multus/multus-p92bt"
Jul 07 16:47:45.779401 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.779148    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f7e6b596-4e4f-47d2-80e6-900c70180ef5-os-release\") pod \"multus-p92bt\" (UID: \"f7e6b596-4e4f-47d2-80e6-900c70180ef5\") " pod="openshift-multus/multus-p92bt"
Jul 07 16:47:45.779401 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.779183    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f7e6b596-4e4f-47d2-80e6-900c70180ef5-host-run-k8s-cni-cncf-io\") pod \"multus-p92bt\" (UID: \"f7e6b596-4e4f-47d2-80e6-900c70180ef5\") " pod="openshift-multus/multus-p92bt"
Jul 07 16:47:45.779401 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.779190    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/816e0df7-c9cf-4303-a920-a90a16ee015c-kubelet-config\") pod \"global-pull-secret-syncer-h5f7f\" (UID: \"816e0df7-c9cf-4303-a920-a90a16ee015c\") " pod="kube-system/global-pull-secret-syncer-h5f7f"
Jul 07 16:47:45.779401 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.779214    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0bb06482-d151-4a1f-8e49-9101ad901e7a-host-slash\") pod \"iptables-alerter-llvkg\" (UID: \"0bb06482-d151-4a1f-8e49-9101ad901e7a\") " pod="openshift-network-operator/iptables-alerter-llvkg"
Jul 07 16:47:45.779401 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.779232    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/816e0df7-c9cf-4303-a920-a90a16ee015c-kubelet-config\") pod \"global-pull-secret-syncer-h5f7f\" (UID: \"816e0df7-c9cf-4303-a920-a90a16ee015c\") " pod="kube-system/global-pull-secret-syncer-h5f7f"
Jul 07 16:47:45.779401 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.779235    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f7e6b596-4e4f-47d2-80e6-900c70180ef5-cni-binary-copy\") pod \"multus-p92bt\" (UID: \"f7e6b596-4e4f-47d2-80e6-900c70180ef5\") " pod="openshift-multus/multus-p92bt"
Jul 07 16:47:45.779401 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.779266    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb2l8\" (UniqueName: \"kubernetes.io/projected/f7e6b596-4e4f-47d2-80e6-900c70180ef5-kube-api-access-rb2l8\") pod \"multus-p92bt\" (UID: \"f7e6b596-4e4f-47d2-80e6-900c70180ef5\") " pod="openshift-multus/multus-p92bt"
Jul 07 16:47:45.779401 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.779291    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/816e0df7-c9cf-4303-a920-a90a16ee015c-original-pull-secret\") pod \"global-pull-secret-syncer-h5f7f\" (UID: \"816e0df7-c9cf-4303-a920-a90a16ee015c\") " pod="kube-system/global-pull-secret-syncer-h5f7f"
Jul 07 16:47:45.779401 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.779312    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f7e6b596-4e4f-47d2-80e6-900c70180ef5-system-cni-dir\") pod \"multus-p92bt\" (UID: \"f7e6b596-4e4f-47d2-80e6-900c70180ef5\") " pod="openshift-multus/multus-p92bt"
Jul 07 16:47:45.779401 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.779341    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7e6b596-4e4f-47d2-80e6-900c70180ef5-host-var-lib-kubelet\") pod \"multus-p92bt\" (UID: \"f7e6b596-4e4f-47d2-80e6-900c70180ef5\") " pod="openshift-multus/multus-p92bt"
Jul 07 16:47:45.779401 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.779364    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f7e6b596-4e4f-47d2-80e6-900c70180ef5-hostroot\") pod \"multus-p92bt\" (UID: \"f7e6b596-4e4f-47d2-80e6-900c70180ef5\") " pod="openshift-multus/multus-p92bt"
Jul 07 16:47:45.779401 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.779392    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0bb06482-d151-4a1f-8e49-9101ad901e7a-iptables-alerter-script\") pod \"iptables-alerter-llvkg\" (UID: \"0bb06482-d151-4a1f-8e49-9101ad901e7a\") " pod="openshift-network-operator/iptables-alerter-llvkg"
Jul 07 16:47:45.780255 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.779416    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f7e6b596-4e4f-47d2-80e6-900c70180ef5-multus-cni-dir\") pod \"multus-p92bt\" (UID: \"f7e6b596-4e4f-47d2-80e6-900c70180ef5\") " pod="openshift-multus/multus-p92bt"
Jul 07 16:47:45.780255 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.779444    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7e6b596-4e4f-47d2-80e6-900c70180ef5-etc-kubernetes\") pod \"multus-p92bt\" (UID: \"f7e6b596-4e4f-47d2-80e6-900c70180ef5\") " pod="openshift-multus/multus-p92bt"
Jul 07 16:47:45.780255 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.779469    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7e6b596-4e4f-47d2-80e6-900c70180ef5-host-var-lib-cni-bin\") pod \"multus-p92bt\" (UID: \"f7e6b596-4e4f-47d2-80e6-900c70180ef5\") " pod="openshift-multus/multus-p92bt"
Jul 07 16:47:45.780255 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.779492    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b5aa7217-bfcc-4c09-8d6f-51bb83cce667-konnectivity-ca\") pod \"konnectivity-agent-snjp4\" (UID: \"b5aa7217-bfcc-4c09-8d6f-51bb83cce667\") " pod="kube-system/konnectivity-agent-snjp4"
Jul 07 16:47:45.780255 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.779514    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f7e6b596-4e4f-47d2-80e6-900c70180ef5-host-var-lib-cni-multus\") pod \"multus-p92bt\" (UID: \"f7e6b596-4e4f-47d2-80e6-900c70180ef5\") " pod="openshift-multus/multus-p92bt"
Jul 07 16:47:45.780255 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.779578    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f7e6b596-4e4f-47d2-80e6-900c70180ef5-host-var-lib-cni-multus\") pod \"multus-p92bt\" (UID: \"f7e6b596-4e4f-47d2-80e6-900c70180ef5\") " pod="openshift-multus/multus-p92bt"
Jul 07 16:47:45.780255 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.779631    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0bb06482-d151-4a1f-8e49-9101ad901e7a-host-slash\") pod \"iptables-alerter-llvkg\" (UID: \"0bb06482-d151-4a1f-8e49-9101ad901e7a\") " pod="openshift-network-operator/iptables-alerter-llvkg"
Jul 07 16:47:45.780255 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:45.779668    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 16:47:45.780255 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.779687    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f7e6b596-4e4f-47d2-80e6-900c70180ef5-multus-daemon-config\") pod \"multus-p92bt\" (UID: \"f7e6b596-4e4f-47d2-80e6-900c70180ef5\") " pod="openshift-multus/multus-p92bt"
Jul 07 16:47:45.780255 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:45.779717    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/816e0df7-c9cf-4303-a920-a90a16ee015c-original-pull-secret podName:816e0df7-c9cf-4303-a920-a90a16ee015c nodeName:}" failed. No retries permitted until 2026-07-07 16:47:46.279700838 +0000 UTC m=+3.056015023 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/816e0df7-c9cf-4303-a920-a90a16ee015c-original-pull-secret") pod "global-pull-secret-syncer-h5f7f" (UID: "816e0df7-c9cf-4303-a920-a90a16ee015c") : object "kube-system"/"original-pull-secret" not registered
Jul 07 16:47:45.780255 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.779750    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f7e6b596-4e4f-47d2-80e6-900c70180ef5-cni-binary-copy\") pod \"multus-p92bt\" (UID: \"f7e6b596-4e4f-47d2-80e6-900c70180ef5\") " pod="openshift-multus/multus-p92bt"
Jul 07 16:47:45.780255 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.779787    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f7e6b596-4e4f-47d2-80e6-900c70180ef5-system-cni-dir\") pod \"multus-p92bt\" (UID: \"f7e6b596-4e4f-47d2-80e6-900c70180ef5\") " pod="openshift-multus/multus-p92bt"
Jul 07 16:47:45.780255 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.779807    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7e6b596-4e4f-47d2-80e6-900c70180ef5-host-var-lib-cni-bin\") pod \"multus-p92bt\" (UID: \"f7e6b596-4e4f-47d2-80e6-900c70180ef5\") " pod="openshift-multus/multus-p92bt"
Jul 07 16:47:45.780255 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.779833    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f7e6b596-4e4f-47d2-80e6-900c70180ef5-multus-cni-dir\") pod \"multus-p92bt\" (UID: \"f7e6b596-4e4f-47d2-80e6-900c70180ef5\") " pod="openshift-multus/multus-p92bt"
Jul 07 16:47:45.780255 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.779843    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7e6b596-4e4f-47d2-80e6-900c70180ef5-etc-kubernetes\") pod \"multus-p92bt\" (UID: \"f7e6b596-4e4f-47d2-80e6-900c70180ef5\") " pod="openshift-multus/multus-p92bt"
Jul 07 16:47:45.780255 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.779868    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f7e6b596-4e4f-47d2-80e6-900c70180ef5-hostroot\") pod \"multus-p92bt\" (UID: \"f7e6b596-4e4f-47d2-80e6-900c70180ef5\") " pod="openshift-multus/multus-p92bt"
Jul 07 16:47:45.780255 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.779871    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7e6b596-4e4f-47d2-80e6-900c70180ef5-host-var-lib-kubelet\") pod \"multus-p92bt\" (UID: \"f7e6b596-4e4f-47d2-80e6-900c70180ef5\") " pod="openshift-multus/multus-p92bt"
Jul 07 16:47:45.780255 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.780252    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0bb06482-d151-4a1f-8e49-9101ad901e7a-iptables-alerter-script\") pod \"iptables-alerter-llvkg\" (UID: \"0bb06482-d151-4a1f-8e49-9101ad901e7a\") " pod="openshift-network-operator/iptables-alerter-llvkg"
Jul 07 16:47:45.780829 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.780339    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b5aa7217-bfcc-4c09-8d6f-51bb83cce667-konnectivity-ca\") pod \"konnectivity-agent-snjp4\" (UID: \"b5aa7217-bfcc-4c09-8d6f-51bb83cce667\") " pod="kube-system/konnectivity-agent-snjp4"
Jul 07 16:47:45.781399 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.781379    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b5aa7217-bfcc-4c09-8d6f-51bb83cce667-agent-certs\") pod \"konnectivity-agent-snjp4\" (UID: \"b5aa7217-bfcc-4c09-8d6f-51bb83cce667\") " pod="kube-system/konnectivity-agent-snjp4"
Jul 07 16:47:45.791388 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.791359    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb2l8\" (UniqueName: \"kubernetes.io/projected/f7e6b596-4e4f-47d2-80e6-900c70180ef5-kube-api-access-rb2l8\") pod \"multus-p92bt\" (UID: \"f7e6b596-4e4f-47d2-80e6-900c70180ef5\") " pod="openshift-multus/multus-p92bt"
Jul 07 16:47:45.792534 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.792516    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmdwc\" (UniqueName: \"kubernetes.io/projected/0bb06482-d151-4a1f-8e49-9101ad901e7a-kube-api-access-qmdwc\") pod \"iptables-alerter-llvkg\" (UID: \"0bb06482-d151-4a1f-8e49-9101ad901e7a\") " pod="openshift-network-operator/iptables-alerter-llvkg"
Jul 07 16:47:45.867370 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.867287    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bvj2w"
Jul 07 16:47:45.874492 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.874469    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-f9r9j"
Jul 07 16:47:45.883771 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.883752    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pqq66"
Jul 07 16:47:45.888900 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.888885    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dbsvg"
Jul 07 16:47:45.894794 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.894776    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:47:45.902525 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.902511    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-86jhq"
Jul 07 16:47:45.908103 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.908087    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-p92bt"
Jul 07 16:47:45.912664 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.912646    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-snjp4"
Jul 07 16:47:45.917258 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:45.917242    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-llvkg"
Jul 07 16:47:46.093496 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:46.093463    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 16:47:46.182326 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:46.182246    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91e42c9a-31b0-42bf-9055-28fd68ab4ba9-metrics-certs\") pod \"network-metrics-daemon-kdcss\" (UID: \"91e42c9a-31b0-42bf-9055-28fd68ab4ba9\") " pod="openshift-multus/network-metrics-daemon-kdcss"
Jul 07 16:47:46.182468 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:46.182363    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:47:46.182468 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:46.182423    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91e42c9a-31b0-42bf-9055-28fd68ab4ba9-metrics-certs podName:91e42c9a-31b0-42bf-9055-28fd68ab4ba9 nodeName:}" failed. No retries permitted until 2026-07-07 16:47:47.182405804 +0000 UTC m=+3.958719989 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/91e42c9a-31b0-42bf-9055-28fd68ab4ba9-metrics-certs") pod "network-metrics-daemon-kdcss" (UID: "91e42c9a-31b0-42bf-9055-28fd68ab4ba9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:47:46.282709 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:46.282670    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpnrf\" (UniqueName: \"kubernetes.io/projected/57008745-859f-49ec-9269-7671fa8164a2-kube-api-access-zpnrf\") pod \"network-check-target-wwb84\" (UID: \"57008745-859f-49ec-9269-7671fa8164a2\") " pod="openshift-network-diagnostics/network-check-target-wwb84"
Jul 07 16:47:46.282885 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:46.282726    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/816e0df7-c9cf-4303-a920-a90a16ee015c-original-pull-secret\") pod \"global-pull-secret-syncer-h5f7f\" (UID: \"816e0df7-c9cf-4303-a920-a90a16ee015c\") " pod="kube-system/global-pull-secret-syncer-h5f7f"
Jul 07 16:47:46.282885 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:46.282880    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 16:47:46.283011 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:46.282882    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 16:47:46.283011 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:46.282949    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/816e0df7-c9cf-4303-a920-a90a16ee015c-original-pull-secret podName:816e0df7-c9cf-4303-a920-a90a16ee015c nodeName:}" failed. No retries permitted until 2026-07-07 16:47:47.282933127 +0000 UTC m=+4.059247314 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/816e0df7-c9cf-4303-a920-a90a16ee015c-original-pull-secret") pod "global-pull-secret-syncer-h5f7f" (UID: "816e0df7-c9cf-4303-a920-a90a16ee015c") : object "kube-system"/"original-pull-secret" not registered
Jul 07 16:47:46.283011 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:46.282968    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 16:47:46.283011 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:46.282987    2578 projected.go:196] Error preparing data for projected volume kube-api-access-zpnrf for pod openshift-network-diagnostics/network-check-target-wwb84: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:47:46.283192 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:46.283052    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57008745-859f-49ec-9269-7671fa8164a2-kube-api-access-zpnrf podName:57008745-859f-49ec-9269-7671fa8164a2 nodeName:}" failed. No retries permitted until 2026-07-07 16:47:47.283033391 +0000 UTC m=+4.059347598 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-zpnrf" (UniqueName: "kubernetes.io/projected/57008745-859f-49ec-9269-7671fa8164a2-kube-api-access-zpnrf") pod "network-check-target-wwb84" (UID: "57008745-859f-49ec-9269-7671fa8164a2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:47:46.439941 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:46.439912    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40a4b335_1bc8_4e56_91bc_2ba0d32bd0ca.slice/crio-473055bc3de101a2976cfb88736a2f1ebf9a97ff10a973982ba37e23878da61f WatchSource:0}: Error finding container 473055bc3de101a2976cfb88736a2f1ebf9a97ff10a973982ba37e23878da61f: Status 404 returned error can't find the container with id 473055bc3de101a2976cfb88736a2f1ebf9a97ff10a973982ba37e23878da61f
Jul 07 16:47:46.440898 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:46.440876    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca3370c2_512b_41bb_a09e_b26cac03572d.slice/crio-1ea8a467abdff7b168de8bba69573e11451aca326ad3ca61f4935bdaec4b3a46 WatchSource:0}: Error finding container 1ea8a467abdff7b168de8bba69573e11451aca326ad3ca61f4935bdaec4b3a46: Status 404 returned error can't find the container with id 1ea8a467abdff7b168de8bba69573e11451aca326ad3ca61f4935bdaec4b3a46
Jul 07 16:47:46.442088 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:46.442054    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5aa7217_bfcc_4c09_8d6f_51bb83cce667.slice/crio-61ce89bda9b01775b104a22b0ed253bfc74acf9c5b73b2793c6595e4dc80c21b WatchSource:0}: Error finding container 61ce89bda9b01775b104a22b0ed253bfc74acf9c5b73b2793c6595e4dc80c21b: Status 404 returned error can't find the container with id 61ce89bda9b01775b104a22b0ed253bfc74acf9c5b73b2793c6595e4dc80c21b
Jul 07 16:47:46.444888 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:46.444870    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26ecad8c_d52d_4d74_b5c1_0fdba353728e.slice/crio-1e296c1075b82c53d6fa6709a89ecebc8c58023fa4b6a718e10196823254adf0 WatchSource:0}: Error finding container 1e296c1075b82c53d6fa6709a89ecebc8c58023fa4b6a718e10196823254adf0: Status 404 returned error can't find the container with id 1e296c1075b82c53d6fa6709a89ecebc8c58023fa4b6a718e10196823254adf0
Jul 07 16:47:46.445818 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:46.445801    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bb06482_d151_4a1f_8e49_9101ad901e7a.slice/crio-92b8973dd0ad95dcc5bd307f4f71fe87d9c7b1a780be8ab4823ff3cc9d19ef48 WatchSource:0}: Error finding container 92b8973dd0ad95dcc5bd307f4f71fe87d9c7b1a780be8ab4823ff3cc9d19ef48: Status 404 returned error can't find the container with id 92b8973dd0ad95dcc5bd307f4f71fe87d9c7b1a780be8ab4823ff3cc9d19ef48
Jul 07 16:47:46.447592 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:46.447567    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60d7869e_bef8_4020_b685_5f597d1031b9.slice/crio-e5712e89e31609dd431d1dd7b795ca34ecb58fd834ac6726b70d6541c4a5064e WatchSource:0}: Error finding container e5712e89e31609dd431d1dd7b795ca34ecb58fd834ac6726b70d6541c4a5064e: Status 404 returned error can't find the container with id e5712e89e31609dd431d1dd7b795ca34ecb58fd834ac6726b70d6541c4a5064e
Jul 07 16:47:46.448182 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:46.448162    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cc86643_1ed6_4849_ad30_923acbf45ab7.slice/crio-55a297afb255810e12e7ad50cc2f00cb7a0e89c257b2178d4932c86a7814f70c WatchSource:0}: Error finding container 55a297afb255810e12e7ad50cc2f00cb7a0e89c257b2178d4932c86a7814f70c: Status 404 returned error can't find the container with id 55a297afb255810e12e7ad50cc2f00cb7a0e89c257b2178d4932c86a7814f70c
Jul 07 16:47:46.449413 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:46.449386    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7e6b596_4e4f_47d2_80e6_900c70180ef5.slice/crio-0bc9cfa7a38d2ced85b55d9d87179ce84be107c2e11877ffd6ad583e1d4a89b8 WatchSource:0}: Error finding container 0bc9cfa7a38d2ced85b55d9d87179ce84be107c2e11877ffd6ad583e1d4a89b8: Status 404 returned error can't find the container with id 0bc9cfa7a38d2ced85b55d9d87179ce84be107c2e11877ffd6ad583e1d4a89b8
Jul 07 16:47:46.450359 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:47:46.450268    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82568637_39b9_4e53_a44f_6c7148d449a0.slice/crio-66b0b7a97775a2352285403f08d25367c9fe5084c7ed3f27267a2326abe9d9de WatchSource:0}: Error finding container 66b0b7a97775a2352285403f08d25367c9fe5084c7ed3f27267a2326abe9d9de: Status 404 returned error can't find the container with id 66b0b7a97775a2352285403f08d25367c9fe5084c7ed3f27267a2326abe9d9de
Jul 07 16:47:46.604911 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:46.604677    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 16:42:45 +0000 UTC" deadline="2027-12-08 05:31:03.295676093 +0000 UTC"
Jul 07 16:47:46.604911 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:46.604907    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12444h43m16.690773199s"
Jul 07 16:47:46.730144 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:46.730116    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-86jhq" event={"ID":"40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca","Type":"ContainerStarted","Data":"473055bc3de101a2976cfb88736a2f1ebf9a97ff10a973982ba37e23878da61f"}
Jul 07 16:47:46.731952 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:46.731930    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-217.ec2.internal" event={"ID":"e0242a47c8f42188a150225d6e13e270","Type":"ContainerStarted","Data":"062071c038b7d0d9b04bae893dd575182c37a77a6972f3664f53ce87a34a54fe"}
Jul 07 16:47:46.732963 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:46.732942    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dbsvg" event={"ID":"60d7869e-bef8-4020-b685-5f597d1031b9","Type":"ContainerStarted","Data":"e5712e89e31609dd431d1dd7b795ca34ecb58fd834ac6726b70d6541c4a5064e"}
Jul 07 16:47:46.733776 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:46.733754    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg" event={"ID":"26ecad8c-d52d-4d74-b5c1-0fdba353728e","Type":"ContainerStarted","Data":"1e296c1075b82c53d6fa6709a89ecebc8c58023fa4b6a718e10196823254adf0"}
Jul 07 16:47:46.734645 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:46.734627    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-snjp4" event={"ID":"b5aa7217-bfcc-4c09-8d6f-51bb83cce667","Type":"ContainerStarted","Data":"61ce89bda9b01775b104a22b0ed253bfc74acf9c5b73b2793c6595e4dc80c21b"}
Jul 07 16:47:46.735645 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:46.735626    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pqq66" event={"ID":"ca3370c2-512b-41bb-a09e-b26cac03572d","Type":"ContainerStarted","Data":"1ea8a467abdff7b168de8bba69573e11451aca326ad3ca61f4935bdaec4b3a46"}
Jul 07 16:47:46.736796 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:46.736771    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-217.ec2.internal" event={"ID":"d3cfd0e1ebd8169b57d9e712a094a397","Type":"ContainerStarted","Data":"5972135f51330f6b0ce98e65ecc302bb7ea1db7c6b83799931901d44692e0beb"}
Jul 07 16:47:46.737751 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:46.737718    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bvj2w" event={"ID":"82568637-39b9-4e53-a44f-6c7148d449a0","Type":"ContainerStarted","Data":"66b0b7a97775a2352285403f08d25367c9fe5084c7ed3f27267a2326abe9d9de"}
Jul 07 16:47:46.738788 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:46.738764    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-p92bt" event={"ID":"f7e6b596-4e4f-47d2-80e6-900c70180ef5","Type":"ContainerStarted","Data":"0bc9cfa7a38d2ced85b55d9d87179ce84be107c2e11877ffd6ad583e1d4a89b8"}
Jul 07 16:47:46.739890 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:46.739709    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-f9r9j" event={"ID":"1cc86643-1ed6-4849-ad30-923acbf45ab7","Type":"ContainerStarted","Data":"55a297afb255810e12e7ad50cc2f00cb7a0e89c257b2178d4932c86a7814f70c"}
Jul 07 16:47:46.740686 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:46.740668    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-llvkg" event={"ID":"0bb06482-d151-4a1f-8e49-9101ad901e7a","Type":"ContainerStarted","Data":"92b8973dd0ad95dcc5bd307f4f71fe87d9c7b1a780be8ab4823ff3cc9d19ef48"}
Jul 07 16:47:46.764439 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:46.764400    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-217.ec2.internal" podStartSLOduration=2.764385368 podStartE2EDuration="2.764385368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:47:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:47:46.747789551 +0000 UTC m=+3.524103777" watchObservedRunningTime="2026-07-07 16:47:46.764385368 +0000 UTC m=+3.540699576"
Jul 07 16:47:47.190611 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:47.190526    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91e42c9a-31b0-42bf-9055-28fd68ab4ba9-metrics-certs\") pod \"network-metrics-daemon-kdcss\" (UID: \"91e42c9a-31b0-42bf-9055-28fd68ab4ba9\") " pod="openshift-multus/network-metrics-daemon-kdcss"
Jul 07 16:47:47.190792 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:47.190690    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:47:47.190792 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:47.190775    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91e42c9a-31b0-42bf-9055-28fd68ab4ba9-metrics-certs podName:91e42c9a-31b0-42bf-9055-28fd68ab4ba9 nodeName:}" failed. No retries permitted until 2026-07-07 16:47:49.190756032 +0000 UTC m=+5.967070220 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/91e42c9a-31b0-42bf-9055-28fd68ab4ba9-metrics-certs") pod "network-metrics-daemon-kdcss" (UID: "91e42c9a-31b0-42bf-9055-28fd68ab4ba9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:47:47.291419 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:47.291380    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpnrf\" (UniqueName: \"kubernetes.io/projected/57008745-859f-49ec-9269-7671fa8164a2-kube-api-access-zpnrf\") pod \"network-check-target-wwb84\" (UID: \"57008745-859f-49ec-9269-7671fa8164a2\") " pod="openshift-network-diagnostics/network-check-target-wwb84"
Jul 07 16:47:47.291602 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:47.291446    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/816e0df7-c9cf-4303-a920-a90a16ee015c-original-pull-secret\") pod \"global-pull-secret-syncer-h5f7f\" (UID: \"816e0df7-c9cf-4303-a920-a90a16ee015c\") " pod="kube-system/global-pull-secret-syncer-h5f7f"
Jul 07 16:47:47.291679 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:47.291617    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 16:47:47.292021 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:47.291743    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/816e0df7-c9cf-4303-a920-a90a16ee015c-original-pull-secret podName:816e0df7-c9cf-4303-a920-a90a16ee015c nodeName:}" failed. No retries permitted until 2026-07-07 16:47:49.291708919 +0000 UTC m=+6.068023106 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/816e0df7-c9cf-4303-a920-a90a16ee015c-original-pull-secret") pod "global-pull-secret-syncer-h5f7f" (UID: "816e0df7-c9cf-4303-a920-a90a16ee015c") : object "kube-system"/"original-pull-secret" not registered
Jul 07 16:47:47.292021 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:47.291896    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 16:47:47.292021 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:47.291921    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 16:47:47.292021 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:47.291935    2578 projected.go:196] Error preparing data for projected volume kube-api-access-zpnrf for pod openshift-network-diagnostics/network-check-target-wwb84: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:47:47.292021 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:47.291993    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57008745-859f-49ec-9269-7671fa8164a2-kube-api-access-zpnrf podName:57008745-859f-49ec-9269-7671fa8164a2 nodeName:}" failed. No retries permitted until 2026-07-07 16:47:49.291977053 +0000 UTC m=+6.068291260 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-zpnrf" (UniqueName: "kubernetes.io/projected/57008745-859f-49ec-9269-7671fa8164a2-kube-api-access-zpnrf") pod "network-check-target-wwb84" (UID: "57008745-859f-49ec-9269-7671fa8164a2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:47:47.605684 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:47.605529    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 16:42:45 +0000 UTC" deadline="2027-12-08 03:14:20.062676462 +0000 UTC"
Jul 07 16:47:47.605684 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:47.605586    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12442h26m32.457096167s"
Jul 07 16:47:47.721771 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:47.721038    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wwb84"
Jul 07 16:47:47.721771 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:47.721166    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wwb84" podUID="57008745-859f-49ec-9269-7671fa8164a2"
Jul 07 16:47:47.721771 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:47.721569    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdcss"
Jul 07 16:47:47.721771 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:47.721670    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdcss" podUID="91e42c9a-31b0-42bf-9055-28fd68ab4ba9"
Jul 07 16:47:47.722410 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:47.722179    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h5f7f"
Jul 07 16:47:47.722410 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:47.722277    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h5f7f" podUID="816e0df7-c9cf-4303-a920-a90a16ee015c"
Jul 07 16:47:47.757200 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:47.756658    2578 generic.go:356] "Generic (PLEG): container finished" podID="d3cfd0e1ebd8169b57d9e712a094a397" containerID="5972135f51330f6b0ce98e65ecc302bb7ea1db7c6b83799931901d44692e0beb" exitCode=0
Jul 07 16:47:47.757200 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:47.756769    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-217.ec2.internal" event={"ID":"d3cfd0e1ebd8169b57d9e712a094a397","Type":"ContainerDied","Data":"5972135f51330f6b0ce98e65ecc302bb7ea1db7c6b83799931901d44692e0beb"}
Jul 07 16:47:48.781407 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:48.780866    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-217.ec2.internal" event={"ID":"d3cfd0e1ebd8169b57d9e712a094a397","Type":"ContainerStarted","Data":"97d423ccb7336191c178b25cce063a446d9a066a37a87f15da9dad21249094da"}
Jul 07 16:47:49.208090 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:49.207425    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91e42c9a-31b0-42bf-9055-28fd68ab4ba9-metrics-certs\") pod \"network-metrics-daemon-kdcss\" (UID: \"91e42c9a-31b0-42bf-9055-28fd68ab4ba9\") " pod="openshift-multus/network-metrics-daemon-kdcss"
Jul 07 16:47:49.208090 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:49.207586    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:47:49.208090 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:49.207656    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91e42c9a-31b0-42bf-9055-28fd68ab4ba9-metrics-certs podName:91e42c9a-31b0-42bf-9055-28fd68ab4ba9 nodeName:}" failed. No retries permitted until 2026-07-07 16:47:53.207629739 +0000 UTC m=+9.983943940 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/91e42c9a-31b0-42bf-9055-28fd68ab4ba9-metrics-certs") pod "network-metrics-daemon-kdcss" (UID: "91e42c9a-31b0-42bf-9055-28fd68ab4ba9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:47:49.309401 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:49.308510    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/816e0df7-c9cf-4303-a920-a90a16ee015c-original-pull-secret\") pod \"global-pull-secret-syncer-h5f7f\" (UID: \"816e0df7-c9cf-4303-a920-a90a16ee015c\") " pod="kube-system/global-pull-secret-syncer-h5f7f"
Jul 07 16:47:49.309401 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:49.308598    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpnrf\" (UniqueName: \"kubernetes.io/projected/57008745-859f-49ec-9269-7671fa8164a2-kube-api-access-zpnrf\") pod \"network-check-target-wwb84\" (UID: \"57008745-859f-49ec-9269-7671fa8164a2\") " pod="openshift-network-diagnostics/network-check-target-wwb84"
Jul 07 16:47:49.309401 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:49.308752    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 16:47:49.309401 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:49.308771    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 16:47:49.309401 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:49.308783    2578 projected.go:196] Error preparing data for projected volume kube-api-access-zpnrf for pod openshift-network-diagnostics/network-check-target-wwb84: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:47:49.309401 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:49.308838    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57008745-859f-49ec-9269-7671fa8164a2-kube-api-access-zpnrf podName:57008745-859f-49ec-9269-7671fa8164a2 nodeName:}" failed. No retries permitted until 2026-07-07 16:47:53.308822314 +0000 UTC m=+10.085136518 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-zpnrf" (UniqueName: "kubernetes.io/projected/57008745-859f-49ec-9269-7671fa8164a2-kube-api-access-zpnrf") pod "network-check-target-wwb84" (UID: "57008745-859f-49ec-9269-7671fa8164a2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:47:49.309401 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:49.309271    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 16:47:49.309401 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:49.309324    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/816e0df7-c9cf-4303-a920-a90a16ee015c-original-pull-secret podName:816e0df7-c9cf-4303-a920-a90a16ee015c nodeName:}" failed. No retries permitted until 2026-07-07 16:47:53.309308438 +0000 UTC m=+10.085622640 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/816e0df7-c9cf-4303-a920-a90a16ee015c-original-pull-secret") pod "global-pull-secret-syncer-h5f7f" (UID: "816e0df7-c9cf-4303-a920-a90a16ee015c") : object "kube-system"/"original-pull-secret" not registered
Jul 07 16:47:49.719969 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:49.719937    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wwb84"
Jul 07 16:47:49.720152 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:49.720126    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wwb84" podUID="57008745-859f-49ec-9269-7671fa8164a2"
Jul 07 16:47:49.720590 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:49.720530    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdcss"
Jul 07 16:47:49.720677 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:49.720648    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdcss" podUID="91e42c9a-31b0-42bf-9055-28fd68ab4ba9"
Jul 07 16:47:49.720752 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:49.720709    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h5f7f"
Jul 07 16:47:49.720842 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:49.720822    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h5f7f" podUID="816e0df7-c9cf-4303-a920-a90a16ee015c"
Jul 07 16:47:51.720199 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:51.720167    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h5f7f"
Jul 07 16:47:51.720630 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:51.720293    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h5f7f" podUID="816e0df7-c9cf-4303-a920-a90a16ee015c"
Jul 07 16:47:51.720630 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:51.720167    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wwb84"
Jul 07 16:47:51.720630 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:51.720438    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wwb84" podUID="57008745-859f-49ec-9269-7671fa8164a2"
Jul 07 16:47:51.720630 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:51.720482    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdcss"
Jul 07 16:47:51.720630 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:51.720622    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdcss" podUID="91e42c9a-31b0-42bf-9055-28fd68ab4ba9"
Jul 07 16:47:53.239039 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:53.239000    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91e42c9a-31b0-42bf-9055-28fd68ab4ba9-metrics-certs\") pod \"network-metrics-daemon-kdcss\" (UID: \"91e42c9a-31b0-42bf-9055-28fd68ab4ba9\") " pod="openshift-multus/network-metrics-daemon-kdcss"
Jul 07 16:47:53.239492 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:53.239170    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:47:53.239492 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:53.239244    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91e42c9a-31b0-42bf-9055-28fd68ab4ba9-metrics-certs podName:91e42c9a-31b0-42bf-9055-28fd68ab4ba9 nodeName:}" failed. No retries permitted until 2026-07-07 16:48:01.239224382 +0000 UTC m=+18.015538568 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/91e42c9a-31b0-42bf-9055-28fd68ab4ba9-metrics-certs") pod "network-metrics-daemon-kdcss" (UID: "91e42c9a-31b0-42bf-9055-28fd68ab4ba9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:47:53.340058 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:53.340018    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpnrf\" (UniqueName: \"kubernetes.io/projected/57008745-859f-49ec-9269-7671fa8164a2-kube-api-access-zpnrf\") pod \"network-check-target-wwb84\" (UID: \"57008745-859f-49ec-9269-7671fa8164a2\") " pod="openshift-network-diagnostics/network-check-target-wwb84"
Jul 07 16:47:53.340218 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:53.340080    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/816e0df7-c9cf-4303-a920-a90a16ee015c-original-pull-secret\") pod \"global-pull-secret-syncer-h5f7f\" (UID: \"816e0df7-c9cf-4303-a920-a90a16ee015c\") " pod="kube-system/global-pull-secret-syncer-h5f7f"
Jul 07 16:47:53.340218 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:53.340162    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 16:47:53.340218 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:53.340174    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 16:47:53.340218 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:53.340184    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 16:47:53.340218 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:53.340196    2578 projected.go:196] Error preparing data for projected volume kube-api-access-zpnrf for pod openshift-network-diagnostics/network-check-target-wwb84: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:47:53.340478 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:53.340238    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57008745-859f-49ec-9269-7671fa8164a2-kube-api-access-zpnrf podName:57008745-859f-49ec-9269-7671fa8164a2 nodeName:}" failed. No retries permitted until 2026-07-07 16:48:01.340219936 +0000 UTC m=+18.116534133 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-zpnrf" (UniqueName: "kubernetes.io/projected/57008745-859f-49ec-9269-7671fa8164a2-kube-api-access-zpnrf") pod "network-check-target-wwb84" (UID: "57008745-859f-49ec-9269-7671fa8164a2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:47:53.340478 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:53.340255    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/816e0df7-c9cf-4303-a920-a90a16ee015c-original-pull-secret podName:816e0df7-c9cf-4303-a920-a90a16ee015c nodeName:}" failed. No retries permitted until 2026-07-07 16:48:01.340246994 +0000 UTC m=+18.116561178 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/816e0df7-c9cf-4303-a920-a90a16ee015c-original-pull-secret") pod "global-pull-secret-syncer-h5f7f" (UID: "816e0df7-c9cf-4303-a920-a90a16ee015c") : object "kube-system"/"original-pull-secret" not registered
Jul 07 16:47:53.721587 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:53.721509    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wwb84"
Jul 07 16:47:53.721750 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:53.721618    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wwb84" podUID="57008745-859f-49ec-9269-7671fa8164a2"
Jul 07 16:47:53.722032 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:53.722013    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdcss"
Jul 07 16:47:53.722301 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:53.722127    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdcss" podUID="91e42c9a-31b0-42bf-9055-28fd68ab4ba9"
Jul 07 16:47:53.722301 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:53.722196    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h5f7f"
Jul 07 16:47:53.722301 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:53.722266    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h5f7f" podUID="816e0df7-c9cf-4303-a920-a90a16ee015c"
Jul 07 16:47:55.720690 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:55.720580    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wwb84"
Jul 07 16:47:55.721196 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:55.720714    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wwb84" podUID="57008745-859f-49ec-9269-7671fa8164a2"
Jul 07 16:47:55.721676 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:55.721472    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h5f7f"
Jul 07 16:47:55.721676 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:55.721623    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h5f7f" podUID="816e0df7-c9cf-4303-a920-a90a16ee015c"
Jul 07 16:47:55.721866 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:55.720580    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdcss"
Jul 07 16:47:55.721914 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:55.721873    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdcss" podUID="91e42c9a-31b0-42bf-9055-28fd68ab4ba9"
Jul 07 16:47:57.720638 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:57.720608    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdcss"
Jul 07 16:47:57.720638 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:57.720625    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h5f7f"
Jul 07 16:47:57.721151 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:57.720758    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdcss" podUID="91e42c9a-31b0-42bf-9055-28fd68ab4ba9"
Jul 07 16:47:57.721151 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:57.720831    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h5f7f" podUID="816e0df7-c9cf-4303-a920-a90a16ee015c"
Jul 07 16:47:57.721151 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:57.720872    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wwb84"
Jul 07 16:47:57.721151 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:57.720965    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wwb84" podUID="57008745-859f-49ec-9269-7671fa8164a2"
Jul 07 16:47:59.720214 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:59.720175    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h5f7f"
Jul 07 16:47:59.720651 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:59.720225    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdcss"
Jul 07 16:47:59.720651 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:59.720308    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h5f7f" podUID="816e0df7-c9cf-4303-a920-a90a16ee015c"
Jul 07 16:47:59.720651 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:47:59.720188    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wwb84"
Jul 07 16:47:59.720651 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:59.720432    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdcss" podUID="91e42c9a-31b0-42bf-9055-28fd68ab4ba9"
Jul 07 16:47:59.720651 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:47:59.720519    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wwb84" podUID="57008745-859f-49ec-9269-7671fa8164a2"
Jul 07 16:48:01.300394 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:01.300367    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91e42c9a-31b0-42bf-9055-28fd68ab4ba9-metrics-certs\") pod \"network-metrics-daemon-kdcss\" (UID: \"91e42c9a-31b0-42bf-9055-28fd68ab4ba9\") " pod="openshift-multus/network-metrics-daemon-kdcss"
Jul 07 16:48:01.300810 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:01.300531    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:48:01.300810 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:01.300611    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91e42c9a-31b0-42bf-9055-28fd68ab4ba9-metrics-certs podName:91e42c9a-31b0-42bf-9055-28fd68ab4ba9 nodeName:}" failed. No retries permitted until 2026-07-07 16:48:17.300588361 +0000 UTC m=+34.076902545 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/91e42c9a-31b0-42bf-9055-28fd68ab4ba9-metrics-certs") pod "network-metrics-daemon-kdcss" (UID: "91e42c9a-31b0-42bf-9055-28fd68ab4ba9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:48:01.400823 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:01.400789    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpnrf\" (UniqueName: \"kubernetes.io/projected/57008745-859f-49ec-9269-7671fa8164a2-kube-api-access-zpnrf\") pod \"network-check-target-wwb84\" (UID: \"57008745-859f-49ec-9269-7671fa8164a2\") " pod="openshift-network-diagnostics/network-check-target-wwb84"
Jul 07 16:48:01.400970 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:01.400831    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/816e0df7-c9cf-4303-a920-a90a16ee015c-original-pull-secret\") pod \"global-pull-secret-syncer-h5f7f\" (UID: \"816e0df7-c9cf-4303-a920-a90a16ee015c\") " pod="kube-system/global-pull-secret-syncer-h5f7f"
Jul 07 16:48:01.400970 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:01.400959    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 16:48:01.401077 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:01.400964    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 16:48:01.401077 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:01.400988    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 16:48:01.401077 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:01.401002    2578 projected.go:196] Error preparing data for projected volume kube-api-access-zpnrf for pod openshift-network-diagnostics/network-check-target-wwb84: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:48:01.401077 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:01.401019    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/816e0df7-c9cf-4303-a920-a90a16ee015c-original-pull-secret podName:816e0df7-c9cf-4303-a920-a90a16ee015c nodeName:}" failed. No retries permitted until 2026-07-07 16:48:17.401001361 +0000 UTC m=+34.177315548 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/816e0df7-c9cf-4303-a920-a90a16ee015c-original-pull-secret") pod "global-pull-secret-syncer-h5f7f" (UID: "816e0df7-c9cf-4303-a920-a90a16ee015c") : object "kube-system"/"original-pull-secret" not registered
Jul 07 16:48:01.401077 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:01.401053    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57008745-859f-49ec-9269-7671fa8164a2-kube-api-access-zpnrf podName:57008745-859f-49ec-9269-7671fa8164a2 nodeName:}" failed. No retries permitted until 2026-07-07 16:48:17.401036686 +0000 UTC m=+34.177350888 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-zpnrf" (UniqueName: "kubernetes.io/projected/57008745-859f-49ec-9269-7671fa8164a2-kube-api-access-zpnrf") pod "network-check-target-wwb84" (UID: "57008745-859f-49ec-9269-7671fa8164a2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:48:01.720413 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:01.720330    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wwb84"
Jul 07 16:48:01.720570 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:01.720330    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdcss"
Jul 07 16:48:01.720570 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:01.720477    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wwb84" podUID="57008745-859f-49ec-9269-7671fa8164a2"
Jul 07 16:48:01.720570 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:01.720335    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h5f7f"
Jul 07 16:48:01.720570 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:01.720563    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdcss" podUID="91e42c9a-31b0-42bf-9055-28fd68ab4ba9"
Jul 07 16:48:01.720845 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:01.720602    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h5f7f" podUID="816e0df7-c9cf-4303-a920-a90a16ee015c"
Jul 07 16:48:03.720598 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:03.720569    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h5f7f"
Jul 07 16:48:03.721059 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:03.720693    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h5f7f" podUID="816e0df7-c9cf-4303-a920-a90a16ee015c"
Jul 07 16:48:03.721059 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:03.720841    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wwb84"
Jul 07 16:48:03.721059 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:03.720976    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wwb84" podUID="57008745-859f-49ec-9269-7671fa8164a2"
Jul 07 16:48:03.721059 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:03.721013    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdcss"
Jul 07 16:48:03.721266 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:03.721169    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdcss" podUID="91e42c9a-31b0-42bf-9055-28fd68ab4ba9"
Jul 07 16:48:05.720256 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:05.719990    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h5f7f"
Jul 07 16:48:05.721033 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:05.719987    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdcss"
Jul 07 16:48:05.721033 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:05.720342    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h5f7f" podUID="816e0df7-c9cf-4303-a920-a90a16ee015c"
Jul 07 16:48:05.721033 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:05.720408    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdcss" podUID="91e42c9a-31b0-42bf-9055-28fd68ab4ba9"
Jul 07 16:48:05.721033 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:05.720043    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wwb84"
Jul 07 16:48:05.721033 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:05.720485    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wwb84" podUID="57008745-859f-49ec-9269-7671fa8164a2"
Jul 07 16:48:05.816634 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:05.815772    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bvj2w" event={"ID":"82568637-39b9-4e53-a44f-6c7148d449a0","Type":"ContainerStarted","Data":"572e58c204e74e17ed151fb423edf5c30906fa50b67f38671374d6be582de68d"}
Jul 07 16:48:05.818193 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:05.818166    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-p92bt" event={"ID":"f7e6b596-4e4f-47d2-80e6-900c70180ef5","Type":"ContainerStarted","Data":"645642341b02e5c59c2eb82f36283097d607d1286305b2af1ebf688bc8a2a1e1"}
Jul 07 16:48:05.819619 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:05.819597    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-f9r9j" event={"ID":"1cc86643-1ed6-4849-ad30-923acbf45ab7","Type":"ContainerStarted","Data":"2029817a79f6cc1191a34ea56d65eed7200a1d1a7313535cd178390f066539a1"}
Jul 07 16:48:05.821194 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:05.821171    2578 generic.go:356] "Generic (PLEG): container finished" podID="40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca" containerID="1791614165a5c0b88f4a49ceeb75d3196e942d87d6b7e92075cf945bc1d520b6" exitCode=0
Jul 07 16:48:05.821281 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:05.821199    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-86jhq" event={"ID":"40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca","Type":"ContainerDied","Data":"1791614165a5c0b88f4a49ceeb75d3196e942d87d6b7e92075cf945bc1d520b6"}
Jul 07 16:48:05.822492 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:05.822471    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dbsvg" event={"ID":"60d7869e-bef8-4020-b685-5f597d1031b9","Type":"ContainerStarted","Data":"28eed11fc1b03a8f9a521f74392846df69552d8893fa82a453affa598cc42a2c"}
Jul 07 16:48:05.824178 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:05.824160    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg" event={"ID":"26ecad8c-d52d-4d74-b5c1-0fdba353728e","Type":"ContainerStarted","Data":"c549904097bdeb175b4464af3abbee6f569cb45aad5b55f7ce5cf69666c45d7b"}
Jul 07 16:48:05.824264 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:05.824219    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg" event={"ID":"26ecad8c-d52d-4d74-b5c1-0fdba353728e","Type":"ContainerStarted","Data":"2b35641a07517c8c522b1daf60b9e6cde8f805e753262fc63689bad3670bc7c2"}
Jul 07 16:48:05.824264 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:05.824251    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg" event={"ID":"26ecad8c-d52d-4d74-b5c1-0fdba353728e","Type":"ContainerStarted","Data":"9c996fe69875e2ac00a0df4755bdb606ec94e1fe45afe0211e9452d040430b00"}
Jul 07 16:48:05.825447 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:05.825431    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-snjp4" event={"ID":"b5aa7217-bfcc-4c09-8d6f-51bb83cce667","Type":"ContainerStarted","Data":"4fe96263fac34b20370bf57b6b2a7860f109d0013f8aa19e0265632243fec01f"}
Jul 07 16:48:05.826667 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:05.826651    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pqq66" event={"ID":"ca3370c2-512b-41bb-a09e-b26cac03572d","Type":"ContainerStarted","Data":"70d44e1ad316ef43508c745848bb95655216ee9f873fd8ca74f6c114421ce63d"}
Jul 07 16:48:05.835331 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:05.835287    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-217.ec2.internal" podStartSLOduration=21.835272131 podStartE2EDuration="21.835272131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:47:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:47:48.798650932 +0000 UTC m=+5.574965156" watchObservedRunningTime="2026-07-07 16:48:05.835272131 +0000 UTC m=+22.611586339"
Jul 07 16:48:05.835657 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:05.835633    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-bvj2w" podStartSLOduration=3.915054989 podStartE2EDuration="22.835619745s" totalImagesPullingTime="18.920564756s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:47:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:48:05.834991056 +0000 UTC m=+22.611305300" watchObservedRunningTime="2026-07-07 16:48:05.835619745 +0000 UTC m=+22.611933951"
Jul 07 16:48:05.880771 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:05.880717    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-dbsvg" podStartSLOduration=3.97857709 podStartE2EDuration="22.880703515s" totalImagesPullingTime="18.902126425s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:47:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:48:05.853523804 +0000 UTC m=+22.629838277" watchObservedRunningTime="2026-07-07 16:48:05.880703515 +0000 UTC m=+22.657017720"
Jul 07 16:48:05.898079 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:05.898041    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-snjp4" podStartSLOduration=4.397265215 podStartE2EDuration="22.898027561s" totalImagesPullingTime="18.500762346s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:47:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:48:05.897609932 +0000 UTC m=+22.673924138" watchObservedRunningTime="2026-07-07 16:48:05.898027561 +0000 UTC m=+22.674341767"
Jul 07 16:48:05.936677 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:05.936471    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-p92bt" podStartSLOduration=4.000857087 podStartE2EDuration="22.936454931s" totalImagesPullingTime="18.935597844s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:47:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:48:05.936026687 +0000 UTC m=+22.712340895" watchObservedRunningTime="2026-07-07 16:48:05.936454931 +0000 UTC m=+22.712769140"
Jul 07 16:48:05.936835 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:05.936712    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-f9r9j" podStartSLOduration=4.008639641 podStartE2EDuration="22.936702936s" totalImagesPullingTime="18.928063295s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:47:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:48:05.917248557 +0000 UTC m=+22.693562762" watchObservedRunningTime="2026-07-07 16:48:05.936702936 +0000 UTC m=+22.713017143"
Jul 07 16:48:06.829826 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:06.829784    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-llvkg" event={"ID":"0bb06482-d151-4a1f-8e49-9101ad901e7a","Type":"ContainerStarted","Data":"49345b233448d164f92301a5d90c35765027b9986d2d8314db3a305bd7fb693b"}
Jul 07 16:48:06.832427 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:06.832405    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg" event={"ID":"26ecad8c-d52d-4d74-b5c1-0fdba353728e","Type":"ContainerStarted","Data":"6ecb95c280ebd5f89a4d8d2475f3e011f83d0aa45e782b09526cf9877e967a11"}
Jul 07 16:48:06.832565 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:06.832478    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg" event={"ID":"26ecad8c-d52d-4d74-b5c1-0fdba353728e","Type":"ContainerStarted","Data":"22fbbff58b5a57bc1fdb89dda9b8a1c6eee8ad55247dd4f1b5fc4aa8bb55acf6"}
Jul 07 16:48:06.832565 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:06.832527    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg" event={"ID":"26ecad8c-d52d-4d74-b5c1-0fdba353728e","Type":"ContainerStarted","Data":"7d35ac44150c7eb2f84e71fd8f5c29d8de0738e5afb5ad8785de95503e0a05c2"}
Jul 07 16:48:06.849172 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:06.849136    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-llvkg" podStartSLOduration=4.923838726 podStartE2EDuration="23.849123756s" totalImagesPullingTime="18.92528503s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:47:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:48:06.848515631 +0000 UTC m=+23.624829839" watchObservedRunningTime="2026-07-07 16:48:06.849123756 +0000 UTC m=+23.625437962"
Jul 07 16:48:07.010303 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:07.010278    2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 07 16:48:07.637719 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:07.637599    2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-07T16:48:07.010298943Z","UUID":"d40f97a6-5ddf-48cc-afbe-1c45e1a63261","Handler":null,"Name":"","Endpoint":""}
Jul 07 16:48:07.639445 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:07.639421    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 07 16:48:07.639593 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:07.639466    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 07 16:48:07.720333 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:07.720306    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wwb84"
Jul 07 16:48:07.720333 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:07.720324    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h5f7f"
Jul 07 16:48:07.720559 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:07.720315    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdcss"
Jul 07 16:48:07.720559 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:07.720420    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wwb84" podUID="57008745-859f-49ec-9269-7671fa8164a2"
Jul 07 16:48:07.720559 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:07.720512    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h5f7f" podUID="816e0df7-c9cf-4303-a920-a90a16ee015c"
Jul 07 16:48:07.720708 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:07.720603    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdcss" podUID="91e42c9a-31b0-42bf-9055-28fd68ab4ba9"
Jul 07 16:48:07.836797 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:07.836763    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pqq66" event={"ID":"ca3370c2-512b-41bb-a09e-b26cac03572d","Type":"ContainerStarted","Data":"052c28913e607c8f23d8ec021f5dcba984662fc4c90f8da2dbcfdab9690738e0"}
Jul 07 16:48:07.896442 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:07.896375    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-snjp4"
Jul 07 16:48:07.897054 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:07.897022    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-snjp4"
Jul 07 16:48:08.842370 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:08.842118    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg" event={"ID":"26ecad8c-d52d-4d74-b5c1-0fdba353728e","Type":"ContainerStarted","Data":"bbdf841d70094f4f0c6bea6af03734751e33366f9c2f44f7ae25bd067a6f4d0e"}
Jul 07 16:48:08.844113 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:08.844084    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pqq66" event={"ID":"ca3370c2-512b-41bb-a09e-b26cac03572d","Type":"ContainerStarted","Data":"1957aaffbfa1b01e2d0b5068cd63856e621974bfa742041cfbe102849ea23139"}
Jul 07 16:48:08.863135 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:08.863094    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pqq66" podStartSLOduration=4.342718513 podStartE2EDuration="25.863081187s" totalImagesPullingTime="21.520362674s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:47:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:48:08.862855211 +0000 UTC m=+25.639169418" watchObservedRunningTime="2026-07-07 16:48:08.863081187 +0000 UTC m=+25.639395393"
Jul 07 16:48:09.720513 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:09.720485    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wwb84"
Jul 07 16:48:09.720513 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:09.720501    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h5f7f"
Jul 07 16:48:09.720766 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:09.720496    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdcss"
Jul 07 16:48:09.720766 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:09.720601    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wwb84" podUID="57008745-859f-49ec-9269-7671fa8164a2"
Jul 07 16:48:09.720766 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:09.720679    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h5f7f" podUID="816e0df7-c9cf-4303-a920-a90a16ee015c"
Jul 07 16:48:09.720932 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:09.720776    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdcss" podUID="91e42c9a-31b0-42bf-9055-28fd68ab4ba9"
Jul 07 16:48:10.674205 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:10.674069    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-f9r9j_1cc86643-1ed6-4849-ad30-923acbf45ab7/dns-node-resolver/0.log"
Jul 07 16:48:10.861150 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:10.861119    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-86jhq" event={"ID":"40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca","Type":"ContainerStarted","Data":"cfc95b2c3b65941e2fdd9d036a4372c9c0fa0194293fa1314f8a09b6eee1912c"}
Jul 07 16:48:10.865200 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:10.865177    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg" event={"ID":"26ecad8c-d52d-4d74-b5c1-0fdba353728e","Type":"ContainerStarted","Data":"e72ef087e11eeeeaac99e01c3cc9f92954b92b8d9df70017a6aa15017ba4996f"}
Jul 07 16:48:10.865451 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:10.865427    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:48:10.865522 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:10.865475    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:48:10.879856 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:10.879832    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:48:10.879959 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:10.879919    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:48:10.915039 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:10.914997    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg" podStartSLOduration=8.880626213 podStartE2EDuration="27.914985012s" totalImagesPullingTime="19.034358799s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:47:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:48:10.914762422 +0000 UTC m=+27.691076647" watchObservedRunningTime="2026-07-07 16:48:10.914985012 +0000 UTC m=+27.691299255"
Jul 07 16:48:11.214804 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:11.214710    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-snjp4"
Jul 07 16:48:11.214967 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:11.214875    2578 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 07 16:48:11.215307 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:11.215291    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-snjp4"
Jul 07 16:48:11.461574 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:11.461548    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dbsvg_60d7869e-bef8-4020-b685-5f597d1031b9/node-ca/0.log"
Jul 07 16:48:11.724305 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:11.724273    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wwb84"
Jul 07 16:48:11.724837 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:11.724394    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wwb84" podUID="57008745-859f-49ec-9269-7671fa8164a2"
Jul 07 16:48:11.724837 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:11.724830    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdcss"
Jul 07 16:48:11.724965 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:11.724933    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdcss" podUID="91e42c9a-31b0-42bf-9055-28fd68ab4ba9"
Jul 07 16:48:11.725030 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:11.724993    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h5f7f"
Jul 07 16:48:11.725083 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:11.725064    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h5f7f" podUID="816e0df7-c9cf-4303-a920-a90a16ee015c"
Jul 07 16:48:11.869429 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:11.869394    2578 generic.go:356] "Generic (PLEG): container finished" podID="40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca" containerID="cfc95b2c3b65941e2fdd9d036a4372c9c0fa0194293fa1314f8a09b6eee1912c" exitCode=0
Jul 07 16:48:11.869632 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:11.869481    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-86jhq" event={"ID":"40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca","Type":"ContainerDied","Data":"cfc95b2c3b65941e2fdd9d036a4372c9c0fa0194293fa1314f8a09b6eee1912c"}
Jul 07 16:48:11.869785 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:11.869770    2578 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 07 16:48:12.822215 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:12.822183    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wwb84"]
Jul 07 16:48:12.822559 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:12.822289    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wwb84"
Jul 07 16:48:12.822559 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:12.822389    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wwb84" podUID="57008745-859f-49ec-9269-7671fa8164a2"
Jul 07 16:48:12.827925 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:12.827652    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kdcss"]
Jul 07 16:48:12.827925 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:12.827781    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdcss"
Jul 07 16:48:12.827925 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:12.827888    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdcss" podUID="91e42c9a-31b0-42bf-9055-28fd68ab4ba9"
Jul 07 16:48:12.828450 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:12.828426    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-h5f7f"]
Jul 07 16:48:12.828533 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:12.828511    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h5f7f"
Jul 07 16:48:12.828615 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:12.828593    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h5f7f" podUID="816e0df7-c9cf-4303-a920-a90a16ee015c"
Jul 07 16:48:12.871701 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:12.871677    2578 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 07 16:48:13.876027 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:13.875989    2578 generic.go:356] "Generic (PLEG): container finished" podID="40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca" containerID="26d9d3a4c7a4570e0c4228c52a6361e05381aa05dd44017d7fbe66ada69c3a3a" exitCode=0
Jul 07 16:48:13.876518 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:13.876035    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-86jhq" event={"ID":"40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca","Type":"ContainerDied","Data":"26d9d3a4c7a4570e0c4228c52a6361e05381aa05dd44017d7fbe66ada69c3a3a"}
Jul 07 16:48:14.720631 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:14.720598    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdcss"
Jul 07 16:48:14.720808 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:14.720634    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wwb84"
Jul 07 16:48:14.720808 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:14.720634    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h5f7f"
Jul 07 16:48:14.720808 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:14.720726    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdcss" podUID="91e42c9a-31b0-42bf-9055-28fd68ab4ba9"
Jul 07 16:48:14.720968 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:14.720815    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wwb84" podUID="57008745-859f-49ec-9269-7671fa8164a2"
Jul 07 16:48:14.720968 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:14.720899    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h5f7f" podUID="816e0df7-c9cf-4303-a920-a90a16ee015c"
Jul 07 16:48:15.603655 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:15.603625    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:48:15.604318 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:15.603839    2578 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 07 16:48:15.620203 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:15.620169    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg" podUID="26ecad8c-d52d-4d74-b5c1-0fdba353728e" containerName="ovnkube-controller" probeResult="failure" output=""
Jul 07 16:48:15.628317 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:15.628291    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg" podUID="26ecad8c-d52d-4d74-b5c1-0fdba353728e" containerName="ovnkube-controller" probeResult="failure" output=""
Jul 07 16:48:15.883546 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:15.883471    2578 generic.go:356] "Generic (PLEG): container finished" podID="40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca" containerID="85e6e6f15b53a2349ca0c51934105da8f8d3120e9b998688f718d555b06c676b" exitCode=0
Jul 07 16:48:15.883546 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:15.883528    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-86jhq" event={"ID":"40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca","Type":"ContainerDied","Data":"85e6e6f15b53a2349ca0c51934105da8f8d3120e9b998688f718d555b06c676b"}
Jul 07 16:48:16.719868 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:16.719834    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wwb84"
Jul 07 16:48:16.720257 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:16.719873    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h5f7f"
Jul 07 16:48:16.720257 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:16.719921    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdcss"
Jul 07 16:48:16.720257 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:16.719927    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wwb84" podUID="57008745-859f-49ec-9269-7671fa8164a2"
Jul 07 16:48:16.720257 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:16.719986    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h5f7f" podUID="816e0df7-c9cf-4303-a920-a90a16ee015c"
Jul 07 16:48:16.720257 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:16.720099    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdcss" podUID="91e42c9a-31b0-42bf-9055-28fd68ab4ba9"
Jul 07 16:48:17.323812 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:17.323780    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91e42c9a-31b0-42bf-9055-28fd68ab4ba9-metrics-certs\") pod \"network-metrics-daemon-kdcss\" (UID: \"91e42c9a-31b0-42bf-9055-28fd68ab4ba9\") " pod="openshift-multus/network-metrics-daemon-kdcss"
Jul 07 16:48:17.324007 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:17.323885    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:48:17.324007 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:17.323956    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91e42c9a-31b0-42bf-9055-28fd68ab4ba9-metrics-certs podName:91e42c9a-31b0-42bf-9055-28fd68ab4ba9 nodeName:}" failed. No retries permitted until 2026-07-07 16:48:49.323934297 +0000 UTC m=+66.100248484 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/91e42c9a-31b0-42bf-9055-28fd68ab4ba9-metrics-certs") pod "network-metrics-daemon-kdcss" (UID: "91e42c9a-31b0-42bf-9055-28fd68ab4ba9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:48:17.424212 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:17.424187    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpnrf\" (UniqueName: \"kubernetes.io/projected/57008745-859f-49ec-9269-7671fa8164a2-kube-api-access-zpnrf\") pod \"network-check-target-wwb84\" (UID: \"57008745-859f-49ec-9269-7671fa8164a2\") " pod="openshift-network-diagnostics/network-check-target-wwb84"
Jul 07 16:48:17.424212 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:17.424222    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/816e0df7-c9cf-4303-a920-a90a16ee015c-original-pull-secret\") pod \"global-pull-secret-syncer-h5f7f\" (UID: \"816e0df7-c9cf-4303-a920-a90a16ee015c\") " pod="kube-system/global-pull-secret-syncer-h5f7f"
Jul 07 16:48:17.424405 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:17.424310    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 16:48:17.424405 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:17.424330    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 16:48:17.424405 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:17.424351    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 16:48:17.424405 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:17.424355    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/816e0df7-c9cf-4303-a920-a90a16ee015c-original-pull-secret podName:816e0df7-c9cf-4303-a920-a90a16ee015c nodeName:}" failed. No retries permitted until 2026-07-07 16:48:49.424342813 +0000 UTC m=+66.200656997 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/816e0df7-c9cf-4303-a920-a90a16ee015c-original-pull-secret") pod "global-pull-secret-syncer-h5f7f" (UID: "816e0df7-c9cf-4303-a920-a90a16ee015c") : object "kube-system"/"original-pull-secret" not registered
Jul 07 16:48:17.424405 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:17.424363    2578 projected.go:196] Error preparing data for projected volume kube-api-access-zpnrf for pod openshift-network-diagnostics/network-check-target-wwb84: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:48:17.424405 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:17.424401    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57008745-859f-49ec-9269-7671fa8164a2-kube-api-access-zpnrf podName:57008745-859f-49ec-9269-7671fa8164a2 nodeName:}" failed. No retries permitted until 2026-07-07 16:48:49.424388252 +0000 UTC m=+66.200702440 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-zpnrf" (UniqueName: "kubernetes.io/projected/57008745-859f-49ec-9269-7671fa8164a2-kube-api-access-zpnrf") pod "network-check-target-wwb84" (UID: "57008745-859f-49ec-9269-7671fa8164a2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:48:18.720105 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:18.720074    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdcss"
Jul 07 16:48:18.720558 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:18.720107    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h5f7f"
Jul 07 16:48:18.720558 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:18.720107    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wwb84"
Jul 07 16:48:18.720558 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:18.720212    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdcss" podUID="91e42c9a-31b0-42bf-9055-28fd68ab4ba9"
Jul 07 16:48:18.720558 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:18.720278    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h5f7f" podUID="816e0df7-c9cf-4303-a920-a90a16ee015c"
Jul 07 16:48:18.720558 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:18.720363    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wwb84" podUID="57008745-859f-49ec-9269-7671fa8164a2"
Jul 07 16:48:20.720066 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:20.720027    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h5f7f"
Jul 07 16:48:20.720490 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:20.720131    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wwb84"
Jul 07 16:48:20.720490 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:20.720245    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wwb84" podUID="57008745-859f-49ec-9269-7671fa8164a2"
Jul 07 16:48:20.720490 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:20.720133    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h5f7f" podUID="816e0df7-c9cf-4303-a920-a90a16ee015c"
Jul 07 16:48:20.720490 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:20.720137    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdcss"
Jul 07 16:48:20.720490 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:20.720361    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdcss" podUID="91e42c9a-31b0-42bf-9055-28fd68ab4ba9"
Jul 07 16:48:21.897967 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:21.897771    2578 generic.go:356] "Generic (PLEG): container finished" podID="40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca" containerID="f46e2f6ab82d79d7f1626ef8dcb753f55d879c4974496c1b2525ad832740546d" exitCode=0
Jul 07 16:48:21.897967 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:21.897852    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-86jhq" event={"ID":"40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca","Type":"ContainerDied","Data":"f46e2f6ab82d79d7f1626ef8dcb753f55d879c4974496c1b2525ad832740546d"}
Jul 07 16:48:22.720471 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:22.720438    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdcss"
Jul 07 16:48:22.720646 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:22.720439    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h5f7f"
Jul 07 16:48:22.720646 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:22.720534    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdcss" podUID="91e42c9a-31b0-42bf-9055-28fd68ab4ba9"
Jul 07 16:48:22.720646 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:22.720551    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wwb84"
Jul 07 16:48:22.720646 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:22.720637    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h5f7f" podUID="816e0df7-c9cf-4303-a920-a90a16ee015c"
Jul 07 16:48:22.720895 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:22.720677    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wwb84" podUID="57008745-859f-49ec-9269-7671fa8164a2"
Jul 07 16:48:22.902967 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:22.902930    2578 generic.go:356] "Generic (PLEG): container finished" podID="40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca" containerID="4880c627c596551671c4b231688888ee3ea7873bbd9a59334576e38d3c515a63" exitCode=0
Jul 07 16:48:22.902967 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:22.902963    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-86jhq" event={"ID":"40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca","Type":"ContainerDied","Data":"4880c627c596551671c4b231688888ee3ea7873bbd9a59334576e38d3c515a63"}
Jul 07 16:48:23.907957 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:23.907922    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-86jhq" event={"ID":"40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca","Type":"ContainerStarted","Data":"75882faa66e5c97dd01b56a607d9c88f220f1347bdbb241a7972f16c365e04cc"}
Jul 07 16:48:23.934554 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:23.934506    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-86jhq" podStartSLOduration=10.265906888 podStartE2EDuration="40.934492513s" totalImagesPullingTime="30.668585625s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:47:43 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:48:23.933940781 +0000 UTC m=+40.710254986" watchObservedRunningTime="2026-07-07 16:48:23.934492513 +0000 UTC m=+40.710806719"
Jul 07 16:48:24.720023 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:24.719994    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h5f7f"
Jul 07 16:48:24.720182 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:24.720088    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wwb84"
Jul 07 16:48:24.720182 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:24.720097    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h5f7f" podUID="816e0df7-c9cf-4303-a920-a90a16ee015c"
Jul 07 16:48:24.720182 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:24.720112    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdcss"
Jul 07 16:48:24.720182 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:24.720169    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wwb84" podUID="57008745-859f-49ec-9269-7671fa8164a2"
Jul 07 16:48:24.720310 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:24.720247    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdcss" podUID="91e42c9a-31b0-42bf-9055-28fd68ab4ba9"
Jul 07 16:48:26.719998 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:26.719971    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wwb84"
Jul 07 16:48:26.720359 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:26.719971    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h5f7f"
Jul 07 16:48:26.720359 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:26.720065    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wwb84" podUID="57008745-859f-49ec-9269-7671fa8164a2"
Jul 07 16:48:26.720359 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:26.719973    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdcss"
Jul 07 16:48:26.720359 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:26.720136    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h5f7f" podUID="816e0df7-c9cf-4303-a920-a90a16ee015c"
Jul 07 16:48:26.720359 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:26.720200    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdcss" podUID="91e42c9a-31b0-42bf-9055-28fd68ab4ba9"
Jul 07 16:48:28.720002 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:28.719968    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h5f7f"
Jul 07 16:48:28.720435 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:28.719970    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wwb84"
Jul 07 16:48:28.720435 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:28.720091    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdcss"
Jul 07 16:48:28.720435 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:28.720079    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h5f7f" podUID="816e0df7-c9cf-4303-a920-a90a16ee015c"
Jul 07 16:48:28.720435 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:28.720169    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wwb84" podUID="57008745-859f-49ec-9269-7671fa8164a2"
Jul 07 16:48:28.720435 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:28.720246    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdcss" podUID="91e42c9a-31b0-42bf-9055-28fd68ab4ba9"
Jul 07 16:48:30.720596 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:30.720568    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdcss"
Jul 07 16:48:30.721024 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:30.720568    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h5f7f"
Jul 07 16:48:30.721024 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:30.720668    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdcss" podUID="91e42c9a-31b0-42bf-9055-28fd68ab4ba9"
Jul 07 16:48:30.721024 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:30.720568    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wwb84"
Jul 07 16:48:30.721024 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:30.720788    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h5f7f" podUID="816e0df7-c9cf-4303-a920-a90a16ee015c"
Jul 07 16:48:30.721024 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:30.720816    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wwb84" podUID="57008745-859f-49ec-9269-7671fa8164a2"
Jul 07 16:48:32.023400 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.023376    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-217.ec2.internal" event="NodeReady"
Jul 07 16:48:32.023847 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.023491    2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 07 16:48:32.093370 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.092102    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-lmvlr"]
Jul 07 16:48:32.094506 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.094479    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lmvlr"
Jul 07 16:48:32.097610 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.097589    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-g82pc\""
Jul 07 16:48:32.097756 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.097719    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 07 16:48:32.097880 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.097768    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 07 16:48:32.098135 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.098119    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 07 16:48:32.108301 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.108283    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lmvlr"]
Jul 07 16:48:32.121329 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.121306    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-t9dk7"]
Jul 07 16:48:32.124598 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.124557    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-t9dk7"
Jul 07 16:48:32.127749 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.127710    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 07 16:48:32.128324 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.128310    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 07 16:48:32.128377 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.128351    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5tvm4\""
Jul 07 16:48:32.128634 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.128622    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 07 16:48:32.128722 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.128709    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 07 16:48:32.137382 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.137362    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-t9dk7"]
Jul 07 16:48:32.140672 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.140654    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b1d1a91-1f92-4209-af86-f3496494d229-data-volume\") pod \"insights-runtime-extractor-t9dk7\" (UID: \"0b1d1a91-1f92-4209-af86-f3496494d229\") " pod="openshift-insights/insights-runtime-extractor-t9dk7"
Jul 07 16:48:32.140778 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.140685    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b1d1a91-1f92-4209-af86-f3496494d229-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t9dk7\" (UID: \"0b1d1a91-1f92-4209-af86-f3496494d229\") " pod="openshift-insights/insights-runtime-extractor-t9dk7"
Jul 07 16:48:32.140778 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.140714    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvd5n\" (UniqueName: \"kubernetes.io/projected/06abc595-8bff-454f-99e4-959c9ffcc06b-kube-api-access-gvd5n\") pod \"ingress-canary-lmvlr\" (UID: \"06abc595-8bff-454f-99e4-959c9ffcc06b\") " pod="openshift-ingress-canary/ingress-canary-lmvlr"
Jul 07 16:48:32.140860 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.140823    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2mwq\" (UniqueName: \"kubernetes.io/projected/0b1d1a91-1f92-4209-af86-f3496494d229-kube-api-access-t2mwq\") pod \"insights-runtime-extractor-t9dk7\" (UID: \"0b1d1a91-1f92-4209-af86-f3496494d229\") " pod="openshift-insights/insights-runtime-extractor-t9dk7"
Jul 07 16:48:32.140860 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.140846    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0b1d1a91-1f92-4209-af86-f3496494d229-crio-socket\") pod \"insights-runtime-extractor-t9dk7\" (UID: \"0b1d1a91-1f92-4209-af86-f3496494d229\") " pod="openshift-insights/insights-runtime-extractor-t9dk7"
Jul 07 16:48:32.140933 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.140889    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/06abc595-8bff-454f-99e4-959c9ffcc06b-cert\") pod \"ingress-canary-lmvlr\" (UID: \"06abc595-8bff-454f-99e4-959c9ffcc06b\") " pod="openshift-ingress-canary/ingress-canary-lmvlr"
Jul 07 16:48:32.140966 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.140946    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b1d1a91-1f92-4209-af86-f3496494d229-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t9dk7\" (UID: \"0b1d1a91-1f92-4209-af86-f3496494d229\") " pod="openshift-insights/insights-runtime-extractor-t9dk7"
Jul 07 16:48:32.191023 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.190990    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-hh2zz"]
Jul 07 16:48:32.195677 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.195663    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hh2zz"
Jul 07 16:48:32.198375 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.198349    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 07 16:48:32.198502 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.198487    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 07 16:48:32.198572 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.198544    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cn8rd\""
Jul 07 16:48:32.207607 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.207581    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hh2zz"]
Jul 07 16:48:32.241710 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.241671    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7fa61b07-1978-4db6-9e3f-0dd7862906ef-config-volume\") pod \"dns-default-hh2zz\" (UID: \"7fa61b07-1978-4db6-9e3f-0dd7862906ef\") " pod="openshift-dns/dns-default-hh2zz"
Jul 07 16:48:32.241837 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.241726    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b1d1a91-1f92-4209-af86-f3496494d229-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t9dk7\" (UID: \"0b1d1a91-1f92-4209-af86-f3496494d229\") " pod="openshift-insights/insights-runtime-extractor-t9dk7"
Jul 07 16:48:32.241837 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.241776    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b1d1a91-1f92-4209-af86-f3496494d229-data-volume\") pod \"insights-runtime-extractor-t9dk7\" (UID: \"0b1d1a91-1f92-4209-af86-f3496494d229\") " pod="openshift-insights/insights-runtime-extractor-t9dk7"
Jul 07 16:48:32.241837 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.241802    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b1d1a91-1f92-4209-af86-f3496494d229-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t9dk7\" (UID: \"0b1d1a91-1f92-4209-af86-f3496494d229\") " pod="openshift-insights/insights-runtime-extractor-t9dk7"
Jul 07 16:48:32.241837 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.241832    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvd5n\" (UniqueName: \"kubernetes.io/projected/06abc595-8bff-454f-99e4-959c9ffcc06b-kube-api-access-gvd5n\") pod \"ingress-canary-lmvlr\" (UID: \"06abc595-8bff-454f-99e4-959c9ffcc06b\") " pod="openshift-ingress-canary/ingress-canary-lmvlr"
Jul 07 16:48:32.242019 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.241857    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7fa61b07-1978-4db6-9e3f-0dd7862906ef-metrics-tls\") pod \"dns-default-hh2zz\" (UID: \"7fa61b07-1978-4db6-9e3f-0dd7862906ef\") " pod="openshift-dns/dns-default-hh2zz"
Jul 07 16:48:32.242075 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.242021    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2mwq\" (UniqueName: \"kubernetes.io/projected/0b1d1a91-1f92-4209-af86-f3496494d229-kube-api-access-t2mwq\") pod \"insights-runtime-extractor-t9dk7\" (UID: \"0b1d1a91-1f92-4209-af86-f3496494d229\") " pod="openshift-insights/insights-runtime-extractor-t9dk7"
Jul 07 16:48:32.242075 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.242049    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fa61b07-1978-4db6-9e3f-0dd7862906ef-tmp-dir\") pod \"dns-default-hh2zz\" (UID: \"7fa61b07-1978-4db6-9e3f-0dd7862906ef\") " pod="openshift-dns/dns-default-hh2zz"
Jul 07 16:48:32.242154 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.242122    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b1d1a91-1f92-4209-af86-f3496494d229-data-volume\") pod \"insights-runtime-extractor-t9dk7\" (UID: \"0b1d1a91-1f92-4209-af86-f3496494d229\") " pod="openshift-insights/insights-runtime-extractor-t9dk7"
Jul 07 16:48:32.242192 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.242168    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwwck\" (UniqueName: \"kubernetes.io/projected/7fa61b07-1978-4db6-9e3f-0dd7862906ef-kube-api-access-hwwck\") pod \"dns-default-hh2zz\" (UID: \"7fa61b07-1978-4db6-9e3f-0dd7862906ef\") " pod="openshift-dns/dns-default-hh2zz"
Jul 07 16:48:32.242238 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.242208    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0b1d1a91-1f92-4209-af86-f3496494d229-crio-socket\") pod \"insights-runtime-extractor-t9dk7\" (UID: \"0b1d1a91-1f92-4209-af86-f3496494d229\") " pod="openshift-insights/insights-runtime-extractor-t9dk7"
Jul 07 16:48:32.242298 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.242279    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/06abc595-8bff-454f-99e4-959c9ffcc06b-cert\") pod \"ingress-canary-lmvlr\" (UID: \"06abc595-8bff-454f-99e4-959c9ffcc06b\") " pod="openshift-ingress-canary/ingress-canary-lmvlr"
Jul 07 16:48:32.242352 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.242299    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b1d1a91-1f92-4209-af86-f3496494d229-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t9dk7\" (UID: \"0b1d1a91-1f92-4209-af86-f3496494d229\") " pod="openshift-insights/insights-runtime-extractor-t9dk7"
Jul 07 16:48:32.242422 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.242406    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0b1d1a91-1f92-4209-af86-f3496494d229-crio-socket\") pod \"insights-runtime-extractor-t9dk7\" (UID: \"0b1d1a91-1f92-4209-af86-f3496494d229\") " pod="openshift-insights/insights-runtime-extractor-t9dk7"
Jul 07 16:48:32.245477 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.245458    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b1d1a91-1f92-4209-af86-f3496494d229-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t9dk7\" (UID: \"0b1d1a91-1f92-4209-af86-f3496494d229\") " pod="openshift-insights/insights-runtime-extractor-t9dk7"
Jul 07 16:48:32.245581 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.245565    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/06abc595-8bff-454f-99e4-959c9ffcc06b-cert\") pod \"ingress-canary-lmvlr\" (UID: \"06abc595-8bff-454f-99e4-959c9ffcc06b\") " pod="openshift-ingress-canary/ingress-canary-lmvlr"
Jul 07 16:48:32.252655 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.252627    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2mwq\" (UniqueName: \"kubernetes.io/projected/0b1d1a91-1f92-4209-af86-f3496494d229-kube-api-access-t2mwq\") pod \"insights-runtime-extractor-t9dk7\" (UID: \"0b1d1a91-1f92-4209-af86-f3496494d229\") " pod="openshift-insights/insights-runtime-extractor-t9dk7"
Jul 07 16:48:32.252813 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.252797    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvd5n\" (UniqueName: \"kubernetes.io/projected/06abc595-8bff-454f-99e4-959c9ffcc06b-kube-api-access-gvd5n\") pod \"ingress-canary-lmvlr\" (UID: \"06abc595-8bff-454f-99e4-959c9ffcc06b\") " pod="openshift-ingress-canary/ingress-canary-lmvlr"
Jul 07 16:48:32.342786 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.342760    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7fa61b07-1978-4db6-9e3f-0dd7862906ef-config-volume\") pod \"dns-default-hh2zz\" (UID: \"7fa61b07-1978-4db6-9e3f-0dd7862906ef\") " pod="openshift-dns/dns-default-hh2zz"
Jul 07 16:48:32.342884 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.342795    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7fa61b07-1978-4db6-9e3f-0dd7862906ef-metrics-tls\") pod \"dns-default-hh2zz\" (UID: \"7fa61b07-1978-4db6-9e3f-0dd7862906ef\") " pod="openshift-dns/dns-default-hh2zz"
Jul 07 16:48:32.342884 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.342838    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fa61b07-1978-4db6-9e3f-0dd7862906ef-tmp-dir\") pod \"dns-default-hh2zz\" (UID: \"7fa61b07-1978-4db6-9e3f-0dd7862906ef\") " pod="openshift-dns/dns-default-hh2zz"
Jul 07 16:48:32.342884 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.342860    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwwck\" (UniqueName: \"kubernetes.io/projected/7fa61b07-1978-4db6-9e3f-0dd7862906ef-kube-api-access-hwwck\") pod \"dns-default-hh2zz\" (UID: \"7fa61b07-1978-4db6-9e3f-0dd7862906ef\") " pod="openshift-dns/dns-default-hh2zz"
Jul 07 16:48:32.343227 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.343203    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fa61b07-1978-4db6-9e3f-0dd7862906ef-tmp-dir\") pod \"dns-default-hh2zz\" (UID: \"7fa61b07-1978-4db6-9e3f-0dd7862906ef\") " pod="openshift-dns/dns-default-hh2zz"
Jul 07 16:48:32.343342 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.343325    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7fa61b07-1978-4db6-9e3f-0dd7862906ef-config-volume\") pod \"dns-default-hh2zz\" (UID: \"7fa61b07-1978-4db6-9e3f-0dd7862906ef\") " pod="openshift-dns/dns-default-hh2zz"
Jul 07 16:48:32.344787 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.344769    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7fa61b07-1978-4db6-9e3f-0dd7862906ef-metrics-tls\") pod \"dns-default-hh2zz\" (UID: \"7fa61b07-1978-4db6-9e3f-0dd7862906ef\") " pod="openshift-dns/dns-default-hh2zz"
Jul 07 16:48:32.351156 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.351141    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwwck\" (UniqueName: \"kubernetes.io/projected/7fa61b07-1978-4db6-9e3f-0dd7862906ef-kube-api-access-hwwck\") pod \"dns-default-hh2zz\" (UID: \"7fa61b07-1978-4db6-9e3f-0dd7862906ef\") " pod="openshift-dns/dns-default-hh2zz"
Jul 07 16:48:32.404142 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.404084    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lmvlr"
Jul 07 16:48:32.432961 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.432944    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-t9dk7"
Jul 07 16:48:32.504540 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.504511    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hh2zz"
Jul 07 16:48:32.604560 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.604384    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-t9dk7"]
Jul 07 16:48:32.617857 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.617792    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lmvlr"]
Jul 07 16:48:32.621267 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:48:32.621242    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06abc595_8bff_454f_99e4_959c9ffcc06b.slice/crio-8261c7cacff258304e2070985042237e4365538d33efbbabfa7c202b3d955ea7 WatchSource:0}: Error finding container 8261c7cacff258304e2070985042237e4365538d33efbbabfa7c202b3d955ea7: Status 404 returned error can't find the container with id 8261c7cacff258304e2070985042237e4365538d33efbbabfa7c202b3d955ea7
Jul 07 16:48:32.658153 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.658093    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hh2zz"]
Jul 07 16:48:32.660550 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:48:32.660525    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fa61b07_1978_4db6_9e3f_0dd7862906ef.slice/crio-bb947cb1b8cb673fa8872a7fb7742afa5a9833df8557a1ed39d97cf48efd7681 WatchSource:0}: Error finding container bb947cb1b8cb673fa8872a7fb7742afa5a9833df8557a1ed39d97cf48efd7681: Status 404 returned error can't find the container with id bb947cb1b8cb673fa8872a7fb7742afa5a9833df8557a1ed39d97cf48efd7681
Jul 07 16:48:32.719788 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.719764    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdcss"
Jul 07 16:48:32.719788 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.719778    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h5f7f"
Jul 07 16:48:32.719915 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.719860    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wwb84"
Jul 07 16:48:32.722571 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.722555    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 07 16:48:32.722671 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.722603    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 07 16:48:32.722671 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.722651    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jwbcf\""
Jul 07 16:48:32.723606 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.723586    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 07 16:48:32.723813 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.723796    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sk64f\""
Jul 07 16:48:32.723884 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.723813    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 07 16:48:32.927669 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.927577    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lmvlr" event={"ID":"06abc595-8bff-454f-99e4-959c9ffcc06b","Type":"ContainerStarted","Data":"8261c7cacff258304e2070985042237e4365538d33efbbabfa7c202b3d955ea7"}
Jul 07 16:48:32.928679 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.928649    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hh2zz" event={"ID":"7fa61b07-1978-4db6-9e3f-0dd7862906ef","Type":"ContainerStarted","Data":"bb947cb1b8cb673fa8872a7fb7742afa5a9833df8557a1ed39d97cf48efd7681"}
Jul 07 16:48:32.929624 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:32.929604    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t9dk7" event={"ID":"0b1d1a91-1f92-4209-af86-f3496494d229","Type":"ContainerStarted","Data":"40610958c9cadf4a1babe314b8dd416fe0881ae16488118b328c0e6272c5f708"}
Jul 07 16:48:36.791340 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:36.791309    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-qtmps"]
Jul 07 16:48:36.794225 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:36.794202    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-qtmps"
Jul 07 16:48:36.796844 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:36.796818    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 07 16:48:36.796844 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:36.796836    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-sl82z\""
Jul 07 16:48:36.804833 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:36.804811    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-qtmps"]
Jul 07 16:48:36.873497 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:36.873458    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c326cb78-d362-4afb-96ef-dde6c98bafc8-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-qtmps\" (UID: \"c326cb78-d362-4afb-96ef-dde6c98bafc8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-qtmps"
Jul 07 16:48:36.974952 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:36.974915    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c326cb78-d362-4afb-96ef-dde6c98bafc8-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-qtmps\" (UID: \"c326cb78-d362-4afb-96ef-dde6c98bafc8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-qtmps"
Jul 07 16:48:36.977478 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:36.977458    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c326cb78-d362-4afb-96ef-dde6c98bafc8-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-qtmps\" (UID: \"c326cb78-d362-4afb-96ef-dde6c98bafc8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-qtmps"
Jul 07 16:48:37.105903 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:37.105861    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-qtmps"
Jul 07 16:48:37.758594 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:37.757727    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-qtmps"]
Jul 07 16:48:37.763913 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:48:37.763853    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc326cb78_d362_4afb_96ef_dde6c98bafc8.slice/crio-74fa75f01f859d2dd0b813dacea9f25ce1b68e1ca511414ec0e80375240c16b9 WatchSource:0}: Error finding container 74fa75f01f859d2dd0b813dacea9f25ce1b68e1ca511414ec0e80375240c16b9: Status 404 returned error can't find the container with id 74fa75f01f859d2dd0b813dacea9f25ce1b68e1ca511414ec0e80375240c16b9
Jul 07 16:48:37.944856 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:37.944773    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lmvlr" event={"ID":"06abc595-8bff-454f-99e4-959c9ffcc06b","Type":"ContainerStarted","Data":"604160bed6b9cc2154edb610ef10a7417901124088a86e430ac70ace1013bf68"}
Jul 07 16:48:37.946524 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:37.946496    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hh2zz" event={"ID":"7fa61b07-1978-4db6-9e3f-0dd7862906ef","Type":"ContainerStarted","Data":"90e571f71a58ba9dc00a56d6ea413ff409d3e3521535f1290b882500e230d738"}
Jul 07 16:48:37.946650 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:37.946547    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hh2zz" event={"ID":"7fa61b07-1978-4db6-9e3f-0dd7862906ef","Type":"ContainerStarted","Data":"114ff114884c17bf870e9905ebc0146ceca56bf14ac120a017b74450f48c939f"}
Jul 07 16:48:37.946650 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:37.946574    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-hh2zz"
Jul 07 16:48:37.947906 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:37.947883    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t9dk7" event={"ID":"0b1d1a91-1f92-4209-af86-f3496494d229","Type":"ContainerStarted","Data":"df462ed65d73486964882c3724f2865e135f012670c027b85ff1f7a9571da59f"}
Jul 07 16:48:37.948946 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:37.948927    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-qtmps" event={"ID":"c326cb78-d362-4afb-96ef-dde6c98bafc8","Type":"ContainerStarted","Data":"74fa75f01f859d2dd0b813dacea9f25ce1b68e1ca511414ec0e80375240c16b9"}
Jul 07 16:48:37.983996 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:37.983948    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-lmvlr" podStartSLOduration=0.966320556 podStartE2EDuration="5.983930005s" totalImagesPullingTime="5.017609449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:48:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:48:37.965230083 +0000 UTC m=+54.741544289" watchObservedRunningTime="2026-07-07 16:48:37.983930005 +0000 UTC m=+54.760244213"
Jul 07 16:48:37.984126 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:37.984046    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-hh2zz" podStartSLOduration=1.029437509 podStartE2EDuration="5.98404098s" totalImagesPullingTime="4.954603471s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:48:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:48:37.983216196 +0000 UTC m=+54.759530416" watchObservedRunningTime="2026-07-07 16:48:37.98404098 +0000 UTC m=+54.760355186"
Jul 07 16:48:38.955957 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:38.955923    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t9dk7" event={"ID":"0b1d1a91-1f92-4209-af86-f3496494d229","Type":"ContainerStarted","Data":"f16aed82003f5ad74aa3d6d2b1135cb8712fa8f0558c6a240959a6de9476e98f"}
Jul 07 16:48:38.957626 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:38.957574    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-qtmps" event={"ID":"c326cb78-d362-4afb-96ef-dde6c98bafc8","Type":"ContainerStarted","Data":"2b43dd104b291b9bc11cde48dc6fae2980fd23a9e40be54514d83b541a096881"}
Jul 07 16:48:38.957976 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:38.957949    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-qtmps"
Jul 07 16:48:38.963070 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:38.963046    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-qtmps"
Jul 07 16:48:38.974725 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:38.974685    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-qtmps" podStartSLOduration=1.917686518 podStartE2EDuration="2.974671344s" totalImagesPullingTime="1.056984826s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:48:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:48:38.973875472 +0000 UTC m=+55.750189703" watchObservedRunningTime="2026-07-07 16:48:38.974671344 +0000 UTC m=+55.750985528"
Jul 07 16:48:39.856093 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:39.856065    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-p7svq"]
Jul 07 16:48:39.869745 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:39.869711    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-p7svq"
Jul 07 16:48:39.873293 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:39.873273    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 07 16:48:39.874366 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:39.874351    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 07 16:48:39.874463 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:39.874447    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 07 16:48:39.876107 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:39.875714    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 07 16:48:39.876107 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:39.875724    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-j7gld\""
Jul 07 16:48:39.876107 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:39.875780    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 07 16:48:39.878966 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:39.878945    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-p7svq"]
Jul 07 16:48:39.899432 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:39.899408    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20a7b5cf-65d2-4a85-88ed-d4b104cf5707-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-p7svq\" (UID: \"20a7b5cf-65d2-4a85-88ed-d4b104cf5707\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-p7svq"
Jul 07 16:48:39.899543 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:39.899464    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/20a7b5cf-65d2-4a85-88ed-d4b104cf5707-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-p7svq\" (UID: \"20a7b5cf-65d2-4a85-88ed-d4b104cf5707\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-p7svq"
Jul 07 16:48:39.899543 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:39.899521    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bqsz\" (UniqueName: \"kubernetes.io/projected/20a7b5cf-65d2-4a85-88ed-d4b104cf5707-kube-api-access-2bqsz\") pod \"prometheus-operator-567f7b8b7c-p7svq\" (UID: \"20a7b5cf-65d2-4a85-88ed-d4b104cf5707\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-p7svq"
Jul 07 16:48:39.899630 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:39.899555    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/20a7b5cf-65d2-4a85-88ed-d4b104cf5707-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-p7svq\" (UID: \"20a7b5cf-65d2-4a85-88ed-d4b104cf5707\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-p7svq"
Jul 07 16:48:40.000361 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:40.000325    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/20a7b5cf-65d2-4a85-88ed-d4b104cf5707-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-p7svq\" (UID: \"20a7b5cf-65d2-4a85-88ed-d4b104cf5707\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-p7svq"
Jul 07 16:48:40.000361 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:40.000361    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bqsz\" (UniqueName: \"kubernetes.io/projected/20a7b5cf-65d2-4a85-88ed-d4b104cf5707-kube-api-access-2bqsz\") pod \"prometheus-operator-567f7b8b7c-p7svq\" (UID: \"20a7b5cf-65d2-4a85-88ed-d4b104cf5707\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-p7svq"
Jul 07 16:48:40.000871 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:40.000395    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/20a7b5cf-65d2-4a85-88ed-d4b104cf5707-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-p7svq\" (UID: \"20a7b5cf-65d2-4a85-88ed-d4b104cf5707\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-p7svq"
Jul 07 16:48:40.000871 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:40.000490    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20a7b5cf-65d2-4a85-88ed-d4b104cf5707-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-p7svq\" (UID: \"20a7b5cf-65d2-4a85-88ed-d4b104cf5707\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-p7svq"
Jul 07 16:48:40.000871 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:40.000601    2578 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 07 16:48:40.000871 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:40.000673    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20a7b5cf-65d2-4a85-88ed-d4b104cf5707-prometheus-operator-tls podName:20a7b5cf-65d2-4a85-88ed-d4b104cf5707 nodeName:}" failed. No retries permitted until 2026-07-07 16:48:40.500654882 +0000 UTC m=+57.276969089 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/20a7b5cf-65d2-4a85-88ed-d4b104cf5707-prometheus-operator-tls") pod "prometheus-operator-567f7b8b7c-p7svq" (UID: "20a7b5cf-65d2-4a85-88ed-d4b104cf5707") : secret "prometheus-operator-tls" not found
Jul 07 16:48:40.001197 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:40.001180    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20a7b5cf-65d2-4a85-88ed-d4b104cf5707-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-p7svq\" (UID: \"20a7b5cf-65d2-4a85-88ed-d4b104cf5707\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-p7svq"
Jul 07 16:48:40.002973 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:40.002953    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/20a7b5cf-65d2-4a85-88ed-d4b104cf5707-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-p7svq\" (UID: \"20a7b5cf-65d2-4a85-88ed-d4b104cf5707\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-p7svq"
Jul 07 16:48:40.010780 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:40.010758    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bqsz\" (UniqueName: \"kubernetes.io/projected/20a7b5cf-65d2-4a85-88ed-d4b104cf5707-kube-api-access-2bqsz\") pod \"prometheus-operator-567f7b8b7c-p7svq\" (UID: \"20a7b5cf-65d2-4a85-88ed-d4b104cf5707\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-p7svq"
Jul 07 16:48:40.452861 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:40.452827    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7475fc7665-d9jpq"]
Jul 07 16:48:40.505118 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:40.505086    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/20a7b5cf-65d2-4a85-88ed-d4b104cf5707-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-p7svq\" (UID: \"20a7b5cf-65d2-4a85-88ed-d4b104cf5707\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-p7svq"
Jul 07 16:48:40.507748 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:40.507699    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/20a7b5cf-65d2-4a85-88ed-d4b104cf5707-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-p7svq\" (UID: \"20a7b5cf-65d2-4a85-88ed-d4b104cf5707\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-p7svq"
Jul 07 16:48:40.510199 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:40.510172    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7475fc7665-d9jpq"]
Jul 07 16:48:40.510313 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:40.510295    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7475fc7665-d9jpq"
Jul 07 16:48:40.514315 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:40.514064    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 07 16:48:40.514315 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:40.514065    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 07 16:48:40.514315 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:40.514147    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 07 16:48:40.514315 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:40.514123    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 07 16:48:40.514315 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:40.514195    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 07 16:48:40.514621 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:40.514565    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 07 16:48:40.514621 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:40.514589    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-54jkp\""
Jul 07 16:48:40.514725 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:40.514646    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 07 16:48:40.605776 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:40.605713    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5790cd94-92b3-46f3-8441-caf5efcb2779-service-ca\") pod \"console-7475fc7665-d9jpq\" (UID: \"5790cd94-92b3-46f3-8441-caf5efcb2779\") " pod="openshift-console/console-7475fc7665-d9jpq"
Jul 07 16:48:40.605776 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:40.605773    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5790cd94-92b3-46f3-8441-caf5efcb2779-oauth-serving-cert\") pod \"console-7475fc7665-d9jpq\" (UID: \"5790cd94-92b3-46f3-8441-caf5efcb2779\") " pod="openshift-console/console-7475fc7665-d9jpq"
Jul 07 16:48:40.606020 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:40.605823    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pczm4\" (UniqueName: \"kubernetes.io/projected/5790cd94-92b3-46f3-8441-caf5efcb2779-kube-api-access-pczm4\") pod \"console-7475fc7665-d9jpq\" (UID: \"5790cd94-92b3-46f3-8441-caf5efcb2779\") " pod="openshift-console/console-7475fc7665-d9jpq"
Jul 07 16:48:40.606020 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:40.605895    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5790cd94-92b3-46f3-8441-caf5efcb2779-console-serving-cert\") pod \"console-7475fc7665-d9jpq\" (UID: \"5790cd94-92b3-46f3-8441-caf5efcb2779\") " pod="openshift-console/console-7475fc7665-d9jpq"
Jul 07 16:48:40.606020 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:40.605955    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5790cd94-92b3-46f3-8441-caf5efcb2779-console-oauth-config\") pod \"console-7475fc7665-d9jpq\" (UID: \"5790cd94-92b3-46f3-8441-caf5efcb2779\") " pod="openshift-console/console-7475fc7665-d9jpq"
Jul 07 16:48:40.606020 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:40.605974    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5790cd94-92b3-46f3-8441-caf5efcb2779-console-config\") pod \"console-7475fc7665-d9jpq\" (UID: \"5790cd94-92b3-46f3-8441-caf5efcb2779\") " pod="openshift-console/console-7475fc7665-d9jpq"
Jul 07 16:48:40.706967 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:40.706899    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5790cd94-92b3-46f3-8441-caf5efcb2779-console-serving-cert\") pod \"console-7475fc7665-d9jpq\" (UID: \"5790cd94-92b3-46f3-8441-caf5efcb2779\") " pod="openshift-console/console-7475fc7665-d9jpq"
Jul 07 16:48:40.706967 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:40.706953    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5790cd94-92b3-46f3-8441-caf5efcb2779-console-oauth-config\") pod \"console-7475fc7665-d9jpq\" (UID: \"5790cd94-92b3-46f3-8441-caf5efcb2779\") " pod="openshift-console/console-7475fc7665-d9jpq"
Jul 07 16:48:40.707176 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:40.706972    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5790cd94-92b3-46f3-8441-caf5efcb2779-console-config\") pod \"console-7475fc7665-d9jpq\" (UID: \"5790cd94-92b3-46f3-8441-caf5efcb2779\") " pod="openshift-console/console-7475fc7665-d9jpq"
Jul 07 16:48:40.707176 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:40.707051    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5790cd94-92b3-46f3-8441-caf5efcb2779-service-ca\") pod \"console-7475fc7665-d9jpq\" (UID: \"5790cd94-92b3-46f3-8441-caf5efcb2779\") " pod="openshift-console/console-7475fc7665-d9jpq"
Jul 07 16:48:40.707176 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:40.707074    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5790cd94-92b3-46f3-8441-caf5efcb2779-oauth-serving-cert\") pod \"console-7475fc7665-d9jpq\" (UID: \"5790cd94-92b3-46f3-8441-caf5efcb2779\") " pod="openshift-console/console-7475fc7665-d9jpq"
Jul 07 16:48:40.707176 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:40.707104    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pczm4\" (UniqueName: \"kubernetes.io/projected/5790cd94-92b3-46f3-8441-caf5efcb2779-kube-api-access-pczm4\") pod \"console-7475fc7665-d9jpq\" (UID: \"5790cd94-92b3-46f3-8441-caf5efcb2779\") " pod="openshift-console/console-7475fc7665-d9jpq"
Jul 07 16:48:40.707834 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:40.707805    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5790cd94-92b3-46f3-8441-caf5efcb2779-service-ca\") pod \"console-7475fc7665-d9jpq\" (UID: \"5790cd94-92b3-46f3-8441-caf5efcb2779\") " pod="openshift-console/console-7475fc7665-d9jpq"
Jul 07 16:48:40.707950 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:40.707924    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5790cd94-92b3-46f3-8441-caf5efcb2779-oauth-serving-cert\") pod \"console-7475fc7665-d9jpq\" (UID: \"5790cd94-92b3-46f3-8441-caf5efcb2779\") " pod="openshift-console/console-7475fc7665-d9jpq"
Jul 07 16:48:40.707950 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:40.707920    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5790cd94-92b3-46f3-8441-caf5efcb2779-console-config\") pod \"console-7475fc7665-d9jpq\" (UID: \"5790cd94-92b3-46f3-8441-caf5efcb2779\") " pod="openshift-console/console-7475fc7665-d9jpq"
Jul 07 16:48:40.709532 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:40.709511    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5790cd94-92b3-46f3-8441-caf5efcb2779-console-oauth-config\") pod \"console-7475fc7665-d9jpq\" (UID: \"5790cd94-92b3-46f3-8441-caf5efcb2779\") " pod="openshift-console/console-7475fc7665-d9jpq"
Jul 07 16:48:40.709633 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:40.709574    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5790cd94-92b3-46f3-8441-caf5efcb2779-console-serving-cert\") pod \"console-7475fc7665-d9jpq\" (UID: \"5790cd94-92b3-46f3-8441-caf5efcb2779\") " pod="openshift-console/console-7475fc7665-d9jpq"
Jul 07 16:48:40.717128 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:40.717106    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pczm4\" (UniqueName: \"kubernetes.io/projected/5790cd94-92b3-46f3-8441-caf5efcb2779-kube-api-access-pczm4\") pod \"console-7475fc7665-d9jpq\" (UID: \"5790cd94-92b3-46f3-8441-caf5efcb2779\") " pod="openshift-console/console-7475fc7665-d9jpq"
Jul 07 16:48:40.779014 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:40.778977    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-p7svq"
Jul 07 16:48:40.823413 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:40.823376    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7475fc7665-d9jpq"
Jul 07 16:48:41.135359 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:41.135328    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7475fc7665-d9jpq"]
Jul 07 16:48:41.139836 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:48:41.139807    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5790cd94_92b3_46f3_8441_caf5efcb2779.slice/crio-a5457d863d8ef6bf26b9fea7da21807ca68ff795971afca9929613415f96c852 WatchSource:0}: Error finding container a5457d863d8ef6bf26b9fea7da21807ca68ff795971afca9929613415f96c852: Status 404 returned error can't find the container with id a5457d863d8ef6bf26b9fea7da21807ca68ff795971afca9929613415f96c852
Jul 07 16:48:41.150099 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:41.150075    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-p7svq"]
Jul 07 16:48:41.155443 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:48:41.155424    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20a7b5cf_65d2_4a85_88ed_d4b104cf5707.slice/crio-fe925c8481e74b3f8eb9369c5eb0059aa18dcd58b2032ead369c002433863386 WatchSource:0}: Error finding container fe925c8481e74b3f8eb9369c5eb0059aa18dcd58b2032ead369c002433863386: Status 404 returned error can't find the container with id fe925c8481e74b3f8eb9369c5eb0059aa18dcd58b2032ead369c002433863386
Jul 07 16:48:41.970272 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:41.970230    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-p7svq" event={"ID":"20a7b5cf-65d2-4a85-88ed-d4b104cf5707","Type":"ContainerStarted","Data":"fe925c8481e74b3f8eb9369c5eb0059aa18dcd58b2032ead369c002433863386"}
Jul 07 16:48:41.973069 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:41.972994    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t9dk7" event={"ID":"0b1d1a91-1f92-4209-af86-f3496494d229","Type":"ContainerStarted","Data":"8f9ec4fbdf995cbfa0c6c0a2991078520d50fde34b2ed31ba2204715a61de723"}
Jul 07 16:48:41.974707 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:41.974675    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7475fc7665-d9jpq" event={"ID":"5790cd94-92b3-46f3-8441-caf5efcb2779","Type":"ContainerStarted","Data":"a5457d863d8ef6bf26b9fea7da21807ca68ff795971afca9929613415f96c852"}
Jul 07 16:48:41.993468 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:41.993412    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-t9dk7" podStartSLOduration=1.590646896 podStartE2EDuration="9.993391213s" totalImagesPullingTime="8.402744317s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:48:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:48:41.992257723 +0000 UTC m=+58.768571973" watchObservedRunningTime="2026-07-07 16:48:41.993391213 +0000 UTC m=+58.769705424"
Jul 07 16:48:42.982363 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:42.982265    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-p7svq" event={"ID":"20a7b5cf-65d2-4a85-88ed-d4b104cf5707","Type":"ContainerStarted","Data":"d8bb680f837858ba90fa1b9332e071a6830f7ccc2ff5c57e7d309b94b967437a"}
Jul 07 16:48:42.982363 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:42.982341    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-p7svq" event={"ID":"20a7b5cf-65d2-4a85-88ed-d4b104cf5707","Type":"ContainerStarted","Data":"8bda554dfe79b0f87dcc1072955998fd0a1db2b84dd0dda7d75877b7b4dcb054"}
Jul 07 16:48:43.001076 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:43.001033    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-p7svq" podStartSLOduration=2.579512219 podStartE2EDuration="4.001017272s" totalImagesPullingTime="1.421505053s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:48:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:48:43.000275183 +0000 UTC m=+59.776589422" watchObservedRunningTime="2026-07-07 16:48:43.001017272 +0000 UTC m=+59.777331477"
Jul 07 16:48:44.991835 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:44.991791    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7475fc7665-d9jpq" event={"ID":"5790cd94-92b3-46f3-8441-caf5efcb2779","Type":"ContainerStarted","Data":"7448d5f51370ad91a6706d48cfc5e17b80ba3e45f6b333d31bfd731d142e27e1"}
Jul 07 16:48:45.018517 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:45.018472    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7475fc7665-d9jpq" podStartSLOduration=2.113245666 podStartE2EDuration="5.018456758s" totalImagesPullingTime="2.905211092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:48:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:48:45.017595837 +0000 UTC m=+61.793910044" watchObservedRunningTime="2026-07-07 16:48:45.018456758 +0000 UTC m=+61.794771034"
Jul 07 16:48:45.279212 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:45.279142    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-nfgmm"]
Jul 07 16:48:45.281321 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:45.281305    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nfgmm"
Jul 07 16:48:45.284225 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:45.284200    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 07 16:48:45.284389 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:45.284342    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5gvkf\""
Jul 07 16:48:45.284389 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:45.284340    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 07 16:48:45.284595 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:45.284583    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 07 16:48:45.345571 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:45.345541    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nfgmm\" (UID: \"b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168\") " pod="openshift-monitoring/node-exporter-nfgmm"
Jul 07 16:48:45.345722 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:45.345582    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168-root\") pod \"node-exporter-nfgmm\" (UID: \"b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168\") " pod="openshift-monitoring/node-exporter-nfgmm"
Jul 07 16:48:45.345722 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:45.345606    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168-node-exporter-accelerators-collector-config\") pod \"node-exporter-nfgmm\" (UID: \"b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168\") " pod="openshift-monitoring/node-exporter-nfgmm"
Jul 07 16:48:45.345722 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:45.345652    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168-metrics-client-ca\") pod \"node-exporter-nfgmm\" (UID: \"b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168\") " pod="openshift-monitoring/node-exporter-nfgmm"
Jul 07 16:48:45.345722 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:45.345688    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fqcx\" (UniqueName: \"kubernetes.io/projected/b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168-kube-api-access-9fqcx\") pod \"node-exporter-nfgmm\" (UID: \"b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168\") " pod="openshift-monitoring/node-exporter-nfgmm"
Jul 07 16:48:45.345722 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:45.345716    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168-sys\") pod \"node-exporter-nfgmm\" (UID: \"b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168\") " pod="openshift-monitoring/node-exporter-nfgmm"
Jul 07 16:48:45.345934 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:45.345748    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168-node-exporter-wtmp\") pod \"node-exporter-nfgmm\" (UID: \"b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168\") " pod="openshift-monitoring/node-exporter-nfgmm"
Jul 07 16:48:45.345934 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:45.345796    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168-node-exporter-textfile\") pod \"node-exporter-nfgmm\" (UID: \"b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168\") " pod="openshift-monitoring/node-exporter-nfgmm"
Jul 07 16:48:45.345934 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:45.345884    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168-node-exporter-tls\") pod \"node-exporter-nfgmm\" (UID: \"b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168\") " pod="openshift-monitoring/node-exporter-nfgmm"
Jul 07 16:48:45.446616 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:45.446580    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168-node-exporter-tls\") pod \"node-exporter-nfgmm\" (UID: \"b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168\") " pod="openshift-monitoring/node-exporter-nfgmm"
Jul 07 16:48:45.446794 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:45.446638    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nfgmm\" (UID: \"b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168\") " pod="openshift-monitoring/node-exporter-nfgmm"
Jul 07 16:48:45.446794 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:45.446674    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168-root\") pod \"node-exporter-nfgmm\" (UID: \"b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168\") " pod="openshift-monitoring/node-exporter-nfgmm"
Jul 07 16:48:45.446794 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:45.446699    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168-node-exporter-accelerators-collector-config\") pod \"node-exporter-nfgmm\" (UID: \"b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168\") " pod="openshift-monitoring/node-exporter-nfgmm"
Jul 07 16:48:45.446794 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:45.446769    2578 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 07 16:48:45.447013 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:45.446806    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168-root\") pod \"node-exporter-nfgmm\" (UID: \"b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168\") " pod="openshift-monitoring/node-exporter-nfgmm"
Jul 07 16:48:45.447013 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:48:45.446829    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168-node-exporter-tls podName:b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168 nodeName:}" failed. No retries permitted until 2026-07-07 16:48:45.946810077 +0000 UTC m=+62.723124271 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168-node-exporter-tls") pod "node-exporter-nfgmm" (UID: "b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168") : secret "node-exporter-tls" not found
Jul 07 16:48:45.447013 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:45.446863    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168-metrics-client-ca\") pod \"node-exporter-nfgmm\" (UID: \"b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168\") " pod="openshift-monitoring/node-exporter-nfgmm"
Jul 07 16:48:45.447013 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:45.446903    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fqcx\" (UniqueName: \"kubernetes.io/projected/b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168-kube-api-access-9fqcx\") pod \"node-exporter-nfgmm\" (UID: \"b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168\") " pod="openshift-monitoring/node-exporter-nfgmm"
Jul 07 16:48:45.447013 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:45.446942    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168-sys\") pod \"node-exporter-nfgmm\" (UID: \"b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168\") " pod="openshift-monitoring/node-exporter-nfgmm"
Jul 07 16:48:45.447013 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:45.446967    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168-node-exporter-wtmp\") pod \"node-exporter-nfgmm\" (UID: \"b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168\") " pod="openshift-monitoring/node-exporter-nfgmm"
Jul 07 16:48:45.447013 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:45.447004    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168-node-exporter-textfile\") pod \"node-exporter-nfgmm\" (UID: \"b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168\") " pod="openshift-monitoring/node-exporter-nfgmm"
Jul 07 16:48:45.447349 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:45.447297    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168-sys\") pod \"node-exporter-nfgmm\" (UID: \"b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168\") " pod="openshift-monitoring/node-exporter-nfgmm"
Jul 07 16:48:45.447349 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:45.447306    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168-node-exporter-accelerators-collector-config\") pod \"node-exporter-nfgmm\" (UID: \"b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168\") " pod="openshift-monitoring/node-exporter-nfgmm"
Jul 07 16:48:45.447432 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:45.447351    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168-node-exporter-textfile\") pod \"node-exporter-nfgmm\" (UID: \"b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168\") " pod="openshift-monitoring/node-exporter-nfgmm"
Jul 07 16:48:45.447432 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:45.447415    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168-node-exporter-wtmp\") pod \"node-exporter-nfgmm\" (UID: \"b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168\") " pod="openshift-monitoring/node-exporter-nfgmm"
Jul 07 16:48:45.447704 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:45.447681    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168-metrics-client-ca\") pod \"node-exporter-nfgmm\" (UID: \"b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168\") " pod="openshift-monitoring/node-exporter-nfgmm"
Jul 07 16:48:45.449551 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:45.449527    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nfgmm\" (UID: \"b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168\") " pod="openshift-monitoring/node-exporter-nfgmm"
Jul 07 16:48:45.456562 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:45.456542    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fqcx\" (UniqueName: \"kubernetes.io/projected/b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168-kube-api-access-9fqcx\") pod \"node-exporter-nfgmm\" (UID: \"b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168\") " pod="openshift-monitoring/node-exporter-nfgmm"
Jul 07 16:48:45.629971 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:45.629948    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bjtgg"
Jul 07 16:48:45.951298 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:45.951215    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168-node-exporter-tls\") pod \"node-exporter-nfgmm\" (UID: \"b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168\") " pod="openshift-monitoring/node-exporter-nfgmm"
Jul 07 16:48:45.953564 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:45.953537    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168-node-exporter-tls\") pod \"node-exporter-nfgmm\" (UID: \"b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168\") " pod="openshift-monitoring/node-exporter-nfgmm"
Jul 07 16:48:46.190295 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:46.190042    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nfgmm"
Jul 07 16:48:46.200632 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:48:46.200604    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5d5b47b_4f72_4fe3_b59e_b2c3f8ab7168.slice/crio-d3ce1eea644ef22d5b743e66fa513e939f7c552355b64c86fd6386afcf80fbf1 WatchSource:0}: Error finding container d3ce1eea644ef22d5b743e66fa513e939f7c552355b64c86fd6386afcf80fbf1: Status 404 returned error can't find the container with id d3ce1eea644ef22d5b743e66fa513e939f7c552355b64c86fd6386afcf80fbf1
Jul 07 16:48:46.339143 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:46.339111    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 07 16:48:46.342214 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:46.342188    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:46.345520 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:46.345498    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 07 16:48:46.345642 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:46.345498    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 07 16:48:46.345642 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:46.345537    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 07 16:48:46.345642 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:46.345531    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 07 16:48:46.345894 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:46.345795    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-m9wrw\""
Jul 07 16:48:46.345894 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:46.345857    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 07 16:48:46.345894 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:46.345864    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 07 16:48:46.345894 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:46.345876    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 07 16:48:46.346126 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:46.345877    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 07 16:48:46.346363 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:46.346348    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 07 16:48:46.359806 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:46.359783    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 07 16:48:46.456325 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:46.456248    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/835f4b2a-30d0-441b-af00-f0b2d166fe7c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:46.456325 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:46.456279    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/835f4b2a-30d0-441b-af00-f0b2d166fe7c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:46.456325 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:46.456298    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/835f4b2a-30d0-441b-af00-f0b2d166fe7c-config-volume\") pod \"alertmanager-main-0\" (UID: \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:46.456325 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:46.456314    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/835f4b2a-30d0-441b-af00-f0b2d166fe7c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:46.456590 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:46.456367    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/835f4b2a-30d0-441b-af00-f0b2d166fe7c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:46.456590 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:46.456404    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/835f4b2a-30d0-441b-af00-f0b2d166fe7c-web-config\") pod \"alertmanager-main-0\" (UID: \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:46.456590 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:46.456427    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/835f4b2a-30d0-441b-af00-f0b2d166fe7c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:46.456590 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:46.456448    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/835f4b2a-30d0-441b-af00-f0b2d166fe7c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:46.456590 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:46.456474    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrbf9\" (UniqueName: \"kubernetes.io/projected/835f4b2a-30d0-441b-af00-f0b2d166fe7c-kube-api-access-zrbf9\") pod \"alertmanager-main-0\" (UID: \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:46.456590 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:46.456523    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/835f4b2a-30d0-441b-af00-f0b2d166fe7c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:46.456590 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:46.456540    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/835f4b2a-30d0-441b-af00-f0b2d166fe7c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:46.456590 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:46.456585    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/835f4b2a-30d0-441b-af00-f0b2d166fe7c-config-out\") pod \"alertmanager-main-0\" (UID: \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:46.456846 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:46.456611    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/835f4b2a-30d0-441b-af00-f0b2d166fe7c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:46.557212 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:46.557180    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/835f4b2a-30d0-441b-af00-f0b2d166fe7c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:46.557212 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:46.557218    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/835f4b2a-30d0-441b-af00-f0b2d166fe7c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:46.557438 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:46.557238    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/835f4b2a-30d0-441b-af00-f0b2d166fe7c-config-out\") pod \"alertmanager-main-0\" (UID: \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:46.557438 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:46.557260    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/835f4b2a-30d0-441b-af00-f0b2d166fe7c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:46.557438 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:46.557356    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/835f4b2a-30d0-441b-af00-f0b2d166fe7c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:46.557438 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:46.557384    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/835f4b2a-30d0-441b-af00-f0b2d166fe7c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:46.557438 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:46.557409    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/835f4b2a-30d0-441b-af00-f0b2d166fe7c-config-volume\") pod \"alertmanager-main-0\" (UID: \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:46.557438 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:46.557430    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/835f4b2a-30d0-441b-af00-f0b2d166fe7c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:46.557721 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:46.557458    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/835f4b2a-30d0-441b-af00-f0b2d166fe7c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:46.557721 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:46.557480    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/835f4b2a-30d0-441b-af00-f0b2d166fe7c-web-config\") pod \"alertmanager-main-0\" (UID: \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:46.557721 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:46.557502    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/835f4b2a-30d0-441b-af00-f0b2d166fe7c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:46.557721 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:46.557533    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/835f4b2a-30d0-441b-af00-f0b2d166fe7c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:46.557721 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:46.557561    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrbf9\" (UniqueName: \"kubernetes.io/projected/835f4b2a-30d0-441b-af00-f0b2d166fe7c-kube-api-access-zrbf9\") pod \"alertmanager-main-0\" (UID: \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:46.557970 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:46.557884    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/835f4b2a-30d0-441b-af00-f0b2d166fe7c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:46.559091 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:46.558712    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/835f4b2a-30d0-441b-af00-f0b2d166fe7c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:46.559091 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:46.558977    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/835f4b2a-30d0-441b-af00-f0b2d166fe7c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:46.560163 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:46.560133    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/835f4b2a-30d0-441b-af00-f0b2d166fe7c-config-out\") pod \"alertmanager-main-0\" (UID: \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:46.560462 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:46.560433    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/835f4b2a-30d0-441b-af00-f0b2d166fe7c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:46.560696 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:46.560629    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/835f4b2a-30d0-441b-af00-f0b2d166fe7c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:46.560696 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:46.560632    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/835f4b2a-30d0-441b-af00-f0b2d166fe7c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:46.561122 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:46.561093    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/835f4b2a-30d0-441b-af00-f0b2d166fe7c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:46.561443 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:46.561419    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/835f4b2a-30d0-441b-af00-f0b2d166fe7c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:46.561538 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:46.561424    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/835f4b2a-30d0-441b-af00-f0b2d166fe7c-config-volume\") pod \"alertmanager-main-0\" (UID: \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:46.561649 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:46.561633    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/835f4b2a-30d0-441b-af00-f0b2d166fe7c-web-config\") pod \"alertmanager-main-0\" (UID: \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:46.562110 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:46.562093    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/835f4b2a-30d0-441b-af00-f0b2d166fe7c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:46.567779 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:46.567757    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrbf9\" (UniqueName: \"kubernetes.io/projected/835f4b2a-30d0-441b-af00-f0b2d166fe7c-kube-api-access-zrbf9\") pod \"alertmanager-main-0\" (UID: \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:46.652472 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:46.652436    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:46.810211 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:46.810115    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 07 16:48:46.811560 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:48:46.811526    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod835f4b2a_30d0_441b_af00_f0b2d166fe7c.slice/crio-fc38479a5312dd0bc36073f6c092a59ed0e56dc985525c043e4b7045dbd4f724 WatchSource:0}: Error finding container fc38479a5312dd0bc36073f6c092a59ed0e56dc985525c043e4b7045dbd4f724: Status 404 returned error can't find the container with id fc38479a5312dd0bc36073f6c092a59ed0e56dc985525c043e4b7045dbd4f724
Jul 07 16:48:46.999082 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:46.999001    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"835f4b2a-30d0-441b-af00-f0b2d166fe7c","Type":"ContainerStarted","Data":"fc38479a5312dd0bc36073f6c092a59ed0e56dc985525c043e4b7045dbd4f724"}
Jul 07 16:48:47.000637 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:47.000598    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nfgmm" event={"ID":"b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168","Type":"ContainerStarted","Data":"d3ce1eea644ef22d5b743e66fa513e939f7c552355b64c86fd6386afcf80fbf1"}
Jul 07 16:48:47.643026 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:47.642995    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-787c6b6b98-7v2dr"]
Jul 07 16:48:47.645579 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:47.645557    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-787c6b6b98-7v2dr"
Jul 07 16:48:47.654611 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:47.654586    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 07 16:48:47.660807 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:47.660787    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-787c6b6b98-7v2dr"]
Jul 07 16:48:47.770850 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:47.770815    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9480c793-ca95-433d-8d2a-9f8fb71e04a1-console-serving-cert\") pod \"console-787c6b6b98-7v2dr\" (UID: \"9480c793-ca95-433d-8d2a-9f8fb71e04a1\") " pod="openshift-console/console-787c6b6b98-7v2dr"
Jul 07 16:48:47.771008 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:47.770879    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9480c793-ca95-433d-8d2a-9f8fb71e04a1-console-oauth-config\") pod \"console-787c6b6b98-7v2dr\" (UID: \"9480c793-ca95-433d-8d2a-9f8fb71e04a1\") " pod="openshift-console/console-787c6b6b98-7v2dr"
Jul 07 16:48:47.771008 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:47.770904    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9480c793-ca95-433d-8d2a-9f8fb71e04a1-trusted-ca-bundle\") pod \"console-787c6b6b98-7v2dr\" (UID: \"9480c793-ca95-433d-8d2a-9f8fb71e04a1\") " pod="openshift-console/console-787c6b6b98-7v2dr"
Jul 07 16:48:47.771008 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:47.770943    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4bs6\" (UniqueName: \"kubernetes.io/projected/9480c793-ca95-433d-8d2a-9f8fb71e04a1-kube-api-access-p4bs6\") pod \"console-787c6b6b98-7v2dr\" (UID: \"9480c793-ca95-433d-8d2a-9f8fb71e04a1\") " pod="openshift-console/console-787c6b6b98-7v2dr"
Jul 07 16:48:47.771008 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:47.770978    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9480c793-ca95-433d-8d2a-9f8fb71e04a1-service-ca\") pod \"console-787c6b6b98-7v2dr\" (UID: \"9480c793-ca95-433d-8d2a-9f8fb71e04a1\") " pod="openshift-console/console-787c6b6b98-7v2dr"
Jul 07 16:48:47.771173 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:47.771025    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9480c793-ca95-433d-8d2a-9f8fb71e04a1-console-config\") pod \"console-787c6b6b98-7v2dr\" (UID: \"9480c793-ca95-433d-8d2a-9f8fb71e04a1\") " pod="openshift-console/console-787c6b6b98-7v2dr"
Jul 07 16:48:47.771173 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:47.771100    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9480c793-ca95-433d-8d2a-9f8fb71e04a1-oauth-serving-cert\") pod \"console-787c6b6b98-7v2dr\" (UID: \"9480c793-ca95-433d-8d2a-9f8fb71e04a1\") " pod="openshift-console/console-787c6b6b98-7v2dr"
Jul 07 16:48:47.872134 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:47.872097    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9480c793-ca95-433d-8d2a-9f8fb71e04a1-console-serving-cert\") pod \"console-787c6b6b98-7v2dr\" (UID: \"9480c793-ca95-433d-8d2a-9f8fb71e04a1\") " pod="openshift-console/console-787c6b6b98-7v2dr"
Jul 07 16:48:47.872322 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:47.872150    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9480c793-ca95-433d-8d2a-9f8fb71e04a1-console-oauth-config\") pod \"console-787c6b6b98-7v2dr\" (UID: \"9480c793-ca95-433d-8d2a-9f8fb71e04a1\") " pod="openshift-console/console-787c6b6b98-7v2dr"
Jul 07 16:48:47.872322 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:47.872171    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9480c793-ca95-433d-8d2a-9f8fb71e04a1-trusted-ca-bundle\") pod \"console-787c6b6b98-7v2dr\" (UID: \"9480c793-ca95-433d-8d2a-9f8fb71e04a1\") " pod="openshift-console/console-787c6b6b98-7v2dr"
Jul 07 16:48:47.872322 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:47.872226    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4bs6\" (UniqueName: \"kubernetes.io/projected/9480c793-ca95-433d-8d2a-9f8fb71e04a1-kube-api-access-p4bs6\") pod \"console-787c6b6b98-7v2dr\" (UID: \"9480c793-ca95-433d-8d2a-9f8fb71e04a1\") " pod="openshift-console/console-787c6b6b98-7v2dr"
Jul 07 16:48:47.872322 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:47.872254    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9480c793-ca95-433d-8d2a-9f8fb71e04a1-service-ca\") pod \"console-787c6b6b98-7v2dr\" (UID: \"9480c793-ca95-433d-8d2a-9f8fb71e04a1\") " pod="openshift-console/console-787c6b6b98-7v2dr"
Jul 07 16:48:47.872624 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:47.872599    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9480c793-ca95-433d-8d2a-9f8fb71e04a1-console-config\") pod \"console-787c6b6b98-7v2dr\" (UID: \"9480c793-ca95-433d-8d2a-9f8fb71e04a1\") " pod="openshift-console/console-787c6b6b98-7v2dr"
Jul 07 16:48:47.872689 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:47.872667    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9480c793-ca95-433d-8d2a-9f8fb71e04a1-oauth-serving-cert\") pod \"console-787c6b6b98-7v2dr\" (UID: \"9480c793-ca95-433d-8d2a-9f8fb71e04a1\") " pod="openshift-console/console-787c6b6b98-7v2dr"
Jul 07 16:48:47.873114 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:47.873090    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9480c793-ca95-433d-8d2a-9f8fb71e04a1-service-ca\") pod \"console-787c6b6b98-7v2dr\" (UID: \"9480c793-ca95-433d-8d2a-9f8fb71e04a1\") " pod="openshift-console/console-787c6b6b98-7v2dr"
Jul 07 16:48:47.873243 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:47.873221    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9480c793-ca95-433d-8d2a-9f8fb71e04a1-trusted-ca-bundle\") pod \"console-787c6b6b98-7v2dr\" (UID: \"9480c793-ca95-433d-8d2a-9f8fb71e04a1\") " pod="openshift-console/console-787c6b6b98-7v2dr"
Jul 07 16:48:47.873366 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:47.873320    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9480c793-ca95-433d-8d2a-9f8fb71e04a1-console-config\") pod \"console-787c6b6b98-7v2dr\" (UID: \"9480c793-ca95-433d-8d2a-9f8fb71e04a1\") " pod="openshift-console/console-787c6b6b98-7v2dr"
Jul 07 16:48:47.873473 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:47.873441    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9480c793-ca95-433d-8d2a-9f8fb71e04a1-oauth-serving-cert\") pod \"console-787c6b6b98-7v2dr\" (UID: \"9480c793-ca95-433d-8d2a-9f8fb71e04a1\") " pod="openshift-console/console-787c6b6b98-7v2dr"
Jul 07 16:48:47.874914 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:47.874893    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9480c793-ca95-433d-8d2a-9f8fb71e04a1-console-oauth-config\") pod \"console-787c6b6b98-7v2dr\" (UID: \"9480c793-ca95-433d-8d2a-9f8fb71e04a1\") " pod="openshift-console/console-787c6b6b98-7v2dr"
Jul 07 16:48:47.875114 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:47.875092    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9480c793-ca95-433d-8d2a-9f8fb71e04a1-console-serving-cert\") pod \"console-787c6b6b98-7v2dr\" (UID: \"9480c793-ca95-433d-8d2a-9f8fb71e04a1\") " pod="openshift-console/console-787c6b6b98-7v2dr"
Jul 07 16:48:47.881587 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:47.881546    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4bs6\" (UniqueName: \"kubernetes.io/projected/9480c793-ca95-433d-8d2a-9f8fb71e04a1-kube-api-access-p4bs6\") pod \"console-787c6b6b98-7v2dr\" (UID: \"9480c793-ca95-433d-8d2a-9f8fb71e04a1\") " pod="openshift-console/console-787c6b6b98-7v2dr"
Jul 07 16:48:47.956390 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:47.956368    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-787c6b6b98-7v2dr"
Jul 07 16:48:47.960038 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:47.960022    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-hh2zz"
Jul 07 16:48:48.006492 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:48.006457    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"835f4b2a-30d0-441b-af00-f0b2d166fe7c","Type":"ContainerStarted","Data":"04e1653d7c7f308c8ff27f69ce74ff84349918f08fe32d4b6b494ea66a196fa4"}
Jul 07 16:48:48.009796 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:48.009772    2578 generic.go:356] "Generic (PLEG): container finished" podID="b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168" containerID="6cbd54ce2a9bb9171775384aeb74a8962b797221cbef14d9872a647cee59e009" exitCode=0
Jul 07 16:48:48.009882 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:48.009828    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nfgmm" event={"ID":"b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168","Type":"ContainerDied","Data":"6cbd54ce2a9bb9171775384aeb74a8962b797221cbef14d9872a647cee59e009"}
Jul 07 16:48:48.094261 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:48.094237    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-787c6b6b98-7v2dr"]
Jul 07 16:48:48.098413 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:48:48.098293    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9480c793_ca95_433d_8d2a_9f8fb71e04a1.slice/crio-7dc4aa0885fbd6ccac89cd1a1a686d0841ccb63d037479826f8e4800d29cf91e WatchSource:0}: Error finding container 7dc4aa0885fbd6ccac89cd1a1a686d0841ccb63d037479826f8e4800d29cf91e: Status 404 returned error can't find the container with id 7dc4aa0885fbd6ccac89cd1a1a686d0841ccb63d037479826f8e4800d29cf91e
Jul 07 16:48:49.017898 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:49.017861    2578 generic.go:356] "Generic (PLEG): container finished" podID="835f4b2a-30d0-441b-af00-f0b2d166fe7c" containerID="04e1653d7c7f308c8ff27f69ce74ff84349918f08fe32d4b6b494ea66a196fa4" exitCode=0
Jul 07 16:48:49.018294 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:49.017941    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"835f4b2a-30d0-441b-af00-f0b2d166fe7c","Type":"ContainerDied","Data":"04e1653d7c7f308c8ff27f69ce74ff84349918f08fe32d4b6b494ea66a196fa4"}
Jul 07 16:48:49.020094 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:49.020068    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nfgmm" event={"ID":"b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168","Type":"ContainerStarted","Data":"4974fc15e098e0a5813f4c2364dea690e742d53ca78cb9397e61f532ee92140d"}
Jul 07 16:48:49.020200 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:49.020108    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nfgmm" event={"ID":"b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168","Type":"ContainerStarted","Data":"4c5f417ad538958e48a3b946bc0dc2a376441116c4171b269b68ff30eac8d79f"}
Jul 07 16:48:49.021562 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:49.021541    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-787c6b6b98-7v2dr" event={"ID":"9480c793-ca95-433d-8d2a-9f8fb71e04a1","Type":"ContainerStarted","Data":"9f383eaa60ef69f3cd46117306b72da0e23ce1f5baf023f1b35ef02e65fe68d4"}
Jul 07 16:48:49.021630 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:49.021579    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-787c6b6b98-7v2dr" event={"ID":"9480c793-ca95-433d-8d2a-9f8fb71e04a1","Type":"ContainerStarted","Data":"7dc4aa0885fbd6ccac89cd1a1a686d0841ccb63d037479826f8e4800d29cf91e"}
Jul 07 16:48:49.070134 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:49.070089    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-nfgmm" podStartSLOduration=3.161267498 podStartE2EDuration="4.070077762s" totalImagesPullingTime="844.55161ms" totalInitContainerRuntime="64.258654ms" isStatefulPod=true podCreationTimestamp="2026-07-07 16:48:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:48:49.069319353 +0000 UTC m=+65.845633559" watchObservedRunningTime="2026-07-07 16:48:49.070077762 +0000 UTC m=+65.846391969"
Jul 07 16:48:49.087981 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:49.087948    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-787c6b6b98-7v2dr" podStartSLOduration=2.0879376020000002 podStartE2EDuration="2.087937602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:48:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:48:49.087417044 +0000 UTC m=+65.863731270" watchObservedRunningTime="2026-07-07 16:48:49.087937602 +0000 UTC m=+65.864251807"
Jul 07 16:48:49.387133 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:49.387092    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91e42c9a-31b0-42bf-9055-28fd68ab4ba9-metrics-certs\") pod \"network-metrics-daemon-kdcss\" (UID: \"91e42c9a-31b0-42bf-9055-28fd68ab4ba9\") " pod="openshift-multus/network-metrics-daemon-kdcss"
Jul 07 16:48:49.389806 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:49.389789    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 07 16:48:49.400118 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:49.400090    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91e42c9a-31b0-42bf-9055-28fd68ab4ba9-metrics-certs\") pod \"network-metrics-daemon-kdcss\" (UID: \"91e42c9a-31b0-42bf-9055-28fd68ab4ba9\") " pod="openshift-multus/network-metrics-daemon-kdcss"
Jul 07 16:48:49.487494 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:49.487461    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpnrf\" (UniqueName: \"kubernetes.io/projected/57008745-859f-49ec-9269-7671fa8164a2-kube-api-access-zpnrf\") pod \"network-check-target-wwb84\" (UID: \"57008745-859f-49ec-9269-7671fa8164a2\") " pod="openshift-network-diagnostics/network-check-target-wwb84"
Jul 07 16:48:49.487662 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:49.487527    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/816e0df7-c9cf-4303-a920-a90a16ee015c-original-pull-secret\") pod \"global-pull-secret-syncer-h5f7f\" (UID: \"816e0df7-c9cf-4303-a920-a90a16ee015c\") " pod="kube-system/global-pull-secret-syncer-h5f7f"
Jul 07 16:48:49.490249 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:49.490235    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 07 16:48:49.490505 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:49.490486    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 07 16:48:49.500624 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:49.500601    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/816e0df7-c9cf-4303-a920-a90a16ee015c-original-pull-secret\") pod \"global-pull-secret-syncer-h5f7f\" (UID: \"816e0df7-c9cf-4303-a920-a90a16ee015c\") " pod="kube-system/global-pull-secret-syncer-h5f7f"
Jul 07 16:48:49.500953 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:49.500940    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 07 16:48:49.511455 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:49.511431    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpnrf\" (UniqueName: \"kubernetes.io/projected/57008745-859f-49ec-9269-7671fa8164a2-kube-api-access-zpnrf\") pod \"network-check-target-wwb84\" (UID: \"57008745-859f-49ec-9269-7671fa8164a2\") " pod="openshift-network-diagnostics/network-check-target-wwb84"
Jul 07 16:48:49.531279 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:49.531250    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7886dfcf5f-r7svd"]
Jul 07 16:48:49.531536 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:49.531255    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sk64f\""
Jul 07 16:48:49.534595 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:49.534573    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7886dfcf5f-r7svd"
Jul 07 16:48:49.537541 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:49.537517    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 07 16:48:49.538843 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:49.538821    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h5f7f"
Jul 07 16:48:49.539139 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:49.539117    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdcss"
Jul 07 16:48:49.539353 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:49.539289    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 07 16:48:49.539353 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:49.539321    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-w5g6j\""
Jul 07 16:48:49.539547 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:49.539367    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 07 16:48:49.539547 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:49.539289    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-6g4tss721l40v\""
Jul 07 16:48:49.539657 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:49.539601    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jwbcf\""
Jul 07 16:48:49.539657 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:49.539602    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 07 16:48:49.545224 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:49.545201    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wwb84"
Jul 07 16:48:49.546873 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:49.546849    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7886dfcf5f-r7svd"]
Jul 07 16:48:49.588637 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:49.588272    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/323eccaa-741e-495a-81a2-6368321ae1db-secret-metrics-server-client-certs\") pod \"metrics-server-7886dfcf5f-r7svd\" (UID: \"323eccaa-741e-495a-81a2-6368321ae1db\") " pod="openshift-monitoring/metrics-server-7886dfcf5f-r7svd"
Jul 07 16:48:49.588637 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:49.588327    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/323eccaa-741e-495a-81a2-6368321ae1db-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7886dfcf5f-r7svd\" (UID: \"323eccaa-741e-495a-81a2-6368321ae1db\") " pod="openshift-monitoring/metrics-server-7886dfcf5f-r7svd"
Jul 07 16:48:49.588637 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:49.588365    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/323eccaa-741e-495a-81a2-6368321ae1db-metrics-server-audit-profiles\") pod \"metrics-server-7886dfcf5f-r7svd\" (UID: \"323eccaa-741e-495a-81a2-6368321ae1db\") " pod="openshift-monitoring/metrics-server-7886dfcf5f-r7svd"
Jul 07 16:48:49.588637 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:49.588392    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/323eccaa-741e-495a-81a2-6368321ae1db-client-ca-bundle\") pod \"metrics-server-7886dfcf5f-r7svd\" (UID: \"323eccaa-741e-495a-81a2-6368321ae1db\") " pod="openshift-monitoring/metrics-server-7886dfcf5f-r7svd"
Jul 07 16:48:49.588637 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:49.588441    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/323eccaa-741e-495a-81a2-6368321ae1db-secret-metrics-server-tls\") pod \"metrics-server-7886dfcf5f-r7svd\" (UID: \"323eccaa-741e-495a-81a2-6368321ae1db\") " pod="openshift-monitoring/metrics-server-7886dfcf5f-r7svd"
Jul 07 16:48:49.588637 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:49.588465    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/323eccaa-741e-495a-81a2-6368321ae1db-audit-log\") pod \"metrics-server-7886dfcf5f-r7svd\" (UID: \"323eccaa-741e-495a-81a2-6368321ae1db\") " pod="openshift-monitoring/metrics-server-7886dfcf5f-r7svd"
Jul 07 16:48:49.588637 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:49.588575    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqsg6\" (UniqueName: \"kubernetes.io/projected/323eccaa-741e-495a-81a2-6368321ae1db-kube-api-access-wqsg6\") pod \"metrics-server-7886dfcf5f-r7svd\" (UID: \"323eccaa-741e-495a-81a2-6368321ae1db\") " pod="openshift-monitoring/metrics-server-7886dfcf5f-r7svd"
Jul 07 16:48:49.690386 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:49.689692    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/323eccaa-741e-495a-81a2-6368321ae1db-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7886dfcf5f-r7svd\" (UID: \"323eccaa-741e-495a-81a2-6368321ae1db\") " pod="openshift-monitoring/metrics-server-7886dfcf5f-r7svd"
Jul 07 16:48:49.690386 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:49.689759    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/323eccaa-741e-495a-81a2-6368321ae1db-metrics-server-audit-profiles\") pod \"metrics-server-7886dfcf5f-r7svd\" (UID: \"323eccaa-741e-495a-81a2-6368321ae1db\") " pod="openshift-monitoring/metrics-server-7886dfcf5f-r7svd"
Jul 07 16:48:49.690386 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:49.689793    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/323eccaa-741e-495a-81a2-6368321ae1db-client-ca-bundle\") pod \"metrics-server-7886dfcf5f-r7svd\" (UID: \"323eccaa-741e-495a-81a2-6368321ae1db\") " pod="openshift-monitoring/metrics-server-7886dfcf5f-r7svd"
Jul 07 16:48:49.690386 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:49.689845    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/323eccaa-741e-495a-81a2-6368321ae1db-secret-metrics-server-tls\") pod \"metrics-server-7886dfcf5f-r7svd\" (UID: \"323eccaa-741e-495a-81a2-6368321ae1db\") " pod="openshift-monitoring/metrics-server-7886dfcf5f-r7svd"
Jul 07 16:48:49.690386 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:49.689873    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/323eccaa-741e-495a-81a2-6368321ae1db-audit-log\") pod \"metrics-server-7886dfcf5f-r7svd\" (UID: \"323eccaa-741e-495a-81a2-6368321ae1db\") " pod="openshift-monitoring/metrics-server-7886dfcf5f-r7svd"
Jul 07 16:48:49.690386 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:49.689915    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqsg6\" (UniqueName: \"kubernetes.io/projected/323eccaa-741e-495a-81a2-6368321ae1db-kube-api-access-wqsg6\") pod \"metrics-server-7886dfcf5f-r7svd\" (UID: \"323eccaa-741e-495a-81a2-6368321ae1db\") " pod="openshift-monitoring/metrics-server-7886dfcf5f-r7svd"
Jul 07 16:48:49.690386 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:49.689983    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/323eccaa-741e-495a-81a2-6368321ae1db-secret-metrics-server-client-certs\") pod \"metrics-server-7886dfcf5f-r7svd\" (UID: \"323eccaa-741e-495a-81a2-6368321ae1db\") " pod="openshift-monitoring/metrics-server-7886dfcf5f-r7svd"
Jul 07 16:48:49.690386 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:49.690307    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/323eccaa-741e-495a-81a2-6368321ae1db-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7886dfcf5f-r7svd\" (UID: \"323eccaa-741e-495a-81a2-6368321ae1db\") " pod="openshift-monitoring/metrics-server-7886dfcf5f-r7svd"
Jul 07 16:48:49.690957 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:49.690915    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/323eccaa-741e-495a-81a2-6368321ae1db-audit-log\") pod \"metrics-server-7886dfcf5f-r7svd\" (UID: \"323eccaa-741e-495a-81a2-6368321ae1db\") " pod="openshift-monitoring/metrics-server-7886dfcf5f-r7svd"
Jul 07 16:48:49.691981 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:49.691954    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/323eccaa-741e-495a-81a2-6368321ae1db-metrics-server-audit-profiles\") pod \"metrics-server-7886dfcf5f-r7svd\" (UID: \"323eccaa-741e-495a-81a2-6368321ae1db\") " pod="openshift-monitoring/metrics-server-7886dfcf5f-r7svd"
Jul 07 16:48:49.693322 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:49.693282    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/323eccaa-741e-495a-81a2-6368321ae1db-secret-metrics-server-client-certs\") pod \"metrics-server-7886dfcf5f-r7svd\" (UID: \"323eccaa-741e-495a-81a2-6368321ae1db\") " pod="openshift-monitoring/metrics-server-7886dfcf5f-r7svd"
Jul 07 16:48:49.693700 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:49.693622    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/323eccaa-741e-495a-81a2-6368321ae1db-secret-metrics-server-tls\") pod \"metrics-server-7886dfcf5f-r7svd\" (UID: \"323eccaa-741e-495a-81a2-6368321ae1db\") " pod="openshift-monitoring/metrics-server-7886dfcf5f-r7svd"
Jul 07 16:48:49.694314 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:49.694250    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/323eccaa-741e-495a-81a2-6368321ae1db-client-ca-bundle\") pod \"metrics-server-7886dfcf5f-r7svd\" (UID: \"323eccaa-741e-495a-81a2-6368321ae1db\") " pod="openshift-monitoring/metrics-server-7886dfcf5f-r7svd"
Jul 07 16:48:49.702532 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:49.702493    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-h5f7f"]
Jul 07 16:48:49.708476 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:48:49.708448    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod816e0df7_c9cf_4303_a920_a90a16ee015c.slice/crio-fca7403bb965ef244549f1ead7c44fdcbb3e0fce4d6e3ad88a0a4f40bf9063ce WatchSource:0}: Error finding container fca7403bb965ef244549f1ead7c44fdcbb3e0fce4d6e3ad88a0a4f40bf9063ce: Status 404 returned error can't find the container with id fca7403bb965ef244549f1ead7c44fdcbb3e0fce4d6e3ad88a0a4f40bf9063ce
Jul 07 16:48:49.713458 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:49.713300    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqsg6\" (UniqueName: \"kubernetes.io/projected/323eccaa-741e-495a-81a2-6368321ae1db-kube-api-access-wqsg6\") pod \"metrics-server-7886dfcf5f-r7svd\" (UID: \"323eccaa-741e-495a-81a2-6368321ae1db\") " pod="openshift-monitoring/metrics-server-7886dfcf5f-r7svd"
Jul 07 16:48:49.729446 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:49.729421    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wwb84"]
Jul 07 16:48:49.732216 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:48:49.732186    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57008745_859f_49ec_9269_7671fa8164a2.slice/crio-12acad2c8c7ecfc4523e46bc225147004282b7acc9faba7d6c9c0ca01b8161e4 WatchSource:0}: Error finding container 12acad2c8c7ecfc4523e46bc225147004282b7acc9faba7d6c9c0ca01b8161e4: Status 404 returned error can't find the container with id 12acad2c8c7ecfc4523e46bc225147004282b7acc9faba7d6c9c0ca01b8161e4
Jul 07 16:48:49.732899 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:49.732863    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kdcss"]
Jul 07 16:48:49.738027 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:48:49.738000    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e42c9a_31b0_42bf_9055_28fd68ab4ba9.slice/crio-ea82443093fc9e7bbba9a96c73e73abd288d6076ed710248d81e62a355b1dd9f WatchSource:0}: Error finding container ea82443093fc9e7bbba9a96c73e73abd288d6076ed710248d81e62a355b1dd9f: Status 404 returned error can't find the container with id ea82443093fc9e7bbba9a96c73e73abd288d6076ed710248d81e62a355b1dd9f
Jul 07 16:48:49.846961 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:49.846933    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7886dfcf5f-r7svd"
Jul 07 16:48:49.980630 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:49.980580    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7886dfcf5f-r7svd"]
Jul 07 16:48:49.983157 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:48:49.983132    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod323eccaa_741e_495a_81a2_6368321ae1db.slice/crio-8d79a679d3e5891ecfd6f9d27099bf86564e6d3b6449a063f4b3885793770216 WatchSource:0}: Error finding container 8d79a679d3e5891ecfd6f9d27099bf86564e6d3b6449a063f4b3885793770216: Status 404 returned error can't find the container with id 8d79a679d3e5891ecfd6f9d27099bf86564e6d3b6449a063f4b3885793770216
Jul 07 16:48:50.028916 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:50.028872    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-h5f7f" event={"ID":"816e0df7-c9cf-4303-a920-a90a16ee015c","Type":"ContainerStarted","Data":"fca7403bb965ef244549f1ead7c44fdcbb3e0fce4d6e3ad88a0a4f40bf9063ce"}
Jul 07 16:48:50.030677 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:50.030637    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7886dfcf5f-r7svd" event={"ID":"323eccaa-741e-495a-81a2-6368321ae1db","Type":"ContainerStarted","Data":"8d79a679d3e5891ecfd6f9d27099bf86564e6d3b6449a063f4b3885793770216"}
Jul 07 16:48:50.032211 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:50.032182    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kdcss" event={"ID":"91e42c9a-31b0-42bf-9055-28fd68ab4ba9","Type":"ContainerStarted","Data":"ea82443093fc9e7bbba9a96c73e73abd288d6076ed710248d81e62a355b1dd9f"}
Jul 07 16:48:50.033362 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:50.033337    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wwb84" event={"ID":"57008745-859f-49ec-9269-7671fa8164a2","Type":"ContainerStarted","Data":"12acad2c8c7ecfc4523e46bc225147004282b7acc9faba7d6c9c0ca01b8161e4"}
Jul 07 16:48:50.471945 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:50.471417    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-86d8764fcc-t7vw9"]
Jul 07 16:48:50.491522 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:50.490271    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-86d8764fcc-t7vw9"
Jul 07 16:48:50.494454 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:50.494291    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 07 16:48:50.498000 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:50.494959    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-66vl4\""
Jul 07 16:48:50.500709 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:50.499845    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 07 16:48:50.500709 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:50.499867    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-86d8764fcc-t7vw9"]
Jul 07 16:48:50.500709 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:50.499974    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 07 16:48:50.500709 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:50.500064    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 07 16:48:50.500709 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:50.500549    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 07 16:48:50.506342 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:50.506310    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 07 16:48:50.598877 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:50.598411    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff8de6df-aa2e-4d3e-bac0-a746f8433868-telemeter-trusted-ca-bundle\") pod \"telemeter-client-86d8764fcc-t7vw9\" (UID: \"ff8de6df-aa2e-4d3e-bac0-a746f8433868\") " pod="openshift-monitoring/telemeter-client-86d8764fcc-t7vw9"
Jul 07 16:48:50.598877 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:50.598479    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff8de6df-aa2e-4d3e-bac0-a746f8433868-metrics-client-ca\") pod \"telemeter-client-86d8764fcc-t7vw9\" (UID: \"ff8de6df-aa2e-4d3e-bac0-a746f8433868\") " pod="openshift-monitoring/telemeter-client-86d8764fcc-t7vw9"
Jul 07 16:48:50.598877 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:50.598510    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ff8de6df-aa2e-4d3e-bac0-a746f8433868-telemeter-client-tls\") pod \"telemeter-client-86d8764fcc-t7vw9\" (UID: \"ff8de6df-aa2e-4d3e-bac0-a746f8433868\") " pod="openshift-monitoring/telemeter-client-86d8764fcc-t7vw9"
Jul 07 16:48:50.598877 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:50.598538    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ff8de6df-aa2e-4d3e-bac0-a746f8433868-federate-client-tls\") pod \"telemeter-client-86d8764fcc-t7vw9\" (UID: \"ff8de6df-aa2e-4d3e-bac0-a746f8433868\") " pod="openshift-monitoring/telemeter-client-86d8764fcc-t7vw9"
Jul 07 16:48:50.598877 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:50.598609    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff8de6df-aa2e-4d3e-bac0-a746f8433868-serving-certs-ca-bundle\") pod \"telemeter-client-86d8764fcc-t7vw9\" (UID: \"ff8de6df-aa2e-4d3e-bac0-a746f8433868\") " pod="openshift-monitoring/telemeter-client-86d8764fcc-t7vw9"
Jul 07 16:48:50.598877 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:50.598637    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmgtl\" (UniqueName: \"kubernetes.io/projected/ff8de6df-aa2e-4d3e-bac0-a746f8433868-kube-api-access-vmgtl\") pod \"telemeter-client-86d8764fcc-t7vw9\" (UID: \"ff8de6df-aa2e-4d3e-bac0-a746f8433868\") " pod="openshift-monitoring/telemeter-client-86d8764fcc-t7vw9"
Jul 07 16:48:50.598877 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:50.598682    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ff8de6df-aa2e-4d3e-bac0-a746f8433868-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-86d8764fcc-t7vw9\" (UID: \"ff8de6df-aa2e-4d3e-bac0-a746f8433868\") " pod="openshift-monitoring/telemeter-client-86d8764fcc-t7vw9"
Jul 07 16:48:50.598877 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:50.598709    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ff8de6df-aa2e-4d3e-bac0-a746f8433868-secret-telemeter-client\") pod \"telemeter-client-86d8764fcc-t7vw9\" (UID: \"ff8de6df-aa2e-4d3e-bac0-a746f8433868\") " pod="openshift-monitoring/telemeter-client-86d8764fcc-t7vw9"
Jul 07 16:48:50.700986 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:50.699599    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmgtl\" (UniqueName: \"kubernetes.io/projected/ff8de6df-aa2e-4d3e-bac0-a746f8433868-kube-api-access-vmgtl\") pod \"telemeter-client-86d8764fcc-t7vw9\" (UID: \"ff8de6df-aa2e-4d3e-bac0-a746f8433868\") " pod="openshift-monitoring/telemeter-client-86d8764fcc-t7vw9"
Jul 07 16:48:50.700986 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:50.699663    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ff8de6df-aa2e-4d3e-bac0-a746f8433868-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-86d8764fcc-t7vw9\" (UID: \"ff8de6df-aa2e-4d3e-bac0-a746f8433868\") " pod="openshift-monitoring/telemeter-client-86d8764fcc-t7vw9"
Jul 07 16:48:50.700986 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:50.699701    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ff8de6df-aa2e-4d3e-bac0-a746f8433868-secret-telemeter-client\") pod \"telemeter-client-86d8764fcc-t7vw9\" (UID: \"ff8de6df-aa2e-4d3e-bac0-a746f8433868\") " pod="openshift-monitoring/telemeter-client-86d8764fcc-t7vw9"
Jul 07 16:48:50.700986 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:50.699755    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff8de6df-aa2e-4d3e-bac0-a746f8433868-telemeter-trusted-ca-bundle\") pod \"telemeter-client-86d8764fcc-t7vw9\" (UID: \"ff8de6df-aa2e-4d3e-bac0-a746f8433868\") " pod="openshift-monitoring/telemeter-client-86d8764fcc-t7vw9"
Jul 07 16:48:50.700986 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:50.699795    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff8de6df-aa2e-4d3e-bac0-a746f8433868-metrics-client-ca\") pod \"telemeter-client-86d8764fcc-t7vw9\" (UID: \"ff8de6df-aa2e-4d3e-bac0-a746f8433868\") " pod="openshift-monitoring/telemeter-client-86d8764fcc-t7vw9"
Jul 07 16:48:50.700986 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:50.699824    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ff8de6df-aa2e-4d3e-bac0-a746f8433868-telemeter-client-tls\") pod \"telemeter-client-86d8764fcc-t7vw9\" (UID: \"ff8de6df-aa2e-4d3e-bac0-a746f8433868\") " pod="openshift-monitoring/telemeter-client-86d8764fcc-t7vw9"
Jul 07 16:48:50.700986 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:50.699849    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ff8de6df-aa2e-4d3e-bac0-a746f8433868-federate-client-tls\") pod \"telemeter-client-86d8764fcc-t7vw9\" (UID: \"ff8de6df-aa2e-4d3e-bac0-a746f8433868\") " pod="openshift-monitoring/telemeter-client-86d8764fcc-t7vw9"
Jul 07 16:48:50.700986 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:50.699916    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff8de6df-aa2e-4d3e-bac0-a746f8433868-serving-certs-ca-bundle\") pod \"telemeter-client-86d8764fcc-t7vw9\" (UID: \"ff8de6df-aa2e-4d3e-bac0-a746f8433868\") " pod="openshift-monitoring/telemeter-client-86d8764fcc-t7vw9"
Jul 07 16:48:50.700986 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:50.700672    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff8de6df-aa2e-4d3e-bac0-a746f8433868-serving-certs-ca-bundle\") pod \"telemeter-client-86d8764fcc-t7vw9\" (UID: \"ff8de6df-aa2e-4d3e-bac0-a746f8433868\") " pod="openshift-monitoring/telemeter-client-86d8764fcc-t7vw9"
Jul 07 16:48:50.700986 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:50.700695    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff8de6df-aa2e-4d3e-bac0-a746f8433868-metrics-client-ca\") pod \"telemeter-client-86d8764fcc-t7vw9\" (UID: \"ff8de6df-aa2e-4d3e-bac0-a746f8433868\") " pod="openshift-monitoring/telemeter-client-86d8764fcc-t7vw9"
Jul 07 16:48:50.701615 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:50.701496    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff8de6df-aa2e-4d3e-bac0-a746f8433868-telemeter-trusted-ca-bundle\") pod \"telemeter-client-86d8764fcc-t7vw9\" (UID: \"ff8de6df-aa2e-4d3e-bac0-a746f8433868\") " pod="openshift-monitoring/telemeter-client-86d8764fcc-t7vw9"
Jul 07 16:48:50.703895 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:50.703848    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ff8de6df-aa2e-4d3e-bac0-a746f8433868-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-86d8764fcc-t7vw9\" (UID: \"ff8de6df-aa2e-4d3e-bac0-a746f8433868\") " pod="openshift-monitoring/telemeter-client-86d8764fcc-t7vw9"
Jul 07 16:48:50.705134 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:50.705089    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ff8de6df-aa2e-4d3e-bac0-a746f8433868-telemeter-client-tls\") pod \"telemeter-client-86d8764fcc-t7vw9\" (UID: \"ff8de6df-aa2e-4d3e-bac0-a746f8433868\") " pod="openshift-monitoring/telemeter-client-86d8764fcc-t7vw9"
Jul 07 16:48:50.706165 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:50.706127    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ff8de6df-aa2e-4d3e-bac0-a746f8433868-federate-client-tls\") pod \"telemeter-client-86d8764fcc-t7vw9\" (UID: \"ff8de6df-aa2e-4d3e-bac0-a746f8433868\") " pod="openshift-monitoring/telemeter-client-86d8764fcc-t7vw9"
Jul 07 16:48:50.706540 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:50.706492    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ff8de6df-aa2e-4d3e-bac0-a746f8433868-secret-telemeter-client\") pod \"telemeter-client-86d8764fcc-t7vw9\" (UID: \"ff8de6df-aa2e-4d3e-bac0-a746f8433868\") " pod="openshift-monitoring/telemeter-client-86d8764fcc-t7vw9"
Jul 07 16:48:50.712927 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:50.712883    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmgtl\" (UniqueName: \"kubernetes.io/projected/ff8de6df-aa2e-4d3e-bac0-a746f8433868-kube-api-access-vmgtl\") pod \"telemeter-client-86d8764fcc-t7vw9\" (UID: \"ff8de6df-aa2e-4d3e-bac0-a746f8433868\") " pod="openshift-monitoring/telemeter-client-86d8764fcc-t7vw9"
Jul 07 16:48:50.810412 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:50.810374    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-86d8764fcc-t7vw9"
Jul 07 16:48:50.824253 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:50.824209    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7475fc7665-d9jpq"
Jul 07 16:48:50.824366 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:50.824273    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7475fc7665-d9jpq"
Jul 07 16:48:50.833464 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:50.833247    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7475fc7665-d9jpq"
Jul 07 16:48:51.046090 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.045886    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7475fc7665-d9jpq"
Jul 07 16:48:51.525543 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.525501    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 16:48:51.532923 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.532829    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:51.539110 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.538253    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 07 16:48:51.539110 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.538320    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 07 16:48:51.539110 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.538532    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 07 16:48:51.539110 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.538771    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 07 16:48:51.539110 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.538784    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 07 16:48:51.539110 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.538936    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-srjsv\""
Jul 07 16:48:51.539110 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.538999    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 07 16:48:51.539579 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.539123    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 07 16:48:51.539579 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.539169    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 07 16:48:51.539579 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.539353    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 07 16:48:51.539579 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.539537    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 07 16:48:51.539896 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.539725    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 07 16:48:51.540410 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.539979    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-34p0926q0hcnv\""
Jul 07 16:48:51.540410 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.540300    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 07 16:48:51.540602 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.540545    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 07 16:48:51.542319 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.542130    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 07 16:48:51.546423 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.546228    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 16:48:51.612506 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.611420    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/242f4d41-bb8d-4488-8477-0de8946569ce-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:51.612506 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.611469    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/242f4d41-bb8d-4488-8477-0de8946569ce-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:51.612506 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.611506    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/242f4d41-bb8d-4488-8477-0de8946569ce-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:51.612506 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.611532    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/242f4d41-bb8d-4488-8477-0de8946569ce-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:51.612506 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.611555    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/242f4d41-bb8d-4488-8477-0de8946569ce-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:51.612506 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.611583    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/242f4d41-bb8d-4488-8477-0de8946569ce-web-config\") pod \"prometheus-k8s-0\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:51.612506 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.611612    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/242f4d41-bb8d-4488-8477-0de8946569ce-config\") pod \"prometheus-k8s-0\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:51.612506 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.611639    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/242f4d41-bb8d-4488-8477-0de8946569ce-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:51.612506 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.611677    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/242f4d41-bb8d-4488-8477-0de8946569ce-config-out\") pod \"prometheus-k8s-0\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:51.612506 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.611707    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/242f4d41-bb8d-4488-8477-0de8946569ce-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:51.612506 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.611751    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/242f4d41-bb8d-4488-8477-0de8946569ce-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:51.612506 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.611781    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/242f4d41-bb8d-4488-8477-0de8946569ce-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:51.612506 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.611803    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/242f4d41-bb8d-4488-8477-0de8946569ce-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:51.612506 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.611847    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/242f4d41-bb8d-4488-8477-0de8946569ce-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:51.612506 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.611900    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/242f4d41-bb8d-4488-8477-0de8946569ce-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:51.612506 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.611931    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/242f4d41-bb8d-4488-8477-0de8946569ce-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:51.613580 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.611976    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/242f4d41-bb8d-4488-8477-0de8946569ce-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:51.613580 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.612811    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/242f4d41-bb8d-4488-8477-0de8946569ce-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:51.613580 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.612846    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b86vd\" (UniqueName: \"kubernetes.io/projected/242f4d41-bb8d-4488-8477-0de8946569ce-kube-api-access-b86vd\") pod \"prometheus-k8s-0\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:51.613580 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.612878    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/242f4d41-bb8d-4488-8477-0de8946569ce-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:51.714127 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.714089    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/242f4d41-bb8d-4488-8477-0de8946569ce-config-out\") pod \"prometheus-k8s-0\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:51.714127 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.714140    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/242f4d41-bb8d-4488-8477-0de8946569ce-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:51.714386 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.714164    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/242f4d41-bb8d-4488-8477-0de8946569ce-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:51.714386 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.714195    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/242f4d41-bb8d-4488-8477-0de8946569ce-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:51.714386 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.714223    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/242f4d41-bb8d-4488-8477-0de8946569ce-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:51.714386 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.714270    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/242f4d41-bb8d-4488-8477-0de8946569ce-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:51.714386 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.714343    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/242f4d41-bb8d-4488-8477-0de8946569ce-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:51.714386 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.714378    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/242f4d41-bb8d-4488-8477-0de8946569ce-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:51.714672 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.714422    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/242f4d41-bb8d-4488-8477-0de8946569ce-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:51.714672 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.714445    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/242f4d41-bb8d-4488-8477-0de8946569ce-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:51.714672 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.714493    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b86vd\" (UniqueName: \"kubernetes.io/projected/242f4d41-bb8d-4488-8477-0de8946569ce-kube-api-access-b86vd\") pod \"prometheus-k8s-0\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:51.714672 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.714522    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/242f4d41-bb8d-4488-8477-0de8946569ce-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:51.714672 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.714629    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/242f4d41-bb8d-4488-8477-0de8946569ce-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:51.714672 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.714665    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/242f4d41-bb8d-4488-8477-0de8946569ce-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:51.715047 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.714701    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/242f4d41-bb8d-4488-8477-0de8946569ce-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:51.715047 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.714727    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/242f4d41-bb8d-4488-8477-0de8946569ce-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:51.715047 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.714779    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/242f4d41-bb8d-4488-8477-0de8946569ce-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:51.715047 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.714807    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/242f4d41-bb8d-4488-8477-0de8946569ce-web-config\") pod \"prometheus-k8s-0\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:51.715047 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.714842    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/242f4d41-bb8d-4488-8477-0de8946569ce-config\") pod \"prometheus-k8s-0\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:51.715047 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.714872    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/242f4d41-bb8d-4488-8477-0de8946569ce-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:51.716842 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.716355    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/242f4d41-bb8d-4488-8477-0de8946569ce-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:51.717702 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.717677    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/242f4d41-bb8d-4488-8477-0de8946569ce-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:51.718297 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.718272    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/242f4d41-bb8d-4488-8477-0de8946569ce-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:51.718770 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.717684    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/242f4d41-bb8d-4488-8477-0de8946569ce-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:51.730137 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.721442    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/242f4d41-bb8d-4488-8477-0de8946569ce-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:51.730137 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.721708    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/242f4d41-bb8d-4488-8477-0de8946569ce-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:51.730137 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.722126    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/242f4d41-bb8d-4488-8477-0de8946569ce-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:51.730137 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.722146    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/242f4d41-bb8d-4488-8477-0de8946569ce-config\") pod \"prometheus-k8s-0\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:51.730137 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.722540    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/242f4d41-bb8d-4488-8477-0de8946569ce-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:51.730137 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.722660    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/242f4d41-bb8d-4488-8477-0de8946569ce-web-config\") pod \"prometheus-k8s-0\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:51.730137 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.723201    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/242f4d41-bb8d-4488-8477-0de8946569ce-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:51.730137 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.725172    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/242f4d41-bb8d-4488-8477-0de8946569ce-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:51.730137 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.727232    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/242f4d41-bb8d-4488-8477-0de8946569ce-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:51.730137 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.728206    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/242f4d41-bb8d-4488-8477-0de8946569ce-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:51.730137 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.728372    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/242f4d41-bb8d-4488-8477-0de8946569ce-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:51.730137 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.729832    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/242f4d41-bb8d-4488-8477-0de8946569ce-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:51.730137 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.729991    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/242f4d41-bb8d-4488-8477-0de8946569ce-config-out\") pod \"prometheus-k8s-0\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:51.731005 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.730261    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/242f4d41-bb8d-4488-8477-0de8946569ce-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:51.731005 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.730563    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/242f4d41-bb8d-4488-8477-0de8946569ce-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:51.731005 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.730957    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b86vd\" (UniqueName: \"kubernetes.io/projected/242f4d41-bb8d-4488-8477-0de8946569ce-kube-api-access-b86vd\") pod \"prometheus-k8s-0\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:51.853811 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:51.853475    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:52.554229 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:52.554182    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 16:48:52.557427 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:52.557404    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-86d8764fcc-t7vw9"]
Jul 07 16:48:53.462085 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:48:53.462045    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod242f4d41_bb8d_4488_8477_0de8946569ce.slice/crio-a05e57a0324445d9f993465b20e1e556684a378750d3c9ade773be37bddc27c8 WatchSource:0}: Error finding container a05e57a0324445d9f993465b20e1e556684a378750d3c9ade773be37bddc27c8: Status 404 returned error can't find the container with id a05e57a0324445d9f993465b20e1e556684a378750d3c9ade773be37bddc27c8
Jul 07 16:48:53.463029 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:48:53.462709    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff8de6df_aa2e_4d3e_bac0_a746f8433868.slice/crio-c48a4c3ab379b1375db47c6d647ed7f853c3add8b95142e2e52dec6398d53166 WatchSource:0}: Error finding container c48a4c3ab379b1375db47c6d647ed7f853c3add8b95142e2e52dec6398d53166: Status 404 returned error can't find the container with id c48a4c3ab379b1375db47c6d647ed7f853c3add8b95142e2e52dec6398d53166
Jul 07 16:48:54.053544 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:54.053509    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"242f4d41-bb8d-4488-8477-0de8946569ce","Type":"ContainerStarted","Data":"a05e57a0324445d9f993465b20e1e556684a378750d3c9ade773be37bddc27c8"}
Jul 07 16:48:54.055465 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:54.055171    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-86d8764fcc-t7vw9" event={"ID":"ff8de6df-aa2e-4d3e-bac0-a746f8433868","Type":"ContainerStarted","Data":"c48a4c3ab379b1375db47c6d647ed7f853c3add8b95142e2e52dec6398d53166"}
Jul 07 16:48:55.064639 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:55.064501    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"835f4b2a-30d0-441b-af00-f0b2d166fe7c","Type":"ContainerStarted","Data":"08b05db6e10b744177a3b6ada620be59f1b0ae73061f00e104f73f9766fd6723"}
Jul 07 16:48:55.067498 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:55.067468    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7886dfcf5f-r7svd" event={"ID":"323eccaa-741e-495a-81a2-6368321ae1db","Type":"ContainerStarted","Data":"88923c6729097df01a5d7f611a2b8d00e474fd2aee5bd8b332a637ca36a44c5c"}
Jul 07 16:48:55.069887 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:55.069845    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kdcss" event={"ID":"91e42c9a-31b0-42bf-9055-28fd68ab4ba9","Type":"ContainerStarted","Data":"6dc3e4d1ec2792ddd7c43b9048226f94bb57d3b79317e0d148aa421727e705cc"}
Jul 07 16:48:55.072122 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:55.072095    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wwb84" event={"ID":"57008745-859f-49ec-9269-7671fa8164a2","Type":"ContainerStarted","Data":"1ab20aedf11317367ca56e25c3e1e8c8104f7d626a4183279bcbd528b00b6ab6"}
Jul 07 16:48:55.072583 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:55.072534    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wwb84"
Jul 07 16:48:55.074349 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:55.074302    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-h5f7f" event={"ID":"816e0df7-c9cf-4303-a920-a90a16ee015c","Type":"ContainerStarted","Data":"bf6183c7d1186ab7fc069a09795d2e7859e3dd3d829bdddad613da609db86f0b"}
Jul 07 16:48:55.076158 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:55.076031    2578 generic.go:356] "Generic (PLEG): container finished" podID="242f4d41-bb8d-4488-8477-0de8946569ce" containerID="e8ae6fc78374b6e07ffb679f868d996001e98e5bab5cc61560ed025b5308f863" exitCode=0
Jul 07 16:48:55.076158 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:55.076066    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"242f4d41-bb8d-4488-8477-0de8946569ce","Type":"ContainerDied","Data":"e8ae6fc78374b6e07ffb679f868d996001e98e5bab5cc61560ed025b5308f863"}
Jul 07 16:48:55.088576 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:55.088528    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7886dfcf5f-r7svd" podStartSLOduration=3.693078867 podStartE2EDuration="6.088510148s" totalImagesPullingTime="2.395431281s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:48:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:48:55.086624527 +0000 UTC m=+71.862938761" watchObservedRunningTime="2026-07-07 16:48:55.088510148 +0000 UTC m=+71.864824356"
Jul 07 16:48:55.114046 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:55.113263    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-wwb84" podStartSLOduration=66.975243605 podStartE2EDuration="1m12.113241336s" totalImagesPullingTime="5.137997731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:47:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:48:55.112219203 +0000 UTC m=+71.888533411" watchObservedRunningTime="2026-07-07 16:48:55.113241336 +0000 UTC m=+71.889555543"
Jul 07 16:48:55.180699 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:55.180616    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-h5f7f" podStartSLOduration=67.015336221 podStartE2EDuration="1m12.180594472s" totalImagesPullingTime="5.165258251s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:47:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:48:55.179594802 +0000 UTC m=+71.955909029" watchObservedRunningTime="2026-07-07 16:48:55.180594472 +0000 UTC m=+71.956908680"
Jul 07 16:48:55.844280 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:55.844247    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-787c6b6b98-7v2dr"]
Jul 07 16:48:56.083361 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:56.083328    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"835f4b2a-30d0-441b-af00-f0b2d166fe7c","Type":"ContainerStarted","Data":"c613ff08b7c96b399889a7b382659b69a64ed3cce71c955c9eb3221e4e0e8388"}
Jul 07 16:48:56.083838 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:56.083416    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"835f4b2a-30d0-441b-af00-f0b2d166fe7c","Type":"ContainerStarted","Data":"7783db8b43c7d9ba2b52fd561c366ab1640876727cf802b178720b09bf5cee32"}
Jul 07 16:48:56.083838 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:56.083455    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"835f4b2a-30d0-441b-af00-f0b2d166fe7c","Type":"ContainerStarted","Data":"34abfee077090afb0e19341d0de872441659de05216970611259dceaa6820141"}
Jul 07 16:48:56.083838 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:56.083491    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"835f4b2a-30d0-441b-af00-f0b2d166fe7c","Type":"ContainerStarted","Data":"4e193be7788020bdc3da9597679ff37023d4e62716efecd6565e0f1163d5c040"}
Jul 07 16:48:56.085224 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:56.085195    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kdcss" event={"ID":"91e42c9a-31b0-42bf-9055-28fd68ab4ba9","Type":"ContainerStarted","Data":"1c6ebea859f96edc145c22bb791552835002d181972f4838e9f551948643480e"}
Jul 07 16:48:56.102885 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:56.102801    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-kdcss" podStartSLOduration=70.464037819 podStartE2EDuration="1m13.102785922s" totalImagesPullingTime="2.638748103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:47:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:48:56.101979992 +0000 UTC m=+72.878294217" watchObservedRunningTime="2026-07-07 16:48:56.102785922 +0000 UTC m=+72.879100132"
Jul 07 16:48:57.093741 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:57.093687    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-86d8764fcc-t7vw9" event={"ID":"ff8de6df-aa2e-4d3e-bac0-a746f8433868","Type":"ContainerStarted","Data":"09d921e55ebdba5a08224bb19c5383bcf38966f3081cff9ea1b48afa53b4aa8e"}
Jul 07 16:48:57.094290 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:57.093800    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-86d8764fcc-t7vw9" event={"ID":"ff8de6df-aa2e-4d3e-bac0-a746f8433868","Type":"ContainerStarted","Data":"23eddb57b5403095b3c3b86baaa84baf3681cf5ef7ae1b80acc99ff080101e61"}
Jul 07 16:48:57.094290 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:57.093821    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-86d8764fcc-t7vw9" event={"ID":"ff8de6df-aa2e-4d3e-bac0-a746f8433868","Type":"ContainerStarted","Data":"1f2e83ba2cfb8f86de4388831c4c28372c692a5a84059e671ea56f2ac60456f9"}
Jul 07 16:48:57.097453 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:57.097419    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"835f4b2a-30d0-441b-af00-f0b2d166fe7c","Type":"ContainerStarted","Data":"06eeedde37279cb18fbaa81647e367fe563bca47a6e394cbc0f6a0e4c34ab420"}
Jul 07 16:48:57.118178 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:57.118118    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-86d8764fcc-t7vw9" podStartSLOduration=3.966579186 podStartE2EDuration="7.118096949s" totalImagesPullingTime="3.151517763s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:48:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:48:57.116422795 +0000 UTC m=+73.892737023" watchObservedRunningTime="2026-07-07 16:48:57.118096949 +0000 UTC m=+73.894411160"
Jul 07 16:48:57.146662 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:57.146602    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=5.339970686 podStartE2EDuration="11.146585485s" totalImagesPullingTime="5.788080459s" totalInitContainerRuntime="18.53434ms" isStatefulPod=false podCreationTimestamp="2026-07-07 16:48:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:48:57.144841052 +0000 UTC m=+73.921155281" watchObservedRunningTime="2026-07-07 16:48:57.146585485 +0000 UTC m=+73.922899691"
Jul 07 16:48:57.957391 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:57.957355    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-787c6b6b98-7v2dr"
Jul 07 16:48:59.109714 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:48:59.109690    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"242f4d41-bb8d-4488-8477-0de8946569ce","Type":"ContainerStarted","Data":"160fa002a7adbf7be0064d29e23e6b4a0d40a877ee39afa9472abc3c1610f244"}
Jul 07 16:49:00.117408 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:00.117373    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"242f4d41-bb8d-4488-8477-0de8946569ce","Type":"ContainerStarted","Data":"0b49853db2492a72058608ec810b494cd313faeface778b08816e20570f88376"}
Jul 07 16:49:02.129849 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:02.129811    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"242f4d41-bb8d-4488-8477-0de8946569ce","Type":"ContainerStarted","Data":"28c09282528f2dafed0372f50b6d295fd7b28d2e392b318ec4ada3ee02649a03"}
Jul 07 16:49:02.130223 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:02.129890    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"242f4d41-bb8d-4488-8477-0de8946569ce","Type":"ContainerStarted","Data":"b8d17cbc6192e7797c7d8c746f009f7cc4a619f591db65b7b5ac0049d702179c"}
Jul 07 16:49:02.130223 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:02.129933    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"242f4d41-bb8d-4488-8477-0de8946569ce","Type":"ContainerStarted","Data":"0da92d3ce34e6c80b754c6ca6bfd37abcb3f3d5a486338eeb982bfd47baf3c7e"}
Jul 07 16:49:02.130223 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:02.129972    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"242f4d41-bb8d-4488-8477-0de8946569ce","Type":"ContainerStarted","Data":"3ae5ceb91f346dbe8f4c9fd4b8e34dbe74d6b4237794432e5bb895641800ff3f"}
Jul 07 16:49:02.161585 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:02.161524    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=5.154679345 podStartE2EDuration="11.161504709s" totalImagesPullingTime="5.973706354s" totalInitContainerRuntime="33.11901ms" isStatefulPod=false podCreationTimestamp="2026-07-07 16:48:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:49:02.157987764 +0000 UTC m=+78.934301970" watchObservedRunningTime="2026-07-07 16:49:02.161504709 +0000 UTC m=+78.937818916"
Jul 07 16:49:06.854184 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:06.854066    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:08.921482 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:08.921448    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7475fc7665-d9jpq"]
Jul 07 16:49:09.847250 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:09.847224    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7886dfcf5f-r7svd"
Jul 07 16:49:09.847405 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:09.847277    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7886dfcf5f-r7svd"
Jul 07 16:49:20.876108 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:20.876027    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-787c6b6b98-7v2dr" podUID="9480c793-ca95-433d-8d2a-9f8fb71e04a1" containerName="console" containerID="cri-o://9f383eaa60ef69f3cd46117306b72da0e23ce1f5baf023f1b35ef02e65fe68d4" gracePeriod=15
Jul 07 16:49:21.117185 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:21.117161    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-787c6b6b98-7v2dr"
Jul 07 16:49:21.188652 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:21.188572    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9480c793-ca95-433d-8d2a-9f8fb71e04a1-oauth-serving-cert\") pod \"9480c793-ca95-433d-8d2a-9f8fb71e04a1\" (UID: \"9480c793-ca95-433d-8d2a-9f8fb71e04a1\") "
Jul 07 16:49:21.188652 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:21.188617    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9480c793-ca95-433d-8d2a-9f8fb71e04a1-console-config\") pod \"9480c793-ca95-433d-8d2a-9f8fb71e04a1\" (UID: \"9480c793-ca95-433d-8d2a-9f8fb71e04a1\") "
Jul 07 16:49:21.188882 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:21.188684    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9480c793-ca95-433d-8d2a-9f8fb71e04a1-service-ca\") pod \"9480c793-ca95-433d-8d2a-9f8fb71e04a1\" (UID: \"9480c793-ca95-433d-8d2a-9f8fb71e04a1\") "
Jul 07 16:49:21.188882 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:21.188718    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9480c793-ca95-433d-8d2a-9f8fb71e04a1-console-oauth-config\") pod \"9480c793-ca95-433d-8d2a-9f8fb71e04a1\" (UID: \"9480c793-ca95-433d-8d2a-9f8fb71e04a1\") "
Jul 07 16:49:21.188882 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:21.188784    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9480c793-ca95-433d-8d2a-9f8fb71e04a1-trusted-ca-bundle\") pod \"9480c793-ca95-433d-8d2a-9f8fb71e04a1\" (UID: \"9480c793-ca95-433d-8d2a-9f8fb71e04a1\") "
Jul 07 16:49:21.188882 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:21.188814    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9480c793-ca95-433d-8d2a-9f8fb71e04a1-console-serving-cert\") pod \"9480c793-ca95-433d-8d2a-9f8fb71e04a1\" (UID: \"9480c793-ca95-433d-8d2a-9f8fb71e04a1\") "
Jul 07 16:49:21.188882 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:21.188843    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4bs6\" (UniqueName: \"kubernetes.io/projected/9480c793-ca95-433d-8d2a-9f8fb71e04a1-kube-api-access-p4bs6\") pod \"9480c793-ca95-433d-8d2a-9f8fb71e04a1\" (UID: \"9480c793-ca95-433d-8d2a-9f8fb71e04a1\") "
Jul 07 16:49:21.189129 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:21.188992    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9480c793-ca95-433d-8d2a-9f8fb71e04a1-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9480c793-ca95-433d-8d2a-9f8fb71e04a1" (UID: "9480c793-ca95-433d-8d2a-9f8fb71e04a1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:49:21.189129 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:21.189036    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9480c793-ca95-433d-8d2a-9f8fb71e04a1-service-ca" (OuterVolumeSpecName: "service-ca") pod "9480c793-ca95-433d-8d2a-9f8fb71e04a1" (UID: "9480c793-ca95-433d-8d2a-9f8fb71e04a1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:49:21.189129 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:21.189100    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9480c793-ca95-433d-8d2a-9f8fb71e04a1-console-config" (OuterVolumeSpecName: "console-config") pod "9480c793-ca95-433d-8d2a-9f8fb71e04a1" (UID: "9480c793-ca95-433d-8d2a-9f8fb71e04a1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:49:21.189282 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:21.189173    2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9480c793-ca95-433d-8d2a-9f8fb71e04a1-oauth-serving-cert\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:49:21.189282 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:21.189192    2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9480c793-ca95-433d-8d2a-9f8fb71e04a1-console-config\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:49:21.189282 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:21.189205    2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9480c793-ca95-433d-8d2a-9f8fb71e04a1-service-ca\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:49:21.189427 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:21.189408    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9480c793-ca95-433d-8d2a-9f8fb71e04a1-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "9480c793-ca95-433d-8d2a-9f8fb71e04a1" (UID: "9480c793-ca95-433d-8d2a-9f8fb71e04a1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:49:21.191307 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:21.191271    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9480c793-ca95-433d-8d2a-9f8fb71e04a1-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9480c793-ca95-433d-8d2a-9f8fb71e04a1" (UID: "9480c793-ca95-433d-8d2a-9f8fb71e04a1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:49:21.191441 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:21.191396    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9480c793-ca95-433d-8d2a-9f8fb71e04a1-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9480c793-ca95-433d-8d2a-9f8fb71e04a1" (UID: "9480c793-ca95-433d-8d2a-9f8fb71e04a1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:49:21.191441 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:21.191398    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9480c793-ca95-433d-8d2a-9f8fb71e04a1-kube-api-access-p4bs6" (OuterVolumeSpecName: "kube-api-access-p4bs6") pod "9480c793-ca95-433d-8d2a-9f8fb71e04a1" (UID: "9480c793-ca95-433d-8d2a-9f8fb71e04a1"). InnerVolumeSpecName "kube-api-access-p4bs6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:49:21.206361 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:21.206334    2578 generic.go:356] "Generic (PLEG): container finished" podID="9480c793-ca95-433d-8d2a-9f8fb71e04a1" containerID="9f383eaa60ef69f3cd46117306b72da0e23ce1f5baf023f1b35ef02e65fe68d4" exitCode=0
Jul 07 16:49:21.206501 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:21.206373    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-787c6b6b98-7v2dr"
Jul 07 16:49:21.206501 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:21.206372    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-787c6b6b98-7v2dr" event={"ID":"9480c793-ca95-433d-8d2a-9f8fb71e04a1","Type":"ContainerDied","Data":"9f383eaa60ef69f3cd46117306b72da0e23ce1f5baf023f1b35ef02e65fe68d4"}
Jul 07 16:49:21.206501 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:21.206496    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-787c6b6b98-7v2dr" event={"ID":"9480c793-ca95-433d-8d2a-9f8fb71e04a1","Type":"ContainerDied","Data":"7dc4aa0885fbd6ccac89cd1a1a686d0841ccb63d037479826f8e4800d29cf91e"}
Jul 07 16:49:21.206687 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:21.206533    2578 scope.go:117] "RemoveContainer" containerID="9f383eaa60ef69f3cd46117306b72da0e23ce1f5baf023f1b35ef02e65fe68d4"
Jul 07 16:49:21.218529 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:21.216384    2578 scope.go:117] "RemoveContainer" containerID="9f383eaa60ef69f3cd46117306b72da0e23ce1f5baf023f1b35ef02e65fe68d4"
Jul 07 16:49:21.218529 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:49:21.217143    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f383eaa60ef69f3cd46117306b72da0e23ce1f5baf023f1b35ef02e65fe68d4\": container with ID starting with 9f383eaa60ef69f3cd46117306b72da0e23ce1f5baf023f1b35ef02e65fe68d4 not found: ID does not exist" containerID="9f383eaa60ef69f3cd46117306b72da0e23ce1f5baf023f1b35ef02e65fe68d4"
Jul 07 16:49:21.218529 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:21.217177    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f383eaa60ef69f3cd46117306b72da0e23ce1f5baf023f1b35ef02e65fe68d4"} err="failed to get container status \"9f383eaa60ef69f3cd46117306b72da0e23ce1f5baf023f1b35ef02e65fe68d4\": rpc error: code = NotFound desc = could not find container \"9f383eaa60ef69f3cd46117306b72da0e23ce1f5baf023f1b35ef02e65fe68d4\": container with ID starting with 9f383eaa60ef69f3cd46117306b72da0e23ce1f5baf023f1b35ef02e65fe68d4 not found: ID does not exist"
Jul 07 16:49:21.228223 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:21.228199    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-787c6b6b98-7v2dr"]
Jul 07 16:49:21.232004 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:21.231980    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-787c6b6b98-7v2dr"]
Jul 07 16:49:21.290304 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:21.290265    2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9480c793-ca95-433d-8d2a-9f8fb71e04a1-console-oauth-config\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:49:21.290304 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:21.290295    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9480c793-ca95-433d-8d2a-9f8fb71e04a1-trusted-ca-bundle\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:49:21.290304 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:21.290309    2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9480c793-ca95-433d-8d2a-9f8fb71e04a1-console-serving-cert\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:49:21.290537 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:21.290321    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4bs6\" (UniqueName: \"kubernetes.io/projected/9480c793-ca95-433d-8d2a-9f8fb71e04a1-kube-api-access-p4bs6\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:49:21.725099 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:21.725069    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9480c793-ca95-433d-8d2a-9f8fb71e04a1" path="/var/lib/kubelet/pods/9480c793-ca95-433d-8d2a-9f8fb71e04a1/volumes"
Jul 07 16:49:26.088414 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:26.088381    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-wwb84"
Jul 07 16:49:29.852660 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:29.852634    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7886dfcf5f-r7svd"
Jul 07 16:49:29.856495 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:29.856473    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7886dfcf5f-r7svd"
Jul 07 16:49:33.941801 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:33.941757    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-7475fc7665-d9jpq" podUID="5790cd94-92b3-46f3-8441-caf5efcb2779" containerName="console" containerID="cri-o://7448d5f51370ad91a6706d48cfc5e17b80ba3e45f6b333d31bfd731d142e27e1" gracePeriod=15
Jul 07 16:49:34.192877 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:34.192823    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7475fc7665-d9jpq"
Jul 07 16:49:34.256293 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:34.256267    2578 generic.go:356] "Generic (PLEG): container finished" podID="5790cd94-92b3-46f3-8441-caf5efcb2779" containerID="7448d5f51370ad91a6706d48cfc5e17b80ba3e45f6b333d31bfd731d142e27e1" exitCode=0
Jul 07 16:49:34.256435 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:34.256305    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7475fc7665-d9jpq" event={"ID":"5790cd94-92b3-46f3-8441-caf5efcb2779","Type":"ContainerDied","Data":"7448d5f51370ad91a6706d48cfc5e17b80ba3e45f6b333d31bfd731d142e27e1"}
Jul 07 16:49:34.256435 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:34.256312    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7475fc7665-d9jpq"
Jul 07 16:49:34.256435 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:34.256340    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7475fc7665-d9jpq" event={"ID":"5790cd94-92b3-46f3-8441-caf5efcb2779","Type":"ContainerDied","Data":"a5457d863d8ef6bf26b9fea7da21807ca68ff795971afca9929613415f96c852"}
Jul 07 16:49:34.256435 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:34.256358    2578 scope.go:117] "RemoveContainer" containerID="7448d5f51370ad91a6706d48cfc5e17b80ba3e45f6b333d31bfd731d142e27e1"
Jul 07 16:49:34.263836 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:34.263818    2578 scope.go:117] "RemoveContainer" containerID="7448d5f51370ad91a6706d48cfc5e17b80ba3e45f6b333d31bfd731d142e27e1"
Jul 07 16:49:34.264080 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:49:34.264062    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7448d5f51370ad91a6706d48cfc5e17b80ba3e45f6b333d31bfd731d142e27e1\": container with ID starting with 7448d5f51370ad91a6706d48cfc5e17b80ba3e45f6b333d31bfd731d142e27e1 not found: ID does not exist" containerID="7448d5f51370ad91a6706d48cfc5e17b80ba3e45f6b333d31bfd731d142e27e1"
Jul 07 16:49:34.264135 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:34.264090    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7448d5f51370ad91a6706d48cfc5e17b80ba3e45f6b333d31bfd731d142e27e1"} err="failed to get container status \"7448d5f51370ad91a6706d48cfc5e17b80ba3e45f6b333d31bfd731d142e27e1\": rpc error: code = NotFound desc = could not find container \"7448d5f51370ad91a6706d48cfc5e17b80ba3e45f6b333d31bfd731d142e27e1\": container with ID starting with 7448d5f51370ad91a6706d48cfc5e17b80ba3e45f6b333d31bfd731d142e27e1 not found: ID does not exist"
Jul 07 16:49:34.297359 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:34.297340    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pczm4\" (UniqueName: \"kubernetes.io/projected/5790cd94-92b3-46f3-8441-caf5efcb2779-kube-api-access-pczm4\") pod \"5790cd94-92b3-46f3-8441-caf5efcb2779\" (UID: \"5790cd94-92b3-46f3-8441-caf5efcb2779\") "
Jul 07 16:49:34.297431 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:34.297370    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5790cd94-92b3-46f3-8441-caf5efcb2779-console-serving-cert\") pod \"5790cd94-92b3-46f3-8441-caf5efcb2779\" (UID: \"5790cd94-92b3-46f3-8441-caf5efcb2779\") "
Jul 07 16:49:34.297431 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:34.297398    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5790cd94-92b3-46f3-8441-caf5efcb2779-oauth-serving-cert\") pod \"5790cd94-92b3-46f3-8441-caf5efcb2779\" (UID: \"5790cd94-92b3-46f3-8441-caf5efcb2779\") "
Jul 07 16:49:34.297431 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:34.297427    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5790cd94-92b3-46f3-8441-caf5efcb2779-console-oauth-config\") pod \"5790cd94-92b3-46f3-8441-caf5efcb2779\" (UID: \"5790cd94-92b3-46f3-8441-caf5efcb2779\") "
Jul 07 16:49:34.297524 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:34.297450    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5790cd94-92b3-46f3-8441-caf5efcb2779-service-ca\") pod \"5790cd94-92b3-46f3-8441-caf5efcb2779\" (UID: \"5790cd94-92b3-46f3-8441-caf5efcb2779\") "
Jul 07 16:49:34.297568 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:34.297520    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5790cd94-92b3-46f3-8441-caf5efcb2779-console-config\") pod \"5790cd94-92b3-46f3-8441-caf5efcb2779\" (UID: \"5790cd94-92b3-46f3-8441-caf5efcb2779\") "
Jul 07 16:49:34.297902 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:34.297877    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5790cd94-92b3-46f3-8441-caf5efcb2779-service-ca" (OuterVolumeSpecName: "service-ca") pod "5790cd94-92b3-46f3-8441-caf5efcb2779" (UID: "5790cd94-92b3-46f3-8441-caf5efcb2779"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:49:34.297902 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:34.297892    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5790cd94-92b3-46f3-8441-caf5efcb2779-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "5790cd94-92b3-46f3-8441-caf5efcb2779" (UID: "5790cd94-92b3-46f3-8441-caf5efcb2779"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:49:34.298032 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:34.297964    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5790cd94-92b3-46f3-8441-caf5efcb2779-console-config" (OuterVolumeSpecName: "console-config") pod "5790cd94-92b3-46f3-8441-caf5efcb2779" (UID: "5790cd94-92b3-46f3-8441-caf5efcb2779"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:49:34.299498 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:34.299470    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5790cd94-92b3-46f3-8441-caf5efcb2779-kube-api-access-pczm4" (OuterVolumeSpecName: "kube-api-access-pczm4") pod "5790cd94-92b3-46f3-8441-caf5efcb2779" (UID: "5790cd94-92b3-46f3-8441-caf5efcb2779"). InnerVolumeSpecName "kube-api-access-pczm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:49:34.299498 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:34.299475    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5790cd94-92b3-46f3-8441-caf5efcb2779-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "5790cd94-92b3-46f3-8441-caf5efcb2779" (UID: "5790cd94-92b3-46f3-8441-caf5efcb2779"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:49:34.299612 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:34.299542    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5790cd94-92b3-46f3-8441-caf5efcb2779-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "5790cd94-92b3-46f3-8441-caf5efcb2779" (UID: "5790cd94-92b3-46f3-8441-caf5efcb2779"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:49:34.398459 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:34.398429    2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5790cd94-92b3-46f3-8441-caf5efcb2779-console-serving-cert\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:49:34.398459 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:34.398454    2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5790cd94-92b3-46f3-8441-caf5efcb2779-oauth-serving-cert\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:49:34.398459 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:34.398463    2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5790cd94-92b3-46f3-8441-caf5efcb2779-console-oauth-config\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:49:34.398632 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:34.398472    2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5790cd94-92b3-46f3-8441-caf5efcb2779-service-ca\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:49:34.398632 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:34.398481    2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5790cd94-92b3-46f3-8441-caf5efcb2779-console-config\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:49:34.398632 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:34.398488    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pczm4\" (UniqueName: \"kubernetes.io/projected/5790cd94-92b3-46f3-8441-caf5efcb2779-kube-api-access-pczm4\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:49:34.577713 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:34.577686    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7475fc7665-d9jpq"]
Jul 07 16:49:34.581544 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:34.581521    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7475fc7665-d9jpq"]
Jul 07 16:49:35.724984 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:35.724952    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5790cd94-92b3-46f3-8441-caf5efcb2779" path="/var/lib/kubelet/pods/5790cd94-92b3-46f3-8441-caf5efcb2779/volumes"
Jul 07 16:49:51.854471 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:51.854440    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:51.873993 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:51.873963    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:52.334991 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:49:52.334967    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:05.621881 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:05.621847    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 07 16:50:05.622416 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:05.622199    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="835f4b2a-30d0-441b-af00-f0b2d166fe7c" containerName="alertmanager" containerID="cri-o://08b05db6e10b744177a3b6ada620be59f1b0ae73061f00e104f73f9766fd6723" gracePeriod=120
Jul 07 16:50:05.622416 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:05.622260    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="835f4b2a-30d0-441b-af00-f0b2d166fe7c" containerName="kube-rbac-proxy-metric" containerID="cri-o://c613ff08b7c96b399889a7b382659b69a64ed3cce71c955c9eb3221e4e0e8388" gracePeriod=120
Jul 07 16:50:05.622416 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:05.622307    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="835f4b2a-30d0-441b-af00-f0b2d166fe7c" containerName="kube-rbac-proxy" containerID="cri-o://7783db8b43c7d9ba2b52fd561c366ab1640876727cf802b178720b09bf5cee32" gracePeriod=120
Jul 07 16:50:05.622416 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:05.622343    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="835f4b2a-30d0-441b-af00-f0b2d166fe7c" containerName="prom-label-proxy" containerID="cri-o://06eeedde37279cb18fbaa81647e367fe563bca47a6e394cbc0f6a0e4c34ab420" gracePeriod=120
Jul 07 16:50:05.622416 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:05.622274    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="835f4b2a-30d0-441b-af00-f0b2d166fe7c" containerName="kube-rbac-proxy-web" containerID="cri-o://34abfee077090afb0e19341d0de872441659de05216970611259dceaa6820141" gracePeriod=120
Jul 07 16:50:05.622668 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:05.622479    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="835f4b2a-30d0-441b-af00-f0b2d166fe7c" containerName="config-reloader" containerID="cri-o://4e193be7788020bdc3da9597679ff37023d4e62716efecd6565e0f1163d5c040" gracePeriod=120
Jul 07 16:50:06.372811 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:06.372779    2578 generic.go:356] "Generic (PLEG): container finished" podID="835f4b2a-30d0-441b-af00-f0b2d166fe7c" containerID="06eeedde37279cb18fbaa81647e367fe563bca47a6e394cbc0f6a0e4c34ab420" exitCode=0
Jul 07 16:50:06.372811 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:06.372804    2578 generic.go:356] "Generic (PLEG): container finished" podID="835f4b2a-30d0-441b-af00-f0b2d166fe7c" containerID="7783db8b43c7d9ba2b52fd561c366ab1640876727cf802b178720b09bf5cee32" exitCode=0
Jul 07 16:50:06.372811 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:06.372812    2578 generic.go:356] "Generic (PLEG): container finished" podID="835f4b2a-30d0-441b-af00-f0b2d166fe7c" containerID="4e193be7788020bdc3da9597679ff37023d4e62716efecd6565e0f1163d5c040" exitCode=0
Jul 07 16:50:06.373041 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:06.372820    2578 generic.go:356] "Generic (PLEG): container finished" podID="835f4b2a-30d0-441b-af00-f0b2d166fe7c" containerID="08b05db6e10b744177a3b6ada620be59f1b0ae73061f00e104f73f9766fd6723" exitCode=0
Jul 07 16:50:06.373041 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:06.372859    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"835f4b2a-30d0-441b-af00-f0b2d166fe7c","Type":"ContainerDied","Data":"06eeedde37279cb18fbaa81647e367fe563bca47a6e394cbc0f6a0e4c34ab420"}
Jul 07 16:50:06.373041 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:06.372924    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"835f4b2a-30d0-441b-af00-f0b2d166fe7c","Type":"ContainerDied","Data":"7783db8b43c7d9ba2b52fd561c366ab1640876727cf802b178720b09bf5cee32"}
Jul 07 16:50:06.373041 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:06.372952    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"835f4b2a-30d0-441b-af00-f0b2d166fe7c","Type":"ContainerDied","Data":"4e193be7788020bdc3da9597679ff37023d4e62716efecd6565e0f1163d5c040"}
Jul 07 16:50:06.373041 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:06.372978    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"835f4b2a-30d0-441b-af00-f0b2d166fe7c","Type":"ContainerDied","Data":"08b05db6e10b744177a3b6ada620be59f1b0ae73061f00e104f73f9766fd6723"}
Jul 07 16:50:06.854227 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:06.854204    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:50:06.866583 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:06.866553    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zrbf9\" (UniqueName: \"kubernetes.io/projected/835f4b2a-30d0-441b-af00-f0b2d166fe7c-kube-api-access-zrbf9\") pod \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\" (UID: \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\") "
Jul 07 16:50:06.866699 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:06.866589    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/835f4b2a-30d0-441b-af00-f0b2d166fe7c-secret-alertmanager-kube-rbac-proxy-web\") pod \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\" (UID: \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\") "
Jul 07 16:50:06.866699 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:06.866615    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/835f4b2a-30d0-441b-af00-f0b2d166fe7c-config-out\") pod \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\" (UID: \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\") "
Jul 07 16:50:06.866699 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:06.866660    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/835f4b2a-30d0-441b-af00-f0b2d166fe7c-alertmanager-main-db\") pod \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\" (UID: \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\") "
Jul 07 16:50:06.867066 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:06.866711    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/835f4b2a-30d0-441b-af00-f0b2d166fe7c-secret-alertmanager-kube-rbac-proxy\") pod \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\" (UID: \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\") "
Jul 07 16:50:06.867066 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:06.866838    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/835f4b2a-30d0-441b-af00-f0b2d166fe7c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\" (UID: \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\") "
Jul 07 16:50:06.867066 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:06.866874    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/835f4b2a-30d0-441b-af00-f0b2d166fe7c-config-volume\") pod \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\" (UID: \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\") "
Jul 07 16:50:06.867066 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:06.866905    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/835f4b2a-30d0-441b-af00-f0b2d166fe7c-tls-assets\") pod \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\" (UID: \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\") "
Jul 07 16:50:06.867066 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:06.866932    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/835f4b2a-30d0-441b-af00-f0b2d166fe7c-metrics-client-ca\") pod \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\" (UID: \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\") "
Jul 07 16:50:06.867066 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:06.866971    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/835f4b2a-30d0-441b-af00-f0b2d166fe7c-secret-alertmanager-main-tls\") pod \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\" (UID: \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\") "
Jul 07 16:50:06.867066 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:06.867015    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/835f4b2a-30d0-441b-af00-f0b2d166fe7c-alertmanager-trusted-ca-bundle\") pod \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\" (UID: \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\") "
Jul 07 16:50:06.867066 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:06.867039    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/835f4b2a-30d0-441b-af00-f0b2d166fe7c-web-config\") pod \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\" (UID: \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\") "
Jul 07 16:50:06.867066 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:06.867052    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/835f4b2a-30d0-441b-af00-f0b2d166fe7c-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "835f4b2a-30d0-441b-af00-f0b2d166fe7c" (UID: "835f4b2a-30d0-441b-af00-f0b2d166fe7c"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:50:06.867494 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:06.867083    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/835f4b2a-30d0-441b-af00-f0b2d166fe7c-cluster-tls-config\") pod \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\" (UID: \"835f4b2a-30d0-441b-af00-f0b2d166fe7c\") "
Jul 07 16:50:06.867494 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:06.867307    2578 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/835f4b2a-30d0-441b-af00-f0b2d166fe7c-alertmanager-main-db\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:50:06.867494 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:06.867367    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/835f4b2a-30d0-441b-af00-f0b2d166fe7c-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "835f4b2a-30d0-441b-af00-f0b2d166fe7c" (UID: "835f4b2a-30d0-441b-af00-f0b2d166fe7c"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:50:06.868177 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:06.867916    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/835f4b2a-30d0-441b-af00-f0b2d166fe7c-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "835f4b2a-30d0-441b-af00-f0b2d166fe7c" (UID: "835f4b2a-30d0-441b-af00-f0b2d166fe7c"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:50:06.869395 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:06.869369    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/835f4b2a-30d0-441b-af00-f0b2d166fe7c-kube-api-access-zrbf9" (OuterVolumeSpecName: "kube-api-access-zrbf9") pod "835f4b2a-30d0-441b-af00-f0b2d166fe7c" (UID: "835f4b2a-30d0-441b-af00-f0b2d166fe7c"). InnerVolumeSpecName "kube-api-access-zrbf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:50:06.869880 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:06.869834    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/835f4b2a-30d0-441b-af00-f0b2d166fe7c-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "835f4b2a-30d0-441b-af00-f0b2d166fe7c" (UID: "835f4b2a-30d0-441b-af00-f0b2d166fe7c"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:50:06.869880 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:06.869872    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/835f4b2a-30d0-441b-af00-f0b2d166fe7c-config-volume" (OuterVolumeSpecName: "config-volume") pod "835f4b2a-30d0-441b-af00-f0b2d166fe7c" (UID: "835f4b2a-30d0-441b-af00-f0b2d166fe7c"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:50:06.870464 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:06.870435    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/835f4b2a-30d0-441b-af00-f0b2d166fe7c-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "835f4b2a-30d0-441b-af00-f0b2d166fe7c" (UID: "835f4b2a-30d0-441b-af00-f0b2d166fe7c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:50:06.870568 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:06.870541    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/835f4b2a-30d0-441b-af00-f0b2d166fe7c-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "835f4b2a-30d0-441b-af00-f0b2d166fe7c" (UID: "835f4b2a-30d0-441b-af00-f0b2d166fe7c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:50:06.870940 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:06.870902    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/835f4b2a-30d0-441b-af00-f0b2d166fe7c-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "835f4b2a-30d0-441b-af00-f0b2d166fe7c" (UID: "835f4b2a-30d0-441b-af00-f0b2d166fe7c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:50:06.870940 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:06.870905    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/835f4b2a-30d0-441b-af00-f0b2d166fe7c-config-out" (OuterVolumeSpecName: "config-out") pod "835f4b2a-30d0-441b-af00-f0b2d166fe7c" (UID: "835f4b2a-30d0-441b-af00-f0b2d166fe7c"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:50:06.871200 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:06.871166    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/835f4b2a-30d0-441b-af00-f0b2d166fe7c-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "835f4b2a-30d0-441b-af00-f0b2d166fe7c" (UID: "835f4b2a-30d0-441b-af00-f0b2d166fe7c"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:50:06.874784 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:06.874690    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/835f4b2a-30d0-441b-af00-f0b2d166fe7c-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "835f4b2a-30d0-441b-af00-f0b2d166fe7c" (UID: "835f4b2a-30d0-441b-af00-f0b2d166fe7c"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:50:06.880909 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:06.880889    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/835f4b2a-30d0-441b-af00-f0b2d166fe7c-web-config" (OuterVolumeSpecName: "web-config") pod "835f4b2a-30d0-441b-af00-f0b2d166fe7c" (UID: "835f4b2a-30d0-441b-af00-f0b2d166fe7c"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:50:06.968002 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:06.967929    2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/835f4b2a-30d0-441b-af00-f0b2d166fe7c-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:50:06.968002 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:06.967953    2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/835f4b2a-30d0-441b-af00-f0b2d166fe7c-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:50:06.968002 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:06.967964    2578 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/835f4b2a-30d0-441b-af00-f0b2d166fe7c-config-volume\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:50:06.968002 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:06.967973    2578 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/835f4b2a-30d0-441b-af00-f0b2d166fe7c-tls-assets\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:50:06.968002 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:06.967982    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/835f4b2a-30d0-441b-af00-f0b2d166fe7c-metrics-client-ca\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:50:06.968002 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:06.967991    2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/835f4b2a-30d0-441b-af00-f0b2d166fe7c-secret-alertmanager-main-tls\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:50:06.968002 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:06.968000    2578 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/835f4b2a-30d0-441b-af00-f0b2d166fe7c-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:50:06.968345 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:06.968009    2578 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/835f4b2a-30d0-441b-af00-f0b2d166fe7c-web-config\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:50:06.968345 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:06.968018    2578 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/835f4b2a-30d0-441b-af00-f0b2d166fe7c-cluster-tls-config\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:50:06.968345 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:06.968026    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrbf9\" (UniqueName: \"kubernetes.io/projected/835f4b2a-30d0-441b-af00-f0b2d166fe7c-kube-api-access-zrbf9\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:50:06.968345 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:06.968034    2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/835f4b2a-30d0-441b-af00-f0b2d166fe7c-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:50:06.968345 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:06.968043    2578 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/835f4b2a-30d0-441b-af00-f0b2d166fe7c-config-out\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:50:07.379923 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.379890    2578 generic.go:356] "Generic (PLEG): container finished" podID="835f4b2a-30d0-441b-af00-f0b2d166fe7c" containerID="c613ff08b7c96b399889a7b382659b69a64ed3cce71c955c9eb3221e4e0e8388" exitCode=0
Jul 07 16:50:07.379923 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.379912    2578 generic.go:356] "Generic (PLEG): container finished" podID="835f4b2a-30d0-441b-af00-f0b2d166fe7c" containerID="34abfee077090afb0e19341d0de872441659de05216970611259dceaa6820141" exitCode=0
Jul 07 16:50:07.380119 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.379931    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"835f4b2a-30d0-441b-af00-f0b2d166fe7c","Type":"ContainerDied","Data":"c613ff08b7c96b399889a7b382659b69a64ed3cce71c955c9eb3221e4e0e8388"}
Jul 07 16:50:07.380119 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.379960    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:50:07.380119 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.379996    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"835f4b2a-30d0-441b-af00-f0b2d166fe7c","Type":"ContainerDied","Data":"34abfee077090afb0e19341d0de872441659de05216970611259dceaa6820141"}
Jul 07 16:50:07.380119 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.380023    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"835f4b2a-30d0-441b-af00-f0b2d166fe7c","Type":"ContainerDied","Data":"fc38479a5312dd0bc36073f6c092a59ed0e56dc985525c043e4b7045dbd4f724"}
Jul 07 16:50:07.380119 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.380054    2578 scope.go:117] "RemoveContainer" containerID="06eeedde37279cb18fbaa81647e367fe563bca47a6e394cbc0f6a0e4c34ab420"
Jul 07 16:50:07.387468 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.387361    2578 scope.go:117] "RemoveContainer" containerID="c613ff08b7c96b399889a7b382659b69a64ed3cce71c955c9eb3221e4e0e8388"
Jul 07 16:50:07.393582 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.393569    2578 scope.go:117] "RemoveContainer" containerID="7783db8b43c7d9ba2b52fd561c366ab1640876727cf802b178720b09bf5cee32"
Jul 07 16:50:07.399398 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.399380    2578 scope.go:117] "RemoveContainer" containerID="34abfee077090afb0e19341d0de872441659de05216970611259dceaa6820141"
Jul 07 16:50:07.403604 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.403582    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 07 16:50:07.406916 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.406895    2578 scope.go:117] "RemoveContainer" containerID="4e193be7788020bdc3da9597679ff37023d4e62716efecd6565e0f1163d5c040"
Jul 07 16:50:07.407818 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.407752    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 07 16:50:07.413525 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.413510    2578 scope.go:117] "RemoveContainer" containerID="08b05db6e10b744177a3b6ada620be59f1b0ae73061f00e104f73f9766fd6723"
Jul 07 16:50:07.419719 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.419703    2578 scope.go:117] "RemoveContainer" containerID="04e1653d7c7f308c8ff27f69ce74ff84349918f08fe32d4b6b494ea66a196fa4"
Jul 07 16:50:07.426038 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.426025    2578 scope.go:117] "RemoveContainer" containerID="06eeedde37279cb18fbaa81647e367fe563bca47a6e394cbc0f6a0e4c34ab420"
Jul 07 16:50:07.426277 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:50:07.426260    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06eeedde37279cb18fbaa81647e367fe563bca47a6e394cbc0f6a0e4c34ab420\": container with ID starting with 06eeedde37279cb18fbaa81647e367fe563bca47a6e394cbc0f6a0e4c34ab420 not found: ID does not exist" containerID="06eeedde37279cb18fbaa81647e367fe563bca47a6e394cbc0f6a0e4c34ab420"
Jul 07 16:50:07.426331 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.426286    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06eeedde37279cb18fbaa81647e367fe563bca47a6e394cbc0f6a0e4c34ab420"} err="failed to get container status \"06eeedde37279cb18fbaa81647e367fe563bca47a6e394cbc0f6a0e4c34ab420\": rpc error: code = NotFound desc = could not find container \"06eeedde37279cb18fbaa81647e367fe563bca47a6e394cbc0f6a0e4c34ab420\": container with ID starting with 06eeedde37279cb18fbaa81647e367fe563bca47a6e394cbc0f6a0e4c34ab420 not found: ID does not exist"
Jul 07 16:50:07.426331 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.426303    2578 scope.go:117] "RemoveContainer" containerID="c613ff08b7c96b399889a7b382659b69a64ed3cce71c955c9eb3221e4e0e8388"
Jul 07 16:50:07.426541 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:50:07.426523    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c613ff08b7c96b399889a7b382659b69a64ed3cce71c955c9eb3221e4e0e8388\": container with ID starting with c613ff08b7c96b399889a7b382659b69a64ed3cce71c955c9eb3221e4e0e8388 not found: ID does not exist" containerID="c613ff08b7c96b399889a7b382659b69a64ed3cce71c955c9eb3221e4e0e8388"
Jul 07 16:50:07.426584 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.426547    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c613ff08b7c96b399889a7b382659b69a64ed3cce71c955c9eb3221e4e0e8388"} err="failed to get container status \"c613ff08b7c96b399889a7b382659b69a64ed3cce71c955c9eb3221e4e0e8388\": rpc error: code = NotFound desc = could not find container \"c613ff08b7c96b399889a7b382659b69a64ed3cce71c955c9eb3221e4e0e8388\": container with ID starting with c613ff08b7c96b399889a7b382659b69a64ed3cce71c955c9eb3221e4e0e8388 not found: ID does not exist"
Jul 07 16:50:07.426584 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.426562    2578 scope.go:117] "RemoveContainer" containerID="7783db8b43c7d9ba2b52fd561c366ab1640876727cf802b178720b09bf5cee32"
Jul 07 16:50:07.426788 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:50:07.426772    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7783db8b43c7d9ba2b52fd561c366ab1640876727cf802b178720b09bf5cee32\": container with ID starting with 7783db8b43c7d9ba2b52fd561c366ab1640876727cf802b178720b09bf5cee32 not found: ID does not exist" containerID="7783db8b43c7d9ba2b52fd561c366ab1640876727cf802b178720b09bf5cee32"
Jul 07 16:50:07.426835 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.426795    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7783db8b43c7d9ba2b52fd561c366ab1640876727cf802b178720b09bf5cee32"} err="failed to get container status \"7783db8b43c7d9ba2b52fd561c366ab1640876727cf802b178720b09bf5cee32\": rpc error: code = NotFound desc = could not find container \"7783db8b43c7d9ba2b52fd561c366ab1640876727cf802b178720b09bf5cee32\": container with ID starting with 7783db8b43c7d9ba2b52fd561c366ab1640876727cf802b178720b09bf5cee32 not found: ID does not exist"
Jul 07 16:50:07.426835 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.426811    2578 scope.go:117] "RemoveContainer" containerID="34abfee077090afb0e19341d0de872441659de05216970611259dceaa6820141"
Jul 07 16:50:07.427043 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:50:07.427027    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34abfee077090afb0e19341d0de872441659de05216970611259dceaa6820141\": container with ID starting with 34abfee077090afb0e19341d0de872441659de05216970611259dceaa6820141 not found: ID does not exist" containerID="34abfee077090afb0e19341d0de872441659de05216970611259dceaa6820141"
Jul 07 16:50:07.427092 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.427047    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34abfee077090afb0e19341d0de872441659de05216970611259dceaa6820141"} err="failed to get container status \"34abfee077090afb0e19341d0de872441659de05216970611259dceaa6820141\": rpc error: code = NotFound desc = could not find container \"34abfee077090afb0e19341d0de872441659de05216970611259dceaa6820141\": container with ID starting with 34abfee077090afb0e19341d0de872441659de05216970611259dceaa6820141 not found: ID does not exist"
Jul 07 16:50:07.427092 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.427061    2578 scope.go:117] "RemoveContainer" containerID="4e193be7788020bdc3da9597679ff37023d4e62716efecd6565e0f1163d5c040"
Jul 07 16:50:07.427244 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:50:07.427229    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e193be7788020bdc3da9597679ff37023d4e62716efecd6565e0f1163d5c040\": container with ID starting with 4e193be7788020bdc3da9597679ff37023d4e62716efecd6565e0f1163d5c040 not found: ID does not exist" containerID="4e193be7788020bdc3da9597679ff37023d4e62716efecd6565e0f1163d5c040"
Jul 07 16:50:07.427283 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.427246    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e193be7788020bdc3da9597679ff37023d4e62716efecd6565e0f1163d5c040"} err="failed to get container status \"4e193be7788020bdc3da9597679ff37023d4e62716efecd6565e0f1163d5c040\": rpc error: code = NotFound desc = could not find container \"4e193be7788020bdc3da9597679ff37023d4e62716efecd6565e0f1163d5c040\": container with ID starting with 4e193be7788020bdc3da9597679ff37023d4e62716efecd6565e0f1163d5c040 not found: ID does not exist"
Jul 07 16:50:07.427283 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.427258    2578 scope.go:117] "RemoveContainer" containerID="08b05db6e10b744177a3b6ada620be59f1b0ae73061f00e104f73f9766fd6723"
Jul 07 16:50:07.427436 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:50:07.427421    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08b05db6e10b744177a3b6ada620be59f1b0ae73061f00e104f73f9766fd6723\": container with ID starting with 08b05db6e10b744177a3b6ada620be59f1b0ae73061f00e104f73f9766fd6723 not found: ID does not exist" containerID="08b05db6e10b744177a3b6ada620be59f1b0ae73061f00e104f73f9766fd6723"
Jul 07 16:50:07.427470 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.427439    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08b05db6e10b744177a3b6ada620be59f1b0ae73061f00e104f73f9766fd6723"} err="failed to get container status \"08b05db6e10b744177a3b6ada620be59f1b0ae73061f00e104f73f9766fd6723\": rpc error: code = NotFound desc = could not find container \"08b05db6e10b744177a3b6ada620be59f1b0ae73061f00e104f73f9766fd6723\": container with ID starting with 08b05db6e10b744177a3b6ada620be59f1b0ae73061f00e104f73f9766fd6723 not found: ID does not exist"
Jul 07 16:50:07.427470 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.427453    2578 scope.go:117] "RemoveContainer" containerID="04e1653d7c7f308c8ff27f69ce74ff84349918f08fe32d4b6b494ea66a196fa4"
Jul 07 16:50:07.427635 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:50:07.427620    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04e1653d7c7f308c8ff27f69ce74ff84349918f08fe32d4b6b494ea66a196fa4\": container with ID starting with 04e1653d7c7f308c8ff27f69ce74ff84349918f08fe32d4b6b494ea66a196fa4 not found: ID does not exist" containerID="04e1653d7c7f308c8ff27f69ce74ff84349918f08fe32d4b6b494ea66a196fa4"
Jul 07 16:50:07.427673 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.427640    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04e1653d7c7f308c8ff27f69ce74ff84349918f08fe32d4b6b494ea66a196fa4"} err="failed to get container status \"04e1653d7c7f308c8ff27f69ce74ff84349918f08fe32d4b6b494ea66a196fa4\": rpc error: code = NotFound desc = could not find container \"04e1653d7c7f308c8ff27f69ce74ff84349918f08fe32d4b6b494ea66a196fa4\": container with ID starting with 04e1653d7c7f308c8ff27f69ce74ff84349918f08fe32d4b6b494ea66a196fa4 not found: ID does not exist"
Jul 07 16:50:07.427673 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.427653    2578 scope.go:117] "RemoveContainer" containerID="06eeedde37279cb18fbaa81647e367fe563bca47a6e394cbc0f6a0e4c34ab420"
Jul 07 16:50:07.427926 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.427909    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06eeedde37279cb18fbaa81647e367fe563bca47a6e394cbc0f6a0e4c34ab420"} err="failed to get container status \"06eeedde37279cb18fbaa81647e367fe563bca47a6e394cbc0f6a0e4c34ab420\": rpc error: code = NotFound desc = could not find container \"06eeedde37279cb18fbaa81647e367fe563bca47a6e394cbc0f6a0e4c34ab420\": container with ID starting with 06eeedde37279cb18fbaa81647e367fe563bca47a6e394cbc0f6a0e4c34ab420 not found: ID does not exist"
Jul 07 16:50:07.427977 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.427927    2578 scope.go:117] "RemoveContainer" containerID="c613ff08b7c96b399889a7b382659b69a64ed3cce71c955c9eb3221e4e0e8388"
Jul 07 16:50:07.428109 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.428094    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c613ff08b7c96b399889a7b382659b69a64ed3cce71c955c9eb3221e4e0e8388"} err="failed to get container status \"c613ff08b7c96b399889a7b382659b69a64ed3cce71c955c9eb3221e4e0e8388\": rpc error: code = NotFound desc = could not find container \"c613ff08b7c96b399889a7b382659b69a64ed3cce71c955c9eb3221e4e0e8388\": container with ID starting with c613ff08b7c96b399889a7b382659b69a64ed3cce71c955c9eb3221e4e0e8388 not found: ID does not exist"
Jul 07 16:50:07.428150 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.428109    2578 scope.go:117] "RemoveContainer" containerID="7783db8b43c7d9ba2b52fd561c366ab1640876727cf802b178720b09bf5cee32"
Jul 07 16:50:07.428294 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.428279    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7783db8b43c7d9ba2b52fd561c366ab1640876727cf802b178720b09bf5cee32"} err="failed to get container status \"7783db8b43c7d9ba2b52fd561c366ab1640876727cf802b178720b09bf5cee32\": rpc error: code = NotFound desc = could not find container \"7783db8b43c7d9ba2b52fd561c366ab1640876727cf802b178720b09bf5cee32\": container with ID starting with 7783db8b43c7d9ba2b52fd561c366ab1640876727cf802b178720b09bf5cee32 not found: ID does not exist"
Jul 07 16:50:07.428335 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.428294    2578 scope.go:117] "RemoveContainer" containerID="34abfee077090afb0e19341d0de872441659de05216970611259dceaa6820141"
Jul 07 16:50:07.428458 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.428444    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34abfee077090afb0e19341d0de872441659de05216970611259dceaa6820141"} err="failed to get container status \"34abfee077090afb0e19341d0de872441659de05216970611259dceaa6820141\": rpc error: code = NotFound desc = could not find container \"34abfee077090afb0e19341d0de872441659de05216970611259dceaa6820141\": container with ID starting with 34abfee077090afb0e19341d0de872441659de05216970611259dceaa6820141 not found: ID does not exist"
Jul 07 16:50:07.428500 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.428458    2578 scope.go:117] "RemoveContainer" containerID="4e193be7788020bdc3da9597679ff37023d4e62716efecd6565e0f1163d5c040"
Jul 07 16:50:07.428658 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.428642    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e193be7788020bdc3da9597679ff37023d4e62716efecd6565e0f1163d5c040"} err="failed to get container status \"4e193be7788020bdc3da9597679ff37023d4e62716efecd6565e0f1163d5c040\": rpc error: code = NotFound desc = could not find container \"4e193be7788020bdc3da9597679ff37023d4e62716efecd6565e0f1163d5c040\": container with ID starting with 4e193be7788020bdc3da9597679ff37023d4e62716efecd6565e0f1163d5c040 not found: ID does not exist"
Jul 07 16:50:07.428658 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.428657    2578 scope.go:117] "RemoveContainer" containerID="08b05db6e10b744177a3b6ada620be59f1b0ae73061f00e104f73f9766fd6723"
Jul 07 16:50:07.428833 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.428817    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08b05db6e10b744177a3b6ada620be59f1b0ae73061f00e104f73f9766fd6723"} err="failed to get container status \"08b05db6e10b744177a3b6ada620be59f1b0ae73061f00e104f73f9766fd6723\": rpc error: code = NotFound desc = could not find container \"08b05db6e10b744177a3b6ada620be59f1b0ae73061f00e104f73f9766fd6723\": container with ID starting with 08b05db6e10b744177a3b6ada620be59f1b0ae73061f00e104f73f9766fd6723 not found: ID does not exist"
Jul 07 16:50:07.428891 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.428833    2578 scope.go:117] "RemoveContainer" containerID="04e1653d7c7f308c8ff27f69ce74ff84349918f08fe32d4b6b494ea66a196fa4"
Jul 07 16:50:07.429044 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.429028    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04e1653d7c7f308c8ff27f69ce74ff84349918f08fe32d4b6b494ea66a196fa4"} err="failed to get container status \"04e1653d7c7f308c8ff27f69ce74ff84349918f08fe32d4b6b494ea66a196fa4\": rpc error: code = NotFound desc = could not find container \"04e1653d7c7f308c8ff27f69ce74ff84349918f08fe32d4b6b494ea66a196fa4\": container with ID starting with 04e1653d7c7f308c8ff27f69ce74ff84349918f08fe32d4b6b494ea66a196fa4 not found: ID does not exist"
Jul 07 16:50:07.432882 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.432863    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 07 16:50:07.433097 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.433085    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="835f4b2a-30d0-441b-af00-f0b2d166fe7c" containerName="kube-rbac-proxy"
Jul 07 16:50:07.433141 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.433098    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="835f4b2a-30d0-441b-af00-f0b2d166fe7c" containerName="kube-rbac-proxy"
Jul 07 16:50:07.433141 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.433106    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5790cd94-92b3-46f3-8441-caf5efcb2779" containerName="console"
Jul 07 16:50:07.433141 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.433111    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5790cd94-92b3-46f3-8441-caf5efcb2779" containerName="console"
Jul 07 16:50:07.433141 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.433120    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9480c793-ca95-433d-8d2a-9f8fb71e04a1" containerName="console"
Jul 07 16:50:07.433141 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.433127    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9480c793-ca95-433d-8d2a-9f8fb71e04a1" containerName="console"
Jul 07 16:50:07.433141 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.433137    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="835f4b2a-30d0-441b-af00-f0b2d166fe7c" containerName="prom-label-proxy"
Jul 07 16:50:07.433141 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.433141    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="835f4b2a-30d0-441b-af00-f0b2d166fe7c" containerName="prom-label-proxy"
Jul 07 16:50:07.433337 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.433149    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="835f4b2a-30d0-441b-af00-f0b2d166fe7c" containerName="kube-rbac-proxy-web"
Jul 07 16:50:07.433337 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.433154    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="835f4b2a-30d0-441b-af00-f0b2d166fe7c" containerName="kube-rbac-proxy-web"
Jul 07 16:50:07.433337 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.433165    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="835f4b2a-30d0-441b-af00-f0b2d166fe7c" containerName="init-config-reloader"
Jul 07 16:50:07.433337 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.433170    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="835f4b2a-30d0-441b-af00-f0b2d166fe7c" containerName="init-config-reloader"
Jul 07 16:50:07.433337 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.433179    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="835f4b2a-30d0-441b-af00-f0b2d166fe7c" containerName="alertmanager"
Jul 07 16:50:07.433337 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.433184    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="835f4b2a-30d0-441b-af00-f0b2d166fe7c" containerName="alertmanager"
Jul 07 16:50:07.433337 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.433190    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="835f4b2a-30d0-441b-af00-f0b2d166fe7c" containerName="config-reloader"
Jul 07 16:50:07.433337 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.433195    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="835f4b2a-30d0-441b-af00-f0b2d166fe7c" containerName="config-reloader"
Jul 07 16:50:07.433337 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.433201    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="835f4b2a-30d0-441b-af00-f0b2d166fe7c" containerName="kube-rbac-proxy-metric"
Jul 07 16:50:07.433337 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.433206    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="835f4b2a-30d0-441b-af00-f0b2d166fe7c" containerName="kube-rbac-proxy-metric"
Jul 07 16:50:07.433337 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.433242    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9480c793-ca95-433d-8d2a-9f8fb71e04a1" containerName="console"
Jul 07 16:50:07.433337 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.433248    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="835f4b2a-30d0-441b-af00-f0b2d166fe7c" containerName="alertmanager"
Jul 07 16:50:07.433337 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.433255    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="835f4b2a-30d0-441b-af00-f0b2d166fe7c" containerName="config-reloader"
Jul 07 16:50:07.433337 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.433259    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="835f4b2a-30d0-441b-af00-f0b2d166fe7c" containerName="kube-rbac-proxy-web"
Jul 07 16:50:07.433337 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.433265    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="835f4b2a-30d0-441b-af00-f0b2d166fe7c" containerName="kube-rbac-proxy-metric"
Jul 07 16:50:07.433337 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.433271    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5790cd94-92b3-46f3-8441-caf5efcb2779" containerName="console"
Jul 07 16:50:07.433337 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.433277    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="835f4b2a-30d0-441b-af00-f0b2d166fe7c" containerName="kube-rbac-proxy"
Jul 07 16:50:07.433337 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.433282    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="835f4b2a-30d0-441b-af00-f0b2d166fe7c" containerName="prom-label-proxy"
Jul 07 16:50:07.437080 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.437041    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:50:07.442095 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.439900    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 07 16:50:07.442095 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.440352    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 07 16:50:07.442633 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.442382    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 07 16:50:07.444550 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.443433    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 07 16:50:07.444550 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.443676    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 07 16:50:07.444550 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.444023    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 07 16:50:07.444550 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.444054    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 07 16:50:07.444550 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.444305    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-m9wrw\""
Jul 07 16:50:07.444834 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.444616    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 07 16:50:07.448453 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.448435    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 07 16:50:07.453248 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.453228    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 07 16:50:07.471236 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.471212    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d5653d04-ba26-49b7-9441-35df51e53991-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d5653d04-ba26-49b7-9441-35df51e53991\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:50:07.471330 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.471247    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d5653d04-ba26-49b7-9441-35df51e53991-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d5653d04-ba26-49b7-9441-35df51e53991\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:50:07.471330 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.471268    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d5653d04-ba26-49b7-9441-35df51e53991-web-config\") pod \"alertmanager-main-0\" (UID: \"d5653d04-ba26-49b7-9441-35df51e53991\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:50:07.471330 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.471323    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d5653d04-ba26-49b7-9441-35df51e53991-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d5653d04-ba26-49b7-9441-35df51e53991\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:50:07.471428 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.471405    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d5653d04-ba26-49b7-9441-35df51e53991-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d5653d04-ba26-49b7-9441-35df51e53991\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:50:07.471464 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.471430    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5653d04-ba26-49b7-9441-35df51e53991-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d5653d04-ba26-49b7-9441-35df51e53991\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:50:07.471464 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.471448    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d5653d04-ba26-49b7-9441-35df51e53991-config-out\") pod \"alertmanager-main-0\" (UID: \"d5653d04-ba26-49b7-9441-35df51e53991\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:50:07.471522 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.471463    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d5653d04-ba26-49b7-9441-35df51e53991-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d5653d04-ba26-49b7-9441-35df51e53991\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:50:07.471522 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.471477    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d5653d04-ba26-49b7-9441-35df51e53991-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d5653d04-ba26-49b7-9441-35df51e53991\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:50:07.471522 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.471519    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5653d04-ba26-49b7-9441-35df51e53991-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d5653d04-ba26-49b7-9441-35df51e53991\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:50:07.471621 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.471547    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thdmb\" (UniqueName: \"kubernetes.io/projected/d5653d04-ba26-49b7-9441-35df51e53991-kube-api-access-thdmb\") pod \"alertmanager-main-0\" (UID: \"d5653d04-ba26-49b7-9441-35df51e53991\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:50:07.471621 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.471601    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d5653d04-ba26-49b7-9441-35df51e53991-config-volume\") pod \"alertmanager-main-0\" (UID: \"d5653d04-ba26-49b7-9441-35df51e53991\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:50:07.471694 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.471661    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d5653d04-ba26-49b7-9441-35df51e53991-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d5653d04-ba26-49b7-9441-35df51e53991\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:50:07.572045 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.572009    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d5653d04-ba26-49b7-9441-35df51e53991-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d5653d04-ba26-49b7-9441-35df51e53991\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:50:07.572045 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.572051    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d5653d04-ba26-49b7-9441-35df51e53991-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d5653d04-ba26-49b7-9441-35df51e53991\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:50:07.572282 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.572074    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d5653d04-ba26-49b7-9441-35df51e53991-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d5653d04-ba26-49b7-9441-35df51e53991\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:50:07.572282 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.572105    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d5653d04-ba26-49b7-9441-35df51e53991-web-config\") pod \"alertmanager-main-0\" (UID: \"d5653d04-ba26-49b7-9441-35df51e53991\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:50:07.572282 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.572124    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d5653d04-ba26-49b7-9441-35df51e53991-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d5653d04-ba26-49b7-9441-35df51e53991\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:50:07.572282 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.572152    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d5653d04-ba26-49b7-9441-35df51e53991-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d5653d04-ba26-49b7-9441-35df51e53991\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:50:07.572282 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.572188    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5653d04-ba26-49b7-9441-35df51e53991-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d5653d04-ba26-49b7-9441-35df51e53991\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:50:07.572518 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.572424    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d5653d04-ba26-49b7-9441-35df51e53991-config-out\") pod \"alertmanager-main-0\" (UID: \"d5653d04-ba26-49b7-9441-35df51e53991\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:50:07.572518 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.572494    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d5653d04-ba26-49b7-9441-35df51e53991-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d5653d04-ba26-49b7-9441-35df51e53991\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:50:07.572624 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.572523    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d5653d04-ba26-49b7-9441-35df51e53991-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d5653d04-ba26-49b7-9441-35df51e53991\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:50:07.572624 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.572582    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5653d04-ba26-49b7-9441-35df51e53991-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d5653d04-ba26-49b7-9441-35df51e53991\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:50:07.572744 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.572612    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thdmb\" (UniqueName: \"kubernetes.io/projected/d5653d04-ba26-49b7-9441-35df51e53991-kube-api-access-thdmb\") pod \"alertmanager-main-0\" (UID: \"d5653d04-ba26-49b7-9441-35df51e53991\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:50:07.572744 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.572658    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d5653d04-ba26-49b7-9441-35df51e53991-config-volume\") pod \"alertmanager-main-0\" (UID: \"d5653d04-ba26-49b7-9441-35df51e53991\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:50:07.572744 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.572687    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d5653d04-ba26-49b7-9441-35df51e53991-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d5653d04-ba26-49b7-9441-35df51e53991\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:50:07.573848 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.573820    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5653d04-ba26-49b7-9441-35df51e53991-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d5653d04-ba26-49b7-9441-35df51e53991\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:50:07.574153 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.574126    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5653d04-ba26-49b7-9441-35df51e53991-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d5653d04-ba26-49b7-9441-35df51e53991\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:50:07.575077 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.575032    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d5653d04-ba26-49b7-9441-35df51e53991-config-out\") pod \"alertmanager-main-0\" (UID: \"d5653d04-ba26-49b7-9441-35df51e53991\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:50:07.575237 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.575209    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d5653d04-ba26-49b7-9441-35df51e53991-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d5653d04-ba26-49b7-9441-35df51e53991\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:50:07.575442 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.575420    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d5653d04-ba26-49b7-9441-35df51e53991-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d5653d04-ba26-49b7-9441-35df51e53991\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:50:07.575846 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.575826    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d5653d04-ba26-49b7-9441-35df51e53991-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d5653d04-ba26-49b7-9441-35df51e53991\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:50:07.575932 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.575888    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d5653d04-ba26-49b7-9441-35df51e53991-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d5653d04-ba26-49b7-9441-35df51e53991\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:50:07.575996 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.575977    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d5653d04-ba26-49b7-9441-35df51e53991-web-config\") pod \"alertmanager-main-0\" (UID: \"d5653d04-ba26-49b7-9441-35df51e53991\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:50:07.576306 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.576287    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d5653d04-ba26-49b7-9441-35df51e53991-config-volume\") pod \"alertmanager-main-0\" (UID: \"d5653d04-ba26-49b7-9441-35df51e53991\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:50:07.576337 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.576292    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d5653d04-ba26-49b7-9441-35df51e53991-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d5653d04-ba26-49b7-9441-35df51e53991\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:50:07.576860 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.576844    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d5653d04-ba26-49b7-9441-35df51e53991-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d5653d04-ba26-49b7-9441-35df51e53991\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:50:07.580976 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.580959    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thdmb\" (UniqueName: \"kubernetes.io/projected/d5653d04-ba26-49b7-9441-35df51e53991-kube-api-access-thdmb\") pod \"alertmanager-main-0\" (UID: \"d5653d04-ba26-49b7-9441-35df51e53991\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:50:07.724899 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.724870    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="835f4b2a-30d0-441b-af00-f0b2d166fe7c" path="/var/lib/kubelet/pods/835f4b2a-30d0-441b-af00-f0b2d166fe7c/volumes"
Jul 07 16:50:07.752035 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.752013    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:50:07.879936 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:07.879911    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 07 16:50:07.882304 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:50:07.882273    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5653d04_ba26_49b7_9441_35df51e53991.slice/crio-783ee041d3b8c8aa2e3ad608a0cdf5a1161eec4f70449c67f8560ec01e9b32e7 WatchSource:0}: Error finding container 783ee041d3b8c8aa2e3ad608a0cdf5a1161eec4f70449c67f8560ec01e9b32e7: Status 404 returned error can't find the container with id 783ee041d3b8c8aa2e3ad608a0cdf5a1161eec4f70449c67f8560ec01e9b32e7
Jul 07 16:50:08.385267 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:08.385234    2578 generic.go:356] "Generic (PLEG): container finished" podID="d5653d04-ba26-49b7-9441-35df51e53991" containerID="de4d9adc2cbef52ce8f44d01718aea6217cba1b7c1f5942c06c3561175be596b" exitCode=0
Jul 07 16:50:08.385448 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:08.385327    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5653d04-ba26-49b7-9441-35df51e53991","Type":"ContainerDied","Data":"de4d9adc2cbef52ce8f44d01718aea6217cba1b7c1f5942c06c3561175be596b"}
Jul 07 16:50:08.385448 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:08.385407    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5653d04-ba26-49b7-9441-35df51e53991","Type":"ContainerStarted","Data":"783ee041d3b8c8aa2e3ad608a0cdf5a1161eec4f70449c67f8560ec01e9b32e7"}
Jul 07 16:50:09.393293 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:09.393262    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5653d04-ba26-49b7-9441-35df51e53991","Type":"ContainerStarted","Data":"dc2d6d04329659db01ee7f928e22aba4194e3969115e8f53df0ecb22996e278a"}
Jul 07 16:50:09.393659 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:09.393329    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5653d04-ba26-49b7-9441-35df51e53991","Type":"ContainerStarted","Data":"71578ec6611ac5dfb4bb4439ac90f88414cf071b2ddebbf1a976fd67ff89c251"}
Jul 07 16:50:09.393659 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:09.393358    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5653d04-ba26-49b7-9441-35df51e53991","Type":"ContainerStarted","Data":"2afaf4853c7a82a73afe81a7fa0a50e46fdcf2c2adedacfe0e69f5976d140118"}
Jul 07 16:50:09.393659 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:09.393390    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5653d04-ba26-49b7-9441-35df51e53991","Type":"ContainerStarted","Data":"0df6c1a2e60e02dfa2742412e59f488b2a2571d512bfa5a24170b5f46d18c5a3"}
Jul 07 16:50:09.393659 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:09.393416    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5653d04-ba26-49b7-9441-35df51e53991","Type":"ContainerStarted","Data":"372219b63d083a7b2ef07bd5c8e0a155eedaab54fd6f1107bd3683195df01d5b"}
Jul 07 16:50:09.393659 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:09.393447    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5653d04-ba26-49b7-9441-35df51e53991","Type":"ContainerStarted","Data":"e61a04c9b28b2bcbeaf894985cc12ce65e78b5a39fa16b171a18300ae4cae9be"}
Jul 07 16:50:09.420011 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:09.419961    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.40369105 podStartE2EDuration="2.419938703s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.247653ms" isStatefulPod=false podCreationTimestamp="2026-07-07 16:50:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:50:09.418814575 +0000 UTC m=+146.195128801" watchObservedRunningTime="2026-07-07 16:50:09.419938703 +0000 UTC m=+146.196252911"
Jul 07 16:50:09.881450 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:09.881416    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 16:50:09.881747 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:09.881702    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="242f4d41-bb8d-4488-8477-0de8946569ce" containerName="prometheus" containerID="cri-o://160fa002a7adbf7be0064d29e23e6b4a0d40a877ee39afa9472abc3c1610f244" gracePeriod=600
Jul 07 16:50:09.881962 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:09.881707    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="242f4d41-bb8d-4488-8477-0de8946569ce" containerName="kube-rbac-proxy" containerID="cri-o://b8d17cbc6192e7797c7d8c746f009f7cc4a619f591db65b7b5ac0049d702179c" gracePeriod=600
Jul 07 16:50:09.882039 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:09.881747    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="242f4d41-bb8d-4488-8477-0de8946569ce" containerName="thanos-sidecar" containerID="cri-o://3ae5ceb91f346dbe8f4c9fd4b8e34dbe74d6b4237794432e5bb895641800ff3f" gracePeriod=600
Jul 07 16:50:09.882159 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:09.881766    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="242f4d41-bb8d-4488-8477-0de8946569ce" containerName="config-reloader" containerID="cri-o://0b49853db2492a72058608ec810b494cd313faeface778b08816e20570f88376" gracePeriod=600
Jul 07 16:50:09.882159 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:09.881816    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="242f4d41-bb8d-4488-8477-0de8946569ce" containerName="kube-rbac-proxy-thanos" containerID="cri-o://28c09282528f2dafed0372f50b6d295fd7b28d2e392b318ec4ada3ee02649a03" gracePeriod=600
Jul 07 16:50:09.882273 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:09.881811    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="242f4d41-bb8d-4488-8477-0de8946569ce" containerName="kube-rbac-proxy-web" containerID="cri-o://0da92d3ce34e6c80b754c6ca6bfd37abcb3f3d5a486338eeb982bfd47baf3c7e" gracePeriod=600
Jul 07 16:50:10.125414 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.125258    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:10.196961 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.196884    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/242f4d41-bb8d-4488-8477-0de8946569ce-prometheus-k8s-rulefiles-1\") pod \"242f4d41-bb8d-4488-8477-0de8946569ce\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") "
Jul 07 16:50:10.196961 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.196921    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/242f4d41-bb8d-4488-8477-0de8946569ce-configmap-serving-certs-ca-bundle\") pod \"242f4d41-bb8d-4488-8477-0de8946569ce\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") "
Jul 07 16:50:10.196961 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.196939    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/242f4d41-bb8d-4488-8477-0de8946569ce-tls-assets\") pod \"242f4d41-bb8d-4488-8477-0de8946569ce\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") "
Jul 07 16:50:10.197228 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.196969    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/242f4d41-bb8d-4488-8477-0de8946569ce-prometheus-k8s-rulefiles-0\") pod \"242f4d41-bb8d-4488-8477-0de8946569ce\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") "
Jul 07 16:50:10.197228 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.196993    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/242f4d41-bb8d-4488-8477-0de8946569ce-prometheus-trusted-ca-bundle\") pod \"242f4d41-bb8d-4488-8477-0de8946569ce\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") "
Jul 07 16:50:10.197228 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.197013    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/242f4d41-bb8d-4488-8477-0de8946569ce-secret-metrics-client-certs\") pod \"242f4d41-bb8d-4488-8477-0de8946569ce\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") "
Jul 07 16:50:10.197228 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.197037    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/242f4d41-bb8d-4488-8477-0de8946569ce-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"242f4d41-bb8d-4488-8477-0de8946569ce\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") "
Jul 07 16:50:10.197228 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.197068    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/242f4d41-bb8d-4488-8477-0de8946569ce-web-config\") pod \"242f4d41-bb8d-4488-8477-0de8946569ce\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") "
Jul 07 16:50:10.197228 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.197107    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/242f4d41-bb8d-4488-8477-0de8946569ce-config-out\") pod \"242f4d41-bb8d-4488-8477-0de8946569ce\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") "
Jul 07 16:50:10.197228 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.197169    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/242f4d41-bb8d-4488-8477-0de8946569ce-configmap-kubelet-serving-ca-bundle\") pod \"242f4d41-bb8d-4488-8477-0de8946569ce\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") "
Jul 07 16:50:10.197228 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.197201    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/242f4d41-bb8d-4488-8477-0de8946569ce-config\") pod \"242f4d41-bb8d-4488-8477-0de8946569ce\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") "
Jul 07 16:50:10.197228 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.197202    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/242f4d41-bb8d-4488-8477-0de8946569ce-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "242f4d41-bb8d-4488-8477-0de8946569ce" (UID: "242f4d41-bb8d-4488-8477-0de8946569ce"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:50:10.197228 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.197225    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/242f4d41-bb8d-4488-8477-0de8946569ce-secret-prometheus-k8s-tls\") pod \"242f4d41-bb8d-4488-8477-0de8946569ce\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") "
Jul 07 16:50:10.197694 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.197260    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/242f4d41-bb8d-4488-8477-0de8946569ce-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"242f4d41-bb8d-4488-8477-0de8946569ce\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") "
Jul 07 16:50:10.197694 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.197277    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/242f4d41-bb8d-4488-8477-0de8946569ce-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "242f4d41-bb8d-4488-8477-0de8946569ce" (UID: "242f4d41-bb8d-4488-8477-0de8946569ce"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:50:10.197694 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.197290    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b86vd\" (UniqueName: \"kubernetes.io/projected/242f4d41-bb8d-4488-8477-0de8946569ce-kube-api-access-b86vd\") pod \"242f4d41-bb8d-4488-8477-0de8946569ce\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") "
Jul 07 16:50:10.197694 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.197352    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/242f4d41-bb8d-4488-8477-0de8946569ce-thanos-prometheus-http-client-file\") pod \"242f4d41-bb8d-4488-8477-0de8946569ce\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") "
Jul 07 16:50:10.197694 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.197384    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/242f4d41-bb8d-4488-8477-0de8946569ce-secret-grpc-tls\") pod \"242f4d41-bb8d-4488-8477-0de8946569ce\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") "
Jul 07 16:50:10.197694 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.197429    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/242f4d41-bb8d-4488-8477-0de8946569ce-prometheus-k8s-rulefiles-2\") pod \"242f4d41-bb8d-4488-8477-0de8946569ce\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") "
Jul 07 16:50:10.197694 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.197458    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/242f4d41-bb8d-4488-8477-0de8946569ce-prometheus-k8s-db\") pod \"242f4d41-bb8d-4488-8477-0de8946569ce\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") "
Jul 07 16:50:10.197694 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.197456    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/242f4d41-bb8d-4488-8477-0de8946569ce-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "242f4d41-bb8d-4488-8477-0de8946569ce" (UID: "242f4d41-bb8d-4488-8477-0de8946569ce"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:50:10.197694 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.197505    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/242f4d41-bb8d-4488-8477-0de8946569ce-configmap-metrics-client-ca\") pod \"242f4d41-bb8d-4488-8477-0de8946569ce\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") "
Jul 07 16:50:10.197694 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.197535    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/242f4d41-bb8d-4488-8477-0de8946569ce-secret-kube-rbac-proxy\") pod \"242f4d41-bb8d-4488-8477-0de8946569ce\" (UID: \"242f4d41-bb8d-4488-8477-0de8946569ce\") "
Jul 07 16:50:10.198234 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.197820    2578 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/242f4d41-bb8d-4488-8477-0de8946569ce-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:50:10.198234 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.197840    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/242f4d41-bb8d-4488-8477-0de8946569ce-prometheus-trusted-ca-bundle\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:50:10.198234 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.197855    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/242f4d41-bb8d-4488-8477-0de8946569ce-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:50:10.198812 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.198608    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/242f4d41-bb8d-4488-8477-0de8946569ce-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "242f4d41-bb8d-4488-8477-0de8946569ce" (UID: "242f4d41-bb8d-4488-8477-0de8946569ce"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:50:10.198812 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.198793    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/242f4d41-bb8d-4488-8477-0de8946569ce-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "242f4d41-bb8d-4488-8477-0de8946569ce" (UID: "242f4d41-bb8d-4488-8477-0de8946569ce"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:50:10.199325 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.199287    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/242f4d41-bb8d-4488-8477-0de8946569ce-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "242f4d41-bb8d-4488-8477-0de8946569ce" (UID: "242f4d41-bb8d-4488-8477-0de8946569ce"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:50:10.199425 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.199385    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/242f4d41-bb8d-4488-8477-0de8946569ce-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "242f4d41-bb8d-4488-8477-0de8946569ce" (UID: "242f4d41-bb8d-4488-8477-0de8946569ce"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:50:10.200680 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.200655    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/242f4d41-bb8d-4488-8477-0de8946569ce-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "242f4d41-bb8d-4488-8477-0de8946569ce" (UID: "242f4d41-bb8d-4488-8477-0de8946569ce"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:50:10.201171 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.201132    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/242f4d41-bb8d-4488-8477-0de8946569ce-kube-api-access-b86vd" (OuterVolumeSpecName: "kube-api-access-b86vd") pod "242f4d41-bb8d-4488-8477-0de8946569ce" (UID: "242f4d41-bb8d-4488-8477-0de8946569ce"). InnerVolumeSpecName "kube-api-access-b86vd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:50:10.201282 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.201166    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/242f4d41-bb8d-4488-8477-0de8946569ce-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "242f4d41-bb8d-4488-8477-0de8946569ce" (UID: "242f4d41-bb8d-4488-8477-0de8946569ce"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:50:10.201912 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.201879    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/242f4d41-bb8d-4488-8477-0de8946569ce-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "242f4d41-bb8d-4488-8477-0de8946569ce" (UID: "242f4d41-bb8d-4488-8477-0de8946569ce"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:50:10.202154 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.202036    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/242f4d41-bb8d-4488-8477-0de8946569ce-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "242f4d41-bb8d-4488-8477-0de8946569ce" (UID: "242f4d41-bb8d-4488-8477-0de8946569ce"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:50:10.202244 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.202204    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/242f4d41-bb8d-4488-8477-0de8946569ce-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "242f4d41-bb8d-4488-8477-0de8946569ce" (UID: "242f4d41-bb8d-4488-8477-0de8946569ce"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:50:10.202402 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.202367    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/242f4d41-bb8d-4488-8477-0de8946569ce-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "242f4d41-bb8d-4488-8477-0de8946569ce" (UID: "242f4d41-bb8d-4488-8477-0de8946569ce"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:50:10.202518 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.202401    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/242f4d41-bb8d-4488-8477-0de8946569ce-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "242f4d41-bb8d-4488-8477-0de8946569ce" (UID: "242f4d41-bb8d-4488-8477-0de8946569ce"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:50:10.202907 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.202884    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/242f4d41-bb8d-4488-8477-0de8946569ce-config" (OuterVolumeSpecName: "config") pod "242f4d41-bb8d-4488-8477-0de8946569ce" (UID: "242f4d41-bb8d-4488-8477-0de8946569ce"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:50:10.203040 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.203021    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/242f4d41-bb8d-4488-8477-0de8946569ce-config-out" (OuterVolumeSpecName: "config-out") pod "242f4d41-bb8d-4488-8477-0de8946569ce" (UID: "242f4d41-bb8d-4488-8477-0de8946569ce"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:50:10.203248 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.203229    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/242f4d41-bb8d-4488-8477-0de8946569ce-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "242f4d41-bb8d-4488-8477-0de8946569ce" (UID: "242f4d41-bb8d-4488-8477-0de8946569ce"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:50:10.204495 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.204474    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/242f4d41-bb8d-4488-8477-0de8946569ce-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "242f4d41-bb8d-4488-8477-0de8946569ce" (UID: "242f4d41-bb8d-4488-8477-0de8946569ce"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:50:10.210817 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.210795    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/242f4d41-bb8d-4488-8477-0de8946569ce-web-config" (OuterVolumeSpecName: "web-config") pod "242f4d41-bb8d-4488-8477-0de8946569ce" (UID: "242f4d41-bb8d-4488-8477-0de8946569ce"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:50:10.298743 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.298712    2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/242f4d41-bb8d-4488-8477-0de8946569ce-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:50:10.298865 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.298757    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b86vd\" (UniqueName: \"kubernetes.io/projected/242f4d41-bb8d-4488-8477-0de8946569ce-kube-api-access-b86vd\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:50:10.298865 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.298768    2578 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/242f4d41-bb8d-4488-8477-0de8946569ce-thanos-prometheus-http-client-file\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:50:10.298865 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.298778    2578 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/242f4d41-bb8d-4488-8477-0de8946569ce-secret-grpc-tls\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:50:10.298865 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.298786    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/242f4d41-bb8d-4488-8477-0de8946569ce-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:50:10.298865 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.298794    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/242f4d41-bb8d-4488-8477-0de8946569ce-prometheus-k8s-db\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:50:10.298865 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.298803    2578 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/242f4d41-bb8d-4488-8477-0de8946569ce-configmap-metrics-client-ca\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:50:10.298865 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.298813    2578 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/242f4d41-bb8d-4488-8477-0de8946569ce-secret-kube-rbac-proxy\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:50:10.298865 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.298820    2578 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/242f4d41-bb8d-4488-8477-0de8946569ce-tls-assets\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:50:10.298865 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.298829    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/242f4d41-bb8d-4488-8477-0de8946569ce-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:50:10.298865 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.298836    2578 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/242f4d41-bb8d-4488-8477-0de8946569ce-secret-metrics-client-certs\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:50:10.298865 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.298845    2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/242f4d41-bb8d-4488-8477-0de8946569ce-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:50:10.298865 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.298854    2578 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/242f4d41-bb8d-4488-8477-0de8946569ce-web-config\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:50:10.298865 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.298862    2578 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/242f4d41-bb8d-4488-8477-0de8946569ce-config-out\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:50:10.298865 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.298870    2578 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/242f4d41-bb8d-4488-8477-0de8946569ce-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:50:10.299261 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.298878    2578 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/242f4d41-bb8d-4488-8477-0de8946569ce-config\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:50:10.299261 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.298888    2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/242f4d41-bb8d-4488-8477-0de8946569ce-secret-prometheus-k8s-tls\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:50:10.400713 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.400684    2578 generic.go:356] "Generic (PLEG): container finished" podID="242f4d41-bb8d-4488-8477-0de8946569ce" containerID="28c09282528f2dafed0372f50b6d295fd7b28d2e392b318ec4ada3ee02649a03" exitCode=0
Jul 07 16:50:10.400713 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.400710    2578 generic.go:356] "Generic (PLEG): container finished" podID="242f4d41-bb8d-4488-8477-0de8946569ce" containerID="b8d17cbc6192e7797c7d8c746f009f7cc4a619f591db65b7b5ac0049d702179c" exitCode=0
Jul 07 16:50:10.400713 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.400719    2578 generic.go:356] "Generic (PLEG): container finished" podID="242f4d41-bb8d-4488-8477-0de8946569ce" containerID="0da92d3ce34e6c80b754c6ca6bfd37abcb3f3d5a486338eeb982bfd47baf3c7e" exitCode=0
Jul 07 16:50:10.401162 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.400726    2578 generic.go:356] "Generic (PLEG): container finished" podID="242f4d41-bb8d-4488-8477-0de8946569ce" containerID="3ae5ceb91f346dbe8f4c9fd4b8e34dbe74d6b4237794432e5bb895641800ff3f" exitCode=0
Jul 07 16:50:10.401162 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.400757    2578 generic.go:356] "Generic (PLEG): container finished" podID="242f4d41-bb8d-4488-8477-0de8946569ce" containerID="0b49853db2492a72058608ec810b494cd313faeface778b08816e20570f88376" exitCode=0
Jul 07 16:50:10.401162 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.400765    2578 generic.go:356] "Generic (PLEG): container finished" podID="242f4d41-bb8d-4488-8477-0de8946569ce" containerID="160fa002a7adbf7be0064d29e23e6b4a0d40a877ee39afa9472abc3c1610f244" exitCode=0
Jul 07 16:50:10.401162 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.400772    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:10.401162 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.400767    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"242f4d41-bb8d-4488-8477-0de8946569ce","Type":"ContainerDied","Data":"28c09282528f2dafed0372f50b6d295fd7b28d2e392b318ec4ada3ee02649a03"}
Jul 07 16:50:10.401162 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.400899    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"242f4d41-bb8d-4488-8477-0de8946569ce","Type":"ContainerDied","Data":"b8d17cbc6192e7797c7d8c746f009f7cc4a619f591db65b7b5ac0049d702179c"}
Jul 07 16:50:10.401162 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.400926    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"242f4d41-bb8d-4488-8477-0de8946569ce","Type":"ContainerDied","Data":"0da92d3ce34e6c80b754c6ca6bfd37abcb3f3d5a486338eeb982bfd47baf3c7e"}
Jul 07 16:50:10.401162 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.400942    2578 scope.go:117] "RemoveContainer" containerID="28c09282528f2dafed0372f50b6d295fd7b28d2e392b318ec4ada3ee02649a03"
Jul 07 16:50:10.401162 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.400952    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"242f4d41-bb8d-4488-8477-0de8946569ce","Type":"ContainerDied","Data":"3ae5ceb91f346dbe8f4c9fd4b8e34dbe74d6b4237794432e5bb895641800ff3f"}
Jul 07 16:50:10.401162 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.400976    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"242f4d41-bb8d-4488-8477-0de8946569ce","Type":"ContainerDied","Data":"0b49853db2492a72058608ec810b494cd313faeface778b08816e20570f88376"}
Jul 07 16:50:10.401162 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.401000    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"242f4d41-bb8d-4488-8477-0de8946569ce","Type":"ContainerDied","Data":"160fa002a7adbf7be0064d29e23e6b4a0d40a877ee39afa9472abc3c1610f244"}
Jul 07 16:50:10.401162 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.401024    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"242f4d41-bb8d-4488-8477-0de8946569ce","Type":"ContainerDied","Data":"a05e57a0324445d9f993465b20e1e556684a378750d3c9ade773be37bddc27c8"}
Jul 07 16:50:10.408306 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.408291    2578 scope.go:117] "RemoveContainer" containerID="b8d17cbc6192e7797c7d8c746f009f7cc4a619f591db65b7b5ac0049d702179c"
Jul 07 16:50:10.415119 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.415102    2578 scope.go:117] "RemoveContainer" containerID="0da92d3ce34e6c80b754c6ca6bfd37abcb3f3d5a486338eeb982bfd47baf3c7e"
Jul 07 16:50:10.421569 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.421555    2578 scope.go:117] "RemoveContainer" containerID="3ae5ceb91f346dbe8f4c9fd4b8e34dbe74d6b4237794432e5bb895641800ff3f"
Jul 07 16:50:10.423868 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.423838    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 16:50:10.428151 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.428130    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 16:50:10.429369 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.429347    2578 scope.go:117] "RemoveContainer" containerID="0b49853db2492a72058608ec810b494cd313faeface778b08816e20570f88376"
Jul 07 16:50:10.435604 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.435589    2578 scope.go:117] "RemoveContainer" containerID="160fa002a7adbf7be0064d29e23e6b4a0d40a877ee39afa9472abc3c1610f244"
Jul 07 16:50:10.442095 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.442078    2578 scope.go:117] "RemoveContainer" containerID="e8ae6fc78374b6e07ffb679f868d996001e98e5bab5cc61560ed025b5308f863"
Jul 07 16:50:10.448227 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.448213    2578 scope.go:117] "RemoveContainer" containerID="28c09282528f2dafed0372f50b6d295fd7b28d2e392b318ec4ada3ee02649a03"
Jul 07 16:50:10.448495 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:50:10.448479    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28c09282528f2dafed0372f50b6d295fd7b28d2e392b318ec4ada3ee02649a03\": container with ID starting with 28c09282528f2dafed0372f50b6d295fd7b28d2e392b318ec4ada3ee02649a03 not found: ID does not exist" containerID="28c09282528f2dafed0372f50b6d295fd7b28d2e392b318ec4ada3ee02649a03"
Jul 07 16:50:10.448552 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.448503    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28c09282528f2dafed0372f50b6d295fd7b28d2e392b318ec4ada3ee02649a03"} err="failed to get container status \"28c09282528f2dafed0372f50b6d295fd7b28d2e392b318ec4ada3ee02649a03\": rpc error: code = NotFound desc = could not find container \"28c09282528f2dafed0372f50b6d295fd7b28d2e392b318ec4ada3ee02649a03\": container with ID starting with 28c09282528f2dafed0372f50b6d295fd7b28d2e392b318ec4ada3ee02649a03 not found: ID does not exist"
Jul 07 16:50:10.448552 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.448520    2578 scope.go:117] "RemoveContainer" containerID="b8d17cbc6192e7797c7d8c746f009f7cc4a619f591db65b7b5ac0049d702179c"
Jul 07 16:50:10.448716 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:50:10.448701    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8d17cbc6192e7797c7d8c746f009f7cc4a619f591db65b7b5ac0049d702179c\": container with ID starting with b8d17cbc6192e7797c7d8c746f009f7cc4a619f591db65b7b5ac0049d702179c not found: ID does not exist" containerID="b8d17cbc6192e7797c7d8c746f009f7cc4a619f591db65b7b5ac0049d702179c"
Jul 07 16:50:10.448787 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.448719    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8d17cbc6192e7797c7d8c746f009f7cc4a619f591db65b7b5ac0049d702179c"} err="failed to get container status \"b8d17cbc6192e7797c7d8c746f009f7cc4a619f591db65b7b5ac0049d702179c\": rpc error: code = NotFound desc = could not find container \"b8d17cbc6192e7797c7d8c746f009f7cc4a619f591db65b7b5ac0049d702179c\": container with ID starting with b8d17cbc6192e7797c7d8c746f009f7cc4a619f591db65b7b5ac0049d702179c not found: ID does not exist"
Jul 07 16:50:10.448787 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.448744    2578 scope.go:117] "RemoveContainer" containerID="0da92d3ce34e6c80b754c6ca6bfd37abcb3f3d5a486338eeb982bfd47baf3c7e"
Jul 07 16:50:10.448977 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:50:10.448959    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0da92d3ce34e6c80b754c6ca6bfd37abcb3f3d5a486338eeb982bfd47baf3c7e\": container with ID starting with 0da92d3ce34e6c80b754c6ca6bfd37abcb3f3d5a486338eeb982bfd47baf3c7e not found: ID does not exist" containerID="0da92d3ce34e6c80b754c6ca6bfd37abcb3f3d5a486338eeb982bfd47baf3c7e"
Jul 07 16:50:10.449022 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.448981    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0da92d3ce34e6c80b754c6ca6bfd37abcb3f3d5a486338eeb982bfd47baf3c7e"} err="failed to get container status \"0da92d3ce34e6c80b754c6ca6bfd37abcb3f3d5a486338eeb982bfd47baf3c7e\": rpc error: code = NotFound desc = could not find container \"0da92d3ce34e6c80b754c6ca6bfd37abcb3f3d5a486338eeb982bfd47baf3c7e\": container with ID starting with 0da92d3ce34e6c80b754c6ca6bfd37abcb3f3d5a486338eeb982bfd47baf3c7e not found: ID does not exist"
Jul 07 16:50:10.449022 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.448993    2578 scope.go:117] "RemoveContainer" containerID="3ae5ceb91f346dbe8f4c9fd4b8e34dbe74d6b4237794432e5bb895641800ff3f"
Jul 07 16:50:10.449207 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:50:10.449190    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ae5ceb91f346dbe8f4c9fd4b8e34dbe74d6b4237794432e5bb895641800ff3f\": container with ID starting with 3ae5ceb91f346dbe8f4c9fd4b8e34dbe74d6b4237794432e5bb895641800ff3f not found: ID does not exist" containerID="3ae5ceb91f346dbe8f4c9fd4b8e34dbe74d6b4237794432e5bb895641800ff3f"
Jul 07 16:50:10.449265 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.449215    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ae5ceb91f346dbe8f4c9fd4b8e34dbe74d6b4237794432e5bb895641800ff3f"} err="failed to get container status \"3ae5ceb91f346dbe8f4c9fd4b8e34dbe74d6b4237794432e5bb895641800ff3f\": rpc error: code = NotFound desc = could not find container \"3ae5ceb91f346dbe8f4c9fd4b8e34dbe74d6b4237794432e5bb895641800ff3f\": container with ID starting with 3ae5ceb91f346dbe8f4c9fd4b8e34dbe74d6b4237794432e5bb895641800ff3f not found: ID does not exist"
Jul 07 16:50:10.449265 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.449237    2578 scope.go:117] "RemoveContainer" containerID="0b49853db2492a72058608ec810b494cd313faeface778b08816e20570f88376"
Jul 07 16:50:10.449446 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:50:10.449428    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b49853db2492a72058608ec810b494cd313faeface778b08816e20570f88376\": container with ID starting with 0b49853db2492a72058608ec810b494cd313faeface778b08816e20570f88376 not found: ID does not exist" containerID="0b49853db2492a72058608ec810b494cd313faeface778b08816e20570f88376"
Jul 07 16:50:10.449479 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.449449    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b49853db2492a72058608ec810b494cd313faeface778b08816e20570f88376"} err="failed to get container status \"0b49853db2492a72058608ec810b494cd313faeface778b08816e20570f88376\": rpc error: code = NotFound desc = could not find container \"0b49853db2492a72058608ec810b494cd313faeface778b08816e20570f88376\": container with ID starting with 0b49853db2492a72058608ec810b494cd313faeface778b08816e20570f88376 not found: ID does not exist"
Jul 07 16:50:10.449479 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.449461    2578 scope.go:117] "RemoveContainer" containerID="160fa002a7adbf7be0064d29e23e6b4a0d40a877ee39afa9472abc3c1610f244"
Jul 07 16:50:10.449651 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:50:10.449635    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"160fa002a7adbf7be0064d29e23e6b4a0d40a877ee39afa9472abc3c1610f244\": container with ID starting with 160fa002a7adbf7be0064d29e23e6b4a0d40a877ee39afa9472abc3c1610f244 not found: ID does not exist" containerID="160fa002a7adbf7be0064d29e23e6b4a0d40a877ee39afa9472abc3c1610f244"
Jul 07 16:50:10.449691 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.449656    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"160fa002a7adbf7be0064d29e23e6b4a0d40a877ee39afa9472abc3c1610f244"} err="failed to get container status \"160fa002a7adbf7be0064d29e23e6b4a0d40a877ee39afa9472abc3c1610f244\": rpc error: code = NotFound desc = could not find container \"160fa002a7adbf7be0064d29e23e6b4a0d40a877ee39afa9472abc3c1610f244\": container with ID starting with 160fa002a7adbf7be0064d29e23e6b4a0d40a877ee39afa9472abc3c1610f244 not found: ID does not exist"
Jul 07 16:50:10.449691 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.449677    2578 scope.go:117] "RemoveContainer" containerID="e8ae6fc78374b6e07ffb679f868d996001e98e5bab5cc61560ed025b5308f863"
Jul 07 16:50:10.449921 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:50:10.449905    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8ae6fc78374b6e07ffb679f868d996001e98e5bab5cc61560ed025b5308f863\": container with ID starting with e8ae6fc78374b6e07ffb679f868d996001e98e5bab5cc61560ed025b5308f863 not found: ID does not exist" containerID="e8ae6fc78374b6e07ffb679f868d996001e98e5bab5cc61560ed025b5308f863"
Jul 07 16:50:10.449977 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.449930    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8ae6fc78374b6e07ffb679f868d996001e98e5bab5cc61560ed025b5308f863"} err="failed to get container status \"e8ae6fc78374b6e07ffb679f868d996001e98e5bab5cc61560ed025b5308f863\": rpc error: code = NotFound desc = could not find container \"e8ae6fc78374b6e07ffb679f868d996001e98e5bab5cc61560ed025b5308f863\": container with ID starting with e8ae6fc78374b6e07ffb679f868d996001e98e5bab5cc61560ed025b5308f863 not found: ID does not exist"
Jul 07 16:50:10.449977 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.449949    2578 scope.go:117] "RemoveContainer" containerID="28c09282528f2dafed0372f50b6d295fd7b28d2e392b318ec4ada3ee02649a03"
Jul 07 16:50:10.450155 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.450140    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28c09282528f2dafed0372f50b6d295fd7b28d2e392b318ec4ada3ee02649a03"} err="failed to get container status \"28c09282528f2dafed0372f50b6d295fd7b28d2e392b318ec4ada3ee02649a03\": rpc error: code = NotFound desc = could not find container \"28c09282528f2dafed0372f50b6d295fd7b28d2e392b318ec4ada3ee02649a03\": container with ID starting with 28c09282528f2dafed0372f50b6d295fd7b28d2e392b318ec4ada3ee02649a03 not found: ID does not exist"
Jul 07 16:50:10.450236 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.450155    2578 scope.go:117] "RemoveContainer" containerID="b8d17cbc6192e7797c7d8c746f009f7cc4a619f591db65b7b5ac0049d702179c"
Jul 07 16:50:10.450342 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.450326    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8d17cbc6192e7797c7d8c746f009f7cc4a619f591db65b7b5ac0049d702179c"} err="failed to get container status \"b8d17cbc6192e7797c7d8c746f009f7cc4a619f591db65b7b5ac0049d702179c\": rpc error: code = NotFound desc = could not find container \"b8d17cbc6192e7797c7d8c746f009f7cc4a619f591db65b7b5ac0049d702179c\": container with ID starting with b8d17cbc6192e7797c7d8c746f009f7cc4a619f591db65b7b5ac0049d702179c not found: ID does not exist"
Jul 07 16:50:10.450380 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.450342    2578 scope.go:117] "RemoveContainer" containerID="0da92d3ce34e6c80b754c6ca6bfd37abcb3f3d5a486338eeb982bfd47baf3c7e"
Jul 07 16:50:10.450537 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.450519    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0da92d3ce34e6c80b754c6ca6bfd37abcb3f3d5a486338eeb982bfd47baf3c7e"} err="failed to get container status \"0da92d3ce34e6c80b754c6ca6bfd37abcb3f3d5a486338eeb982bfd47baf3c7e\": rpc error: code = NotFound desc = could not find container \"0da92d3ce34e6c80b754c6ca6bfd37abcb3f3d5a486338eeb982bfd47baf3c7e\": container with ID starting with 0da92d3ce34e6c80b754c6ca6bfd37abcb3f3d5a486338eeb982bfd47baf3c7e not found: ID does not exist"
Jul 07 16:50:10.450577 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.450537    2578 scope.go:117] "RemoveContainer" containerID="3ae5ceb91f346dbe8f4c9fd4b8e34dbe74d6b4237794432e5bb895641800ff3f"
Jul 07 16:50:10.450707 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.450692    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ae5ceb91f346dbe8f4c9fd4b8e34dbe74d6b4237794432e5bb895641800ff3f"} err="failed to get container status \"3ae5ceb91f346dbe8f4c9fd4b8e34dbe74d6b4237794432e5bb895641800ff3f\": rpc error: code = NotFound desc = could not find container \"3ae5ceb91f346dbe8f4c9fd4b8e34dbe74d6b4237794432e5bb895641800ff3f\": container with ID starting with 3ae5ceb91f346dbe8f4c9fd4b8e34dbe74d6b4237794432e5bb895641800ff3f not found: ID does not exist"
Jul 07 16:50:10.450774 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.450707    2578 scope.go:117] "RemoveContainer" containerID="0b49853db2492a72058608ec810b494cd313faeface778b08816e20570f88376"
Jul 07 16:50:10.450944 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.450925    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b49853db2492a72058608ec810b494cd313faeface778b08816e20570f88376"} err="failed to get container status \"0b49853db2492a72058608ec810b494cd313faeface778b08816e20570f88376\": rpc error: code = NotFound desc = could not find container \"0b49853db2492a72058608ec810b494cd313faeface778b08816e20570f88376\": container with ID starting with 0b49853db2492a72058608ec810b494cd313faeface778b08816e20570f88376 not found: ID does not exist"
Jul 07 16:50:10.450986 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.450945    2578 scope.go:117] "RemoveContainer" containerID="160fa002a7adbf7be0064d29e23e6b4a0d40a877ee39afa9472abc3c1610f244"
Jul 07 16:50:10.451182 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.451158    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"160fa002a7adbf7be0064d29e23e6b4a0d40a877ee39afa9472abc3c1610f244"} err="failed to get container status \"160fa002a7adbf7be0064d29e23e6b4a0d40a877ee39afa9472abc3c1610f244\": rpc error: code = NotFound desc = could not find container \"160fa002a7adbf7be0064d29e23e6b4a0d40a877ee39afa9472abc3c1610f244\": container with ID starting with 160fa002a7adbf7be0064d29e23e6b4a0d40a877ee39afa9472abc3c1610f244 not found: ID does not exist"
Jul 07 16:50:10.451378 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.451182    2578 scope.go:117] "RemoveContainer" containerID="e8ae6fc78374b6e07ffb679f868d996001e98e5bab5cc61560ed025b5308f863"
Jul 07 16:50:10.451504 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.451466    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8ae6fc78374b6e07ffb679f868d996001e98e5bab5cc61560ed025b5308f863"} err="failed to get container status \"e8ae6fc78374b6e07ffb679f868d996001e98e5bab5cc61560ed025b5308f863\": rpc error: code = NotFound desc = could not find container \"e8ae6fc78374b6e07ffb679f868d996001e98e5bab5cc61560ed025b5308f863\": container with ID starting with e8ae6fc78374b6e07ffb679f868d996001e98e5bab5cc61560ed025b5308f863 not found: ID does not exist"
Jul 07 16:50:10.451504 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.451491    2578 scope.go:117] "RemoveContainer" containerID="28c09282528f2dafed0372f50b6d295fd7b28d2e392b318ec4ada3ee02649a03"
Jul 07 16:50:10.451926 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.451853    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28c09282528f2dafed0372f50b6d295fd7b28d2e392b318ec4ada3ee02649a03"} err="failed to get container status \"28c09282528f2dafed0372f50b6d295fd7b28d2e392b318ec4ada3ee02649a03\": rpc error: code = NotFound desc = could not find container \"28c09282528f2dafed0372f50b6d295fd7b28d2e392b318ec4ada3ee02649a03\": container with ID starting with 28c09282528f2dafed0372f50b6d295fd7b28d2e392b318ec4ada3ee02649a03 not found: ID does not exist"
Jul 07 16:50:10.451926 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.451882    2578 scope.go:117] "RemoveContainer" containerID="b8d17cbc6192e7797c7d8c746f009f7cc4a619f591db65b7b5ac0049d702179c"
Jul 07 16:50:10.452199 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.452180    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8d17cbc6192e7797c7d8c746f009f7cc4a619f591db65b7b5ac0049d702179c"} err="failed to get container status \"b8d17cbc6192e7797c7d8c746f009f7cc4a619f591db65b7b5ac0049d702179c\": rpc error: code = NotFound desc = could not find container \"b8d17cbc6192e7797c7d8c746f009f7cc4a619f591db65b7b5ac0049d702179c\": container with ID starting with b8d17cbc6192e7797c7d8c746f009f7cc4a619f591db65b7b5ac0049d702179c not found: ID does not exist"
Jul 07 16:50:10.452199 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.452199    2578 scope.go:117] "RemoveContainer" containerID="0da92d3ce34e6c80b754c6ca6bfd37abcb3f3d5a486338eeb982bfd47baf3c7e"
Jul 07 16:50:10.452457 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.452438    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0da92d3ce34e6c80b754c6ca6bfd37abcb3f3d5a486338eeb982bfd47baf3c7e"} err="failed to get container status \"0da92d3ce34e6c80b754c6ca6bfd37abcb3f3d5a486338eeb982bfd47baf3c7e\": rpc error: code = NotFound desc = could not find container \"0da92d3ce34e6c80b754c6ca6bfd37abcb3f3d5a486338eeb982bfd47baf3c7e\": container with ID starting with 0da92d3ce34e6c80b754c6ca6bfd37abcb3f3d5a486338eeb982bfd47baf3c7e not found: ID does not exist"
Jul 07 16:50:10.452519 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.452458    2578 scope.go:117] "RemoveContainer" containerID="3ae5ceb91f346dbe8f4c9fd4b8e34dbe74d6b4237794432e5bb895641800ff3f"
Jul 07 16:50:10.452665 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.452640    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ae5ceb91f346dbe8f4c9fd4b8e34dbe74d6b4237794432e5bb895641800ff3f"} err="failed to get container status \"3ae5ceb91f346dbe8f4c9fd4b8e34dbe74d6b4237794432e5bb895641800ff3f\": rpc error: code = NotFound desc = could not find container \"3ae5ceb91f346dbe8f4c9fd4b8e34dbe74d6b4237794432e5bb895641800ff3f\": container with ID starting with 3ae5ceb91f346dbe8f4c9fd4b8e34dbe74d6b4237794432e5bb895641800ff3f not found: ID does not exist"
Jul 07 16:50:10.452721 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.452666    2578 scope.go:117] "RemoveContainer" containerID="0b49853db2492a72058608ec810b494cd313faeface778b08816e20570f88376"
Jul 07 16:50:10.452870 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.452852    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b49853db2492a72058608ec810b494cd313faeface778b08816e20570f88376"} err="failed to get container status \"0b49853db2492a72058608ec810b494cd313faeface778b08816e20570f88376\": rpc error: code = NotFound desc = could not find container \"0b49853db2492a72058608ec810b494cd313faeface778b08816e20570f88376\": container with ID starting with 0b49853db2492a72058608ec810b494cd313faeface778b08816e20570f88376 not found: ID does not exist"
Jul 07 16:50:10.452929 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.452870    2578 scope.go:117] "RemoveContainer" containerID="160fa002a7adbf7be0064d29e23e6b4a0d40a877ee39afa9472abc3c1610f244"
Jul 07 16:50:10.453052 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.453036    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"160fa002a7adbf7be0064d29e23e6b4a0d40a877ee39afa9472abc3c1610f244"} err="failed to get container status \"160fa002a7adbf7be0064d29e23e6b4a0d40a877ee39afa9472abc3c1610f244\": rpc error: code = NotFound desc = could not find container \"160fa002a7adbf7be0064d29e23e6b4a0d40a877ee39afa9472abc3c1610f244\": container with ID starting with 160fa002a7adbf7be0064d29e23e6b4a0d40a877ee39afa9472abc3c1610f244 not found: ID does not exist"
Jul 07 16:50:10.453052 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.453051    2578 scope.go:117] "RemoveContainer" containerID="e8ae6fc78374b6e07ffb679f868d996001e98e5bab5cc61560ed025b5308f863"
Jul 07 16:50:10.453257 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.453240    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8ae6fc78374b6e07ffb679f868d996001e98e5bab5cc61560ed025b5308f863"} err="failed to get container status \"e8ae6fc78374b6e07ffb679f868d996001e98e5bab5cc61560ed025b5308f863\": rpc error: code = NotFound desc = could not find container \"e8ae6fc78374b6e07ffb679f868d996001e98e5bab5cc61560ed025b5308f863\": container with ID starting with e8ae6fc78374b6e07ffb679f868d996001e98e5bab5cc61560ed025b5308f863 not found: ID does not exist"
Jul 07 16:50:10.453309 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.453258    2578 scope.go:117] "RemoveContainer" containerID="28c09282528f2dafed0372f50b6d295fd7b28d2e392b318ec4ada3ee02649a03"
Jul 07 16:50:10.453356 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.453323    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 16:50:10.453467 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.453448    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28c09282528f2dafed0372f50b6d295fd7b28d2e392b318ec4ada3ee02649a03"} err="failed to get container status \"28c09282528f2dafed0372f50b6d295fd7b28d2e392b318ec4ada3ee02649a03\": rpc error: code = NotFound desc = could not find container \"28c09282528f2dafed0372f50b6d295fd7b28d2e392b318ec4ada3ee02649a03\": container with ID starting with 28c09282528f2dafed0372f50b6d295fd7b28d2e392b318ec4ada3ee02649a03 not found: ID does not exist"
Jul 07 16:50:10.453524 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.453468    2578 scope.go:117] "RemoveContainer" containerID="b8d17cbc6192e7797c7d8c746f009f7cc4a619f591db65b7b5ac0049d702179c"
Jul 07 16:50:10.453651 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.453636    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="242f4d41-bb8d-4488-8477-0de8946569ce" containerName="init-config-reloader"
Jul 07 16:50:10.453704 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.453657    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="242f4d41-bb8d-4488-8477-0de8946569ce" containerName="init-config-reloader"
Jul 07 16:50:10.453704 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.453673    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="242f4d41-bb8d-4488-8477-0de8946569ce" containerName="kube-rbac-proxy-thanos"
Jul 07 16:50:10.453704 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.453667    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8d17cbc6192e7797c7d8c746f009f7cc4a619f591db65b7b5ac0049d702179c"} err="failed to get container status \"b8d17cbc6192e7797c7d8c746f009f7cc4a619f591db65b7b5ac0049d702179c\": rpc error: code = NotFound desc = could not find container \"b8d17cbc6192e7797c7d8c746f009f7cc4a619f591db65b7b5ac0049d702179c\": container with ID starting with b8d17cbc6192e7797c7d8c746f009f7cc4a619f591db65b7b5ac0049d702179c not found: ID does not exist"
Jul 07 16:50:10.453704 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.453693    2578 scope.go:117] "RemoveContainer" containerID="0da92d3ce34e6c80b754c6ca6bfd37abcb3f3d5a486338eeb982bfd47baf3c7e"
Jul 07 16:50:10.453879 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.453682    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="242f4d41-bb8d-4488-8477-0de8946569ce" containerName="kube-rbac-proxy-thanos"
Jul 07 16:50:10.453879 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.453767    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="242f4d41-bb8d-4488-8477-0de8946569ce" containerName="kube-rbac-proxy"
Jul 07 16:50:10.453879 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.453776    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="242f4d41-bb8d-4488-8477-0de8946569ce" containerName="kube-rbac-proxy"
Jul 07 16:50:10.453879 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.453784    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="242f4d41-bb8d-4488-8477-0de8946569ce" containerName="kube-rbac-proxy-web"
Jul 07 16:50:10.453879 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.453789    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="242f4d41-bb8d-4488-8477-0de8946569ce" containerName="kube-rbac-proxy-web"
Jul 07 16:50:10.453879 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.453798    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="242f4d41-bb8d-4488-8477-0de8946569ce" containerName="thanos-sidecar"
Jul 07 16:50:10.453879 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.453809    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="242f4d41-bb8d-4488-8477-0de8946569ce" containerName="thanos-sidecar"
Jul 07 16:50:10.453879 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.453823    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="242f4d41-bb8d-4488-8477-0de8946569ce" containerName="prometheus"
Jul 07 16:50:10.453879 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.453828    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="242f4d41-bb8d-4488-8477-0de8946569ce" containerName="prometheus"
Jul 07 16:50:10.453879 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.453838    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="242f4d41-bb8d-4488-8477-0de8946569ce" containerName="config-reloader"
Jul 07 16:50:10.453879 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.453846    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="242f4d41-bb8d-4488-8477-0de8946569ce" containerName="config-reloader"
Jul 07 16:50:10.454261 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.453915    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="242f4d41-bb8d-4488-8477-0de8946569ce" containerName="config-reloader"
Jul 07 16:50:10.454261 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.453923    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="242f4d41-bb8d-4488-8477-0de8946569ce" containerName="thanos-sidecar"
Jul 07 16:50:10.454261 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.453929    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="242f4d41-bb8d-4488-8477-0de8946569ce" containerName="kube-rbac-proxy-thanos"
Jul 07 16:50:10.454261 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.453936    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="242f4d41-bb8d-4488-8477-0de8946569ce" containerName="prometheus"
Jul 07 16:50:10.454261 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.453931    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0da92d3ce34e6c80b754c6ca6bfd37abcb3f3d5a486338eeb982bfd47baf3c7e"} err="failed to get container status \"0da92d3ce34e6c80b754c6ca6bfd37abcb3f3d5a486338eeb982bfd47baf3c7e\": rpc error: code = NotFound desc = could not find container \"0da92d3ce34e6c80b754c6ca6bfd37abcb3f3d5a486338eeb982bfd47baf3c7e\": container with ID starting with 0da92d3ce34e6c80b754c6ca6bfd37abcb3f3d5a486338eeb982bfd47baf3c7e not found: ID does not exist"
Jul 07 16:50:10.454261 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.453948    2578 scope.go:117] "RemoveContainer" containerID="3ae5ceb91f346dbe8f4c9fd4b8e34dbe74d6b4237794432e5bb895641800ff3f"
Jul 07 16:50:10.454261 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.453941    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="242f4d41-bb8d-4488-8477-0de8946569ce" containerName="kube-rbac-proxy-web"
Jul 07 16:50:10.454261 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.453998    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="242f4d41-bb8d-4488-8477-0de8946569ce" containerName="kube-rbac-proxy"
Jul 07 16:50:10.454261 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.454203    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ae5ceb91f346dbe8f4c9fd4b8e34dbe74d6b4237794432e5bb895641800ff3f"} err="failed to get container status \"3ae5ceb91f346dbe8f4c9fd4b8e34dbe74d6b4237794432e5bb895641800ff3f\": rpc error: code = NotFound desc = could not find container \"3ae5ceb91f346dbe8f4c9fd4b8e34dbe74d6b4237794432e5bb895641800ff3f\": container with ID starting with 3ae5ceb91f346dbe8f4c9fd4b8e34dbe74d6b4237794432e5bb895641800ff3f not found: ID does not exist"
Jul 07 16:50:10.454261 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.454227    2578 scope.go:117] "RemoveContainer" containerID="0b49853db2492a72058608ec810b494cd313faeface778b08816e20570f88376"
Jul 07 16:50:10.454633 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.454452    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b49853db2492a72058608ec810b494cd313faeface778b08816e20570f88376"} err="failed to get container status \"0b49853db2492a72058608ec810b494cd313faeface778b08816e20570f88376\": rpc error: code = NotFound desc = could not find container \"0b49853db2492a72058608ec810b494cd313faeface778b08816e20570f88376\": container with ID starting with 0b49853db2492a72058608ec810b494cd313faeface778b08816e20570f88376 not found: ID does not exist"
Jul 07 16:50:10.454633 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.454464    2578 scope.go:117] "RemoveContainer" containerID="160fa002a7adbf7be0064d29e23e6b4a0d40a877ee39afa9472abc3c1610f244"
Jul 07 16:50:10.454721 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.454669    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"160fa002a7adbf7be0064d29e23e6b4a0d40a877ee39afa9472abc3c1610f244"} err="failed to get container status \"160fa002a7adbf7be0064d29e23e6b4a0d40a877ee39afa9472abc3c1610f244\": rpc error: code = NotFound desc = could not find container \"160fa002a7adbf7be0064d29e23e6b4a0d40a877ee39afa9472abc3c1610f244\": container with ID starting with 160fa002a7adbf7be0064d29e23e6b4a0d40a877ee39afa9472abc3c1610f244 not found: ID does not exist"
Jul 07 16:50:10.454721 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.454687    2578 scope.go:117] "RemoveContainer" containerID="e8ae6fc78374b6e07ffb679f868d996001e98e5bab5cc61560ed025b5308f863"
Jul 07 16:50:10.455013 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.454989    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8ae6fc78374b6e07ffb679f868d996001e98e5bab5cc61560ed025b5308f863"} err="failed to get container status \"e8ae6fc78374b6e07ffb679f868d996001e98e5bab5cc61560ed025b5308f863\": rpc error: code = NotFound desc = could not find container \"e8ae6fc78374b6e07ffb679f868d996001e98e5bab5cc61560ed025b5308f863\": container with ID starting with e8ae6fc78374b6e07ffb679f868d996001e98e5bab5cc61560ed025b5308f863 not found: ID does not exist"
Jul 07 16:50:10.455112 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.455015    2578 scope.go:117] "RemoveContainer" containerID="28c09282528f2dafed0372f50b6d295fd7b28d2e392b318ec4ada3ee02649a03"
Jul 07 16:50:10.455311 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.455284    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28c09282528f2dafed0372f50b6d295fd7b28d2e392b318ec4ada3ee02649a03"} err="failed to get container status \"28c09282528f2dafed0372f50b6d295fd7b28d2e392b318ec4ada3ee02649a03\": rpc error: code = NotFound desc = could not find container \"28c09282528f2dafed0372f50b6d295fd7b28d2e392b318ec4ada3ee02649a03\": container with ID starting with 28c09282528f2dafed0372f50b6d295fd7b28d2e392b318ec4ada3ee02649a03 not found: ID does not exist"
Jul 07 16:50:10.455311 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.455310    2578 scope.go:117] "RemoveContainer" containerID="b8d17cbc6192e7797c7d8c746f009f7cc4a619f591db65b7b5ac0049d702179c"
Jul 07 16:50:10.455550 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.455534    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8d17cbc6192e7797c7d8c746f009f7cc4a619f591db65b7b5ac0049d702179c"} err="failed to get container status \"b8d17cbc6192e7797c7d8c746f009f7cc4a619f591db65b7b5ac0049d702179c\": rpc error: code = NotFound desc = could not find container \"b8d17cbc6192e7797c7d8c746f009f7cc4a619f591db65b7b5ac0049d702179c\": container with ID starting with b8d17cbc6192e7797c7d8c746f009f7cc4a619f591db65b7b5ac0049d702179c not found: ID does not exist"
Jul 07 16:50:10.455550 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.455550    2578 scope.go:117] "RemoveContainer" containerID="0da92d3ce34e6c80b754c6ca6bfd37abcb3f3d5a486338eeb982bfd47baf3c7e"
Jul 07 16:50:10.455814 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.455787    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0da92d3ce34e6c80b754c6ca6bfd37abcb3f3d5a486338eeb982bfd47baf3c7e"} err="failed to get container status \"0da92d3ce34e6c80b754c6ca6bfd37abcb3f3d5a486338eeb982bfd47baf3c7e\": rpc error: code = NotFound desc = could not find container \"0da92d3ce34e6c80b754c6ca6bfd37abcb3f3d5a486338eeb982bfd47baf3c7e\": container with ID starting with 0da92d3ce34e6c80b754c6ca6bfd37abcb3f3d5a486338eeb982bfd47baf3c7e not found: ID does not exist"
Jul 07 16:50:10.455923 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.455815    2578 scope.go:117] "RemoveContainer" containerID="3ae5ceb91f346dbe8f4c9fd4b8e34dbe74d6b4237794432e5bb895641800ff3f"
Jul 07 16:50:10.456050 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.456029    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ae5ceb91f346dbe8f4c9fd4b8e34dbe74d6b4237794432e5bb895641800ff3f"} err="failed to get container status \"3ae5ceb91f346dbe8f4c9fd4b8e34dbe74d6b4237794432e5bb895641800ff3f\": rpc error: code = NotFound desc = could not find container \"3ae5ceb91f346dbe8f4c9fd4b8e34dbe74d6b4237794432e5bb895641800ff3f\": container with ID starting with 3ae5ceb91f346dbe8f4c9fd4b8e34dbe74d6b4237794432e5bb895641800ff3f not found: ID does not exist"
Jul 07 16:50:10.456084 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.456051    2578 scope.go:117] "RemoveContainer" containerID="0b49853db2492a72058608ec810b494cd313faeface778b08816e20570f88376"
Jul 07 16:50:10.456289 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.456271    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b49853db2492a72058608ec810b494cd313faeface778b08816e20570f88376"} err="failed to get container status \"0b49853db2492a72058608ec810b494cd313faeface778b08816e20570f88376\": rpc error: code = NotFound desc = could not find container \"0b49853db2492a72058608ec810b494cd313faeface778b08816e20570f88376\": container with ID starting with 0b49853db2492a72058608ec810b494cd313faeface778b08816e20570f88376 not found: ID does not exist"
Jul 07 16:50:10.456328 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.456299    2578 scope.go:117] "RemoveContainer" containerID="160fa002a7adbf7be0064d29e23e6b4a0d40a877ee39afa9472abc3c1610f244"
Jul 07 16:50:10.456544 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.456525    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"160fa002a7adbf7be0064d29e23e6b4a0d40a877ee39afa9472abc3c1610f244"} err="failed to get container status \"160fa002a7adbf7be0064d29e23e6b4a0d40a877ee39afa9472abc3c1610f244\": rpc error: code = NotFound desc = could not find container \"160fa002a7adbf7be0064d29e23e6b4a0d40a877ee39afa9472abc3c1610f244\": container with ID starting with 160fa002a7adbf7be0064d29e23e6b4a0d40a877ee39afa9472abc3c1610f244 not found: ID does not exist"
Jul 07 16:50:10.456607 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.456546    2578 scope.go:117] "RemoveContainer" containerID="e8ae6fc78374b6e07ffb679f868d996001e98e5bab5cc61560ed025b5308f863"
Jul 07 16:50:10.456779 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.456763    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8ae6fc78374b6e07ffb679f868d996001e98e5bab5cc61560ed025b5308f863"} err="failed to get container status \"e8ae6fc78374b6e07ffb679f868d996001e98e5bab5cc61560ed025b5308f863\": rpc error: code = NotFound desc = could not find container \"e8ae6fc78374b6e07ffb679f868d996001e98e5bab5cc61560ed025b5308f863\": container with ID starting with e8ae6fc78374b6e07ffb679f868d996001e98e5bab5cc61560ed025b5308f863 not found: ID does not exist"
Jul 07 16:50:10.456850 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.456781    2578 scope.go:117] "RemoveContainer" containerID="28c09282528f2dafed0372f50b6d295fd7b28d2e392b318ec4ada3ee02649a03"
Jul 07 16:50:10.457006 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.456982    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28c09282528f2dafed0372f50b6d295fd7b28d2e392b318ec4ada3ee02649a03"} err="failed to get container status \"28c09282528f2dafed0372f50b6d295fd7b28d2e392b318ec4ada3ee02649a03\": rpc error: code = NotFound desc = could not find container \"28c09282528f2dafed0372f50b6d295fd7b28d2e392b318ec4ada3ee02649a03\": container with ID starting with 28c09282528f2dafed0372f50b6d295fd7b28d2e392b318ec4ada3ee02649a03 not found: ID does not exist"
Jul 07 16:50:10.457006 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.457001    2578 scope.go:117] "RemoveContainer" containerID="b8d17cbc6192e7797c7d8c746f009f7cc4a619f591db65b7b5ac0049d702179c"
Jul 07 16:50:10.457175 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.457161    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8d17cbc6192e7797c7d8c746f009f7cc4a619f591db65b7b5ac0049d702179c"} err="failed to get container status \"b8d17cbc6192e7797c7d8c746f009f7cc4a619f591db65b7b5ac0049d702179c\": rpc error: code = NotFound desc = could not find container \"b8d17cbc6192e7797c7d8c746f009f7cc4a619f591db65b7b5ac0049d702179c\": container with ID starting with b8d17cbc6192e7797c7d8c746f009f7cc4a619f591db65b7b5ac0049d702179c not found: ID does not exist"
Jul 07 16:50:10.457220 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.457175    2578 scope.go:117] "RemoveContainer" containerID="0da92d3ce34e6c80b754c6ca6bfd37abcb3f3d5a486338eeb982bfd47baf3c7e"
Jul 07 16:50:10.457358 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.457342    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0da92d3ce34e6c80b754c6ca6bfd37abcb3f3d5a486338eeb982bfd47baf3c7e"} err="failed to get container status \"0da92d3ce34e6c80b754c6ca6bfd37abcb3f3d5a486338eeb982bfd47baf3c7e\": rpc error: code = NotFound desc = could not find container \"0da92d3ce34e6c80b754c6ca6bfd37abcb3f3d5a486338eeb982bfd47baf3c7e\": container with ID starting with 0da92d3ce34e6c80b754c6ca6bfd37abcb3f3d5a486338eeb982bfd47baf3c7e not found: ID does not exist"
Jul 07 16:50:10.457404 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.457359    2578 scope.go:117] "RemoveContainer" containerID="3ae5ceb91f346dbe8f4c9fd4b8e34dbe74d6b4237794432e5bb895641800ff3f"
Jul 07 16:50:10.457469 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.457453    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:10.457875 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.457640    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ae5ceb91f346dbe8f4c9fd4b8e34dbe74d6b4237794432e5bb895641800ff3f"} err="failed to get container status \"3ae5ceb91f346dbe8f4c9fd4b8e34dbe74d6b4237794432e5bb895641800ff3f\": rpc error: code = NotFound desc = could not find container \"3ae5ceb91f346dbe8f4c9fd4b8e34dbe74d6b4237794432e5bb895641800ff3f\": container with ID starting with 3ae5ceb91f346dbe8f4c9fd4b8e34dbe74d6b4237794432e5bb895641800ff3f not found: ID does not exist"
Jul 07 16:50:10.457875 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.457661    2578 scope.go:117] "RemoveContainer" containerID="0b49853db2492a72058608ec810b494cd313faeface778b08816e20570f88376"
Jul 07 16:50:10.458079 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.458058    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b49853db2492a72058608ec810b494cd313faeface778b08816e20570f88376"} err="failed to get container status \"0b49853db2492a72058608ec810b494cd313faeface778b08816e20570f88376\": rpc error: code = NotFound desc = could not find container \"0b49853db2492a72058608ec810b494cd313faeface778b08816e20570f88376\": container with ID starting with 0b49853db2492a72058608ec810b494cd313faeface778b08816e20570f88376 not found: ID does not exist"
Jul 07 16:50:10.458134 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.458083    2578 scope.go:117] "RemoveContainer" containerID="160fa002a7adbf7be0064d29e23e6b4a0d40a877ee39afa9472abc3c1610f244"
Jul 07 16:50:10.458364 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.458319    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"160fa002a7adbf7be0064d29e23e6b4a0d40a877ee39afa9472abc3c1610f244"} err="failed to get container status \"160fa002a7adbf7be0064d29e23e6b4a0d40a877ee39afa9472abc3c1610f244\": rpc error: code = NotFound desc = could not find container \"160fa002a7adbf7be0064d29e23e6b4a0d40a877ee39afa9472abc3c1610f244\": container with ID starting with 160fa002a7adbf7be0064d29e23e6b4a0d40a877ee39afa9472abc3c1610f244 not found: ID does not exist"
Jul 07 16:50:10.458404 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.458367    2578 scope.go:117] "RemoveContainer" containerID="e8ae6fc78374b6e07ffb679f868d996001e98e5bab5cc61560ed025b5308f863"
Jul 07 16:50:10.458706 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.458687    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8ae6fc78374b6e07ffb679f868d996001e98e5bab5cc61560ed025b5308f863"} err="failed to get container status \"e8ae6fc78374b6e07ffb679f868d996001e98e5bab5cc61560ed025b5308f863\": rpc error: code = NotFound desc = could not find container \"e8ae6fc78374b6e07ffb679f868d996001e98e5bab5cc61560ed025b5308f863\": container with ID starting with e8ae6fc78374b6e07ffb679f868d996001e98e5bab5cc61560ed025b5308f863 not found: ID does not exist"
Jul 07 16:50:10.460832 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.460810    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 07 16:50:10.463024 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.461290    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 07 16:50:10.463024 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.461320    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 07 16:50:10.463024 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.461592    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-srjsv\""
Jul 07 16:50:10.463024 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.461753    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 07 16:50:10.463024 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.461866    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 07 16:50:10.463024 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.461931    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 07 16:50:10.463024 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.461965    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 07 16:50:10.463024 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.462039    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-34p0926q0hcnv\""
Jul 07 16:50:10.463024 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.462402    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 07 16:50:10.463024 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.462414    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 07 16:50:10.463024 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.462660    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 07 16:50:10.463889 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.463398    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 07 16:50:10.463889 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.463462    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 07 16:50:10.466860 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.465758    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 07 16:50:10.468483 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.468361    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 07 16:50:10.475463 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.475445    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 16:50:10.499427 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.499408    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29004849-731a-420f-bb78-7150391d28c3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"29004849-731a-420f-bb78-7150391d28c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:10.499511 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.499441    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/29004849-731a-420f-bb78-7150391d28c3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"29004849-731a-420f-bb78-7150391d28c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:10.499511 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.499471    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/29004849-731a-420f-bb78-7150391d28c3-config\") pod \"prometheus-k8s-0\" (UID: \"29004849-731a-420f-bb78-7150391d28c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:10.499511 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.499494    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfj8n\" (UniqueName: \"kubernetes.io/projected/29004849-731a-420f-bb78-7150391d28c3-kube-api-access-lfj8n\") pod \"prometheus-k8s-0\" (UID: \"29004849-731a-420f-bb78-7150391d28c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:10.499647 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.499575    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/29004849-731a-420f-bb78-7150391d28c3-web-config\") pod \"prometheus-k8s-0\" (UID: \"29004849-731a-420f-bb78-7150391d28c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:10.499701 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.499641    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29004849-731a-420f-bb78-7150391d28c3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"29004849-731a-420f-bb78-7150391d28c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:10.499767 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.499715    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/29004849-731a-420f-bb78-7150391d28c3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"29004849-731a-420f-bb78-7150391d28c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:10.499826 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.499773    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29004849-731a-420f-bb78-7150391d28c3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"29004849-731a-420f-bb78-7150391d28c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:10.499826 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.499817    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/29004849-731a-420f-bb78-7150391d28c3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"29004849-731a-420f-bb78-7150391d28c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:10.499919 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.499904    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/29004849-731a-420f-bb78-7150391d28c3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"29004849-731a-420f-bb78-7150391d28c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:10.499969 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.499932    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29004849-731a-420f-bb78-7150391d28c3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"29004849-731a-420f-bb78-7150391d28c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:10.499969 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.499948    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/29004849-731a-420f-bb78-7150391d28c3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"29004849-731a-420f-bb78-7150391d28c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:10.500070 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.499999    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/29004849-731a-420f-bb78-7150391d28c3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"29004849-731a-420f-bb78-7150391d28c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:10.500165 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.500148    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/29004849-731a-420f-bb78-7150391d28c3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"29004849-731a-420f-bb78-7150391d28c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:10.500217 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.500182    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/29004849-731a-420f-bb78-7150391d28c3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"29004849-731a-420f-bb78-7150391d28c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:10.500271 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.500251    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/29004849-731a-420f-bb78-7150391d28c3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"29004849-731a-420f-bb78-7150391d28c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:10.500330 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.500270    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/29004849-731a-420f-bb78-7150391d28c3-config-out\") pod \"prometheus-k8s-0\" (UID: \"29004849-731a-420f-bb78-7150391d28c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:10.500330 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.500287    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/29004849-731a-420f-bb78-7150391d28c3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"29004849-731a-420f-bb78-7150391d28c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:10.500330 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.500304    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/29004849-731a-420f-bb78-7150391d28c3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"29004849-731a-420f-bb78-7150391d28c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:10.500330 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.500322    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/29004849-731a-420f-bb78-7150391d28c3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"29004849-731a-420f-bb78-7150391d28c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:10.601620 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.601590    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/29004849-731a-420f-bb78-7150391d28c3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"29004849-731a-420f-bb78-7150391d28c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:10.601786 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.601630    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29004849-731a-420f-bb78-7150391d28c3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"29004849-731a-420f-bb78-7150391d28c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:10.601786 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.601655    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/29004849-731a-420f-bb78-7150391d28c3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"29004849-731a-420f-bb78-7150391d28c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:10.601786 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.601680    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/29004849-731a-420f-bb78-7150391d28c3-config\") pod \"prometheus-k8s-0\" (UID: \"29004849-731a-420f-bb78-7150391d28c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:10.601786 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.601701    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfj8n\" (UniqueName: \"kubernetes.io/projected/29004849-731a-420f-bb78-7150391d28c3-kube-api-access-lfj8n\") pod \"prometheus-k8s-0\" (UID: \"29004849-731a-420f-bb78-7150391d28c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:10.601979 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.601841    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/29004849-731a-420f-bb78-7150391d28c3-web-config\") pod \"prometheus-k8s-0\" (UID: \"29004849-731a-420f-bb78-7150391d28c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:10.601979 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.601882    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29004849-731a-420f-bb78-7150391d28c3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"29004849-731a-420f-bb78-7150391d28c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:10.601979 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.601924    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/29004849-731a-420f-bb78-7150391d28c3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"29004849-731a-420f-bb78-7150391d28c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:10.601979 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.601957    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29004849-731a-420f-bb78-7150391d28c3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"29004849-731a-420f-bb78-7150391d28c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:10.602186 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.601991    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/29004849-731a-420f-bb78-7150391d28c3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"29004849-731a-420f-bb78-7150391d28c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:10.602186 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.602021    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/29004849-731a-420f-bb78-7150391d28c3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"29004849-731a-420f-bb78-7150391d28c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:10.602186 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.602049    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29004849-731a-420f-bb78-7150391d28c3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"29004849-731a-420f-bb78-7150391d28c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:10.602186 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.602074    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/29004849-731a-420f-bb78-7150391d28c3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"29004849-731a-420f-bb78-7150391d28c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:10.602186 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.602103    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/29004849-731a-420f-bb78-7150391d28c3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"29004849-731a-420f-bb78-7150391d28c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:10.602186 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.602156    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/29004849-731a-420f-bb78-7150391d28c3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"29004849-731a-420f-bb78-7150391d28c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:10.602495 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.602187    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/29004849-731a-420f-bb78-7150391d28c3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"29004849-731a-420f-bb78-7150391d28c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:10.602495 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.602223    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/29004849-731a-420f-bb78-7150391d28c3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"29004849-731a-420f-bb78-7150391d28c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:10.602495 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.602249    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/29004849-731a-420f-bb78-7150391d28c3-config-out\") pod \"prometheus-k8s-0\" (UID: \"29004849-731a-420f-bb78-7150391d28c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:10.602495 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.602268    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/29004849-731a-420f-bb78-7150391d28c3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"29004849-731a-420f-bb78-7150391d28c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:10.602495 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.602276    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/29004849-731a-420f-bb78-7150391d28c3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"29004849-731a-420f-bb78-7150391d28c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:10.602495 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.602312    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/29004849-731a-420f-bb78-7150391d28c3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"29004849-731a-420f-bb78-7150391d28c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:10.602786 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.602656    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29004849-731a-420f-bb78-7150391d28c3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"29004849-731a-420f-bb78-7150391d28c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:10.602786 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.602672    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29004849-731a-420f-bb78-7150391d28c3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"29004849-731a-420f-bb78-7150391d28c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:10.603615 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.603588    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29004849-731a-420f-bb78-7150391d28c3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"29004849-731a-420f-bb78-7150391d28c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:10.603924 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.603896    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/29004849-731a-420f-bb78-7150391d28c3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"29004849-731a-420f-bb78-7150391d28c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:10.604450 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.604426    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29004849-731a-420f-bb78-7150391d28c3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"29004849-731a-420f-bb78-7150391d28c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:10.604859 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.604802    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/29004849-731a-420f-bb78-7150391d28c3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"29004849-731a-420f-bb78-7150391d28c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:10.604949 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.604934    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/29004849-731a-420f-bb78-7150391d28c3-web-config\") pod \"prometheus-k8s-0\" (UID: \"29004849-731a-420f-bb78-7150391d28c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:10.605025 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.605012    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/29004849-731a-420f-bb78-7150391d28c3-config\") pod \"prometheus-k8s-0\" (UID: \"29004849-731a-420f-bb78-7150391d28c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:10.605163 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.605124    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/29004849-731a-420f-bb78-7150391d28c3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"29004849-731a-420f-bb78-7150391d28c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:10.605414 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.605390    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/29004849-731a-420f-bb78-7150391d28c3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"29004849-731a-420f-bb78-7150391d28c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:10.605880 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.605857    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/29004849-731a-420f-bb78-7150391d28c3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"29004849-731a-420f-bb78-7150391d28c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:10.606888 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.606864    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/29004849-731a-420f-bb78-7150391d28c3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"29004849-731a-420f-bb78-7150391d28c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:10.606980 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.606910    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/29004849-731a-420f-bb78-7150391d28c3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"29004849-731a-420f-bb78-7150391d28c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:10.606980 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.606954    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/29004849-731a-420f-bb78-7150391d28c3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"29004849-731a-420f-bb78-7150391d28c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:10.607066 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.606998    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/29004849-731a-420f-bb78-7150391d28c3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"29004849-731a-420f-bb78-7150391d28c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:10.607066 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.606998    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/29004849-731a-420f-bb78-7150391d28c3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"29004849-731a-420f-bb78-7150391d28c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:10.607518 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.607501    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/29004849-731a-420f-bb78-7150391d28c3-config-out\") pod \"prometheus-k8s-0\" (UID: \"29004849-731a-420f-bb78-7150391d28c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:10.608221 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.608206    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/29004849-731a-420f-bb78-7150391d28c3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"29004849-731a-420f-bb78-7150391d28c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:10.610510 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.610488    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfj8n\" (UniqueName: \"kubernetes.io/projected/29004849-731a-420f-bb78-7150391d28c3-kube-api-access-lfj8n\") pod \"prometheus-k8s-0\" (UID: \"29004849-731a-420f-bb78-7150391d28c3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:10.771103 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.771045    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:10.902867 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:10.902840    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 16:50:10.906396 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:50:10.906273    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29004849_731a_420f_bb78_7150391d28c3.slice/crio-e0ee772d36ec7635ff46e9fb7dd028091587c424de574158a5765a8886f16b54 WatchSource:0}: Error finding container e0ee772d36ec7635ff46e9fb7dd028091587c424de574158a5765a8886f16b54: Status 404 returned error can't find the container with id e0ee772d36ec7635ff46e9fb7dd028091587c424de574158a5765a8886f16b54
Jul 07 16:50:11.406851 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:11.406817    2578 generic.go:356] "Generic (PLEG): container finished" podID="29004849-731a-420f-bb78-7150391d28c3" containerID="20bdffa4d2a9cf62eb2e34952cce86716440fa5f56e97a6900ed15cc8bab7aa3" exitCode=0
Jul 07 16:50:11.407290 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:11.406883    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"29004849-731a-420f-bb78-7150391d28c3","Type":"ContainerDied","Data":"20bdffa4d2a9cf62eb2e34952cce86716440fa5f56e97a6900ed15cc8bab7aa3"}
Jul 07 16:50:11.407290 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:11.406951    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"29004849-731a-420f-bb78-7150391d28c3","Type":"ContainerStarted","Data":"e0ee772d36ec7635ff46e9fb7dd028091587c424de574158a5765a8886f16b54"}
Jul 07 16:50:11.726406 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:11.726376    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="242f4d41-bb8d-4488-8477-0de8946569ce" path="/var/lib/kubelet/pods/242f4d41-bb8d-4488-8477-0de8946569ce/volumes"
Jul 07 16:50:12.415074 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:12.415034    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"29004849-731a-420f-bb78-7150391d28c3","Type":"ContainerStarted","Data":"d04f02cec824604e74f8580df29d0daf71edbc3deaebe7dfd53f9ffb69505571"}
Jul 07 16:50:12.415419 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:12.415098    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"29004849-731a-420f-bb78-7150391d28c3","Type":"ContainerStarted","Data":"2651fffed7c27004f60ef4f0f685ff3f5ba89d5b64281e8885ffff81bcc085b9"}
Jul 07 16:50:12.415419 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:12.415124    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"29004849-731a-420f-bb78-7150391d28c3","Type":"ContainerStarted","Data":"1f9aa892f1e6c4580a68d45a17e57b226d1afbe16d29a459755b8abc52f75450"}
Jul 07 16:50:12.415419 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:12.415148    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"29004849-731a-420f-bb78-7150391d28c3","Type":"ContainerStarted","Data":"8647909f82310050f3501e0faf72b775320672d10bbca75bd1acce9871746972"}
Jul 07 16:50:12.415419 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:12.415172    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"29004849-731a-420f-bb78-7150391d28c3","Type":"ContainerStarted","Data":"599772c6efda9f87c449b42a7628163810b4f48447526aae19667bd276cea00b"}
Jul 07 16:50:12.415419 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:12.415194    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"29004849-731a-420f-bb78-7150391d28c3","Type":"ContainerStarted","Data":"525b8af7c77321df9892a3768ada3c4032018c1bed69f768c10756a7cc2879bd"}
Jul 07 16:50:12.443501 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:12.443440    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.421587879 podStartE2EDuration="2.443419022s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.831143ms" isStatefulPod=false podCreationTimestamp="2026-07-07 16:50:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:50:12.440631343 +0000 UTC m=+149.216945574" watchObservedRunningTime="2026-07-07 16:50:12.443419022 +0000 UTC m=+149.219733230"
Jul 07 16:50:15.771409 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:50:15.771376    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:51:10.771869 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:51:10.771839    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:51:10.787351 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:51:10.787329    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:51:11.649983 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:51:11.649958    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:52:43.586818 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:52:43.586793    2578 kubelet.go:1648] "Image garbage collection succeeded"
Jul 07 16:54:02.471837 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:02.471761    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7989c6f6f8-ghf7m"]
Jul 07 16:54:02.473865 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:02.473849    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7989c6f6f8-ghf7m"
Jul 07 16:54:02.476746 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:02.476716    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 07 16:54:02.476933 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:02.476915    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 07 16:54:02.477087 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:02.477071    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 07 16:54:02.477947 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:02.477933    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 07 16:54:02.478148 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:02.478129    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 07 16:54:02.478249 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:02.478178    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 07 16:54:02.478249 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:02.478179    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-54jkp\""
Jul 07 16:54:02.478408 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:02.478396    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 07 16:54:02.482466 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:02.482450    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 07 16:54:02.486051 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:02.486031    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7989c6f6f8-ghf7m"]
Jul 07 16:54:02.557055 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:02.557025    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab62667f-7d8d-476a-b39f-bf86daaa600c-trusted-ca-bundle\") pod \"console-7989c6f6f8-ghf7m\" (UID: \"ab62667f-7d8d-476a-b39f-bf86daaa600c\") " pod="openshift-console/console-7989c6f6f8-ghf7m"
Jul 07 16:54:02.557055 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:02.557052    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ab62667f-7d8d-476a-b39f-bf86daaa600c-service-ca\") pod \"console-7989c6f6f8-ghf7m\" (UID: \"ab62667f-7d8d-476a-b39f-bf86daaa600c\") " pod="openshift-console/console-7989c6f6f8-ghf7m"
Jul 07 16:54:02.557281 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:02.557072    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlh4f\" (UniqueName: \"kubernetes.io/projected/ab62667f-7d8d-476a-b39f-bf86daaa600c-kube-api-access-vlh4f\") pod \"console-7989c6f6f8-ghf7m\" (UID: \"ab62667f-7d8d-476a-b39f-bf86daaa600c\") " pod="openshift-console/console-7989c6f6f8-ghf7m"
Jul 07 16:54:02.557281 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:02.557090    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ab62667f-7d8d-476a-b39f-bf86daaa600c-console-serving-cert\") pod \"console-7989c6f6f8-ghf7m\" (UID: \"ab62667f-7d8d-476a-b39f-bf86daaa600c\") " pod="openshift-console/console-7989c6f6f8-ghf7m"
Jul 07 16:54:02.557281 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:02.557192    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ab62667f-7d8d-476a-b39f-bf86daaa600c-console-oauth-config\") pod \"console-7989c6f6f8-ghf7m\" (UID: \"ab62667f-7d8d-476a-b39f-bf86daaa600c\") " pod="openshift-console/console-7989c6f6f8-ghf7m"
Jul 07 16:54:02.557399 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:02.557286    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ab62667f-7d8d-476a-b39f-bf86daaa600c-oauth-serving-cert\") pod \"console-7989c6f6f8-ghf7m\" (UID: \"ab62667f-7d8d-476a-b39f-bf86daaa600c\") " pod="openshift-console/console-7989c6f6f8-ghf7m"
Jul 07 16:54:02.557399 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:02.557321    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ab62667f-7d8d-476a-b39f-bf86daaa600c-console-config\") pod \"console-7989c6f6f8-ghf7m\" (UID: \"ab62667f-7d8d-476a-b39f-bf86daaa600c\") " pod="openshift-console/console-7989c6f6f8-ghf7m"
Jul 07 16:54:02.658347 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:02.658305    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ab62667f-7d8d-476a-b39f-bf86daaa600c-oauth-serving-cert\") pod \"console-7989c6f6f8-ghf7m\" (UID: \"ab62667f-7d8d-476a-b39f-bf86daaa600c\") " pod="openshift-console/console-7989c6f6f8-ghf7m"
Jul 07 16:54:02.658347 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:02.658348    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ab62667f-7d8d-476a-b39f-bf86daaa600c-console-config\") pod \"console-7989c6f6f8-ghf7m\" (UID: \"ab62667f-7d8d-476a-b39f-bf86daaa600c\") " pod="openshift-console/console-7989c6f6f8-ghf7m"
Jul 07 16:54:02.658578 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:02.658372    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab62667f-7d8d-476a-b39f-bf86daaa600c-trusted-ca-bundle\") pod \"console-7989c6f6f8-ghf7m\" (UID: \"ab62667f-7d8d-476a-b39f-bf86daaa600c\") " pod="openshift-console/console-7989c6f6f8-ghf7m"
Jul 07 16:54:02.658578 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:02.658397    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ab62667f-7d8d-476a-b39f-bf86daaa600c-service-ca\") pod \"console-7989c6f6f8-ghf7m\" (UID: \"ab62667f-7d8d-476a-b39f-bf86daaa600c\") " pod="openshift-console/console-7989c6f6f8-ghf7m"
Jul 07 16:54:02.658578 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:02.658500    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlh4f\" (UniqueName: \"kubernetes.io/projected/ab62667f-7d8d-476a-b39f-bf86daaa600c-kube-api-access-vlh4f\") pod \"console-7989c6f6f8-ghf7m\" (UID: \"ab62667f-7d8d-476a-b39f-bf86daaa600c\") " pod="openshift-console/console-7989c6f6f8-ghf7m"
Jul 07 16:54:02.658578 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:02.658536    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ab62667f-7d8d-476a-b39f-bf86daaa600c-console-serving-cert\") pod \"console-7989c6f6f8-ghf7m\" (UID: \"ab62667f-7d8d-476a-b39f-bf86daaa600c\") " pod="openshift-console/console-7989c6f6f8-ghf7m"
Jul 07 16:54:02.658578 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:02.658570    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ab62667f-7d8d-476a-b39f-bf86daaa600c-console-oauth-config\") pod \"console-7989c6f6f8-ghf7m\" (UID: \"ab62667f-7d8d-476a-b39f-bf86daaa600c\") " pod="openshift-console/console-7989c6f6f8-ghf7m"
Jul 07 16:54:02.659153 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:02.659126    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ab62667f-7d8d-476a-b39f-bf86daaa600c-console-config\") pod \"console-7989c6f6f8-ghf7m\" (UID: \"ab62667f-7d8d-476a-b39f-bf86daaa600c\") " pod="openshift-console/console-7989c6f6f8-ghf7m"
Jul 07 16:54:02.659326 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:02.659301    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ab62667f-7d8d-476a-b39f-bf86daaa600c-oauth-serving-cert\") pod \"console-7989c6f6f8-ghf7m\" (UID: \"ab62667f-7d8d-476a-b39f-bf86daaa600c\") " pod="openshift-console/console-7989c6f6f8-ghf7m"
Jul 07 16:54:02.659557 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:02.659142    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ab62667f-7d8d-476a-b39f-bf86daaa600c-service-ca\") pod \"console-7989c6f6f8-ghf7m\" (UID: \"ab62667f-7d8d-476a-b39f-bf86daaa600c\") " pod="openshift-console/console-7989c6f6f8-ghf7m"
Jul 07 16:54:02.659557 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:02.659546    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab62667f-7d8d-476a-b39f-bf86daaa600c-trusted-ca-bundle\") pod \"console-7989c6f6f8-ghf7m\" (UID: \"ab62667f-7d8d-476a-b39f-bf86daaa600c\") " pod="openshift-console/console-7989c6f6f8-ghf7m"
Jul 07 16:54:02.661089 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:02.661071    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ab62667f-7d8d-476a-b39f-bf86daaa600c-console-oauth-config\") pod \"console-7989c6f6f8-ghf7m\" (UID: \"ab62667f-7d8d-476a-b39f-bf86daaa600c\") " pod="openshift-console/console-7989c6f6f8-ghf7m"
Jul 07 16:54:02.661607 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:02.661583    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ab62667f-7d8d-476a-b39f-bf86daaa600c-console-serving-cert\") pod \"console-7989c6f6f8-ghf7m\" (UID: \"ab62667f-7d8d-476a-b39f-bf86daaa600c\") " pod="openshift-console/console-7989c6f6f8-ghf7m"
Jul 07 16:54:02.667217 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:02.667189    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlh4f\" (UniqueName: \"kubernetes.io/projected/ab62667f-7d8d-476a-b39f-bf86daaa600c-kube-api-access-vlh4f\") pod \"console-7989c6f6f8-ghf7m\" (UID: \"ab62667f-7d8d-476a-b39f-bf86daaa600c\") " pod="openshift-console/console-7989c6f6f8-ghf7m"
Jul 07 16:54:02.783613 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:02.783518    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7989c6f6f8-ghf7m"
Jul 07 16:54:02.907071 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:02.907039    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7989c6f6f8-ghf7m"]
Jul 07 16:54:02.910232 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:54:02.910202    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab62667f_7d8d_476a_b39f_bf86daaa600c.slice/crio-dc1479ff5572b46701b6bdf34f895c9997a2426ce568af9e5bf62bded361d045 WatchSource:0}: Error finding container dc1479ff5572b46701b6bdf34f895c9997a2426ce568af9e5bf62bded361d045: Status 404 returned error can't find the container with id dc1479ff5572b46701b6bdf34f895c9997a2426ce568af9e5bf62bded361d045
Jul 07 16:54:03.235891 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:03.235853    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7989c6f6f8-ghf7m" event={"ID":"ab62667f-7d8d-476a-b39f-bf86daaa600c","Type":"ContainerStarted","Data":"f6b9aa5990e317cff147b41133ab87b6be18a150972540b452de1381cadf7994"}
Jul 07 16:54:03.236091 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:03.235913    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7989c6f6f8-ghf7m" event={"ID":"ab62667f-7d8d-476a-b39f-bf86daaa600c","Type":"ContainerStarted","Data":"dc1479ff5572b46701b6bdf34f895c9997a2426ce568af9e5bf62bded361d045"}
Jul 07 16:54:03.255628 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:03.255581    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7989c6f6f8-ghf7m" podStartSLOduration=1.255565751 podStartE2EDuration="1.255565751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:54:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:54:03.255160574 +0000 UTC m=+380.031474809" watchObservedRunningTime="2026-07-07 16:54:03.255565751 +0000 UTC m=+380.031879956"
Jul 07 16:54:09.084414 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:09.084383    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-fd65d7d6-8hr2z"]
Jul 07 16:54:09.087717 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:09.087698    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-8hr2z"
Jul 07 16:54:09.090470 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:09.090442    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-mdqmn\""
Jul 07 16:54:09.090470 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:09.090444    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 07 16:54:09.090470 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:09.090444    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 07 16:54:09.090686 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:09.090480    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 07 16:54:09.098873 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:09.098854    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-8hr2z"]
Jul 07 16:54:09.114603 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:09.114580    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x6ht\" (UniqueName: \"kubernetes.io/projected/a40fe1eb-57b9-4726-a4b4-17353bde6339-kube-api-access-5x6ht\") pod \"model-serving-api-fd65d7d6-8hr2z\" (UID: \"a40fe1eb-57b9-4726-a4b4-17353bde6339\") " pod="kserve/model-serving-api-fd65d7d6-8hr2z"
Jul 07 16:54:09.114700 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:09.114656    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a40fe1eb-57b9-4726-a4b4-17353bde6339-tls-certs\") pod \"model-serving-api-fd65d7d6-8hr2z\" (UID: \"a40fe1eb-57b9-4726-a4b4-17353bde6339\") " pod="kserve/model-serving-api-fd65d7d6-8hr2z"
Jul 07 16:54:09.215993 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:09.215970    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a40fe1eb-57b9-4726-a4b4-17353bde6339-tls-certs\") pod \"model-serving-api-fd65d7d6-8hr2z\" (UID: \"a40fe1eb-57b9-4726-a4b4-17353bde6339\") " pod="kserve/model-serving-api-fd65d7d6-8hr2z"
Jul 07 16:54:09.216102 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:09.216011    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x6ht\" (UniqueName: \"kubernetes.io/projected/a40fe1eb-57b9-4726-a4b4-17353bde6339-kube-api-access-5x6ht\") pod \"model-serving-api-fd65d7d6-8hr2z\" (UID: \"a40fe1eb-57b9-4726-a4b4-17353bde6339\") " pod="kserve/model-serving-api-fd65d7d6-8hr2z"
Jul 07 16:54:09.218295 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:09.218270    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a40fe1eb-57b9-4726-a4b4-17353bde6339-tls-certs\") pod \"model-serving-api-fd65d7d6-8hr2z\" (UID: \"a40fe1eb-57b9-4726-a4b4-17353bde6339\") " pod="kserve/model-serving-api-fd65d7d6-8hr2z"
Jul 07 16:54:09.224256 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:09.224227    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x6ht\" (UniqueName: \"kubernetes.io/projected/a40fe1eb-57b9-4726-a4b4-17353bde6339-kube-api-access-5x6ht\") pod \"model-serving-api-fd65d7d6-8hr2z\" (UID: \"a40fe1eb-57b9-4726-a4b4-17353bde6339\") " pod="kserve/model-serving-api-fd65d7d6-8hr2z"
Jul 07 16:54:09.398299 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:09.398228    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-8hr2z"
Jul 07 16:54:09.521095 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:09.521074    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-8hr2z"]
Jul 07 16:54:09.523288 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:54:09.523260    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda40fe1eb_57b9_4726_a4b4_17353bde6339.slice/crio-30a8a39d87fb81462f02e919a811b572c39533f7684a01840f11b93ca54e414f WatchSource:0}: Error finding container 30a8a39d87fb81462f02e919a811b572c39533f7684a01840f11b93ca54e414f: Status 404 returned error can't find the container with id 30a8a39d87fb81462f02e919a811b572c39533f7684a01840f11b93ca54e414f
Jul 07 16:54:09.525195 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:09.525178    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 16:54:10.262997 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:10.262958    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-8hr2z" event={"ID":"a40fe1eb-57b9-4726-a4b4-17353bde6339","Type":"ContainerStarted","Data":"30a8a39d87fb81462f02e919a811b572c39533f7684a01840f11b93ca54e414f"}
Jul 07 16:54:12.272962 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:12.272928    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-8hr2z" event={"ID":"a40fe1eb-57b9-4726-a4b4-17353bde6339","Type":"ContainerStarted","Data":"8e2227baf18f12ba63c5a8665f7c169cf5aa96bb0ba41491ae961c6de847e4c9"}
Jul 07 16:54:12.273326 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:12.273059    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-fd65d7d6-8hr2z"
Jul 07 16:54:12.290056 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:12.290006    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-fd65d7d6-8hr2z" podStartSLOduration=1.184468635 podStartE2EDuration="3.289992613s" totalImagesPullingTime="2.105523978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:54:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:54:12.288795035 +0000 UTC m=+389.065109262" watchObservedRunningTime="2026-07-07 16:54:12.289992613 +0000 UTC m=+389.066306818"
Jul 07 16:54:12.784511 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:12.784482    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7989c6f6f8-ghf7m"
Jul 07 16:54:12.784707 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:12.784562    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7989c6f6f8-ghf7m"
Jul 07 16:54:12.789072 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:12.789052    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7989c6f6f8-ghf7m"
Jul 07 16:54:13.280844 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:13.280814    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7989c6f6f8-ghf7m"
Jul 07 16:54:23.281367 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:23.281343    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-fd65d7d6-8hr2z"
Jul 07 16:54:24.913924 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:24.913888    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-rpclz"]
Jul 07 16:54:24.917563 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:24.917542    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-rpclz"
Jul 07 16:54:24.920212 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:24.920191    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-9s599\""
Jul 07 16:54:24.920286 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:24.920251    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 07 16:54:24.924557 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:24.924303    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-rpclz"]
Jul 07 16:54:25.042631 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:25.042597    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b9pm\" (UniqueName: \"kubernetes.io/projected/65414cbc-76b2-4db6-9ceb-df027e3ae9c0-kube-api-access-2b9pm\") pod \"s3-init-rpclz\" (UID: \"65414cbc-76b2-4db6-9ceb-df027e3ae9c0\") " pod="kserve/s3-init-rpclz"
Jul 07 16:54:25.042816 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:25.042638    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65414cbc-76b2-4db6-9ceb-df027e3ae9c0-model-cache\") pod \"s3-init-rpclz\" (UID: \"65414cbc-76b2-4db6-9ceb-df027e3ae9c0\") " pod="kserve/s3-init-rpclz"
Jul 07 16:54:25.143412 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:25.143375    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b9pm\" (UniqueName: \"kubernetes.io/projected/65414cbc-76b2-4db6-9ceb-df027e3ae9c0-kube-api-access-2b9pm\") pod \"s3-init-rpclz\" (UID: \"65414cbc-76b2-4db6-9ceb-df027e3ae9c0\") " pod="kserve/s3-init-rpclz"
Jul 07 16:54:25.143556 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:25.143420    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65414cbc-76b2-4db6-9ceb-df027e3ae9c0-model-cache\") pod \"s3-init-rpclz\" (UID: \"65414cbc-76b2-4db6-9ceb-df027e3ae9c0\") " pod="kserve/s3-init-rpclz"
Jul 07 16:54:25.143828 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:25.143811    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65414cbc-76b2-4db6-9ceb-df027e3ae9c0-model-cache\") pod \"s3-init-rpclz\" (UID: \"65414cbc-76b2-4db6-9ceb-df027e3ae9c0\") " pod="kserve/s3-init-rpclz"
Jul 07 16:54:25.151765 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:25.151728    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b9pm\" (UniqueName: \"kubernetes.io/projected/65414cbc-76b2-4db6-9ceb-df027e3ae9c0-kube-api-access-2b9pm\") pod \"s3-init-rpclz\" (UID: \"65414cbc-76b2-4db6-9ceb-df027e3ae9c0\") " pod="kserve/s3-init-rpclz"
Jul 07 16:54:25.240683 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:25.240614    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-rpclz"
Jul 07 16:54:25.364938 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:25.364906    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-rpclz"]
Jul 07 16:54:25.367120 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:54:25.367102    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65414cbc_76b2_4db6_9ceb_df027e3ae9c0.slice/crio-b1cb2d6a5d8ed3760f8c244dda72d8b2f54862b23350b440811e80c747dedda1 WatchSource:0}: Error finding container b1cb2d6a5d8ed3760f8c244dda72d8b2f54862b23350b440811e80c747dedda1: Status 404 returned error can't find the container with id b1cb2d6a5d8ed3760f8c244dda72d8b2f54862b23350b440811e80c747dedda1
Jul 07 16:54:26.325713 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:26.325684    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-rpclz" event={"ID":"65414cbc-76b2-4db6-9ceb-df027e3ae9c0","Type":"ContainerStarted","Data":"b1cb2d6a5d8ed3760f8c244dda72d8b2f54862b23350b440811e80c747dedda1"}
Jul 07 16:54:29.340764 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:29.340713    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-rpclz" event={"ID":"65414cbc-76b2-4db6-9ceb-df027e3ae9c0","Type":"ContainerStarted","Data":"f87951917197f7204e260f85238ec47ec0e4c2c0d8552037edba57fba888507c"}
Jul 07 16:54:30.345725 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:30.345688    2578 generic.go:356] "Generic (PLEG): container finished" podID="65414cbc-76b2-4db6-9ceb-df027e3ae9c0" containerID="f87951917197f7204e260f85238ec47ec0e4c2c0d8552037edba57fba888507c" exitCode=0
Jul 07 16:54:30.346089 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:30.345755    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-rpclz" event={"ID":"65414cbc-76b2-4db6-9ceb-df027e3ae9c0","Type":"ContainerDied","Data":"f87951917197f7204e260f85238ec47ec0e4c2c0d8552037edba57fba888507c"}
Jul 07 16:54:35.375136 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:35.375099    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-rpclz" event={"ID":"65414cbc-76b2-4db6-9ceb-df027e3ae9c0","Type":"ContainerStarted","Data":"2a1438faf60612da6d4b2e3ee81d69ceb0fea60b4e27da12de0a6ade19af975a"}
Jul 07 16:54:35.393387 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:54:35.393338    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-rpclz" podStartSLOduration=1.95225183 podStartE2EDuration="11.393324186s" totalImagesPullingTime="7.920643376s" totalInitContainerRuntime="1.52042898s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:54:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:54:35.391665809 +0000 UTC m=+412.167980017" watchObservedRunningTime="2026-07-07 16:54:35.393324186 +0000 UTC m=+412.169638412"
Jul 07 16:55:40.622070 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:40.622038    2578 generic.go:356] "Generic (PLEG): container finished" podID="65414cbc-76b2-4db6-9ceb-df027e3ae9c0" containerID="2a1438faf60612da6d4b2e3ee81d69ceb0fea60b4e27da12de0a6ade19af975a" exitCode=0
Jul 07 16:55:40.622070 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:40.622077    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-rpclz" event={"ID":"65414cbc-76b2-4db6-9ceb-df027e3ae9c0","Type":"ContainerDied","Data":"2a1438faf60612da6d4b2e3ee81d69ceb0fea60b4e27da12de0a6ade19af975a"}
Jul 07 16:55:41.746932 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:41.746911    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-rpclz"
Jul 07 16:55:41.879475 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:41.879393    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65414cbc-76b2-4db6-9ceb-df027e3ae9c0-model-cache\") pod \"65414cbc-76b2-4db6-9ceb-df027e3ae9c0\" (UID: \"65414cbc-76b2-4db6-9ceb-df027e3ae9c0\") "
Jul 07 16:55:41.879475 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:41.879474    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2b9pm\" (UniqueName: \"kubernetes.io/projected/65414cbc-76b2-4db6-9ceb-df027e3ae9c0-kube-api-access-2b9pm\") pod \"65414cbc-76b2-4db6-9ceb-df027e3ae9c0\" (UID: \"65414cbc-76b2-4db6-9ceb-df027e3ae9c0\") "
Jul 07 16:55:41.881636 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:41.881609    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65414cbc-76b2-4db6-9ceb-df027e3ae9c0-kube-api-access-2b9pm" (OuterVolumeSpecName: "kube-api-access-2b9pm") pod "65414cbc-76b2-4db6-9ceb-df027e3ae9c0" (UID: "65414cbc-76b2-4db6-9ceb-df027e3ae9c0"). InnerVolumeSpecName "kube-api-access-2b9pm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:55:41.897664 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:41.897641    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65414cbc-76b2-4db6-9ceb-df027e3ae9c0-model-cache" (OuterVolumeSpecName: "model-cache") pod "65414cbc-76b2-4db6-9ceb-df027e3ae9c0" (UID: "65414cbc-76b2-4db6-9ceb-df027e3ae9c0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:55:41.980201 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:41.980172    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2b9pm\" (UniqueName: \"kubernetes.io/projected/65414cbc-76b2-4db6-9ceb-df027e3ae9c0-kube-api-access-2b9pm\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:55:41.980201 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:41.980196    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65414cbc-76b2-4db6-9ceb-df027e3ae9c0-model-cache\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:55:42.631344 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:42.631315    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-rpclz" event={"ID":"65414cbc-76b2-4db6-9ceb-df027e3ae9c0","Type":"ContainerDied","Data":"b1cb2d6a5d8ed3760f8c244dda72d8b2f54862b23350b440811e80c747dedda1"}
Jul 07 16:55:42.631344 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:42.631348    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-rpclz"
Jul 07 16:55:42.631344 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:42.631354    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1cb2d6a5d8ed3760f8c244dda72d8b2f54862b23350b440811e80c747dedda1"
Jul 07 16:55:54.749440 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:54.749409    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c"]
Jul 07 16:55:54.749821 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:54.749706    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65414cbc-76b2-4db6-9ceb-df027e3ae9c0" containerName="s3-init"
Jul 07 16:55:54.749821 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:54.749716    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="65414cbc-76b2-4db6-9ceb-df027e3ae9c0" containerName="s3-init"
Jul 07 16:55:54.749821 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:54.749728    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65414cbc-76b2-4db6-9ceb-df027e3ae9c0" containerName="download-hf-model"
Jul 07 16:55:54.749821 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:54.749747    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="65414cbc-76b2-4db6-9ceb-df027e3ae9c0" containerName="download-hf-model"
Jul 07 16:55:54.749821 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:54.749797    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="65414cbc-76b2-4db6-9ceb-df027e3ae9c0" containerName="s3-init"
Jul 07 16:55:54.753032 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:54.753018    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c"
Jul 07 16:55:54.755824 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:54.755804    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 07 16:55:54.755937 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:54.755842    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-predictor-serving-cert\""
Jul 07 16:55:54.755937 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:54.755859    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\""
Jul 07 16:55:54.755937 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:54.755851    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 07 16:55:54.755937 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:54.755908    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 07 16:55:54.756220 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:54.756207    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 07 16:55:54.757015 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:54.757001    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lvhm6\""
Jul 07 16:55:54.765231 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:54.765212    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c"]
Jul 07 16:55:54.779988 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:54.779968    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw42s\" (UniqueName: \"kubernetes.io/projected/06d394d3-ceb2-4d89-ab7d-46cc9a6f299c-kube-api-access-lw42s\") pod \"isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c\" (UID: \"06d394d3-ceb2-4d89-ab7d-46cc9a6f299c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c"
Jul 07 16:55:54.780088 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:54.780017    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06d394d3-ceb2-4d89-ab7d-46cc9a6f299c-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c\" (UID: \"06d394d3-ceb2-4d89-ab7d-46cc9a6f299c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c"
Jul 07 16:55:54.780088 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:54.780056    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06d394d3-ceb2-4d89-ab7d-46cc9a6f299c-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c\" (UID: \"06d394d3-ceb2-4d89-ab7d-46cc9a6f299c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c"
Jul 07 16:55:54.780217 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:54.780086    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06d394d3-ceb2-4d89-ab7d-46cc9a6f299c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c\" (UID: \"06d394d3-ceb2-4d89-ab7d-46cc9a6f299c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c"
Jul 07 16:55:54.780217 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:54.780154    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06d394d3-ceb2-4d89-ab7d-46cc9a6f299c-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c\" (UID: \"06d394d3-ceb2-4d89-ab7d-46cc9a6f299c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c"
Jul 07 16:55:54.880527 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:54.880501    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw42s\" (UniqueName: \"kubernetes.io/projected/06d394d3-ceb2-4d89-ab7d-46cc9a6f299c-kube-api-access-lw42s\") pod \"isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c\" (UID: \"06d394d3-ceb2-4d89-ab7d-46cc9a6f299c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c"
Jul 07 16:55:54.880675 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:54.880540    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06d394d3-ceb2-4d89-ab7d-46cc9a6f299c-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c\" (UID: \"06d394d3-ceb2-4d89-ab7d-46cc9a6f299c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c"
Jul 07 16:55:54.880675 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:54.880562    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06d394d3-ceb2-4d89-ab7d-46cc9a6f299c-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c\" (UID: \"06d394d3-ceb2-4d89-ab7d-46cc9a6f299c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c"
Jul 07 16:55:54.880675 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:54.880593    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06d394d3-ceb2-4d89-ab7d-46cc9a6f299c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c\" (UID: \"06d394d3-ceb2-4d89-ab7d-46cc9a6f299c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c"
Jul 07 16:55:54.880675 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:54.880639    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06d394d3-ceb2-4d89-ab7d-46cc9a6f299c-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c\" (UID: \"06d394d3-ceb2-4d89-ab7d-46cc9a6f299c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c"
Jul 07 16:55:54.880675 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:55:54.880666    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-serving-cert: secret "isvc-sklearn-graph-1-predictor-serving-cert" not found
Jul 07 16:55:54.880971 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:55:54.880754    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06d394d3-ceb2-4d89-ab7d-46cc9a6f299c-proxy-tls podName:06d394d3-ceb2-4d89-ab7d-46cc9a6f299c nodeName:}" failed. No retries permitted until 2026-07-07 16:55:55.380717698 +0000 UTC m=+492.157031882 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/06d394d3-ceb2-4d89-ab7d-46cc9a6f299c-proxy-tls") pod "isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c" (UID: "06d394d3-ceb2-4d89-ab7d-46cc9a6f299c") : secret "isvc-sklearn-graph-1-predictor-serving-cert" not found
Jul 07 16:55:54.881057 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:54.881040    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06d394d3-ceb2-4d89-ab7d-46cc9a6f299c-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c\" (UID: \"06d394d3-ceb2-4d89-ab7d-46cc9a6f299c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c"
Jul 07 16:55:54.881218 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:54.881201    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06d394d3-ceb2-4d89-ab7d-46cc9a6f299c-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c\" (UID: \"06d394d3-ceb2-4d89-ab7d-46cc9a6f299c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c"
Jul 07 16:55:54.881308 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:54.881290    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06d394d3-ceb2-4d89-ab7d-46cc9a6f299c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c\" (UID: \"06d394d3-ceb2-4d89-ab7d-46cc9a6f299c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c"
Jul 07 16:55:54.888840 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:54.888817    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw42s\" (UniqueName: \"kubernetes.io/projected/06d394d3-ceb2-4d89-ab7d-46cc9a6f299c-kube-api-access-lw42s\") pod \"isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c\" (UID: \"06d394d3-ceb2-4d89-ab7d-46cc9a6f299c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c"
Jul 07 16:55:55.038395 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:55.038320    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j884b"]
Jul 07 16:55:55.041856 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:55.041840    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j884b"
Jul 07 16:55:55.044617 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:55.044593    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-predictor-serving-cert\""
Jul 07 16:55:55.044617 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:55.044599    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-kube-rbac-proxy-sar-config\""
Jul 07 16:55:55.051244 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:55.051223    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j884b"]
Jul 07 16:55:55.082543 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:55.082520    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78832dba-ce1d-49cd-98f0-aaf5fd68a882-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-j884b\" (UID: \"78832dba-ce1d-49cd-98f0-aaf5fd68a882\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j884b"
Jul 07 16:55:55.082682 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:55.082556    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/78832dba-ce1d-49cd-98f0-aaf5fd68a882-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-j884b\" (UID: \"78832dba-ce1d-49cd-98f0-aaf5fd68a882\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j884b"
Jul 07 16:55:55.082682 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:55.082582    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78832dba-ce1d-49cd-98f0-aaf5fd68a882-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-j884b\" (UID: \"78832dba-ce1d-49cd-98f0-aaf5fd68a882\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j884b"
Jul 07 16:55:55.082790 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:55.082677    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sp52\" (UniqueName: \"kubernetes.io/projected/78832dba-ce1d-49cd-98f0-aaf5fd68a882-kube-api-access-6sp52\") pod \"isvc-xgboost-graph-predictor-555fc69d56-j884b\" (UID: \"78832dba-ce1d-49cd-98f0-aaf5fd68a882\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j884b"
Jul 07 16:55:55.082790 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:55.082726    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78832dba-ce1d-49cd-98f0-aaf5fd68a882-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-j884b\" (UID: \"78832dba-ce1d-49cd-98f0-aaf5fd68a882\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j884b"
Jul 07 16:55:55.134602 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:55.134574    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ae852-predictor-585cbdcfd8-ncnq7"]
Jul 07 16:55:55.137741 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:55.137716    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ae852-predictor-585cbdcfd8-ncnq7"
Jul 07 16:55:55.140514 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:55.140489    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-ae852-predictor-serving-cert\""
Jul 07 16:55:55.140514 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:55.140492    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-ae852-kube-rbac-proxy-sar-config\""
Jul 07 16:55:55.147674 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:55.147655    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ae852-predictor-585cbdcfd8-ncnq7"]
Jul 07 16:55:55.183167 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:55.183144    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/78832dba-ce1d-49cd-98f0-aaf5fd68a882-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-j884b\" (UID: \"78832dba-ce1d-49cd-98f0-aaf5fd68a882\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j884b"
Jul 07 16:55:55.183278 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:55.183182    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78832dba-ce1d-49cd-98f0-aaf5fd68a882-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-j884b\" (UID: \"78832dba-ce1d-49cd-98f0-aaf5fd68a882\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j884b"
Jul 07 16:55:55.183278 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:55.183219    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rwxt\" (UniqueName: \"kubernetes.io/projected/d354b738-62f3-4850-9f1c-f6d5d2febb90-kube-api-access-2rwxt\") pod \"error-404-isvc-ae852-predictor-585cbdcfd8-ncnq7\" (UID: \"d354b738-62f3-4850-9f1c-f6d5d2febb90\") " pod="kserve-ci-e2e-test/error-404-isvc-ae852-predictor-585cbdcfd8-ncnq7"
Jul 07 16:55:55.183278 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:55.183260    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sp52\" (UniqueName: \"kubernetes.io/projected/78832dba-ce1d-49cd-98f0-aaf5fd68a882-kube-api-access-6sp52\") pod \"isvc-xgboost-graph-predictor-555fc69d56-j884b\" (UID: \"78832dba-ce1d-49cd-98f0-aaf5fd68a882\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j884b"
Jul 07 16:55:55.183423 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:55.183314    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d354b738-62f3-4850-9f1c-f6d5d2febb90-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-ae852-predictor-585cbdcfd8-ncnq7\" (UID: \"d354b738-62f3-4850-9f1c-f6d5d2febb90\") " pod="kserve-ci-e2e-test/error-404-isvc-ae852-predictor-585cbdcfd8-ncnq7"
Jul 07 16:55:55.183423 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:55.183341    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78832dba-ce1d-49cd-98f0-aaf5fd68a882-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-j884b\" (UID: \"78832dba-ce1d-49cd-98f0-aaf5fd68a882\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j884b"
Jul 07 16:55:55.183423 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:55.183413    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d354b738-62f3-4850-9f1c-f6d5d2febb90-proxy-tls\") pod \"error-404-isvc-ae852-predictor-585cbdcfd8-ncnq7\" (UID: \"d354b738-62f3-4850-9f1c-f6d5d2febb90\") " pod="kserve-ci-e2e-test/error-404-isvc-ae852-predictor-585cbdcfd8-ncnq7"
Jul 07 16:55:55.183575 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:55.183452    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78832dba-ce1d-49cd-98f0-aaf5fd68a882-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-j884b\" (UID: \"78832dba-ce1d-49cd-98f0-aaf5fd68a882\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j884b"
Jul 07 16:55:55.183857 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:55.183829    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78832dba-ce1d-49cd-98f0-aaf5fd68a882-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-j884b\" (UID: \"78832dba-ce1d-49cd-98f0-aaf5fd68a882\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j884b"
Jul 07 16:55:55.183961 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:55.183922    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78832dba-ce1d-49cd-98f0-aaf5fd68a882-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-j884b\" (UID: \"78832dba-ce1d-49cd-98f0-aaf5fd68a882\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j884b"
Jul 07 16:55:55.183961 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:55.183927    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/78832dba-ce1d-49cd-98f0-aaf5fd68a882-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-j884b\" (UID: \"78832dba-ce1d-49cd-98f0-aaf5fd68a882\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j884b"
Jul 07 16:55:55.185673 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:55.185654    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78832dba-ce1d-49cd-98f0-aaf5fd68a882-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-j884b\" (UID: \"78832dba-ce1d-49cd-98f0-aaf5fd68a882\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j884b"
Jul 07 16:55:55.193821 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:55.193802    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sp52\" (UniqueName: \"kubernetes.io/projected/78832dba-ce1d-49cd-98f0-aaf5fd68a882-kube-api-access-6sp52\") pod \"isvc-xgboost-graph-predictor-555fc69d56-j884b\" (UID: \"78832dba-ce1d-49cd-98f0-aaf5fd68a882\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j884b"
Jul 07 16:55:55.238782 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:55.238759    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c"]
Jul 07 16:55:55.242306 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:55.242292    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c"
Jul 07 16:55:55.244897 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:55.244881    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-predictor-serving-cert\""
Jul 07 16:55:55.244976 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:55.244929    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\""
Jul 07 16:55:55.252854 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:55.252834    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c"]
Jul 07 16:55:55.284726 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:55.284702    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2131e33-d451-46cd-bf08-3f122bb41553-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c\" (UID: \"b2131e33-d451-46cd-bf08-3f122bb41553\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c"
Jul 07 16:55:55.284835 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:55.284750    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2131e33-d451-46cd-bf08-3f122bb41553-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c\" (UID: \"b2131e33-d451-46cd-bf08-3f122bb41553\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c"
Jul 07 16:55:55.284835 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:55.284796    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rwxt\" (UniqueName: \"kubernetes.io/projected/d354b738-62f3-4850-9f1c-f6d5d2febb90-kube-api-access-2rwxt\") pod \"error-404-isvc-ae852-predictor-585cbdcfd8-ncnq7\" (UID: \"d354b738-62f3-4850-9f1c-f6d5d2febb90\") " pod="kserve-ci-e2e-test/error-404-isvc-ae852-predictor-585cbdcfd8-ncnq7"
Jul 07 16:55:55.284904 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:55.284840    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2131e33-d451-46cd-bf08-3f122bb41553-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c\" (UID: \"b2131e33-d451-46cd-bf08-3f122bb41553\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c"
Jul 07 16:55:55.284904 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:55.284870    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27s44\" (UniqueName: \"kubernetes.io/projected/b2131e33-d451-46cd-bf08-3f122bb41553-kube-api-access-27s44\") pod \"isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c\" (UID: \"b2131e33-d451-46cd-bf08-3f122bb41553\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c"
Jul 07 16:55:55.284973 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:55.284904    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d354b738-62f3-4850-9f1c-f6d5d2febb90-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-ae852-predictor-585cbdcfd8-ncnq7\" (UID: \"d354b738-62f3-4850-9f1c-f6d5d2febb90\") " pod="kserve-ci-e2e-test/error-404-isvc-ae852-predictor-585cbdcfd8-ncnq7"
Jul 07 16:55:55.284973 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:55.284965    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d354b738-62f3-4850-9f1c-f6d5d2febb90-proxy-tls\") pod \"error-404-isvc-ae852-predictor-585cbdcfd8-ncnq7\" (UID: \"d354b738-62f3-4850-9f1c-f6d5d2febb90\") " pod="kserve-ci-e2e-test/error-404-isvc-ae852-predictor-585cbdcfd8-ncnq7"
Jul 07 16:55:55.285048 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:55.284987    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2131e33-d451-46cd-bf08-3f122bb41553-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c\" (UID: \"b2131e33-d451-46cd-bf08-3f122bb41553\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c"
Jul 07 16:55:55.285087 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:55:55.285064    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-ae852-predictor-serving-cert: secret "error-404-isvc-ae852-predictor-serving-cert" not found
Jul 07 16:55:55.285134 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:55:55.285116    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d354b738-62f3-4850-9f1c-f6d5d2febb90-proxy-tls podName:d354b738-62f3-4850-9f1c-f6d5d2febb90 nodeName:}" failed. No retries permitted until 2026-07-07 16:55:55.78510265 +0000 UTC m=+492.561416837 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d354b738-62f3-4850-9f1c-f6d5d2febb90-proxy-tls") pod "error-404-isvc-ae852-predictor-585cbdcfd8-ncnq7" (UID: "d354b738-62f3-4850-9f1c-f6d5d2febb90") : secret "error-404-isvc-ae852-predictor-serving-cert" not found
Jul 07 16:55:55.285444 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:55.285428    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d354b738-62f3-4850-9f1c-f6d5d2febb90-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-ae852-predictor-585cbdcfd8-ncnq7\" (UID: \"d354b738-62f3-4850-9f1c-f6d5d2febb90\") " pod="kserve-ci-e2e-test/error-404-isvc-ae852-predictor-585cbdcfd8-ncnq7"
Jul 07 16:55:55.295987 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:55.295968    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rwxt\" (UniqueName: \"kubernetes.io/projected/d354b738-62f3-4850-9f1c-f6d5d2febb90-kube-api-access-2rwxt\") pod \"error-404-isvc-ae852-predictor-585cbdcfd8-ncnq7\" (UID: \"d354b738-62f3-4850-9f1c-f6d5d2febb90\") " pod="kserve-ci-e2e-test/error-404-isvc-ae852-predictor-585cbdcfd8-ncnq7"
Jul 07 16:55:55.351559 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:55.351535    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j884b"
Jul 07 16:55:55.385601 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:55.385579    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06d394d3-ceb2-4d89-ab7d-46cc9a6f299c-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c\" (UID: \"06d394d3-ceb2-4d89-ab7d-46cc9a6f299c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c"
Jul 07 16:55:55.385710 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:55.385620    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2131e33-d451-46cd-bf08-3f122bb41553-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c\" (UID: \"b2131e33-d451-46cd-bf08-3f122bb41553\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c"
Jul 07 16:55:55.385710 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:55.385663    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2131e33-d451-46cd-bf08-3f122bb41553-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c\" (UID: \"b2131e33-d451-46cd-bf08-3f122bb41553\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c"
Jul 07 16:55:55.385710 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:55.385690    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2131e33-d451-46cd-bf08-3f122bb41553-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c\" (UID: \"b2131e33-d451-46cd-bf08-3f122bb41553\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c"
Jul 07 16:55:55.385885 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:55.385725    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2131e33-d451-46cd-bf08-3f122bb41553-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c\" (UID: \"b2131e33-d451-46cd-bf08-3f122bb41553\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c"
Jul 07 16:55:55.385885 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:55.385795    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27s44\" (UniqueName: \"kubernetes.io/projected/b2131e33-d451-46cd-bf08-3f122bb41553-kube-api-access-27s44\") pod \"isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c\" (UID: \"b2131e33-d451-46cd-bf08-3f122bb41553\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c"
Jul 07 16:55:55.385885 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:55:55.385803    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-serving-cert: secret "isvc-sklearn-graph-2-predictor-serving-cert" not found
Jul 07 16:55:55.385885 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:55:55.385873    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2131e33-d451-46cd-bf08-3f122bb41553-proxy-tls podName:b2131e33-d451-46cd-bf08-3f122bb41553 nodeName:}" failed. No retries permitted until 2026-07-07 16:55:55.88585329 +0000 UTC m=+492.662167475 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b2131e33-d451-46cd-bf08-3f122bb41553-proxy-tls") pod "isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c" (UID: "b2131e33-d451-46cd-bf08-3f122bb41553") : secret "isvc-sklearn-graph-2-predictor-serving-cert" not found
Jul 07 16:55:55.386230 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:55.386197    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2131e33-d451-46cd-bf08-3f122bb41553-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c\" (UID: \"b2131e33-d451-46cd-bf08-3f122bb41553\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c"
Jul 07 16:55:55.386365 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:55.386346    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2131e33-d451-46cd-bf08-3f122bb41553-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c\" (UID: \"b2131e33-d451-46cd-bf08-3f122bb41553\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c"
Jul 07 16:55:55.386444 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:55.386425    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2131e33-d451-46cd-bf08-3f122bb41553-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c\" (UID: \"b2131e33-d451-46cd-bf08-3f122bb41553\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c"
Jul 07 16:55:55.388507 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:55.388483    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06d394d3-ceb2-4d89-ab7d-46cc9a6f299c-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c\" (UID: \"06d394d3-ceb2-4d89-ab7d-46cc9a6f299c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c"
Jul 07 16:55:55.393936 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:55.393918    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27s44\" (UniqueName: \"kubernetes.io/projected/b2131e33-d451-46cd-bf08-3f122bb41553-kube-api-access-27s44\") pod \"isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c\" (UID: \"b2131e33-d451-46cd-bf08-3f122bb41553\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c"
Jul 07 16:55:55.476252 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:55.476228    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j884b"]
Jul 07 16:55:55.478728 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:55:55.478706    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78832dba_ce1d_49cd_98f0_aaf5fd68a882.slice/crio-590032dfd7e8577fb5824fa3c0310716e9b583dd7d4d85f0d0c766a5e0fdfb24 WatchSource:0}: Error finding container 590032dfd7e8577fb5824fa3c0310716e9b583dd7d4d85f0d0c766a5e0fdfb24: Status 404 returned error can't find the container with id 590032dfd7e8577fb5824fa3c0310716e9b583dd7d4d85f0d0c766a5e0fdfb24
Jul 07 16:55:55.662845 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:55.662727    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c"
Jul 07 16:55:55.682758 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:55.682711    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j884b" event={"ID":"78832dba-ce1d-49cd-98f0-aaf5fd68a882","Type":"ContainerStarted","Data":"df1a51e793742ed94166a8d943d13ab9636cb72b808376bf9c712c5c6a442d3a"}
Jul 07 16:55:55.682881 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:55.682803    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j884b" event={"ID":"78832dba-ce1d-49cd-98f0-aaf5fd68a882","Type":"ContainerStarted","Data":"590032dfd7e8577fb5824fa3c0310716e9b583dd7d4d85f0d0c766a5e0fdfb24"}
Jul 07 16:55:55.789753 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:55.789701    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d354b738-62f3-4850-9f1c-f6d5d2febb90-proxy-tls\") pod \"error-404-isvc-ae852-predictor-585cbdcfd8-ncnq7\" (UID: \"d354b738-62f3-4850-9f1c-f6d5d2febb90\") " pod="kserve-ci-e2e-test/error-404-isvc-ae852-predictor-585cbdcfd8-ncnq7"
Jul 07 16:55:55.791240 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:55.791220    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c"]
Jul 07 16:55:55.792081 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:55.792063    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d354b738-62f3-4850-9f1c-f6d5d2febb90-proxy-tls\") pod \"error-404-isvc-ae852-predictor-585cbdcfd8-ncnq7\" (UID: \"d354b738-62f3-4850-9f1c-f6d5d2febb90\") " pod="kserve-ci-e2e-test/error-404-isvc-ae852-predictor-585cbdcfd8-ncnq7"
Jul 07 16:55:55.794136 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:55:55.794114    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06d394d3_ceb2_4d89_ab7d_46cc9a6f299c.slice/crio-72a6e1acf2b46b1e537019cc85a4f6e83bba1c8619d52fdbed6bc197ad7199d1 WatchSource:0}: Error finding container 72a6e1acf2b46b1e537019cc85a4f6e83bba1c8619d52fdbed6bc197ad7199d1: Status 404 returned error can't find the container with id 72a6e1acf2b46b1e537019cc85a4f6e83bba1c8619d52fdbed6bc197ad7199d1
Jul 07 16:55:55.890820 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:55.890785    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2131e33-d451-46cd-bf08-3f122bb41553-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c\" (UID: \"b2131e33-d451-46cd-bf08-3f122bb41553\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c"
Jul 07 16:55:55.892966 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:55.892938    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2131e33-d451-46cd-bf08-3f122bb41553-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c\" (UID: \"b2131e33-d451-46cd-bf08-3f122bb41553\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c"
Jul 07 16:55:56.047978 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:56.047949    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ae852-predictor-585cbdcfd8-ncnq7"
Jul 07 16:55:56.153148 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:56.153118    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c"
Jul 07 16:55:56.182931 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:56.182907    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ae852-predictor-585cbdcfd8-ncnq7"]
Jul 07 16:55:56.184424 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:55:56.184399    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd354b738_62f3_4850_9f1c_f6d5d2febb90.slice/crio-e3bd1b4186c358bc239fb669fa70321b84dcc86bd27ba54d4302062a269ac202 WatchSource:0}: Error finding container e3bd1b4186c358bc239fb669fa70321b84dcc86bd27ba54d4302062a269ac202: Status 404 returned error can't find the container with id e3bd1b4186c358bc239fb669fa70321b84dcc86bd27ba54d4302062a269ac202
Jul 07 16:55:56.286087 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:56.286059    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c"]
Jul 07 16:55:56.287083 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:55:56.287044    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2131e33_d451_46cd_bf08_3f122bb41553.slice/crio-b579cb72a284d01b2063f4c01a47f08c2b60824ad7f4fc627ad680ffa716010b WatchSource:0}: Error finding container b579cb72a284d01b2063f4c01a47f08c2b60824ad7f4fc627ad680ffa716010b: Status 404 returned error can't find the container with id b579cb72a284d01b2063f4c01a47f08c2b60824ad7f4fc627ad680ffa716010b
Jul 07 16:55:56.688067 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:56.687976    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c" event={"ID":"06d394d3-ceb2-4d89-ab7d-46cc9a6f299c","Type":"ContainerStarted","Data":"7535bfa7037261c681a9c5985287c8b97e9b2c45f243bd78a9a95ee64c51e86a"}
Jul 07 16:55:56.688229 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:56.688083    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c" event={"ID":"06d394d3-ceb2-4d89-ab7d-46cc9a6f299c","Type":"ContainerStarted","Data":"72a6e1acf2b46b1e537019cc85a4f6e83bba1c8619d52fdbed6bc197ad7199d1"}
Jul 07 16:55:56.689199 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:56.689170    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ae852-predictor-585cbdcfd8-ncnq7" event={"ID":"d354b738-62f3-4850-9f1c-f6d5d2febb90","Type":"ContainerStarted","Data":"e3bd1b4186c358bc239fb669fa70321b84dcc86bd27ba54d4302062a269ac202"}
Jul 07 16:55:56.690702 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:56.690678    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c" event={"ID":"b2131e33-d451-46cd-bf08-3f122bb41553","Type":"ContainerStarted","Data":"5367576d7eb6b3816d8ab0a4963872bc0ab6f3556aefd7f7f875849255ceb86b"}
Jul 07 16:55:56.690815 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:55:56.690726    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c" event={"ID":"b2131e33-d451-46cd-bf08-3f122bb41553","Type":"ContainerStarted","Data":"b579cb72a284d01b2063f4c01a47f08c2b60824ad7f4fc627ad680ffa716010b"}
Jul 07 16:56:01.721224 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:56:01.721195    2578 generic.go:356] "Generic (PLEG): container finished" podID="b2131e33-d451-46cd-bf08-3f122bb41553" containerID="5367576d7eb6b3816d8ab0a4963872bc0ab6f3556aefd7f7f875849255ceb86b" exitCode=0
Jul 07 16:56:01.722691 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:56:01.722670    2578 generic.go:356] "Generic (PLEG): container finished" podID="78832dba-ce1d-49cd-98f0-aaf5fd68a882" containerID="df1a51e793742ed94166a8d943d13ab9636cb72b808376bf9c712c5c6a442d3a" exitCode=0
Jul 07 16:56:01.724042 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:56:01.724021    2578 generic.go:356] "Generic (PLEG): container finished" podID="06d394d3-ceb2-4d89-ab7d-46cc9a6f299c" containerID="7535bfa7037261c681a9c5985287c8b97e9b2c45f243bd78a9a95ee64c51e86a" exitCode=0
Jul 07 16:56:01.724425 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:56:01.724401    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c" event={"ID":"b2131e33-d451-46cd-bf08-3f122bb41553","Type":"ContainerDied","Data":"5367576d7eb6b3816d8ab0a4963872bc0ab6f3556aefd7f7f875849255ceb86b"}
Jul 07 16:56:01.724503 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:56:01.724466    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j884b" event={"ID":"78832dba-ce1d-49cd-98f0-aaf5fd68a882","Type":"ContainerDied","Data":"df1a51e793742ed94166a8d943d13ab9636cb72b808376bf9c712c5c6a442d3a"}
Jul 07 16:56:01.724560 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:56:01.724505    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c" event={"ID":"06d394d3-ceb2-4d89-ab7d-46cc9a6f299c","Type":"ContainerDied","Data":"7535bfa7037261c681a9c5985287c8b97e9b2c45f243bd78a9a95ee64c51e86a"}
Jul 07 16:56:15.808683 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:56:15.808650    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ae852-predictor-585cbdcfd8-ncnq7" event={"ID":"d354b738-62f3-4850-9f1c-f6d5d2febb90","Type":"ContainerStarted","Data":"ba17662726074a05616beb8d2981824f98a9c806d1a8a1e0f8eaf15de7ca94a8"}
Jul 07 16:56:15.810610 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:56:15.810548    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c" event={"ID":"b2131e33-d451-46cd-bf08-3f122bb41553","Type":"ContainerStarted","Data":"9eecdbf6abac693170ffee3a9249443c11916af56ea916dd9096d93c68ce2329"}
Jul 07 16:56:15.812765 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:56:15.812718    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c" event={"ID":"06d394d3-ceb2-4d89-ab7d-46cc9a6f299c","Type":"ContainerStarted","Data":"c88b4aa99079c3e62f7cccc9d84337ac3f695a54e8b58af399079ca6406e14e1"}
Jul 07 16:56:19.844163 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:56:19.844121    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c" event={"ID":"06d394d3-ceb2-4d89-ab7d-46cc9a6f299c","Type":"ContainerStarted","Data":"8c291739ea67673b2dfbf5b1c6fdbddf02ddf8002a1cf4ae8c5ad6547711f9c7"}
Jul 07 16:56:19.844630 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:56:19.844232    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c"
Jul 07 16:56:19.845939 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:56:19.845914    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ae852-predictor-585cbdcfd8-ncnq7" event={"ID":"d354b738-62f3-4850-9f1c-f6d5d2febb90","Type":"ContainerStarted","Data":"c5a530b85d53f388af6e6c3a42c2b13c6ad88d132eadb480819a58335acf5cbd"}
Jul 07 16:56:19.846061 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:56:19.845963    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-ae852-predictor-585cbdcfd8-ncnq7"
Jul 07 16:56:19.846061 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:56:19.845984    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-ae852-predictor-585cbdcfd8-ncnq7"
Jul 07 16:56:19.847258 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:56:19.847217    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ae852-predictor-585cbdcfd8-ncnq7" podUID="d354b738-62f3-4850-9f1c-f6d5d2febb90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 16:56:19.848844 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:56:19.848347    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c" event={"ID":"b2131e33-d451-46cd-bf08-3f122bb41553","Type":"ContainerStarted","Data":"bf2954c22b1aee2cfc4bd2af22b492bf344f30321431e03c242f7da81341f303"}
Jul 07 16:56:19.848844 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:56:19.848470    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c"
Jul 07 16:56:19.864516 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:56:19.864478    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c" podStartSLOduration=4.5100017569999995 podStartE2EDuration="25.864465334s" totalImagesPullingTime="17.662268357s" totalInitContainerRuntime="3.69219522s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:55:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:56:19.863720206 +0000 UTC m=+516.640034412" watchObservedRunningTime="2026-07-07 16:56:19.864465334 +0000 UTC m=+516.640779539"
Jul 07 16:56:19.888718 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:56:19.888681    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-ae852-predictor-585cbdcfd8-ncnq7" podStartSLOduration=1.6799897000000001 podStartE2EDuration="24.888669089s" totalImagesPullingTime="23.208679389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:55:55 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:56:19.88741727 +0000 UTC m=+516.663731477" watchObservedRunningTime="2026-07-07 16:56:19.888669089 +0000 UTC m=+516.664983294"
Jul 07 16:56:19.908217 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:56:19.908173    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c" podStartSLOduration=3.195274585 podStartE2EDuration="24.908160293s" totalImagesPullingTime="17.701349274s" totalInitContainerRuntime="4.011536434s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:55:55 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:56:19.906271327 +0000 UTC m=+516.682585545" watchObservedRunningTime="2026-07-07 16:56:19.908160293 +0000 UTC m=+516.684474499"
Jul 07 16:56:20.854373 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:56:20.854331    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ae852-predictor-585cbdcfd8-ncnq7" podUID="d354b738-62f3-4850-9f1c-f6d5d2febb90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 16:56:20.854900 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:56:20.854776    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c"
Jul 07 16:56:20.854900 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:56:20.854825    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c"
Jul 07 16:56:20.855476 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:56:20.855443    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c" podUID="b2131e33-d451-46cd-bf08-3f122bb41553" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 16:56:20.855603 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:56:20.855581    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c" podUID="06d394d3-ceb2-4d89-ab7d-46cc9a6f299c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 07 16:56:21.857560 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:56:21.857523    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c" podUID="06d394d3-ceb2-4d89-ab7d-46cc9a6f299c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 07 16:56:21.857972 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:56:21.857665    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c" podUID="b2131e33-d451-46cd-bf08-3f122bb41553" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 16:56:25.860897 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:56:25.860861    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-ae852-predictor-585cbdcfd8-ncnq7"
Jul 07 16:56:25.861466 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:56:25.861438    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ae852-predictor-585cbdcfd8-ncnq7" podUID="d354b738-62f3-4850-9f1c-f6d5d2febb90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 16:56:26.862698 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:56:26.862673    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c"
Jul 07 16:56:26.863149 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:56:26.863064    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c"
Jul 07 16:56:26.863149 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:56:26.863132    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c" podUID="06d394d3-ceb2-4d89-ab7d-46cc9a6f299c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 07 16:56:26.863564 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:56:26.863517    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c" podUID="b2131e33-d451-46cd-bf08-3f122bb41553" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 16:56:35.861962 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:56:35.861874    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ae852-predictor-585cbdcfd8-ncnq7" podUID="d354b738-62f3-4850-9f1c-f6d5d2febb90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 16:56:36.863821 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:56:36.863724    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c" podUID="06d394d3-ceb2-4d89-ab7d-46cc9a6f299c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 07 16:56:36.863821 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:56:36.863762    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c" podUID="b2131e33-d451-46cd-bf08-3f122bb41553" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 16:56:36.928291 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:56:36.928247    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j884b" event={"ID":"78832dba-ce1d-49cd-98f0-aaf5fd68a882","Type":"ContainerStarted","Data":"3e8fbb6844cd7d584a374fcf875a03da72ea4b9330973c9ee626cf4928cbab49"}
Jul 07 16:56:36.928461 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:56:36.928331    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j884b" event={"ID":"78832dba-ce1d-49cd-98f0-aaf5fd68a882","Type":"ContainerStarted","Data":"a07247ee6d9a3ee667239885cfcf77c1c0c68a48588a2a3457841bd4a3540595"}
Jul 07 16:56:36.928622 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:56:36.928557    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j884b"
Jul 07 16:56:36.949037 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:56:36.948989    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j884b" podStartSLOduration=3.369748167 podStartE2EDuration="41.948974298s" totalImagesPullingTime="34.818727118s" totalInitContainerRuntime="3.760499013s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:55:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:56:36.948213415 +0000 UTC m=+533.724527623" watchObservedRunningTime="2026-07-07 16:56:36.948974298 +0000 UTC m=+533.725288575"
Jul 07 16:56:37.935109 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:56:37.935028    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j884b"
Jul 07 16:56:37.936522 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:56:37.936489    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j884b" podUID="78832dba-ce1d-49cd-98f0-aaf5fd68a882" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 07 16:56:38.939720 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:56:38.939686    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j884b" podUID="78832dba-ce1d-49cd-98f0-aaf5fd68a882" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 07 16:56:43.944302 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:56:43.944270    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j884b"
Jul 07 16:56:43.944801 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:56:43.944773    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j884b" podUID="78832dba-ce1d-49cd-98f0-aaf5fd68a882" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 07 16:56:45.861544 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:56:45.861505    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ae852-predictor-585cbdcfd8-ncnq7" podUID="d354b738-62f3-4850-9f1c-f6d5d2febb90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 16:56:46.863307 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:56:46.863267    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c" podUID="06d394d3-ceb2-4d89-ab7d-46cc9a6f299c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 07 16:56:46.863716 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:56:46.863402    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c" podUID="b2131e33-d451-46cd-bf08-3f122bb41553" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 16:56:53.945626 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:56:53.945585    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j884b" podUID="78832dba-ce1d-49cd-98f0-aaf5fd68a882" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 07 16:56:55.861935 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:56:55.861901    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ae852-predictor-585cbdcfd8-ncnq7" podUID="d354b738-62f3-4850-9f1c-f6d5d2febb90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 16:56:56.863804 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:56:56.863768    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c" podUID="06d394d3-ceb2-4d89-ab7d-46cc9a6f299c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 07 16:56:56.863804 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:56:56.863801    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c" podUID="b2131e33-d451-46cd-bf08-3f122bb41553" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 16:57:03.944881 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:03.944837    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j884b" podUID="78832dba-ce1d-49cd-98f0-aaf5fd68a882" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 07 16:57:05.862112 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:05.862077    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ae852-predictor-585cbdcfd8-ncnq7" podUID="d354b738-62f3-4850-9f1c-f6d5d2febb90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 16:57:06.863130 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:06.863092    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c" podUID="06d394d3-ceb2-4d89-ab7d-46cc9a6f299c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 07 16:57:06.863600 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:06.863550    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c" podUID="b2131e33-d451-46cd-bf08-3f122bb41553" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 16:57:13.945690 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:13.945650    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j884b" podUID="78832dba-ce1d-49cd-98f0-aaf5fd68a882" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 07 16:57:15.862305 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:15.862274    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-ae852-predictor-585cbdcfd8-ncnq7"
Jul 07 16:57:16.863547 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:16.863513    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c" podUID="b2131e33-d451-46cd-bf08-3f122bb41553" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 16:57:16.863949 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:16.863514    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c" podUID="06d394d3-ceb2-4d89-ab7d-46cc9a6f299c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 07 16:57:23.945419 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:23.945376    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j884b" podUID="78832dba-ce1d-49cd-98f0-aaf5fd68a882" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 07 16:57:26.863523 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:26.863489    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c" podUID="06d394d3-ceb2-4d89-ab7d-46cc9a6f299c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 07 16:57:26.863937 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:26.863552    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c" podUID="b2131e33-d451-46cd-bf08-3f122bb41553" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 16:57:27.720477 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:27.720437    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c" podUID="06d394d3-ceb2-4d89-ab7d-46cc9a6f299c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 07 16:57:33.944706 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:33.944671    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j884b" podUID="78832dba-ce1d-49cd-98f0-aaf5fd68a882" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 07 16:57:36.863922 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:36.863887    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c"
Jul 07 16:57:37.724941 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:37.724913    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c"
Jul 07 16:57:39.289681 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:39.289649    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bb9ec-predictor-546444857b-wjtqx"]
Jul 07 16:57:39.291879 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:39.291865    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-bb9ec-predictor-546444857b-wjtqx"
Jul 07 16:57:39.294215 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:39.294200    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-bb9ec-predictor-serving-cert\""
Jul 07 16:57:39.294215 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:39.294206    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-bb9ec-kube-rbac-proxy-sar-config\""
Jul 07 16:57:39.302536 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:39.302512    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bb9ec-predictor-546444857b-wjtqx"]
Jul 07 16:57:39.345308 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:39.345277    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ae852-predictor-585cbdcfd8-ncnq7"]
Jul 07 16:57:39.345548 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:39.345519    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-ae852-predictor-585cbdcfd8-ncnq7" podUID="d354b738-62f3-4850-9f1c-f6d5d2febb90" containerName="kserve-container" containerID="cri-o://ba17662726074a05616beb8d2981824f98a9c806d1a8a1e0f8eaf15de7ca94a8" gracePeriod=30
Jul 07 16:57:39.345663 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:39.345546    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-ae852-predictor-585cbdcfd8-ncnq7" podUID="d354b738-62f3-4850-9f1c-f6d5d2febb90" containerName="kube-rbac-proxy" containerID="cri-o://c5a530b85d53f388af6e6c3a42c2b13c6ad88d132eadb480819a58335acf5cbd" gracePeriod=30
Jul 07 16:57:39.445269 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:39.445239    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b007968f-9b0f-4578-9700-41c6e197d572-proxy-tls\") pod \"error-404-isvc-bb9ec-predictor-546444857b-wjtqx\" (UID: \"b007968f-9b0f-4578-9700-41c6e197d572\") " pod="kserve-ci-e2e-test/error-404-isvc-bb9ec-predictor-546444857b-wjtqx"
Jul 07 16:57:39.445420 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:39.445274    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvxjt\" (UniqueName: \"kubernetes.io/projected/b007968f-9b0f-4578-9700-41c6e197d572-kube-api-access-zvxjt\") pod \"error-404-isvc-bb9ec-predictor-546444857b-wjtqx\" (UID: \"b007968f-9b0f-4578-9700-41c6e197d572\") " pod="kserve-ci-e2e-test/error-404-isvc-bb9ec-predictor-546444857b-wjtqx"
Jul 07 16:57:39.445420 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:39.445374    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b007968f-9b0f-4578-9700-41c6e197d572-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-bb9ec-predictor-546444857b-wjtqx\" (UID: \"b007968f-9b0f-4578-9700-41c6e197d572\") " pod="kserve-ci-e2e-test/error-404-isvc-bb9ec-predictor-546444857b-wjtqx"
Jul 07 16:57:39.546761 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:39.546715    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b007968f-9b0f-4578-9700-41c6e197d572-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-bb9ec-predictor-546444857b-wjtqx\" (UID: \"b007968f-9b0f-4578-9700-41c6e197d572\") " pod="kserve-ci-e2e-test/error-404-isvc-bb9ec-predictor-546444857b-wjtqx"
Jul 07 16:57:39.546910 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:39.546801    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b007968f-9b0f-4578-9700-41c6e197d572-proxy-tls\") pod \"error-404-isvc-bb9ec-predictor-546444857b-wjtqx\" (UID: \"b007968f-9b0f-4578-9700-41c6e197d572\") " pod="kserve-ci-e2e-test/error-404-isvc-bb9ec-predictor-546444857b-wjtqx"
Jul 07 16:57:39.546910 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:39.546825    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvxjt\" (UniqueName: \"kubernetes.io/projected/b007968f-9b0f-4578-9700-41c6e197d572-kube-api-access-zvxjt\") pod \"error-404-isvc-bb9ec-predictor-546444857b-wjtqx\" (UID: \"b007968f-9b0f-4578-9700-41c6e197d572\") " pod="kserve-ci-e2e-test/error-404-isvc-bb9ec-predictor-546444857b-wjtqx"
Jul 07 16:57:39.547366 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:39.547347    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b007968f-9b0f-4578-9700-41c6e197d572-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-bb9ec-predictor-546444857b-wjtqx\" (UID: \"b007968f-9b0f-4578-9700-41c6e197d572\") " pod="kserve-ci-e2e-test/error-404-isvc-bb9ec-predictor-546444857b-wjtqx"
Jul 07 16:57:39.549410 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:39.549387    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b007968f-9b0f-4578-9700-41c6e197d572-proxy-tls\") pod \"error-404-isvc-bb9ec-predictor-546444857b-wjtqx\" (UID: \"b007968f-9b0f-4578-9700-41c6e197d572\") " pod="kserve-ci-e2e-test/error-404-isvc-bb9ec-predictor-546444857b-wjtqx"
Jul 07 16:57:39.555370 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:39.555350    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvxjt\" (UniqueName: \"kubernetes.io/projected/b007968f-9b0f-4578-9700-41c6e197d572-kube-api-access-zvxjt\") pod \"error-404-isvc-bb9ec-predictor-546444857b-wjtqx\" (UID: \"b007968f-9b0f-4578-9700-41c6e197d572\") " pod="kserve-ci-e2e-test/error-404-isvc-bb9ec-predictor-546444857b-wjtqx"
Jul 07 16:57:39.602324 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:39.602296    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-bb9ec-predictor-546444857b-wjtqx"
Jul 07 16:57:39.731984 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:57:39.731954    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb007968f_9b0f_4578_9700_41c6e197d572.slice/crio-a8ce6f2c5cac6c5c38485a4ef1035c95fd2f9aa1dc8d05a2affbb4c44153b1f7 WatchSource:0}: Error finding container a8ce6f2c5cac6c5c38485a4ef1035c95fd2f9aa1dc8d05a2affbb4c44153b1f7: Status 404 returned error can't find the container with id a8ce6f2c5cac6c5c38485a4ef1035c95fd2f9aa1dc8d05a2affbb4c44153b1f7
Jul 07 16:57:39.733584 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:39.733562    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bb9ec-predictor-546444857b-wjtqx"]
Jul 07 16:57:40.193985 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:40.193894    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bb9ec-predictor-546444857b-wjtqx" event={"ID":"b007968f-9b0f-4578-9700-41c6e197d572","Type":"ContainerStarted","Data":"804545ff129d2687d2a04f770bf8d072a6bfa32bd67cc3c00f20539ce128fcc0"}
Jul 07 16:57:40.193985 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:40.193964    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bb9ec-predictor-546444857b-wjtqx" event={"ID":"b007968f-9b0f-4578-9700-41c6e197d572","Type":"ContainerStarted","Data":"a094c8a119bf569c12dcd2e16d4bdcd61a369224cddbb5ec7b52357a5bb7b374"}
Jul 07 16:57:40.194184 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:40.194000    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-bb9ec-predictor-546444857b-wjtqx"
Jul 07 16:57:40.194184 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:40.194060    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bb9ec-predictor-546444857b-wjtqx" event={"ID":"b007968f-9b0f-4578-9700-41c6e197d572","Type":"ContainerStarted","Data":"a8ce6f2c5cac6c5c38485a4ef1035c95fd2f9aa1dc8d05a2affbb4c44153b1f7"}
Jul 07 16:57:40.195631 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:40.195607    2578 generic.go:356] "Generic (PLEG): container finished" podID="d354b738-62f3-4850-9f1c-f6d5d2febb90" containerID="c5a530b85d53f388af6e6c3a42c2b13c6ad88d132eadb480819a58335acf5cbd" exitCode=2
Jul 07 16:57:40.195769 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:40.195639    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ae852-predictor-585cbdcfd8-ncnq7" event={"ID":"d354b738-62f3-4850-9f1c-f6d5d2febb90","Type":"ContainerDied","Data":"c5a530b85d53f388af6e6c3a42c2b13c6ad88d132eadb480819a58335acf5cbd"}
Jul 07 16:57:40.210914 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:40.210871    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-bb9ec-predictor-546444857b-wjtqx" podStartSLOduration=1.210857743 podStartE2EDuration="1.210857743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:57:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:57:40.209040389 +0000 UTC m=+596.985354597" watchObservedRunningTime="2026-07-07 16:57:40.210857743 +0000 UTC m=+596.987172012"
Jul 07 16:57:40.855291 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:40.855251    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ae852-predictor-585cbdcfd8-ncnq7" podUID="d354b738-62f3-4850-9f1c-f6d5d2febb90" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jul 07 16:57:41.200356 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:41.200271    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-bb9ec-predictor-546444857b-wjtqx"
Jul 07 16:57:41.201117 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:41.201088    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bb9ec-predictor-546444857b-wjtqx" podUID="b007968f-9b0f-4578-9700-41c6e197d572" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 07 16:57:42.205037 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:42.204994    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bb9ec-predictor-546444857b-wjtqx" podUID="b007968f-9b0f-4578-9700-41c6e197d572" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 07 16:57:42.491080 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:42.491057    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ae852-predictor-585cbdcfd8-ncnq7"
Jul 07 16:57:42.572675 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:42.572643    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d354b738-62f3-4850-9f1c-f6d5d2febb90-proxy-tls\") pod \"d354b738-62f3-4850-9f1c-f6d5d2febb90\" (UID: \"d354b738-62f3-4850-9f1c-f6d5d2febb90\") "
Jul 07 16:57:42.572828 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:42.572716    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2rwxt\" (UniqueName: \"kubernetes.io/projected/d354b738-62f3-4850-9f1c-f6d5d2febb90-kube-api-access-2rwxt\") pod \"d354b738-62f3-4850-9f1c-f6d5d2febb90\" (UID: \"d354b738-62f3-4850-9f1c-f6d5d2febb90\") "
Jul 07 16:57:42.572828 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:42.572762    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d354b738-62f3-4850-9f1c-f6d5d2febb90-kube-rbac-proxy-sar-config\") pod \"d354b738-62f3-4850-9f1c-f6d5d2febb90\" (UID: \"d354b738-62f3-4850-9f1c-f6d5d2febb90\") "
Jul 07 16:57:42.573160 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:42.573134    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d354b738-62f3-4850-9f1c-f6d5d2febb90-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d354b738-62f3-4850-9f1c-f6d5d2febb90" (UID: "d354b738-62f3-4850-9f1c-f6d5d2febb90"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:57:42.574846 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:42.574823    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d354b738-62f3-4850-9f1c-f6d5d2febb90-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d354b738-62f3-4850-9f1c-f6d5d2febb90" (UID: "d354b738-62f3-4850-9f1c-f6d5d2febb90"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:57:42.574924 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:42.574872    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d354b738-62f3-4850-9f1c-f6d5d2febb90-kube-api-access-2rwxt" (OuterVolumeSpecName: "kube-api-access-2rwxt") pod "d354b738-62f3-4850-9f1c-f6d5d2febb90" (UID: "d354b738-62f3-4850-9f1c-f6d5d2febb90"). InnerVolumeSpecName "kube-api-access-2rwxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:57:42.673590 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:42.673557    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d354b738-62f3-4850-9f1c-f6d5d2febb90-proxy-tls\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:57:42.673590 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:42.673587    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rwxt\" (UniqueName: \"kubernetes.io/projected/d354b738-62f3-4850-9f1c-f6d5d2febb90-kube-api-access-2rwxt\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:57:42.673590 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:42.673597    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d354b738-62f3-4850-9f1c-f6d5d2febb90-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:57:43.210597 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:43.210564    2578 generic.go:356] "Generic (PLEG): container finished" podID="d354b738-62f3-4850-9f1c-f6d5d2febb90" containerID="ba17662726074a05616beb8d2981824f98a9c806d1a8a1e0f8eaf15de7ca94a8" exitCode=0
Jul 07 16:57:43.210996 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:43.210613    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ae852-predictor-585cbdcfd8-ncnq7"
Jul 07 16:57:43.210996 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:43.210644    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ae852-predictor-585cbdcfd8-ncnq7" event={"ID":"d354b738-62f3-4850-9f1c-f6d5d2febb90","Type":"ContainerDied","Data":"ba17662726074a05616beb8d2981824f98a9c806d1a8a1e0f8eaf15de7ca94a8"}
Jul 07 16:57:43.210996 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:43.210712    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ae852-predictor-585cbdcfd8-ncnq7" event={"ID":"d354b738-62f3-4850-9f1c-f6d5d2febb90","Type":"ContainerDied","Data":"e3bd1b4186c358bc239fb669fa70321b84dcc86bd27ba54d4302062a269ac202"}
Jul 07 16:57:43.210996 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:43.210767    2578 scope.go:117] "RemoveContainer" containerID="c5a530b85d53f388af6e6c3a42c2b13c6ad88d132eadb480819a58335acf5cbd"
Jul 07 16:57:43.219662 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:43.219647    2578 scope.go:117] "RemoveContainer" containerID="ba17662726074a05616beb8d2981824f98a9c806d1a8a1e0f8eaf15de7ca94a8"
Jul 07 16:57:43.227141 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:43.227125    2578 scope.go:117] "RemoveContainer" containerID="c5a530b85d53f388af6e6c3a42c2b13c6ad88d132eadb480819a58335acf5cbd"
Jul 07 16:57:43.227408 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:57:43.227387    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5a530b85d53f388af6e6c3a42c2b13c6ad88d132eadb480819a58335acf5cbd\": container with ID starting with c5a530b85d53f388af6e6c3a42c2b13c6ad88d132eadb480819a58335acf5cbd not found: ID does not exist" containerID="c5a530b85d53f388af6e6c3a42c2b13c6ad88d132eadb480819a58335acf5cbd"
Jul 07 16:57:43.227477 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:43.227414    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5a530b85d53f388af6e6c3a42c2b13c6ad88d132eadb480819a58335acf5cbd"} err="failed to get container status \"c5a530b85d53f388af6e6c3a42c2b13c6ad88d132eadb480819a58335acf5cbd\": rpc error: code = NotFound desc = could not find container \"c5a530b85d53f388af6e6c3a42c2b13c6ad88d132eadb480819a58335acf5cbd\": container with ID starting with c5a530b85d53f388af6e6c3a42c2b13c6ad88d132eadb480819a58335acf5cbd not found: ID does not exist"
Jul 07 16:57:43.227477 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:43.227430    2578 scope.go:117] "RemoveContainer" containerID="ba17662726074a05616beb8d2981824f98a9c806d1a8a1e0f8eaf15de7ca94a8"
Jul 07 16:57:43.227680 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:57:43.227650    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba17662726074a05616beb8d2981824f98a9c806d1a8a1e0f8eaf15de7ca94a8\": container with ID starting with ba17662726074a05616beb8d2981824f98a9c806d1a8a1e0f8eaf15de7ca94a8 not found: ID does not exist" containerID="ba17662726074a05616beb8d2981824f98a9c806d1a8a1e0f8eaf15de7ca94a8"
Jul 07 16:57:43.227721 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:43.227689    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba17662726074a05616beb8d2981824f98a9c806d1a8a1e0f8eaf15de7ca94a8"} err="failed to get container status \"ba17662726074a05616beb8d2981824f98a9c806d1a8a1e0f8eaf15de7ca94a8\": rpc error: code = NotFound desc = could not find container \"ba17662726074a05616beb8d2981824f98a9c806d1a8a1e0f8eaf15de7ca94a8\": container with ID starting with ba17662726074a05616beb8d2981824f98a9c806d1a8a1e0f8eaf15de7ca94a8 not found: ID does not exist"
Jul 07 16:57:43.233242 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:43.233217    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ae852-predictor-585cbdcfd8-ncnq7"]
Jul 07 16:57:43.238345 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:43.238314    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ae852-predictor-585cbdcfd8-ncnq7"]
Jul 07 16:57:43.725632 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:43.725601    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d354b738-62f3-4850-9f1c-f6d5d2febb90" path="/var/lib/kubelet/pods/d354b738-62f3-4850-9f1c-f6d5d2febb90/volumes"
Jul 07 16:57:43.946202 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:43.946118    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j884b"
Jul 07 16:57:47.209907 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:47.209880    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-bb9ec-predictor-546444857b-wjtqx"
Jul 07 16:57:47.210356 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:47.210312    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bb9ec-predictor-546444857b-wjtqx" podUID="b007968f-9b0f-4578-9700-41c6e197d572" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 07 16:57:57.210888 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:57:57.210852    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bb9ec-predictor-546444857b-wjtqx" podUID="b007968f-9b0f-4578-9700-41c6e197d572" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 07 16:58:07.210450 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:07.210362    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bb9ec-predictor-546444857b-wjtqx" podUID="b007968f-9b0f-4578-9700-41c6e197d572" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 07 16:58:14.952082 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:14.952050    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c"]
Jul 07 16:58:14.952469 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:14.952285    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c" podUID="06d394d3-ceb2-4d89-ab7d-46cc9a6f299c" containerName="kserve-container" containerID="cri-o://c88b4aa99079c3e62f7cccc9d84337ac3f695a54e8b58af399079ca6406e14e1" gracePeriod=30
Jul 07 16:58:14.952469 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:14.952330    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c" podUID="06d394d3-ceb2-4d89-ab7d-46cc9a6f299c" containerName="kube-rbac-proxy" containerID="cri-o://8c291739ea67673b2dfbf5b1c6fdbddf02ddf8002a1cf4ae8c5ad6547711f9c7" gracePeriod=30
Jul 07 16:58:15.056521 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:15.056481    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j884b"]
Jul 07 16:58:15.056795 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:15.056765    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j884b" podUID="78832dba-ce1d-49cd-98f0-aaf5fd68a882" containerName="kube-rbac-proxy" containerID="cri-o://3e8fbb6844cd7d584a374fcf875a03da72ea4b9330973c9ee626cf4928cbab49" gracePeriod=30
Jul 07 16:58:15.056891 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:15.056725    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j884b" podUID="78832dba-ce1d-49cd-98f0-aaf5fd68a882" containerName="kserve-container" containerID="cri-o://a07247ee6d9a3ee667239885cfcf77c1c0c68a48588a2a3457841bd4a3540595" gracePeriod=30
Jul 07 16:58:15.088667 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:15.088636    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-23477-predictor-747f9595bd-qjnx8"]
Jul 07 16:58:15.089043 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:15.089026    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d354b738-62f3-4850-9f1c-f6d5d2febb90" containerName="kserve-container"
Jul 07 16:58:15.089129 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:15.089046    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d354b738-62f3-4850-9f1c-f6d5d2febb90" containerName="kserve-container"
Jul 07 16:58:15.089129 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:15.089072    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d354b738-62f3-4850-9f1c-f6d5d2febb90" containerName="kube-rbac-proxy"
Jul 07 16:58:15.089129 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:15.089083    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d354b738-62f3-4850-9f1c-f6d5d2febb90" containerName="kube-rbac-proxy"
Jul 07 16:58:15.089279 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:15.089182    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d354b738-62f3-4850-9f1c-f6d5d2febb90" containerName="kube-rbac-proxy"
Jul 07 16:58:15.089279 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:15.089196    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d354b738-62f3-4850-9f1c-f6d5d2febb90" containerName="kserve-container"
Jul 07 16:58:15.091654 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:15.091637    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-23477-predictor-747f9595bd-qjnx8"
Jul 07 16:58:15.094346 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:15.094326    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-23477-kube-rbac-proxy-sar-config\""
Jul 07 16:58:15.094346 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:15.094340    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-23477-predictor-serving-cert\""
Jul 07 16:58:15.099412 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:15.099389    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-23477-predictor-747f9595bd-qjnx8"]
Jul 07 16:58:15.141309 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:15.141282    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c"]
Jul 07 16:58:15.141566 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:15.141542    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c" podUID="b2131e33-d451-46cd-bf08-3f122bb41553" containerName="kserve-container" containerID="cri-o://9eecdbf6abac693170ffee3a9249443c11916af56ea916dd9096d93c68ce2329" gracePeriod=30
Jul 07 16:58:15.141646 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:15.141560    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c" podUID="b2131e33-d451-46cd-bf08-3f122bb41553" containerName="kube-rbac-proxy" containerID="cri-o://bf2954c22b1aee2cfc4bd2af22b492bf344f30321431e03c242f7da81341f303" gracePeriod=30
Jul 07 16:58:15.154191 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:15.154170    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33b0417b-b6c3-47ab-85d3-5231073cc49e-proxy-tls\") pod \"error-404-isvc-23477-predictor-747f9595bd-qjnx8\" (UID: \"33b0417b-b6c3-47ab-85d3-5231073cc49e\") " pod="kserve-ci-e2e-test/error-404-isvc-23477-predictor-747f9595bd-qjnx8"
Jul 07 16:58:15.154275 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:15.154209    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h44x\" (UniqueName: \"kubernetes.io/projected/33b0417b-b6c3-47ab-85d3-5231073cc49e-kube-api-access-5h44x\") pod \"error-404-isvc-23477-predictor-747f9595bd-qjnx8\" (UID: \"33b0417b-b6c3-47ab-85d3-5231073cc49e\") " pod="kserve-ci-e2e-test/error-404-isvc-23477-predictor-747f9595bd-qjnx8"
Jul 07 16:58:15.154328 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:15.154288    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33b0417b-b6c3-47ab-85d3-5231073cc49e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-23477-predictor-747f9595bd-qjnx8\" (UID: \"33b0417b-b6c3-47ab-85d3-5231073cc49e\") " pod="kserve-ci-e2e-test/error-404-isvc-23477-predictor-747f9595bd-qjnx8"
Jul 07 16:58:15.254968 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:15.254944    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33b0417b-b6c3-47ab-85d3-5231073cc49e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-23477-predictor-747f9595bd-qjnx8\" (UID: \"33b0417b-b6c3-47ab-85d3-5231073cc49e\") " pod="kserve-ci-e2e-test/error-404-isvc-23477-predictor-747f9595bd-qjnx8"
Jul 07 16:58:15.255052 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:15.255006    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33b0417b-b6c3-47ab-85d3-5231073cc49e-proxy-tls\") pod \"error-404-isvc-23477-predictor-747f9595bd-qjnx8\" (UID: \"33b0417b-b6c3-47ab-85d3-5231073cc49e\") " pod="kserve-ci-e2e-test/error-404-isvc-23477-predictor-747f9595bd-qjnx8"
Jul 07 16:58:15.255052 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:15.255029    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h44x\" (UniqueName: \"kubernetes.io/projected/33b0417b-b6c3-47ab-85d3-5231073cc49e-kube-api-access-5h44x\") pod \"error-404-isvc-23477-predictor-747f9595bd-qjnx8\" (UID: \"33b0417b-b6c3-47ab-85d3-5231073cc49e\") " pod="kserve-ci-e2e-test/error-404-isvc-23477-predictor-747f9595bd-qjnx8"
Jul 07 16:58:15.255167 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:58:15.255146    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-23477-predictor-serving-cert: secret "error-404-isvc-23477-predictor-serving-cert" not found
Jul 07 16:58:15.255234 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:58:15.255224    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33b0417b-b6c3-47ab-85d3-5231073cc49e-proxy-tls podName:33b0417b-b6c3-47ab-85d3-5231073cc49e nodeName:}" failed. No retries permitted until 2026-07-07 16:58:15.755207481 +0000 UTC m=+632.531521665 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/33b0417b-b6c3-47ab-85d3-5231073cc49e-proxy-tls") pod "error-404-isvc-23477-predictor-747f9595bd-qjnx8" (UID: "33b0417b-b6c3-47ab-85d3-5231073cc49e") : secret "error-404-isvc-23477-predictor-serving-cert" not found
Jul 07 16:58:15.255532 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:15.255514    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33b0417b-b6c3-47ab-85d3-5231073cc49e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-23477-predictor-747f9595bd-qjnx8\" (UID: \"33b0417b-b6c3-47ab-85d3-5231073cc49e\") " pod="kserve-ci-e2e-test/error-404-isvc-23477-predictor-747f9595bd-qjnx8"
Jul 07 16:58:15.263603 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:15.263583    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h44x\" (UniqueName: \"kubernetes.io/projected/33b0417b-b6c3-47ab-85d3-5231073cc49e-kube-api-access-5h44x\") pod \"error-404-isvc-23477-predictor-747f9595bd-qjnx8\" (UID: \"33b0417b-b6c3-47ab-85d3-5231073cc49e\") " pod="kserve-ci-e2e-test/error-404-isvc-23477-predictor-747f9595bd-qjnx8"
Jul 07 16:58:15.344145 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:15.344113    2578 generic.go:356] "Generic (PLEG): container finished" podID="b2131e33-d451-46cd-bf08-3f122bb41553" containerID="bf2954c22b1aee2cfc4bd2af22b492bf344f30321431e03c242f7da81341f303" exitCode=2
Jul 07 16:58:15.344303 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:15.344188    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c" event={"ID":"b2131e33-d451-46cd-bf08-3f122bb41553","Type":"ContainerDied","Data":"bf2954c22b1aee2cfc4bd2af22b492bf344f30321431e03c242f7da81341f303"}
Jul 07 16:58:15.346157 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:15.346139    2578 generic.go:356] "Generic (PLEG): container finished" podID="78832dba-ce1d-49cd-98f0-aaf5fd68a882" containerID="3e8fbb6844cd7d584a374fcf875a03da72ea4b9330973c9ee626cf4928cbab49" exitCode=2
Jul 07 16:58:15.346225 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:15.346205    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j884b" event={"ID":"78832dba-ce1d-49cd-98f0-aaf5fd68a882","Type":"ContainerDied","Data":"3e8fbb6844cd7d584a374fcf875a03da72ea4b9330973c9ee626cf4928cbab49"}
Jul 07 16:58:15.348071 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:15.348053    2578 generic.go:356] "Generic (PLEG): container finished" podID="06d394d3-ceb2-4d89-ab7d-46cc9a6f299c" containerID="8c291739ea67673b2dfbf5b1c6fdbddf02ddf8002a1cf4ae8c5ad6547711f9c7" exitCode=2
Jul 07 16:58:15.348162 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:15.348091    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c" event={"ID":"06d394d3-ceb2-4d89-ab7d-46cc9a6f299c","Type":"ContainerDied","Data":"8c291739ea67673b2dfbf5b1c6fdbddf02ddf8002a1cf4ae8c5ad6547711f9c7"}
Jul 07 16:58:15.757984 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:15.757953    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33b0417b-b6c3-47ab-85d3-5231073cc49e-proxy-tls\") pod \"error-404-isvc-23477-predictor-747f9595bd-qjnx8\" (UID: \"33b0417b-b6c3-47ab-85d3-5231073cc49e\") " pod="kserve-ci-e2e-test/error-404-isvc-23477-predictor-747f9595bd-qjnx8"
Jul 07 16:58:15.760255 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:15.760233    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33b0417b-b6c3-47ab-85d3-5231073cc49e-proxy-tls\") pod \"error-404-isvc-23477-predictor-747f9595bd-qjnx8\" (UID: \"33b0417b-b6c3-47ab-85d3-5231073cc49e\") " pod="kserve-ci-e2e-test/error-404-isvc-23477-predictor-747f9595bd-qjnx8"
Jul 07 16:58:16.001648 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:16.001614    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-23477-predictor-747f9595bd-qjnx8"
Jul 07 16:58:16.126174 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:16.126133    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-23477-predictor-747f9595bd-qjnx8"]
Jul 07 16:58:16.129280 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:58:16.129235    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33b0417b_b6c3_47ab_85d3_5231073cc49e.slice/crio-f08823fc937df4e26f16ef6504e416fec68dbe18d5963c2fed068d195b37e62f WatchSource:0}: Error finding container f08823fc937df4e26f16ef6504e416fec68dbe18d5963c2fed068d195b37e62f: Status 404 returned error can't find the container with id f08823fc937df4e26f16ef6504e416fec68dbe18d5963c2fed068d195b37e62f
Jul 07 16:58:16.353476 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:16.353443    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-23477-predictor-747f9595bd-qjnx8" event={"ID":"33b0417b-b6c3-47ab-85d3-5231073cc49e","Type":"ContainerStarted","Data":"b94578533cfba63618877e261f5f5ff5b1411bac60304f100ba838723fb6cc5c"}
Jul 07 16:58:16.353655 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:16.353506    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-23477-predictor-747f9595bd-qjnx8" event={"ID":"33b0417b-b6c3-47ab-85d3-5231073cc49e","Type":"ContainerStarted","Data":"39a6f50003be77e1f9bc8fbd76acf9f6ab59574a4b1053748a1d3853f1d4875e"}
Jul 07 16:58:16.353655 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:16.353534    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-23477-predictor-747f9595bd-qjnx8" event={"ID":"33b0417b-b6c3-47ab-85d3-5231073cc49e","Type":"ContainerStarted","Data":"f08823fc937df4e26f16ef6504e416fec68dbe18d5963c2fed068d195b37e62f"}
Jul 07 16:58:16.353655 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:16.353600    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-23477-predictor-747f9595bd-qjnx8"
Jul 07 16:58:16.370894 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:16.370854    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-23477-predictor-747f9595bd-qjnx8" podStartSLOduration=1.3708398480000001 podStartE2EDuration="1.370839848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:58:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:58:16.370606207 +0000 UTC m=+633.146920441" watchObservedRunningTime="2026-07-07 16:58:16.370839848 +0000 UTC m=+633.147154057"
Jul 07 16:58:16.858286 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:16.858248    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c" podUID="06d394d3-ceb2-4d89-ab7d-46cc9a6f299c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Jul 07 16:58:16.858465 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:16.858248    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c" podUID="b2131e33-d451-46cd-bf08-3f122bb41553" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jul 07 16:58:16.863655 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:16.863634    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c" podUID="b2131e33-d451-46cd-bf08-3f122bb41553" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 16:58:17.210793 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:17.210688    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bb9ec-predictor-546444857b-wjtqx" podUID="b007968f-9b0f-4578-9700-41c6e197d572" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 07 16:58:17.357228 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:17.357189    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-23477-predictor-747f9595bd-qjnx8"
Jul 07 16:58:17.358635 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:17.358605    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-23477-predictor-747f9595bd-qjnx8" podUID="33b0417b-b6c3-47ab-85d3-5231073cc49e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 07 16:58:17.720587 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:17.720548    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c" podUID="06d394d3-ceb2-4d89-ab7d-46cc9a6f299c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 07 16:58:18.361628 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:18.361588    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-23477-predictor-747f9595bd-qjnx8" podUID="33b0417b-b6c3-47ab-85d3-5231073cc49e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 07 16:58:18.699393 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:18.699370    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j884b"
Jul 07 16:58:18.786237 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:18.786202    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78832dba-ce1d-49cd-98f0-aaf5fd68a882-kserve-provision-location\") pod \"78832dba-ce1d-49cd-98f0-aaf5fd68a882\" (UID: \"78832dba-ce1d-49cd-98f0-aaf5fd68a882\") "
Jul 07 16:58:18.786409 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:18.786286    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78832dba-ce1d-49cd-98f0-aaf5fd68a882-proxy-tls\") pod \"78832dba-ce1d-49cd-98f0-aaf5fd68a882\" (UID: \"78832dba-ce1d-49cd-98f0-aaf5fd68a882\") "
Jul 07 16:58:18.786409 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:18.786323    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78832dba-ce1d-49cd-98f0-aaf5fd68a882-kube-rbac-proxy-sar-config\") pod \"78832dba-ce1d-49cd-98f0-aaf5fd68a882\" (UID: \"78832dba-ce1d-49cd-98f0-aaf5fd68a882\") "
Jul 07 16:58:18.786409 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:18.786350    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/78832dba-ce1d-49cd-98f0-aaf5fd68a882-cabundle-cert\") pod \"78832dba-ce1d-49cd-98f0-aaf5fd68a882\" (UID: \"78832dba-ce1d-49cd-98f0-aaf5fd68a882\") "
Jul 07 16:58:18.786409 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:18.786391    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6sp52\" (UniqueName: \"kubernetes.io/projected/78832dba-ce1d-49cd-98f0-aaf5fd68a882-kube-api-access-6sp52\") pod \"78832dba-ce1d-49cd-98f0-aaf5fd68a882\" (UID: \"78832dba-ce1d-49cd-98f0-aaf5fd68a882\") "
Jul 07 16:58:18.786662 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:18.786571    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78832dba-ce1d-49cd-98f0-aaf5fd68a882-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "78832dba-ce1d-49cd-98f0-aaf5fd68a882" (UID: "78832dba-ce1d-49cd-98f0-aaf5fd68a882"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:58:18.786662 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:18.786654    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78832dba-ce1d-49cd-98f0-aaf5fd68a882-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "78832dba-ce1d-49cd-98f0-aaf5fd68a882" (UID: "78832dba-ce1d-49cd-98f0-aaf5fd68a882"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:58:18.786806 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:18.786783    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78832dba-ce1d-49cd-98f0-aaf5fd68a882-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "78832dba-ce1d-49cd-98f0-aaf5fd68a882" (UID: "78832dba-ce1d-49cd-98f0-aaf5fd68a882"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:58:18.788317 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:18.788299    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78832dba-ce1d-49cd-98f0-aaf5fd68a882-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "78832dba-ce1d-49cd-98f0-aaf5fd68a882" (UID: "78832dba-ce1d-49cd-98f0-aaf5fd68a882"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:58:18.788380 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:18.788357    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78832dba-ce1d-49cd-98f0-aaf5fd68a882-kube-api-access-6sp52" (OuterVolumeSpecName: "kube-api-access-6sp52") pod "78832dba-ce1d-49cd-98f0-aaf5fd68a882" (UID: "78832dba-ce1d-49cd-98f0-aaf5fd68a882"). InnerVolumeSpecName "kube-api-access-6sp52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:58:18.888020 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:18.887942    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78832dba-ce1d-49cd-98f0-aaf5fd68a882-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:58:18.888020 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:18.887969    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/78832dba-ce1d-49cd-98f0-aaf5fd68a882-cabundle-cert\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:58:18.888020 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:18.887979    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sp52\" (UniqueName: \"kubernetes.io/projected/78832dba-ce1d-49cd-98f0-aaf5fd68a882-kube-api-access-6sp52\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:58:18.888020 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:18.887989    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78832dba-ce1d-49cd-98f0-aaf5fd68a882-kserve-provision-location\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:58:18.888020 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:18.887998    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78832dba-ce1d-49cd-98f0-aaf5fd68a882-proxy-tls\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:58:19.366992 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:19.366961    2578 generic.go:356] "Generic (PLEG): container finished" podID="78832dba-ce1d-49cd-98f0-aaf5fd68a882" containerID="a07247ee6d9a3ee667239885cfcf77c1c0c68a48588a2a3457841bd4a3540595" exitCode=0
Jul 07 16:58:19.367362 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:19.367015    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j884b"
Jul 07 16:58:19.367362 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:19.367037    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j884b" event={"ID":"78832dba-ce1d-49cd-98f0-aaf5fd68a882","Type":"ContainerDied","Data":"a07247ee6d9a3ee667239885cfcf77c1c0c68a48588a2a3457841bd4a3540595"}
Jul 07 16:58:19.367362 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:19.367091    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j884b" event={"ID":"78832dba-ce1d-49cd-98f0-aaf5fd68a882","Type":"ContainerDied","Data":"590032dfd7e8577fb5824fa3c0310716e9b583dd7d4d85f0d0c766a5e0fdfb24"}
Jul 07 16:58:19.367362 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:19.367119    2578 scope.go:117] "RemoveContainer" containerID="3e8fbb6844cd7d584a374fcf875a03da72ea4b9330973c9ee626cf4928cbab49"
Jul 07 16:58:19.375588 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:19.375572    2578 scope.go:117] "RemoveContainer" containerID="a07247ee6d9a3ee667239885cfcf77c1c0c68a48588a2a3457841bd4a3540595"
Jul 07 16:58:19.382601 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:19.382586    2578 scope.go:117] "RemoveContainer" containerID="df1a51e793742ed94166a8d943d13ab9636cb72b808376bf9c712c5c6a442d3a"
Jul 07 16:58:19.388362 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:19.388341    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j884b"]
Jul 07 16:58:19.389964 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:19.389949    2578 scope.go:117] "RemoveContainer" containerID="3e8fbb6844cd7d584a374fcf875a03da72ea4b9330973c9ee626cf4928cbab49"
Jul 07 16:58:19.390180 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:58:19.390162    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e8fbb6844cd7d584a374fcf875a03da72ea4b9330973c9ee626cf4928cbab49\": container with ID starting with 3e8fbb6844cd7d584a374fcf875a03da72ea4b9330973c9ee626cf4928cbab49 not found: ID does not exist" containerID="3e8fbb6844cd7d584a374fcf875a03da72ea4b9330973c9ee626cf4928cbab49"
Jul 07 16:58:19.390228 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:19.390190    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e8fbb6844cd7d584a374fcf875a03da72ea4b9330973c9ee626cf4928cbab49"} err="failed to get container status \"3e8fbb6844cd7d584a374fcf875a03da72ea4b9330973c9ee626cf4928cbab49\": rpc error: code = NotFound desc = could not find container \"3e8fbb6844cd7d584a374fcf875a03da72ea4b9330973c9ee626cf4928cbab49\": container with ID starting with 3e8fbb6844cd7d584a374fcf875a03da72ea4b9330973c9ee626cf4928cbab49 not found: ID does not exist"
Jul 07 16:58:19.390228 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:19.390208    2578 scope.go:117] "RemoveContainer" containerID="a07247ee6d9a3ee667239885cfcf77c1c0c68a48588a2a3457841bd4a3540595"
Jul 07 16:58:19.390442 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:58:19.390425    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a07247ee6d9a3ee667239885cfcf77c1c0c68a48588a2a3457841bd4a3540595\": container with ID starting with a07247ee6d9a3ee667239885cfcf77c1c0c68a48588a2a3457841bd4a3540595 not found: ID does not exist" containerID="a07247ee6d9a3ee667239885cfcf77c1c0c68a48588a2a3457841bd4a3540595"
Jul 07 16:58:19.390493 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:19.390448    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a07247ee6d9a3ee667239885cfcf77c1c0c68a48588a2a3457841bd4a3540595"} err="failed to get container status \"a07247ee6d9a3ee667239885cfcf77c1c0c68a48588a2a3457841bd4a3540595\": rpc error: code = NotFound desc = could not find container \"a07247ee6d9a3ee667239885cfcf77c1c0c68a48588a2a3457841bd4a3540595\": container with ID starting with a07247ee6d9a3ee667239885cfcf77c1c0c68a48588a2a3457841bd4a3540595 not found: ID does not exist"
Jul 07 16:58:19.390493 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:19.390463    2578 scope.go:117] "RemoveContainer" containerID="df1a51e793742ed94166a8d943d13ab9636cb72b808376bf9c712c5c6a442d3a"
Jul 07 16:58:19.390685 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:58:19.390670    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df1a51e793742ed94166a8d943d13ab9636cb72b808376bf9c712c5c6a442d3a\": container with ID starting with df1a51e793742ed94166a8d943d13ab9636cb72b808376bf9c712c5c6a442d3a not found: ID does not exist" containerID="df1a51e793742ed94166a8d943d13ab9636cb72b808376bf9c712c5c6a442d3a"
Jul 07 16:58:19.390722 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:19.390689    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df1a51e793742ed94166a8d943d13ab9636cb72b808376bf9c712c5c6a442d3a"} err="failed to get container status \"df1a51e793742ed94166a8d943d13ab9636cb72b808376bf9c712c5c6a442d3a\": rpc error: code = NotFound desc = could not find container \"df1a51e793742ed94166a8d943d13ab9636cb72b808376bf9c712c5c6a442d3a\": container with ID starting with df1a51e793742ed94166a8d943d13ab9636cb72b808376bf9c712c5c6a442d3a not found: ID does not exist"
Jul 07 16:58:19.393369 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:19.393350    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j884b"]
Jul 07 16:58:19.725166 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:19.725092    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78832dba-ce1d-49cd-98f0-aaf5fd68a882" path="/var/lib/kubelet/pods/78832dba-ce1d-49cd-98f0-aaf5fd68a882/volumes"
Jul 07 16:58:20.592845 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:20.592826    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c"
Jul 07 16:58:20.702632 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:20.702552    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06d394d3-ceb2-4d89-ab7d-46cc9a6f299c-proxy-tls\") pod \"06d394d3-ceb2-4d89-ab7d-46cc9a6f299c\" (UID: \"06d394d3-ceb2-4d89-ab7d-46cc9a6f299c\") "
Jul 07 16:58:20.702632 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:20.702616    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lw42s\" (UniqueName: \"kubernetes.io/projected/06d394d3-ceb2-4d89-ab7d-46cc9a6f299c-kube-api-access-lw42s\") pod \"06d394d3-ceb2-4d89-ab7d-46cc9a6f299c\" (UID: \"06d394d3-ceb2-4d89-ab7d-46cc9a6f299c\") "
Jul 07 16:58:20.702632 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:20.702645    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06d394d3-ceb2-4d89-ab7d-46cc9a6f299c-kube-rbac-proxy-sar-config\") pod \"06d394d3-ceb2-4d89-ab7d-46cc9a6f299c\" (UID: \"06d394d3-ceb2-4d89-ab7d-46cc9a6f299c\") "
Jul 07 16:58:20.702929 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:20.702685    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06d394d3-ceb2-4d89-ab7d-46cc9a6f299c-cabundle-cert\") pod \"06d394d3-ceb2-4d89-ab7d-46cc9a6f299c\" (UID: \"06d394d3-ceb2-4d89-ab7d-46cc9a6f299c\") "
Jul 07 16:58:20.702929 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:20.702705    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06d394d3-ceb2-4d89-ab7d-46cc9a6f299c-kserve-provision-location\") pod \"06d394d3-ceb2-4d89-ab7d-46cc9a6f299c\" (UID: \"06d394d3-ceb2-4d89-ab7d-46cc9a6f299c\") "
Jul 07 16:58:20.703070 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:20.703048    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06d394d3-ceb2-4d89-ab7d-46cc9a6f299c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "06d394d3-ceb2-4d89-ab7d-46cc9a6f299c" (UID: "06d394d3-ceb2-4d89-ab7d-46cc9a6f299c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:58:20.703126 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:20.703073    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06d394d3-ceb2-4d89-ab7d-46cc9a6f299c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "06d394d3-ceb2-4d89-ab7d-46cc9a6f299c" (UID: "06d394d3-ceb2-4d89-ab7d-46cc9a6f299c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:58:20.703126 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:20.703086    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06d394d3-ceb2-4d89-ab7d-46cc9a6f299c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "06d394d3-ceb2-4d89-ab7d-46cc9a6f299c" (UID: "06d394d3-ceb2-4d89-ab7d-46cc9a6f299c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:58:20.704669 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:20.704650    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06d394d3-ceb2-4d89-ab7d-46cc9a6f299c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "06d394d3-ceb2-4d89-ab7d-46cc9a6f299c" (UID: "06d394d3-ceb2-4d89-ab7d-46cc9a6f299c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:58:20.704755 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:20.704703    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06d394d3-ceb2-4d89-ab7d-46cc9a6f299c-kube-api-access-lw42s" (OuterVolumeSpecName: "kube-api-access-lw42s") pod "06d394d3-ceb2-4d89-ab7d-46cc9a6f299c" (UID: "06d394d3-ceb2-4d89-ab7d-46cc9a6f299c"). InnerVolumeSpecName "kube-api-access-lw42s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:58:20.803695 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:20.803654    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06d394d3-ceb2-4d89-ab7d-46cc9a6f299c-cabundle-cert\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:58:20.803695 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:20.803685    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06d394d3-ceb2-4d89-ab7d-46cc9a6f299c-kserve-provision-location\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:58:20.803695 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:20.803698    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06d394d3-ceb2-4d89-ab7d-46cc9a6f299c-proxy-tls\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:58:20.803928 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:20.803711    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lw42s\" (UniqueName: \"kubernetes.io/projected/06d394d3-ceb2-4d89-ab7d-46cc9a6f299c-kube-api-access-lw42s\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:58:20.803928 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:20.803726    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06d394d3-ceb2-4d89-ab7d-46cc9a6f299c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:58:20.878075 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:20.878046    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c"
Jul 07 16:58:21.005329 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:21.005246    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2131e33-d451-46cd-bf08-3f122bb41553-kserve-provision-location\") pod \"b2131e33-d451-46cd-bf08-3f122bb41553\" (UID: \"b2131e33-d451-46cd-bf08-3f122bb41553\") "
Jul 07 16:58:21.005329 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:21.005288    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2131e33-d451-46cd-bf08-3f122bb41553-proxy-tls\") pod \"b2131e33-d451-46cd-bf08-3f122bb41553\" (UID: \"b2131e33-d451-46cd-bf08-3f122bb41553\") "
Jul 07 16:58:21.005329 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:21.005313    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2131e33-d451-46cd-bf08-3f122bb41553-cabundle-cert\") pod \"b2131e33-d451-46cd-bf08-3f122bb41553\" (UID: \"b2131e33-d451-46cd-bf08-3f122bb41553\") "
Jul 07 16:58:21.005610 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:21.005339    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2131e33-d451-46cd-bf08-3f122bb41553-kube-rbac-proxy-sar-config\") pod \"b2131e33-d451-46cd-bf08-3f122bb41553\" (UID: \"b2131e33-d451-46cd-bf08-3f122bb41553\") "
Jul 07 16:58:21.005610 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:21.005406    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-27s44\" (UniqueName: \"kubernetes.io/projected/b2131e33-d451-46cd-bf08-3f122bb41553-kube-api-access-27s44\") pod \"b2131e33-d451-46cd-bf08-3f122bb41553\" (UID: \"b2131e33-d451-46cd-bf08-3f122bb41553\") "
Jul 07 16:58:21.005610 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:21.005570    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2131e33-d451-46cd-bf08-3f122bb41553-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b2131e33-d451-46cd-bf08-3f122bb41553" (UID: "b2131e33-d451-46cd-bf08-3f122bb41553"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:58:21.005792 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:21.005663    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2131e33-d451-46cd-bf08-3f122bb41553-kserve-provision-location\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:58:21.005792 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:21.005701    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2131e33-d451-46cd-bf08-3f122bb41553-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b2131e33-d451-46cd-bf08-3f122bb41553" (UID: "b2131e33-d451-46cd-bf08-3f122bb41553"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:58:21.005792 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:21.005722    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2131e33-d451-46cd-bf08-3f122bb41553-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b2131e33-d451-46cd-bf08-3f122bb41553" (UID: "b2131e33-d451-46cd-bf08-3f122bb41553"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:58:21.007368 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:21.007351    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2131e33-d451-46cd-bf08-3f122bb41553-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b2131e33-d451-46cd-bf08-3f122bb41553" (UID: "b2131e33-d451-46cd-bf08-3f122bb41553"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:58:21.007368 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:21.007357    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2131e33-d451-46cd-bf08-3f122bb41553-kube-api-access-27s44" (OuterVolumeSpecName: "kube-api-access-27s44") pod "b2131e33-d451-46cd-bf08-3f122bb41553" (UID: "b2131e33-d451-46cd-bf08-3f122bb41553"). InnerVolumeSpecName "kube-api-access-27s44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:58:21.106390 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:21.106362    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2131e33-d451-46cd-bf08-3f122bb41553-proxy-tls\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:58:21.106390 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:21.106386    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2131e33-d451-46cd-bf08-3f122bb41553-cabundle-cert\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:58:21.106390 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:21.106395    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2131e33-d451-46cd-bf08-3f122bb41553-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:58:21.106581 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:21.106404    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27s44\" (UniqueName: \"kubernetes.io/projected/b2131e33-d451-46cd-bf08-3f122bb41553-kube-api-access-27s44\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 16:58:21.377867 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:21.377834    2578 generic.go:356] "Generic (PLEG): container finished" podID="b2131e33-d451-46cd-bf08-3f122bb41553" containerID="9eecdbf6abac693170ffee3a9249443c11916af56ea916dd9096d93c68ce2329" exitCode=0
Jul 07 16:58:21.378061 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:21.377893    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c"
Jul 07 16:58:21.378061 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:21.377927    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c" event={"ID":"b2131e33-d451-46cd-bf08-3f122bb41553","Type":"ContainerDied","Data":"9eecdbf6abac693170ffee3a9249443c11916af56ea916dd9096d93c68ce2329"}
Jul 07 16:58:21.378061 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:21.377999    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c" event={"ID":"b2131e33-d451-46cd-bf08-3f122bb41553","Type":"ContainerDied","Data":"b579cb72a284d01b2063f4c01a47f08c2b60824ad7f4fc627ad680ffa716010b"}
Jul 07 16:58:21.378061 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:21.378035    2578 scope.go:117] "RemoveContainer" containerID="bf2954c22b1aee2cfc4bd2af22b492bf344f30321431e03c242f7da81341f303"
Jul 07 16:58:21.379819 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:21.379800    2578 generic.go:356] "Generic (PLEG): container finished" podID="06d394d3-ceb2-4d89-ab7d-46cc9a6f299c" containerID="c88b4aa99079c3e62f7cccc9d84337ac3f695a54e8b58af399079ca6406e14e1" exitCode=0
Jul 07 16:58:21.379922 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:21.379864    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c" event={"ID":"06d394d3-ceb2-4d89-ab7d-46cc9a6f299c","Type":"ContainerDied","Data":"c88b4aa99079c3e62f7cccc9d84337ac3f695a54e8b58af399079ca6406e14e1"}
Jul 07 16:58:21.379922 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:21.379877    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c"
Jul 07 16:58:21.379922 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:21.379907    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c" event={"ID":"06d394d3-ceb2-4d89-ab7d-46cc9a6f299c","Type":"ContainerDied","Data":"72a6e1acf2b46b1e537019cc85a4f6e83bba1c8619d52fdbed6bc197ad7199d1"}
Jul 07 16:58:21.387809 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:21.387792    2578 scope.go:117] "RemoveContainer" containerID="9eecdbf6abac693170ffee3a9249443c11916af56ea916dd9096d93c68ce2329"
Jul 07 16:58:21.395231 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:21.395216    2578 scope.go:117] "RemoveContainer" containerID="5367576d7eb6b3816d8ab0a4963872bc0ab6f3556aefd7f7f875849255ceb86b"
Jul 07 16:58:21.402346 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:21.402331    2578 scope.go:117] "RemoveContainer" containerID="bf2954c22b1aee2cfc4bd2af22b492bf344f30321431e03c242f7da81341f303"
Jul 07 16:58:21.402620 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:58:21.402604    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf2954c22b1aee2cfc4bd2af22b492bf344f30321431e03c242f7da81341f303\": container with ID starting with bf2954c22b1aee2cfc4bd2af22b492bf344f30321431e03c242f7da81341f303 not found: ID does not exist" containerID="bf2954c22b1aee2cfc4bd2af22b492bf344f30321431e03c242f7da81341f303"
Jul 07 16:58:21.402659 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:21.402630    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf2954c22b1aee2cfc4bd2af22b492bf344f30321431e03c242f7da81341f303"} err="failed to get container status \"bf2954c22b1aee2cfc4bd2af22b492bf344f30321431e03c242f7da81341f303\": rpc error: code = NotFound desc = could not find container \"bf2954c22b1aee2cfc4bd2af22b492bf344f30321431e03c242f7da81341f303\": container with ID starting with bf2954c22b1aee2cfc4bd2af22b492bf344f30321431e03c242f7da81341f303 not found: ID does not exist"
Jul 07 16:58:21.402659 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:21.402651    2578 scope.go:117] "RemoveContainer" containerID="9eecdbf6abac693170ffee3a9249443c11916af56ea916dd9096d93c68ce2329"
Jul 07 16:58:21.402936 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:58:21.402914    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9eecdbf6abac693170ffee3a9249443c11916af56ea916dd9096d93c68ce2329\": container with ID starting with 9eecdbf6abac693170ffee3a9249443c11916af56ea916dd9096d93c68ce2329 not found: ID does not exist" containerID="9eecdbf6abac693170ffee3a9249443c11916af56ea916dd9096d93c68ce2329"
Jul 07 16:58:21.403038 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:21.402943    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9eecdbf6abac693170ffee3a9249443c11916af56ea916dd9096d93c68ce2329"} err="failed to get container status \"9eecdbf6abac693170ffee3a9249443c11916af56ea916dd9096d93c68ce2329\": rpc error: code = NotFound desc = could not find container \"9eecdbf6abac693170ffee3a9249443c11916af56ea916dd9096d93c68ce2329\": container with ID starting with 9eecdbf6abac693170ffee3a9249443c11916af56ea916dd9096d93c68ce2329 not found: ID does not exist"
Jul 07 16:58:21.403038 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:21.402961    2578 scope.go:117] "RemoveContainer" containerID="5367576d7eb6b3816d8ab0a4963872bc0ab6f3556aefd7f7f875849255ceb86b"
Jul 07 16:58:21.403251 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:21.403225    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c"]
Jul 07 16:58:21.403329 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:58:21.403306    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5367576d7eb6b3816d8ab0a4963872bc0ab6f3556aefd7f7f875849255ceb86b\": container with ID starting with 5367576d7eb6b3816d8ab0a4963872bc0ab6f3556aefd7f7f875849255ceb86b not found: ID does not exist" containerID="5367576d7eb6b3816d8ab0a4963872bc0ab6f3556aefd7f7f875849255ceb86b"
Jul 07 16:58:21.403397 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:21.403333    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5367576d7eb6b3816d8ab0a4963872bc0ab6f3556aefd7f7f875849255ceb86b"} err="failed to get container status \"5367576d7eb6b3816d8ab0a4963872bc0ab6f3556aefd7f7f875849255ceb86b\": rpc error: code = NotFound desc = could not find container \"5367576d7eb6b3816d8ab0a4963872bc0ab6f3556aefd7f7f875849255ceb86b\": container with ID starting with 5367576d7eb6b3816d8ab0a4963872bc0ab6f3556aefd7f7f875849255ceb86b not found: ID does not exist"
Jul 07 16:58:21.403397 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:21.403354    2578 scope.go:117] "RemoveContainer" containerID="8c291739ea67673b2dfbf5b1c6fdbddf02ddf8002a1cf4ae8c5ad6547711f9c7"
Jul 07 16:58:21.405346 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:21.405330    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7fddfc6c5c-gr22c"]
Jul 07 16:58:21.411054 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:21.411040    2578 scope.go:117] "RemoveContainer" containerID="c88b4aa99079c3e62f7cccc9d84337ac3f695a54e8b58af399079ca6406e14e1"
Jul 07 16:58:21.416153 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:21.416133    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c"]
Jul 07 16:58:21.420511 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:21.420492    2578 scope.go:117] "RemoveContainer" containerID="7535bfa7037261c681a9c5985287c8b97e9b2c45f243bd78a9a95ee64c51e86a"
Jul 07 16:58:21.420602 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:21.420572    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5f7864bfc5-pc98c"]
Jul 07 16:58:21.427698 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:21.427680    2578 scope.go:117] "RemoveContainer" containerID="8c291739ea67673b2dfbf5b1c6fdbddf02ddf8002a1cf4ae8c5ad6547711f9c7"
Jul 07 16:58:21.427968 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:58:21.427949    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c291739ea67673b2dfbf5b1c6fdbddf02ddf8002a1cf4ae8c5ad6547711f9c7\": container with ID starting with 8c291739ea67673b2dfbf5b1c6fdbddf02ddf8002a1cf4ae8c5ad6547711f9c7 not found: ID does not exist" containerID="8c291739ea67673b2dfbf5b1c6fdbddf02ddf8002a1cf4ae8c5ad6547711f9c7"
Jul 07 16:58:21.428028 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:21.427978    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c291739ea67673b2dfbf5b1c6fdbddf02ddf8002a1cf4ae8c5ad6547711f9c7"} err="failed to get container status \"8c291739ea67673b2dfbf5b1c6fdbddf02ddf8002a1cf4ae8c5ad6547711f9c7\": rpc error: code = NotFound desc = could not find container \"8c291739ea67673b2dfbf5b1c6fdbddf02ddf8002a1cf4ae8c5ad6547711f9c7\": container with ID starting with 8c291739ea67673b2dfbf5b1c6fdbddf02ddf8002a1cf4ae8c5ad6547711f9c7 not found: ID does not exist"
Jul 07 16:58:21.428028 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:21.428002    2578 scope.go:117] "RemoveContainer" containerID="c88b4aa99079c3e62f7cccc9d84337ac3f695a54e8b58af399079ca6406e14e1"
Jul 07 16:58:21.428282 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:58:21.428259    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c88b4aa99079c3e62f7cccc9d84337ac3f695a54e8b58af399079ca6406e14e1\": container with ID starting with c88b4aa99079c3e62f7cccc9d84337ac3f695a54e8b58af399079ca6406e14e1 not found: ID does not exist" containerID="c88b4aa99079c3e62f7cccc9d84337ac3f695a54e8b58af399079ca6406e14e1"
Jul 07 16:58:21.428326 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:21.428290    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c88b4aa99079c3e62f7cccc9d84337ac3f695a54e8b58af399079ca6406e14e1"} err="failed to get container status \"c88b4aa99079c3e62f7cccc9d84337ac3f695a54e8b58af399079ca6406e14e1\": rpc error: code = NotFound desc = could not find container \"c88b4aa99079c3e62f7cccc9d84337ac3f695a54e8b58af399079ca6406e14e1\": container with ID starting with c88b4aa99079c3e62f7cccc9d84337ac3f695a54e8b58af399079ca6406e14e1 not found: ID does not exist"
Jul 07 16:58:21.428326 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:21.428312    2578 scope.go:117] "RemoveContainer" containerID="7535bfa7037261c681a9c5985287c8b97e9b2c45f243bd78a9a95ee64c51e86a"
Jul 07 16:58:21.428585 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:58:21.428569    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7535bfa7037261c681a9c5985287c8b97e9b2c45f243bd78a9a95ee64c51e86a\": container with ID starting with 7535bfa7037261c681a9c5985287c8b97e9b2c45f243bd78a9a95ee64c51e86a not found: ID does not exist" containerID="7535bfa7037261c681a9c5985287c8b97e9b2c45f243bd78a9a95ee64c51e86a"
Jul 07 16:58:21.428673 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:21.428594    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7535bfa7037261c681a9c5985287c8b97e9b2c45f243bd78a9a95ee64c51e86a"} err="failed to get container status \"7535bfa7037261c681a9c5985287c8b97e9b2c45f243bd78a9a95ee64c51e86a\": rpc error: code = NotFound desc = could not find container \"7535bfa7037261c681a9c5985287c8b97e9b2c45f243bd78a9a95ee64c51e86a\": container with ID starting with 7535bfa7037261c681a9c5985287c8b97e9b2c45f243bd78a9a95ee64c51e86a not found: ID does not exist"
Jul 07 16:58:21.724775 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:21.724690    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06d394d3-ceb2-4d89-ab7d-46cc9a6f299c" path="/var/lib/kubelet/pods/06d394d3-ceb2-4d89-ab7d-46cc9a6f299c/volumes"
Jul 07 16:58:21.725179 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:21.725166    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2131e33-d451-46cd-bf08-3f122bb41553" path="/var/lib/kubelet/pods/b2131e33-d451-46cd-bf08-3f122bb41553/volumes"
Jul 07 16:58:23.365916 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:23.365890    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-23477-predictor-747f9595bd-qjnx8"
Jul 07 16:58:23.367190 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:23.366390    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-23477-predictor-747f9595bd-qjnx8" podUID="33b0417b-b6c3-47ab-85d3-5231073cc49e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 07 16:58:27.211348 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:27.211318    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-bb9ec-predictor-546444857b-wjtqx"
Jul 07 16:58:33.367165 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:33.367129    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-23477-predictor-747f9595bd-qjnx8" podUID="33b0417b-b6c3-47ab-85d3-5231073cc49e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 07 16:58:39.215329 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:39.215298    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-bb9ec-7fc9d47dd-vj2b5"]
Jul 07 16:58:39.215817 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:39.215628    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78832dba-ce1d-49cd-98f0-aaf5fd68a882" containerName="storage-initializer"
Jul 07 16:58:39.215817 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:39.215639    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="78832dba-ce1d-49cd-98f0-aaf5fd68a882" containerName="storage-initializer"
Jul 07 16:58:39.215817 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:39.215666    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06d394d3-ceb2-4d89-ab7d-46cc9a6f299c" containerName="storage-initializer"
Jul 07 16:58:39.215817 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:39.215672    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="06d394d3-ceb2-4d89-ab7d-46cc9a6f299c" containerName="storage-initializer"
Jul 07 16:58:39.215817 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:39.215681    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2131e33-d451-46cd-bf08-3f122bb41553" containerName="storage-initializer"
Jul 07 16:58:39.215817 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:39.215686    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2131e33-d451-46cd-bf08-3f122bb41553" containerName="storage-initializer"
Jul 07 16:58:39.215817 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:39.215692    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2131e33-d451-46cd-bf08-3f122bb41553" containerName="kserve-container"
Jul 07 16:58:39.215817 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:39.215697    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2131e33-d451-46cd-bf08-3f122bb41553" containerName="kserve-container"
Jul 07 16:58:39.215817 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:39.215703    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2131e33-d451-46cd-bf08-3f122bb41553" containerName="kube-rbac-proxy"
Jul 07 16:58:39.215817 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:39.215708    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2131e33-d451-46cd-bf08-3f122bb41553" containerName="kube-rbac-proxy"
Jul 07 16:58:39.215817 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:39.215715    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06d394d3-ceb2-4d89-ab7d-46cc9a6f299c" containerName="kube-rbac-proxy"
Jul 07 16:58:39.215817 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:39.215720    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="06d394d3-ceb2-4d89-ab7d-46cc9a6f299c" containerName="kube-rbac-proxy"
Jul 07 16:58:39.215817 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:39.215725    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78832dba-ce1d-49cd-98f0-aaf5fd68a882" containerName="kube-rbac-proxy"
Jul 07 16:58:39.215817 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:39.215746    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="78832dba-ce1d-49cd-98f0-aaf5fd68a882" containerName="kube-rbac-proxy"
Jul 07 16:58:39.215817 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:39.215757    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06d394d3-ceb2-4d89-ab7d-46cc9a6f299c" containerName="kserve-container"
Jul 07 16:58:39.215817 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:39.215763    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="06d394d3-ceb2-4d89-ab7d-46cc9a6f299c" containerName="kserve-container"
Jul 07 16:58:39.215817 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:39.215769    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78832dba-ce1d-49cd-98f0-aaf5fd68a882" containerName="kserve-container"
Jul 07 16:58:39.215817 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:39.215776    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="78832dba-ce1d-49cd-98f0-aaf5fd68a882" containerName="kserve-container"
Jul 07 16:58:39.215817 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:39.215824    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="78832dba-ce1d-49cd-98f0-aaf5fd68a882" containerName="kube-rbac-proxy"
Jul 07 16:58:39.216602 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:39.215832    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="06d394d3-ceb2-4d89-ab7d-46cc9a6f299c" containerName="kube-rbac-proxy"
Jul 07 16:58:39.216602 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:39.215838    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2131e33-d451-46cd-bf08-3f122bb41553" containerName="kserve-container"
Jul 07 16:58:39.216602 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:39.215845    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="06d394d3-ceb2-4d89-ab7d-46cc9a6f299c" containerName="kserve-container"
Jul 07 16:58:39.216602 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:39.215851    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2131e33-d451-46cd-bf08-3f122bb41553" containerName="kube-rbac-proxy"
Jul 07 16:58:39.216602 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:39.215856    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="78832dba-ce1d-49cd-98f0-aaf5fd68a882" containerName="kserve-container"
Jul 07 16:58:39.217709 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:39.217695    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-bb9ec-7fc9d47dd-vj2b5"
Jul 07 16:58:39.220271 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:39.220254    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-bb9ec-serving-cert\""
Jul 07 16:58:39.220377 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:39.220281    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-bb9ec-kube-rbac-proxy-sar-config\""
Jul 07 16:58:39.226008 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:39.225983    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-bb9ec-7fc9d47dd-vj2b5"]
Jul 07 16:58:39.364114 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:39.364083    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abb124b1-6682-43b7-9631-ee6671f24552-proxy-tls\") pod \"switch-graph-bb9ec-7fc9d47dd-vj2b5\" (UID: \"abb124b1-6682-43b7-9631-ee6671f24552\") " pod="kserve-ci-e2e-test/switch-graph-bb9ec-7fc9d47dd-vj2b5"
Jul 07 16:58:39.364283 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:39.364124    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/abb124b1-6682-43b7-9631-ee6671f24552-openshift-service-ca-bundle\") pod \"switch-graph-bb9ec-7fc9d47dd-vj2b5\" (UID: \"abb124b1-6682-43b7-9631-ee6671f24552\") " pod="kserve-ci-e2e-test/switch-graph-bb9ec-7fc9d47dd-vj2b5"
Jul 07 16:58:39.465283 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:39.465251    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abb124b1-6682-43b7-9631-ee6671f24552-proxy-tls\") pod \"switch-graph-bb9ec-7fc9d47dd-vj2b5\" (UID: \"abb124b1-6682-43b7-9631-ee6671f24552\") " pod="kserve-ci-e2e-test/switch-graph-bb9ec-7fc9d47dd-vj2b5"
Jul 07 16:58:39.465476 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:39.465300    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/abb124b1-6682-43b7-9631-ee6671f24552-openshift-service-ca-bundle\") pod \"switch-graph-bb9ec-7fc9d47dd-vj2b5\" (UID: \"abb124b1-6682-43b7-9631-ee6671f24552\") " pod="kserve-ci-e2e-test/switch-graph-bb9ec-7fc9d47dd-vj2b5"
Jul 07 16:58:39.465956 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:39.465936    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/abb124b1-6682-43b7-9631-ee6671f24552-openshift-service-ca-bundle\") pod \"switch-graph-bb9ec-7fc9d47dd-vj2b5\" (UID: \"abb124b1-6682-43b7-9631-ee6671f24552\") " pod="kserve-ci-e2e-test/switch-graph-bb9ec-7fc9d47dd-vj2b5"
Jul 07 16:58:39.467664 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:39.467637    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abb124b1-6682-43b7-9631-ee6671f24552-proxy-tls\") pod \"switch-graph-bb9ec-7fc9d47dd-vj2b5\" (UID: \"abb124b1-6682-43b7-9631-ee6671f24552\") " pod="kserve-ci-e2e-test/switch-graph-bb9ec-7fc9d47dd-vj2b5"
Jul 07 16:58:39.528437 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:39.528397    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-bb9ec-7fc9d47dd-vj2b5"
Jul 07 16:58:39.657992 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:39.657958    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-bb9ec-7fc9d47dd-vj2b5"]
Jul 07 16:58:39.662098 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:58:39.662071    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabb124b1_6682_43b7_9631_ee6671f24552.slice/crio-4228c70be9f20475dae8d85a3d9c2d5d089802bba03fcce6db3d85b4f6eda965 WatchSource:0}: Error finding container 4228c70be9f20475dae8d85a3d9c2d5d089802bba03fcce6db3d85b4f6eda965: Status 404 returned error can't find the container with id 4228c70be9f20475dae8d85a3d9c2d5d089802bba03fcce6db3d85b4f6eda965
Jul 07 16:58:40.459067 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:40.459027    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-bb9ec-7fc9d47dd-vj2b5" event={"ID":"abb124b1-6682-43b7-9631-ee6671f24552","Type":"ContainerStarted","Data":"4228c70be9f20475dae8d85a3d9c2d5d089802bba03fcce6db3d85b4f6eda965"}
Jul 07 16:58:42.469173 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:42.469087    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-bb9ec-7fc9d47dd-vj2b5" event={"ID":"abb124b1-6682-43b7-9631-ee6671f24552","Type":"ContainerStarted","Data":"e4e9a25800560897e9c1c0772702fe46259ba45569a685f0b1389be6af5783cb"}
Jul 07 16:58:42.469173 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:42.469154    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-bb9ec-7fc9d47dd-vj2b5"
Jul 07 16:58:42.486285 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:42.486240    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-bb9ec-7fc9d47dd-vj2b5" podStartSLOduration=1.073498987 podStartE2EDuration="3.486224876s" totalImagesPullingTime="2.412725889s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:58:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:58:42.484416844 +0000 UTC m=+659.260731051" watchObservedRunningTime="2026-07-07 16:58:42.486224876 +0000 UTC m=+659.262539082"
Jul 07 16:58:43.366947 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:43.366910    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-23477-predictor-747f9595bd-qjnx8" podUID="33b0417b-b6c3-47ab-85d3-5231073cc49e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 07 16:58:48.481036 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:48.481011    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-bb9ec-7fc9d47dd-vj2b5"
Jul 07 16:58:53.366760 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:58:53.366707    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-23477-predictor-747f9595bd-qjnx8" podUID="33b0417b-b6c3-47ab-85d3-5231073cc49e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 07 16:59:03.367220 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:59:03.367194    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-23477-predictor-747f9595bd-qjnx8"
Jul 07 16:59:14.990687 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:59:14.990655    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-23477-5bdcf4b799-f8gp5"]
Jul 07 16:59:14.993911 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:59:14.993895    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-23477-5bdcf4b799-f8gp5"
Jul 07 16:59:14.996286 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:59:14.996263    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-23477-serving-cert\""
Jul 07 16:59:14.996286 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:59:14.996277    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-23477-kube-rbac-proxy-sar-config\""
Jul 07 16:59:15.001640 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:59:15.001620    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-23477-5bdcf4b799-f8gp5"]
Jul 07 16:59:15.065613 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:59:15.065576    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0257ed48-2835-4a88-9564-8728d5196d1a-proxy-tls\") pod \"sequence-graph-23477-5bdcf4b799-f8gp5\" (UID: \"0257ed48-2835-4a88-9564-8728d5196d1a\") " pod="kserve-ci-e2e-test/sequence-graph-23477-5bdcf4b799-f8gp5"
Jul 07 16:59:15.065799 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:59:15.065683    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0257ed48-2835-4a88-9564-8728d5196d1a-openshift-service-ca-bundle\") pod \"sequence-graph-23477-5bdcf4b799-f8gp5\" (UID: \"0257ed48-2835-4a88-9564-8728d5196d1a\") " pod="kserve-ci-e2e-test/sequence-graph-23477-5bdcf4b799-f8gp5"
Jul 07 16:59:15.166922 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:59:15.166888    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0257ed48-2835-4a88-9564-8728d5196d1a-openshift-service-ca-bundle\") pod \"sequence-graph-23477-5bdcf4b799-f8gp5\" (UID: \"0257ed48-2835-4a88-9564-8728d5196d1a\") " pod="kserve-ci-e2e-test/sequence-graph-23477-5bdcf4b799-f8gp5"
Jul 07 16:59:15.167077 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:59:15.166940    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0257ed48-2835-4a88-9564-8728d5196d1a-proxy-tls\") pod \"sequence-graph-23477-5bdcf4b799-f8gp5\" (UID: \"0257ed48-2835-4a88-9564-8728d5196d1a\") " pod="kserve-ci-e2e-test/sequence-graph-23477-5bdcf4b799-f8gp5"
Jul 07 16:59:15.167077 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:59:15.167035    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-23477-serving-cert: secret "sequence-graph-23477-serving-cert" not found
Jul 07 16:59:15.167156 ip-10-0-136-217 kubenswrapper[2578]: E0707 16:59:15.167114    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0257ed48-2835-4a88-9564-8728d5196d1a-proxy-tls podName:0257ed48-2835-4a88-9564-8728d5196d1a nodeName:}" failed. No retries permitted until 2026-07-07 16:59:15.667097567 +0000 UTC m=+692.443411751 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0257ed48-2835-4a88-9564-8728d5196d1a-proxy-tls") pod "sequence-graph-23477-5bdcf4b799-f8gp5" (UID: "0257ed48-2835-4a88-9564-8728d5196d1a") : secret "sequence-graph-23477-serving-cert" not found
Jul 07 16:59:15.167472 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:59:15.167454    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0257ed48-2835-4a88-9564-8728d5196d1a-openshift-service-ca-bundle\") pod \"sequence-graph-23477-5bdcf4b799-f8gp5\" (UID: \"0257ed48-2835-4a88-9564-8728d5196d1a\") " pod="kserve-ci-e2e-test/sequence-graph-23477-5bdcf4b799-f8gp5"
Jul 07 16:59:15.672423 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:59:15.672391    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0257ed48-2835-4a88-9564-8728d5196d1a-proxy-tls\") pod \"sequence-graph-23477-5bdcf4b799-f8gp5\" (UID: \"0257ed48-2835-4a88-9564-8728d5196d1a\") " pod="kserve-ci-e2e-test/sequence-graph-23477-5bdcf4b799-f8gp5"
Jul 07 16:59:15.674596 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:59:15.674576    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0257ed48-2835-4a88-9564-8728d5196d1a-proxy-tls\") pod \"sequence-graph-23477-5bdcf4b799-f8gp5\" (UID: \"0257ed48-2835-4a88-9564-8728d5196d1a\") " pod="kserve-ci-e2e-test/sequence-graph-23477-5bdcf4b799-f8gp5"
Jul 07 16:59:15.904695 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:59:15.904661    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-23477-5bdcf4b799-f8gp5"
Jul 07 16:59:16.027621 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:59:16.027589    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-23477-5bdcf4b799-f8gp5"]
Jul 07 16:59:16.029931 ip-10-0-136-217 kubenswrapper[2578]: W0707 16:59:16.029906    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0257ed48_2835_4a88_9564_8728d5196d1a.slice/crio-cc98df596b8fb4f81b3828c84a02e7f9a3af6e9409910b76ceabd9fc8eec509e WatchSource:0}: Error finding container cc98df596b8fb4f81b3828c84a02e7f9a3af6e9409910b76ceabd9fc8eec509e: Status 404 returned error can't find the container with id cc98df596b8fb4f81b3828c84a02e7f9a3af6e9409910b76ceabd9fc8eec509e
Jul 07 16:59:16.602930 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:59:16.602898    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-23477-5bdcf4b799-f8gp5" event={"ID":"0257ed48-2835-4a88-9564-8728d5196d1a","Type":"ContainerStarted","Data":"2483a280ea316b0f4840099cd1395ef96db46a349cad8bf953c84ac077eadf74"}
Jul 07 16:59:16.603110 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:59:16.602949    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-23477-5bdcf4b799-f8gp5" event={"ID":"0257ed48-2835-4a88-9564-8728d5196d1a","Type":"ContainerStarted","Data":"cc98df596b8fb4f81b3828c84a02e7f9a3af6e9409910b76ceabd9fc8eec509e"}
Jul 07 16:59:16.603110 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:59:16.603013    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-23477-5bdcf4b799-f8gp5"
Jul 07 16:59:16.619551 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:59:16.619509    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-23477-5bdcf4b799-f8gp5" podStartSLOduration=2.6194949149999998 podStartE2EDuration="2.619494915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:59:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:59:16.619090065 +0000 UTC m=+693.395404272" watchObservedRunningTime="2026-07-07 16:59:16.619494915 +0000 UTC m=+693.395809157"
Jul 07 16:59:22.613312 ip-10-0-136-217 kubenswrapper[2578]: I0707 16:59:22.613285    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-23477-5bdcf4b799-f8gp5"
Jul 07 17:06:53.821412 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:06:53.821382    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-bb9ec-7fc9d47dd-vj2b5"]
Jul 07 17:06:53.823956 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:06:53.821614    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-bb9ec-7fc9d47dd-vj2b5" podUID="abb124b1-6682-43b7-9631-ee6671f24552" containerName="switch-graph-bb9ec" containerID="cri-o://e4e9a25800560897e9c1c0772702fe46259ba45569a685f0b1389be6af5783cb" gracePeriod=30
Jul 07 17:06:54.141754 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:06:54.141661    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2614e-predictor-d84c5d694-86rrx"]
Jul 07 17:06:54.145015 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:06:54.145000    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2614e-predictor-d84c5d694-86rrx"
Jul 07 17:06:54.147803 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:06:54.147781    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-2614e-predictor-serving-cert\""
Jul 07 17:06:54.148650 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:06:54.148632    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-2614e-kube-rbac-proxy-sar-config\""
Jul 07 17:06:54.157884 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:06:54.157866    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2614e-predictor-d84c5d694-86rrx"]
Jul 07 17:06:54.192875 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:06:54.192852    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bb9ec-predictor-546444857b-wjtqx"]
Jul 07 17:06:54.193060 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:06:54.193035    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-bb9ec-predictor-546444857b-wjtqx" podUID="b007968f-9b0f-4578-9700-41c6e197d572" containerName="kserve-container" containerID="cri-o://a094c8a119bf569c12dcd2e16d4bdcd61a369224cddbb5ec7b52357a5bb7b374" gracePeriod=30
Jul 07 17:06:54.193196 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:06:54.193083    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-bb9ec-predictor-546444857b-wjtqx" podUID="b007968f-9b0f-4578-9700-41c6e197d572" containerName="kube-rbac-proxy" containerID="cri-o://804545ff129d2687d2a04f770bf8d072a6bfa32bd67cc3c00f20539ce128fcc0" gracePeriod=30
Jul 07 17:06:54.222313 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:06:54.222288    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/287a25a6-c71c-4528-9176-a0e3e5bad741-proxy-tls\") pod \"error-404-isvc-2614e-predictor-d84c5d694-86rrx\" (UID: \"287a25a6-c71c-4528-9176-a0e3e5bad741\") " pod="kserve-ci-e2e-test/error-404-isvc-2614e-predictor-d84c5d694-86rrx"
Jul 07 17:06:54.222423 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:06:54.222318    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2szvf\" (UniqueName: \"kubernetes.io/projected/287a25a6-c71c-4528-9176-a0e3e5bad741-kube-api-access-2szvf\") pod \"error-404-isvc-2614e-predictor-d84c5d694-86rrx\" (UID: \"287a25a6-c71c-4528-9176-a0e3e5bad741\") " pod="kserve-ci-e2e-test/error-404-isvc-2614e-predictor-d84c5d694-86rrx"
Jul 07 17:06:54.222423 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:06:54.222395    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/287a25a6-c71c-4528-9176-a0e3e5bad741-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-2614e-predictor-d84c5d694-86rrx\" (UID: \"287a25a6-c71c-4528-9176-a0e3e5bad741\") " pod="kserve-ci-e2e-test/error-404-isvc-2614e-predictor-d84c5d694-86rrx"
Jul 07 17:06:54.323623 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:06:54.323592    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/287a25a6-c71c-4528-9176-a0e3e5bad741-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-2614e-predictor-d84c5d694-86rrx\" (UID: \"287a25a6-c71c-4528-9176-a0e3e5bad741\") " pod="kserve-ci-e2e-test/error-404-isvc-2614e-predictor-d84c5d694-86rrx"
Jul 07 17:06:54.323756 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:06:54.323693    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/287a25a6-c71c-4528-9176-a0e3e5bad741-proxy-tls\") pod \"error-404-isvc-2614e-predictor-d84c5d694-86rrx\" (UID: \"287a25a6-c71c-4528-9176-a0e3e5bad741\") " pod="kserve-ci-e2e-test/error-404-isvc-2614e-predictor-d84c5d694-86rrx"
Jul 07 17:06:54.323756 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:06:54.323722    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2szvf\" (UniqueName: \"kubernetes.io/projected/287a25a6-c71c-4528-9176-a0e3e5bad741-kube-api-access-2szvf\") pod \"error-404-isvc-2614e-predictor-d84c5d694-86rrx\" (UID: \"287a25a6-c71c-4528-9176-a0e3e5bad741\") " pod="kserve-ci-e2e-test/error-404-isvc-2614e-predictor-d84c5d694-86rrx"
Jul 07 17:06:54.323883 ip-10-0-136-217 kubenswrapper[2578]: E0707 17:06:54.323846    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-2614e-predictor-serving-cert: secret "error-404-isvc-2614e-predictor-serving-cert" not found
Jul 07 17:06:54.323940 ip-10-0-136-217 kubenswrapper[2578]: E0707 17:06:54.323911    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/287a25a6-c71c-4528-9176-a0e3e5bad741-proxy-tls podName:287a25a6-c71c-4528-9176-a0e3e5bad741 nodeName:}" failed. No retries permitted until 2026-07-07 17:06:54.823895408 +0000 UTC m=+1151.600209592 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/287a25a6-c71c-4528-9176-a0e3e5bad741-proxy-tls") pod "error-404-isvc-2614e-predictor-d84c5d694-86rrx" (UID: "287a25a6-c71c-4528-9176-a0e3e5bad741") : secret "error-404-isvc-2614e-predictor-serving-cert" not found
Jul 07 17:06:54.324233 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:06:54.324214    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/287a25a6-c71c-4528-9176-a0e3e5bad741-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-2614e-predictor-d84c5d694-86rrx\" (UID: \"287a25a6-c71c-4528-9176-a0e3e5bad741\") " pod="kserve-ci-e2e-test/error-404-isvc-2614e-predictor-d84c5d694-86rrx"
Jul 07 17:06:54.333566 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:06:54.333550    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2szvf\" (UniqueName: \"kubernetes.io/projected/287a25a6-c71c-4528-9176-a0e3e5bad741-kube-api-access-2szvf\") pod \"error-404-isvc-2614e-predictor-d84c5d694-86rrx\" (UID: \"287a25a6-c71c-4528-9176-a0e3e5bad741\") " pod="kserve-ci-e2e-test/error-404-isvc-2614e-predictor-d84c5d694-86rrx"
Jul 07 17:06:54.399540 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:06:54.399482    2578 generic.go:356] "Generic (PLEG): container finished" podID="b007968f-9b0f-4578-9700-41c6e197d572" containerID="804545ff129d2687d2a04f770bf8d072a6bfa32bd67cc3c00f20539ce128fcc0" exitCode=2
Jul 07 17:06:54.399642 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:06:54.399550    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bb9ec-predictor-546444857b-wjtqx" event={"ID":"b007968f-9b0f-4578-9700-41c6e197d572","Type":"ContainerDied","Data":"804545ff129d2687d2a04f770bf8d072a6bfa32bd67cc3c00f20539ce128fcc0"}
Jul 07 17:06:54.827350 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:06:54.827319    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/287a25a6-c71c-4528-9176-a0e3e5bad741-proxy-tls\") pod \"error-404-isvc-2614e-predictor-d84c5d694-86rrx\" (UID: \"287a25a6-c71c-4528-9176-a0e3e5bad741\") " pod="kserve-ci-e2e-test/error-404-isvc-2614e-predictor-d84c5d694-86rrx"
Jul 07 17:06:54.829655 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:06:54.829633    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/287a25a6-c71c-4528-9176-a0e3e5bad741-proxy-tls\") pod \"error-404-isvc-2614e-predictor-d84c5d694-86rrx\" (UID: \"287a25a6-c71c-4528-9176-a0e3e5bad741\") " pod="kserve-ci-e2e-test/error-404-isvc-2614e-predictor-d84c5d694-86rrx"
Jul 07 17:06:55.054904 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:06:55.054867    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2614e-predictor-d84c5d694-86rrx"
Jul 07 17:06:55.185090 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:06:55.185068    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2614e-predictor-d84c5d694-86rrx"]
Jul 07 17:06:55.187049 ip-10-0-136-217 kubenswrapper[2578]: W0707 17:06:55.187024    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod287a25a6_c71c_4528_9176_a0e3e5bad741.slice/crio-4692a2b2a4d48e0a4f5be0d9ad1fbd9137eebaeffe53c47079c0b50063bdd13f WatchSource:0}: Error finding container 4692a2b2a4d48e0a4f5be0d9ad1fbd9137eebaeffe53c47079c0b50063bdd13f: Status 404 returned error can't find the container with id 4692a2b2a4d48e0a4f5be0d9ad1fbd9137eebaeffe53c47079c0b50063bdd13f
Jul 07 17:06:55.405726 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:06:55.405637    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2614e-predictor-d84c5d694-86rrx" event={"ID":"287a25a6-c71c-4528-9176-a0e3e5bad741","Type":"ContainerStarted","Data":"c76d3837d97d0d42d009dc99b885502952a83e0a74459b61630a22ad36c775f8"}
Jul 07 17:06:55.405726 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:06:55.405693    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2614e-predictor-d84c5d694-86rrx" event={"ID":"287a25a6-c71c-4528-9176-a0e3e5bad741","Type":"ContainerStarted","Data":"2d313822db60f6bb20ad77b2af06b1a0a02723b9209a2b9e8d0d256d72f2a8a7"}
Jul 07 17:06:55.405726 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:06:55.405717    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-2614e-predictor-d84c5d694-86rrx"
Jul 07 17:06:55.405977 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:06:55.405750    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2614e-predictor-d84c5d694-86rrx" event={"ID":"287a25a6-c71c-4528-9176-a0e3e5bad741","Type":"ContainerStarted","Data":"4692a2b2a4d48e0a4f5be0d9ad1fbd9137eebaeffe53c47079c0b50063bdd13f"}
Jul 07 17:06:55.405977 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:06:55.405800    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-2614e-predictor-d84c5d694-86rrx"
Jul 07 17:06:55.407207 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:06:55.407185    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2614e-predictor-d84c5d694-86rrx" podUID="287a25a6-c71c-4528-9176-a0e3e5bad741" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 07 17:06:55.436086 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:06:55.436038    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-2614e-predictor-d84c5d694-86rrx" podStartSLOduration=1.436023678 podStartE2EDuration="1.436023678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:06:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:06:55.432213472 +0000 UTC m=+1152.208527678" watchObservedRunningTime="2026-07-07 17:06:55.436023678 +0000 UTC m=+1152.212337883"
Jul 07 17:06:56.410103 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:06:56.410058    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2614e-predictor-d84c5d694-86rrx" podUID="287a25a6-c71c-4528-9176-a0e3e5bad741" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 07 17:06:57.234551 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:06:57.234529    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-bb9ec-predictor-546444857b-wjtqx"
Jul 07 17:06:57.351026 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:06:57.350998    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b007968f-9b0f-4578-9700-41c6e197d572-proxy-tls\") pod \"b007968f-9b0f-4578-9700-41c6e197d572\" (UID: \"b007968f-9b0f-4578-9700-41c6e197d572\") "
Jul 07 17:06:57.351026 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:06:57.351029    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b007968f-9b0f-4578-9700-41c6e197d572-kube-rbac-proxy-sar-config\") pod \"b007968f-9b0f-4578-9700-41c6e197d572\" (UID: \"b007968f-9b0f-4578-9700-41c6e197d572\") "
Jul 07 17:06:57.351242 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:06:57.351090    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zvxjt\" (UniqueName: \"kubernetes.io/projected/b007968f-9b0f-4578-9700-41c6e197d572-kube-api-access-zvxjt\") pod \"b007968f-9b0f-4578-9700-41c6e197d572\" (UID: \"b007968f-9b0f-4578-9700-41c6e197d572\") "
Jul 07 17:06:57.351487 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:06:57.351456    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b007968f-9b0f-4578-9700-41c6e197d572-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b007968f-9b0f-4578-9700-41c6e197d572" (UID: "b007968f-9b0f-4578-9700-41c6e197d572"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:06:57.353195 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:06:57.353174    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b007968f-9b0f-4578-9700-41c6e197d572-kube-api-access-zvxjt" (OuterVolumeSpecName: "kube-api-access-zvxjt") pod "b007968f-9b0f-4578-9700-41c6e197d572" (UID: "b007968f-9b0f-4578-9700-41c6e197d572"). InnerVolumeSpecName "kube-api-access-zvxjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:06:57.353287 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:06:57.353256    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b007968f-9b0f-4578-9700-41c6e197d572-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b007968f-9b0f-4578-9700-41c6e197d572" (UID: "b007968f-9b0f-4578-9700-41c6e197d572"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 17:06:57.415309 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:06:57.415272    2578 generic.go:356] "Generic (PLEG): container finished" podID="b007968f-9b0f-4578-9700-41c6e197d572" containerID="a094c8a119bf569c12dcd2e16d4bdcd61a369224cddbb5ec7b52357a5bb7b374" exitCode=0
Jul 07 17:06:57.415711 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:06:57.415330    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-bb9ec-predictor-546444857b-wjtqx"
Jul 07 17:06:57.415711 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:06:57.415355    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bb9ec-predictor-546444857b-wjtqx" event={"ID":"b007968f-9b0f-4578-9700-41c6e197d572","Type":"ContainerDied","Data":"a094c8a119bf569c12dcd2e16d4bdcd61a369224cddbb5ec7b52357a5bb7b374"}
Jul 07 17:06:57.415711 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:06:57.415412    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bb9ec-predictor-546444857b-wjtqx" event={"ID":"b007968f-9b0f-4578-9700-41c6e197d572","Type":"ContainerDied","Data":"a8ce6f2c5cac6c5c38485a4ef1035c95fd2f9aa1dc8d05a2affbb4c44153b1f7"}
Jul 07 17:06:57.415711 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:06:57.415439    2578 scope.go:117] "RemoveContainer" containerID="804545ff129d2687d2a04f770bf8d072a6bfa32bd67cc3c00f20539ce128fcc0"
Jul 07 17:06:57.423724 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:06:57.423707    2578 scope.go:117] "RemoveContainer" containerID="a094c8a119bf569c12dcd2e16d4bdcd61a369224cddbb5ec7b52357a5bb7b374"
Jul 07 17:06:57.431229 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:06:57.431210    2578 scope.go:117] "RemoveContainer" containerID="804545ff129d2687d2a04f770bf8d072a6bfa32bd67cc3c00f20539ce128fcc0"
Jul 07 17:06:57.431500 ip-10-0-136-217 kubenswrapper[2578]: E0707 17:06:57.431481    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"804545ff129d2687d2a04f770bf8d072a6bfa32bd67cc3c00f20539ce128fcc0\": container with ID starting with 804545ff129d2687d2a04f770bf8d072a6bfa32bd67cc3c00f20539ce128fcc0 not found: ID does not exist" containerID="804545ff129d2687d2a04f770bf8d072a6bfa32bd67cc3c00f20539ce128fcc0"
Jul 07 17:06:57.431549 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:06:57.431507    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"804545ff129d2687d2a04f770bf8d072a6bfa32bd67cc3c00f20539ce128fcc0"} err="failed to get container status \"804545ff129d2687d2a04f770bf8d072a6bfa32bd67cc3c00f20539ce128fcc0\": rpc error: code = NotFound desc = could not find container \"804545ff129d2687d2a04f770bf8d072a6bfa32bd67cc3c00f20539ce128fcc0\": container with ID starting with 804545ff129d2687d2a04f770bf8d072a6bfa32bd67cc3c00f20539ce128fcc0 not found: ID does not exist"
Jul 07 17:06:57.431549 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:06:57.431523    2578 scope.go:117] "RemoveContainer" containerID="a094c8a119bf569c12dcd2e16d4bdcd61a369224cddbb5ec7b52357a5bb7b374"
Jul 07 17:06:57.431745 ip-10-0-136-217 kubenswrapper[2578]: E0707 17:06:57.431717    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a094c8a119bf569c12dcd2e16d4bdcd61a369224cddbb5ec7b52357a5bb7b374\": container with ID starting with a094c8a119bf569c12dcd2e16d4bdcd61a369224cddbb5ec7b52357a5bb7b374 not found: ID does not exist" containerID="a094c8a119bf569c12dcd2e16d4bdcd61a369224cddbb5ec7b52357a5bb7b374"
Jul 07 17:06:57.431808 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:06:57.431759    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a094c8a119bf569c12dcd2e16d4bdcd61a369224cddbb5ec7b52357a5bb7b374"} err="failed to get container status \"a094c8a119bf569c12dcd2e16d4bdcd61a369224cddbb5ec7b52357a5bb7b374\": rpc error: code = NotFound desc = could not find container \"a094c8a119bf569c12dcd2e16d4bdcd61a369224cddbb5ec7b52357a5bb7b374\": container with ID starting with a094c8a119bf569c12dcd2e16d4bdcd61a369224cddbb5ec7b52357a5bb7b374 not found: ID does not exist"
Jul 07 17:06:57.439014 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:06:57.438992    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bb9ec-predictor-546444857b-wjtqx"]
Jul 07 17:06:57.444045 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:06:57.444024    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bb9ec-predictor-546444857b-wjtqx"]
Jul 07 17:06:57.451905 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:06:57.451884    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvxjt\" (UniqueName: \"kubernetes.io/projected/b007968f-9b0f-4578-9700-41c6e197d572-kube-api-access-zvxjt\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 17:06:57.451905 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:06:57.451904    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b007968f-9b0f-4578-9700-41c6e197d572-proxy-tls\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 17:06:57.452009 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:06:57.451914    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b007968f-9b0f-4578-9700-41c6e197d572-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 17:06:57.725458 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:06:57.725384    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b007968f-9b0f-4578-9700-41c6e197d572" path="/var/lib/kubelet/pods/b007968f-9b0f-4578-9700-41c6e197d572/volumes"
Jul 07 17:06:58.205656 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:06:58.205602    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bb9ec-predictor-546444857b-wjtqx" podUID="b007968f-9b0f-4578-9700-41c6e197d572" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": context deadline exceeded"
Jul 07 17:06:58.210852 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:06:58.210824    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bb9ec-predictor-546444857b-wjtqx" podUID="b007968f-9b0f-4578-9700-41c6e197d572" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: i/o timeout"
Jul 07 17:06:58.479840 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:06:58.479745    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-bb9ec-7fc9d47dd-vj2b5" podUID="abb124b1-6682-43b7-9631-ee6671f24552" containerName="switch-graph-bb9ec" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:07:01.414619 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:01.414584    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-2614e-predictor-d84c5d694-86rrx"
Jul 07 17:07:01.415165 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:01.415140    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2614e-predictor-d84c5d694-86rrx" podUID="287a25a6-c71c-4528-9176-a0e3e5bad741" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 07 17:07:03.479606 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:03.479571    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-bb9ec-7fc9d47dd-vj2b5" podUID="abb124b1-6682-43b7-9631-ee6671f24552" containerName="switch-graph-bb9ec" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:07:08.480075 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:08.479988    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-bb9ec-7fc9d47dd-vj2b5" podUID="abb124b1-6682-43b7-9631-ee6671f24552" containerName="switch-graph-bb9ec" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:07:08.480482 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:08.480112    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-bb9ec-7fc9d47dd-vj2b5"
Jul 07 17:07:11.415917 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:11.415877    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2614e-predictor-d84c5d694-86rrx" podUID="287a25a6-c71c-4528-9176-a0e3e5bad741" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 07 17:07:13.479632 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:13.479599    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-bb9ec-7fc9d47dd-vj2b5" podUID="abb124b1-6682-43b7-9631-ee6671f24552" containerName="switch-graph-bb9ec" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:07:18.479039 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:18.478996    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-bb9ec-7fc9d47dd-vj2b5" podUID="abb124b1-6682-43b7-9631-ee6671f24552" containerName="switch-graph-bb9ec" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:07:21.416051 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:21.416015    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2614e-predictor-d84c5d694-86rrx" podUID="287a25a6-c71c-4528-9176-a0e3e5bad741" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 07 17:07:23.480192 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:23.480153    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-bb9ec-7fc9d47dd-vj2b5" podUID="abb124b1-6682-43b7-9631-ee6671f24552" containerName="switch-graph-bb9ec" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:07:23.968183 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:23.968158    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-bb9ec-7fc9d47dd-vj2b5"
Jul 07 17:07:24.083800 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:24.083695    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/abb124b1-6682-43b7-9631-ee6671f24552-openshift-service-ca-bundle\") pod \"abb124b1-6682-43b7-9631-ee6671f24552\" (UID: \"abb124b1-6682-43b7-9631-ee6671f24552\") "
Jul 07 17:07:24.083800 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:24.083772    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abb124b1-6682-43b7-9631-ee6671f24552-proxy-tls\") pod \"abb124b1-6682-43b7-9631-ee6671f24552\" (UID: \"abb124b1-6682-43b7-9631-ee6671f24552\") "
Jul 07 17:07:24.084112 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:24.084087    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abb124b1-6682-43b7-9631-ee6671f24552-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "abb124b1-6682-43b7-9631-ee6671f24552" (UID: "abb124b1-6682-43b7-9631-ee6671f24552"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:07:24.085811 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:24.085791    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abb124b1-6682-43b7-9631-ee6671f24552-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "abb124b1-6682-43b7-9631-ee6671f24552" (UID: "abb124b1-6682-43b7-9631-ee6671f24552"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 17:07:24.185324 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:24.185283    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/abb124b1-6682-43b7-9631-ee6671f24552-openshift-service-ca-bundle\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 17:07:24.185324 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:24.185320    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abb124b1-6682-43b7-9631-ee6671f24552-proxy-tls\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 17:07:24.533696 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:24.533621    2578 generic.go:356] "Generic (PLEG): container finished" podID="abb124b1-6682-43b7-9631-ee6671f24552" containerID="e4e9a25800560897e9c1c0772702fe46259ba45569a685f0b1389be6af5783cb" exitCode=0
Jul 07 17:07:24.533696 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:24.533670    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-bb9ec-7fc9d47dd-vj2b5"
Jul 07 17:07:24.534268 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:24.533704    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-bb9ec-7fc9d47dd-vj2b5" event={"ID":"abb124b1-6682-43b7-9631-ee6671f24552","Type":"ContainerDied","Data":"e4e9a25800560897e9c1c0772702fe46259ba45569a685f0b1389be6af5783cb"}
Jul 07 17:07:24.534268 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:24.533772    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-bb9ec-7fc9d47dd-vj2b5" event={"ID":"abb124b1-6682-43b7-9631-ee6671f24552","Type":"ContainerDied","Data":"4228c70be9f20475dae8d85a3d9c2d5d089802bba03fcce6db3d85b4f6eda965"}
Jul 07 17:07:24.534268 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:24.533794    2578 scope.go:117] "RemoveContainer" containerID="e4e9a25800560897e9c1c0772702fe46259ba45569a685f0b1389be6af5783cb"
Jul 07 17:07:24.542644 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:24.542628    2578 scope.go:117] "RemoveContainer" containerID="e4e9a25800560897e9c1c0772702fe46259ba45569a685f0b1389be6af5783cb"
Jul 07 17:07:24.542900 ip-10-0-136-217 kubenswrapper[2578]: E0707 17:07:24.542879    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4e9a25800560897e9c1c0772702fe46259ba45569a685f0b1389be6af5783cb\": container with ID starting with e4e9a25800560897e9c1c0772702fe46259ba45569a685f0b1389be6af5783cb not found: ID does not exist" containerID="e4e9a25800560897e9c1c0772702fe46259ba45569a685f0b1389be6af5783cb"
Jul 07 17:07:24.542984 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:24.542910    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4e9a25800560897e9c1c0772702fe46259ba45569a685f0b1389be6af5783cb"} err="failed to get container status \"e4e9a25800560897e9c1c0772702fe46259ba45569a685f0b1389be6af5783cb\": rpc error: code = NotFound desc = could not find container \"e4e9a25800560897e9c1c0772702fe46259ba45569a685f0b1389be6af5783cb\": container with ID starting with e4e9a25800560897e9c1c0772702fe46259ba45569a685f0b1389be6af5783cb not found: ID does not exist"
Jul 07 17:07:24.556384 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:24.556353    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-bb9ec-7fc9d47dd-vj2b5"]
Jul 07 17:07:24.562757 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:24.562721    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-bb9ec-7fc9d47dd-vj2b5"]
Jul 07 17:07:25.725487 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:25.725445    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abb124b1-6682-43b7-9631-ee6671f24552" path="/var/lib/kubelet/pods/abb124b1-6682-43b7-9631-ee6671f24552/volumes"
Jul 07 17:07:29.671342 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:29.671313    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-23477-5bdcf4b799-f8gp5"]
Jul 07 17:07:29.672164 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:29.671521    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-23477-5bdcf4b799-f8gp5" podUID="0257ed48-2835-4a88-9564-8728d5196d1a" containerName="sequence-graph-23477" containerID="cri-o://2483a280ea316b0f4840099cd1395ef96db46a349cad8bf953c84ac077eadf74" gracePeriod=30
Jul 07 17:07:30.022280 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:30.022191    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8411f-predictor-c4654dd58-qkkp6"]
Jul 07 17:07:30.022605 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:30.022589    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abb124b1-6682-43b7-9631-ee6671f24552" containerName="switch-graph-bb9ec"
Jul 07 17:07:30.022688 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:30.022609    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="abb124b1-6682-43b7-9631-ee6671f24552" containerName="switch-graph-bb9ec"
Jul 07 17:07:30.022688 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:30.022622    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b007968f-9b0f-4578-9700-41c6e197d572" containerName="kserve-container"
Jul 07 17:07:30.022688 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:30.022629    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b007968f-9b0f-4578-9700-41c6e197d572" containerName="kserve-container"
Jul 07 17:07:30.022688 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:30.022680    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b007968f-9b0f-4578-9700-41c6e197d572" containerName="kube-rbac-proxy"
Jul 07 17:07:30.022923 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:30.022691    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b007968f-9b0f-4578-9700-41c6e197d572" containerName="kube-rbac-proxy"
Jul 07 17:07:30.022923 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:30.022788    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b007968f-9b0f-4578-9700-41c6e197d572" containerName="kserve-container"
Jul 07 17:07:30.022923 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:30.022805    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="abb124b1-6682-43b7-9631-ee6671f24552" containerName="switch-graph-bb9ec"
Jul 07 17:07:30.022923 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:30.022814    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b007968f-9b0f-4578-9700-41c6e197d572" containerName="kube-rbac-proxy"
Jul 07 17:07:30.027202 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:30.027183    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8411f-predictor-c4654dd58-qkkp6"
Jul 07 17:07:30.031472 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:30.031442    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-8411f-predictor-serving-cert\""
Jul 07 17:07:30.031580 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:30.031487    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-8411f-kube-rbac-proxy-sar-config\""
Jul 07 17:07:30.048230 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:30.048202    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8411f-predictor-c4654dd58-qkkp6"]
Jul 07 17:07:30.094996 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:30.094967    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-23477-predictor-747f9595bd-qjnx8"]
Jul 07 17:07:30.095202 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:30.095176    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-23477-predictor-747f9595bd-qjnx8" podUID="33b0417b-b6c3-47ab-85d3-5231073cc49e" containerName="kserve-container" containerID="cri-o://39a6f50003be77e1f9bc8fbd76acf9f6ab59574a4b1053748a1d3853f1d4875e" gracePeriod=30
Jul 07 17:07:30.095263 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:30.095211    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-23477-predictor-747f9595bd-qjnx8" podUID="33b0417b-b6c3-47ab-85d3-5231073cc49e" containerName="kube-rbac-proxy" containerID="cri-o://b94578533cfba63618877e261f5f5ff5b1411bac60304f100ba838723fb6cc5c" gracePeriod=30
Jul 07 17:07:30.134846 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:30.134823    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/179eb22e-f62d-4e1c-9377-0fa1ce2f634f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8411f-predictor-c4654dd58-qkkp6\" (UID: \"179eb22e-f62d-4e1c-9377-0fa1ce2f634f\") " pod="kserve-ci-e2e-test/error-404-isvc-8411f-predictor-c4654dd58-qkkp6"
Jul 07 17:07:30.134974 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:30.134856    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/179eb22e-f62d-4e1c-9377-0fa1ce2f634f-proxy-tls\") pod \"error-404-isvc-8411f-predictor-c4654dd58-qkkp6\" (UID: \"179eb22e-f62d-4e1c-9377-0fa1ce2f634f\") " pod="kserve-ci-e2e-test/error-404-isvc-8411f-predictor-c4654dd58-qkkp6"
Jul 07 17:07:30.134974 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:30.134897    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52zzn\" (UniqueName: \"kubernetes.io/projected/179eb22e-f62d-4e1c-9377-0fa1ce2f634f-kube-api-access-52zzn\") pod \"error-404-isvc-8411f-predictor-c4654dd58-qkkp6\" (UID: \"179eb22e-f62d-4e1c-9377-0fa1ce2f634f\") " pod="kserve-ci-e2e-test/error-404-isvc-8411f-predictor-c4654dd58-qkkp6"
Jul 07 17:07:30.235851 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:30.235801    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/179eb22e-f62d-4e1c-9377-0fa1ce2f634f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8411f-predictor-c4654dd58-qkkp6\" (UID: \"179eb22e-f62d-4e1c-9377-0fa1ce2f634f\") " pod="kserve-ci-e2e-test/error-404-isvc-8411f-predictor-c4654dd58-qkkp6"
Jul 07 17:07:30.235851 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:30.235863    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/179eb22e-f62d-4e1c-9377-0fa1ce2f634f-proxy-tls\") pod \"error-404-isvc-8411f-predictor-c4654dd58-qkkp6\" (UID: \"179eb22e-f62d-4e1c-9377-0fa1ce2f634f\") " pod="kserve-ci-e2e-test/error-404-isvc-8411f-predictor-c4654dd58-qkkp6"
Jul 07 17:07:30.236086 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:30.235903    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52zzn\" (UniqueName: \"kubernetes.io/projected/179eb22e-f62d-4e1c-9377-0fa1ce2f634f-kube-api-access-52zzn\") pod \"error-404-isvc-8411f-predictor-c4654dd58-qkkp6\" (UID: \"179eb22e-f62d-4e1c-9377-0fa1ce2f634f\") " pod="kserve-ci-e2e-test/error-404-isvc-8411f-predictor-c4654dd58-qkkp6"
Jul 07 17:07:30.236086 ip-10-0-136-217 kubenswrapper[2578]: E0707 17:07:30.236032    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-8411f-predictor-serving-cert: secret "error-404-isvc-8411f-predictor-serving-cert" not found
Jul 07 17:07:30.236202 ip-10-0-136-217 kubenswrapper[2578]: E0707 17:07:30.236122    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/179eb22e-f62d-4e1c-9377-0fa1ce2f634f-proxy-tls podName:179eb22e-f62d-4e1c-9377-0fa1ce2f634f nodeName:}" failed. No retries permitted until 2026-07-07 17:07:30.736099533 +0000 UTC m=+1187.512413730 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/179eb22e-f62d-4e1c-9377-0fa1ce2f634f-proxy-tls") pod "error-404-isvc-8411f-predictor-c4654dd58-qkkp6" (UID: "179eb22e-f62d-4e1c-9377-0fa1ce2f634f") : secret "error-404-isvc-8411f-predictor-serving-cert" not found
Jul 07 17:07:30.236479 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:30.236458    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/179eb22e-f62d-4e1c-9377-0fa1ce2f634f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8411f-predictor-c4654dd58-qkkp6\" (UID: \"179eb22e-f62d-4e1c-9377-0fa1ce2f634f\") " pod="kserve-ci-e2e-test/error-404-isvc-8411f-predictor-c4654dd58-qkkp6"
Jul 07 17:07:30.247167 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:30.247147    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52zzn\" (UniqueName: \"kubernetes.io/projected/179eb22e-f62d-4e1c-9377-0fa1ce2f634f-kube-api-access-52zzn\") pod \"error-404-isvc-8411f-predictor-c4654dd58-qkkp6\" (UID: \"179eb22e-f62d-4e1c-9377-0fa1ce2f634f\") " pod="kserve-ci-e2e-test/error-404-isvc-8411f-predictor-c4654dd58-qkkp6"
Jul 07 17:07:30.561507 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:30.561472    2578 generic.go:356] "Generic (PLEG): container finished" podID="33b0417b-b6c3-47ab-85d3-5231073cc49e" containerID="b94578533cfba63618877e261f5f5ff5b1411bac60304f100ba838723fb6cc5c" exitCode=2
Jul 07 17:07:30.561669 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:30.561547    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-23477-predictor-747f9595bd-qjnx8" event={"ID":"33b0417b-b6c3-47ab-85d3-5231073cc49e","Type":"ContainerDied","Data":"b94578533cfba63618877e261f5f5ff5b1411bac60304f100ba838723fb6cc5c"}
Jul 07 17:07:30.738917 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:30.738883    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/179eb22e-f62d-4e1c-9377-0fa1ce2f634f-proxy-tls\") pod \"error-404-isvc-8411f-predictor-c4654dd58-qkkp6\" (UID: \"179eb22e-f62d-4e1c-9377-0fa1ce2f634f\") " pod="kserve-ci-e2e-test/error-404-isvc-8411f-predictor-c4654dd58-qkkp6"
Jul 07 17:07:30.741184 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:30.741165    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/179eb22e-f62d-4e1c-9377-0fa1ce2f634f-proxy-tls\") pod \"error-404-isvc-8411f-predictor-c4654dd58-qkkp6\" (UID: \"179eb22e-f62d-4e1c-9377-0fa1ce2f634f\") " pod="kserve-ci-e2e-test/error-404-isvc-8411f-predictor-c4654dd58-qkkp6"
Jul 07 17:07:30.937924 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:30.937833    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8411f-predictor-c4654dd58-qkkp6"
Jul 07 17:07:31.071477 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:31.071448    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8411f-predictor-c4654dd58-qkkp6"]
Jul 07 17:07:31.073434 ip-10-0-136-217 kubenswrapper[2578]: W0707 17:07:31.073407    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod179eb22e_f62d_4e1c_9377_0fa1ce2f634f.slice/crio-8a95d697d1b013b313b07c9afa623d9aa3fa6a51a601f4845d6a0e6c24e88c04 WatchSource:0}: Error finding container 8a95d697d1b013b313b07c9afa623d9aa3fa6a51a601f4845d6a0e6c24e88c04: Status 404 returned error can't find the container with id 8a95d697d1b013b313b07c9afa623d9aa3fa6a51a601f4845d6a0e6c24e88c04
Jul 07 17:07:31.415975 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:31.415937    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2614e-predictor-d84c5d694-86rrx" podUID="287a25a6-c71c-4528-9176-a0e3e5bad741" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 07 17:07:31.568233 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:31.568202    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8411f-predictor-c4654dd58-qkkp6" event={"ID":"179eb22e-f62d-4e1c-9377-0fa1ce2f634f","Type":"ContainerStarted","Data":"e3af5f93dd02579a481d27cf215898490d14b3fc903681e7d0b88e3f444000de"}
Jul 07 17:07:31.568397 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:31.568254    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8411f-predictor-c4654dd58-qkkp6" event={"ID":"179eb22e-f62d-4e1c-9377-0fa1ce2f634f","Type":"ContainerStarted","Data":"eb92591c2949c7374d4b80806befe06522f8e41fb5ffe63ad50083f68f6c550b"}
Jul 07 17:07:31.568397 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:31.568274    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8411f-predictor-c4654dd58-qkkp6" event={"ID":"179eb22e-f62d-4e1c-9377-0fa1ce2f634f","Type":"ContainerStarted","Data":"8a95d697d1b013b313b07c9afa623d9aa3fa6a51a601f4845d6a0e6c24e88c04"}
Jul 07 17:07:31.568397 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:31.568361    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-8411f-predictor-c4654dd58-qkkp6"
Jul 07 17:07:31.587431 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:31.587391    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-8411f-predictor-c4654dd58-qkkp6" podStartSLOduration=2.587376835 podStartE2EDuration="2.587376835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:07:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:07:31.586226661 +0000 UTC m=+1188.362540887" watchObservedRunningTime="2026-07-07 17:07:31.587376835 +0000 UTC m=+1188.363691042"
Jul 07 17:07:32.572876 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:32.572843    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-8411f-predictor-c4654dd58-qkkp6"
Jul 07 17:07:32.573771 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:32.573719    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8411f-predictor-c4654dd58-qkkp6" podUID="179eb22e-f62d-4e1c-9377-0fa1ce2f634f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 07 17:07:32.612203 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:32.612173    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-23477-5bdcf4b799-f8gp5" podUID="0257ed48-2835-4a88-9564-8728d5196d1a" containerName="sequence-graph-23477" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:07:33.340303 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:33.340273    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-23477-predictor-747f9595bd-qjnx8"
Jul 07 17:07:33.461307 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:33.461222    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5h44x\" (UniqueName: \"kubernetes.io/projected/33b0417b-b6c3-47ab-85d3-5231073cc49e-kube-api-access-5h44x\") pod \"33b0417b-b6c3-47ab-85d3-5231073cc49e\" (UID: \"33b0417b-b6c3-47ab-85d3-5231073cc49e\") "
Jul 07 17:07:33.461461 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:33.461316    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33b0417b-b6c3-47ab-85d3-5231073cc49e-kube-rbac-proxy-sar-config\") pod \"33b0417b-b6c3-47ab-85d3-5231073cc49e\" (UID: \"33b0417b-b6c3-47ab-85d3-5231073cc49e\") "
Jul 07 17:07:33.461461 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:33.461371    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33b0417b-b6c3-47ab-85d3-5231073cc49e-proxy-tls\") pod \"33b0417b-b6c3-47ab-85d3-5231073cc49e\" (UID: \"33b0417b-b6c3-47ab-85d3-5231073cc49e\") "
Jul 07 17:07:33.461673 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:33.461651    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33b0417b-b6c3-47ab-85d3-5231073cc49e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "33b0417b-b6c3-47ab-85d3-5231073cc49e" (UID: "33b0417b-b6c3-47ab-85d3-5231073cc49e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:07:33.463209 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:33.463181    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33b0417b-b6c3-47ab-85d3-5231073cc49e-kube-api-access-5h44x" (OuterVolumeSpecName: "kube-api-access-5h44x") pod "33b0417b-b6c3-47ab-85d3-5231073cc49e" (UID: "33b0417b-b6c3-47ab-85d3-5231073cc49e"). InnerVolumeSpecName "kube-api-access-5h44x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:07:33.463316 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:33.463269    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33b0417b-b6c3-47ab-85d3-5231073cc49e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "33b0417b-b6c3-47ab-85d3-5231073cc49e" (UID: "33b0417b-b6c3-47ab-85d3-5231073cc49e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 17:07:33.562433 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:33.562390    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5h44x\" (UniqueName: \"kubernetes.io/projected/33b0417b-b6c3-47ab-85d3-5231073cc49e-kube-api-access-5h44x\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 17:07:33.562433 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:33.562431    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33b0417b-b6c3-47ab-85d3-5231073cc49e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 17:07:33.562433 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:33.562441    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33b0417b-b6c3-47ab-85d3-5231073cc49e-proxy-tls\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 17:07:33.586380 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:33.586345    2578 generic.go:356] "Generic (PLEG): container finished" podID="33b0417b-b6c3-47ab-85d3-5231073cc49e" containerID="39a6f50003be77e1f9bc8fbd76acf9f6ab59574a4b1053748a1d3853f1d4875e" exitCode=0
Jul 07 17:07:33.586834 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:33.586405    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-23477-predictor-747f9595bd-qjnx8"
Jul 07 17:07:33.586834 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:33.586443    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-23477-predictor-747f9595bd-qjnx8" event={"ID":"33b0417b-b6c3-47ab-85d3-5231073cc49e","Type":"ContainerDied","Data":"39a6f50003be77e1f9bc8fbd76acf9f6ab59574a4b1053748a1d3853f1d4875e"}
Jul 07 17:07:33.586834 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:33.586527    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-23477-predictor-747f9595bd-qjnx8" event={"ID":"33b0417b-b6c3-47ab-85d3-5231073cc49e","Type":"ContainerDied","Data":"f08823fc937df4e26f16ef6504e416fec68dbe18d5963c2fed068d195b37e62f"}
Jul 07 17:07:33.586834 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:33.586564    2578 scope.go:117] "RemoveContainer" containerID="b94578533cfba63618877e261f5f5ff5b1411bac60304f100ba838723fb6cc5c"
Jul 07 17:07:33.587027 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:33.586986    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8411f-predictor-c4654dd58-qkkp6" podUID="179eb22e-f62d-4e1c-9377-0fa1ce2f634f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 07 17:07:33.595216 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:33.595197    2578 scope.go:117] "RemoveContainer" containerID="39a6f50003be77e1f9bc8fbd76acf9f6ab59574a4b1053748a1d3853f1d4875e"
Jul 07 17:07:33.602722 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:33.602707    2578 scope.go:117] "RemoveContainer" containerID="b94578533cfba63618877e261f5f5ff5b1411bac60304f100ba838723fb6cc5c"
Jul 07 17:07:33.602967 ip-10-0-136-217 kubenswrapper[2578]: E0707 17:07:33.602951    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b94578533cfba63618877e261f5f5ff5b1411bac60304f100ba838723fb6cc5c\": container with ID starting with b94578533cfba63618877e261f5f5ff5b1411bac60304f100ba838723fb6cc5c not found: ID does not exist" containerID="b94578533cfba63618877e261f5f5ff5b1411bac60304f100ba838723fb6cc5c"
Jul 07 17:07:33.603044 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:33.602976    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b94578533cfba63618877e261f5f5ff5b1411bac60304f100ba838723fb6cc5c"} err="failed to get container status \"b94578533cfba63618877e261f5f5ff5b1411bac60304f100ba838723fb6cc5c\": rpc error: code = NotFound desc = could not find container \"b94578533cfba63618877e261f5f5ff5b1411bac60304f100ba838723fb6cc5c\": container with ID starting with b94578533cfba63618877e261f5f5ff5b1411bac60304f100ba838723fb6cc5c not found: ID does not exist"
Jul 07 17:07:33.603044 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:33.602996    2578 scope.go:117] "RemoveContainer" containerID="39a6f50003be77e1f9bc8fbd76acf9f6ab59574a4b1053748a1d3853f1d4875e"
Jul 07 17:07:33.603210 ip-10-0-136-217 kubenswrapper[2578]: E0707 17:07:33.603192    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39a6f50003be77e1f9bc8fbd76acf9f6ab59574a4b1053748a1d3853f1d4875e\": container with ID starting with 39a6f50003be77e1f9bc8fbd76acf9f6ab59574a4b1053748a1d3853f1d4875e not found: ID does not exist" containerID="39a6f50003be77e1f9bc8fbd76acf9f6ab59574a4b1053748a1d3853f1d4875e"
Jul 07 17:07:33.603248 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:33.603216    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39a6f50003be77e1f9bc8fbd76acf9f6ab59574a4b1053748a1d3853f1d4875e"} err="failed to get container status \"39a6f50003be77e1f9bc8fbd76acf9f6ab59574a4b1053748a1d3853f1d4875e\": rpc error: code = NotFound desc = could not find container \"39a6f50003be77e1f9bc8fbd76acf9f6ab59574a4b1053748a1d3853f1d4875e\": container with ID starting with 39a6f50003be77e1f9bc8fbd76acf9f6ab59574a4b1053748a1d3853f1d4875e not found: ID does not exist"
Jul 07 17:07:33.612594 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:33.612572    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-23477-predictor-747f9595bd-qjnx8"]
Jul 07 17:07:33.615802 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:33.615784    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-23477-predictor-747f9595bd-qjnx8"]
Jul 07 17:07:33.725110 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:33.725037    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33b0417b-b6c3-47ab-85d3-5231073cc49e" path="/var/lib/kubelet/pods/33b0417b-b6c3-47ab-85d3-5231073cc49e/volumes"
Jul 07 17:07:37.612509 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:37.612474    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-23477-5bdcf4b799-f8gp5" podUID="0257ed48-2835-4a88-9564-8728d5196d1a" containerName="sequence-graph-23477" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:07:38.592436 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:38.592402    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-8411f-predictor-c4654dd58-qkkp6"
Jul 07 17:07:38.593055 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:38.593028    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8411f-predictor-c4654dd58-qkkp6" podUID="179eb22e-f62d-4e1c-9377-0fa1ce2f634f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 07 17:07:41.415831 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:41.415794    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2614e-predictor-d84c5d694-86rrx" podUID="287a25a6-c71c-4528-9176-a0e3e5bad741" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 07 17:07:42.612461 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:42.612421    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-23477-5bdcf4b799-f8gp5" podUID="0257ed48-2835-4a88-9564-8728d5196d1a" containerName="sequence-graph-23477" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:07:42.612872 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:42.612530    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-23477-5bdcf4b799-f8gp5"
Jul 07 17:07:47.611950 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:47.611912    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-23477-5bdcf4b799-f8gp5" podUID="0257ed48-2835-4a88-9564-8728d5196d1a" containerName="sequence-graph-23477" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:07:48.593462 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:48.593421    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8411f-predictor-c4654dd58-qkkp6" podUID="179eb22e-f62d-4e1c-9377-0fa1ce2f634f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 07 17:07:51.415880 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:51.415845    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-2614e-predictor-d84c5d694-86rrx"
Jul 07 17:07:52.612398 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:52.612356    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-23477-5bdcf4b799-f8gp5" podUID="0257ed48-2835-4a88-9564-8728d5196d1a" containerName="sequence-graph-23477" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:07:54.072545 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:54.072515    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-2614e-67d7467f6-786nc"]
Jul 07 17:07:54.072942 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:54.072878    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33b0417b-b6c3-47ab-85d3-5231073cc49e" containerName="kube-rbac-proxy"
Jul 07 17:07:54.072942 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:54.072890    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="33b0417b-b6c3-47ab-85d3-5231073cc49e" containerName="kube-rbac-proxy"
Jul 07 17:07:54.072942 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:54.072899    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33b0417b-b6c3-47ab-85d3-5231073cc49e" containerName="kserve-container"
Jul 07 17:07:54.072942 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:54.072905    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="33b0417b-b6c3-47ab-85d3-5231073cc49e" containerName="kserve-container"
Jul 07 17:07:54.073070 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:54.072953    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="33b0417b-b6c3-47ab-85d3-5231073cc49e" containerName="kserve-container"
Jul 07 17:07:54.073070 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:54.072959    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="33b0417b-b6c3-47ab-85d3-5231073cc49e" containerName="kube-rbac-proxy"
Jul 07 17:07:54.077381 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:54.077367    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-2614e-67d7467f6-786nc"
Jul 07 17:07:54.079761 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:54.079728    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-2614e-serving-cert\""
Jul 07 17:07:54.079849 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:54.079757    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-2614e-kube-rbac-proxy-sar-config\""
Jul 07 17:07:54.082854 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:54.082833    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-2614e-67d7467f6-786nc"]
Jul 07 17:07:54.240046 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:54.240015    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e9f1188-b89b-445b-ab8c-e21ccb68cafb-openshift-service-ca-bundle\") pod \"ensemble-graph-2614e-67d7467f6-786nc\" (UID: \"0e9f1188-b89b-445b-ab8c-e21ccb68cafb\") " pod="kserve-ci-e2e-test/ensemble-graph-2614e-67d7467f6-786nc"
Jul 07 17:07:54.240218 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:54.240056    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e9f1188-b89b-445b-ab8c-e21ccb68cafb-proxy-tls\") pod \"ensemble-graph-2614e-67d7467f6-786nc\" (UID: \"0e9f1188-b89b-445b-ab8c-e21ccb68cafb\") " pod="kserve-ci-e2e-test/ensemble-graph-2614e-67d7467f6-786nc"
Jul 07 17:07:54.340687 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:54.340599    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e9f1188-b89b-445b-ab8c-e21ccb68cafb-openshift-service-ca-bundle\") pod \"ensemble-graph-2614e-67d7467f6-786nc\" (UID: \"0e9f1188-b89b-445b-ab8c-e21ccb68cafb\") " pod="kserve-ci-e2e-test/ensemble-graph-2614e-67d7467f6-786nc"
Jul 07 17:07:54.340687 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:54.340649    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e9f1188-b89b-445b-ab8c-e21ccb68cafb-proxy-tls\") pod \"ensemble-graph-2614e-67d7467f6-786nc\" (UID: \"0e9f1188-b89b-445b-ab8c-e21ccb68cafb\") " pod="kserve-ci-e2e-test/ensemble-graph-2614e-67d7467f6-786nc"
Jul 07 17:07:54.341255 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:54.341227    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e9f1188-b89b-445b-ab8c-e21ccb68cafb-openshift-service-ca-bundle\") pod \"ensemble-graph-2614e-67d7467f6-786nc\" (UID: \"0e9f1188-b89b-445b-ab8c-e21ccb68cafb\") " pod="kserve-ci-e2e-test/ensemble-graph-2614e-67d7467f6-786nc"
Jul 07 17:07:54.343083 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:54.343062    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e9f1188-b89b-445b-ab8c-e21ccb68cafb-proxy-tls\") pod \"ensemble-graph-2614e-67d7467f6-786nc\" (UID: \"0e9f1188-b89b-445b-ab8c-e21ccb68cafb\") " pod="kserve-ci-e2e-test/ensemble-graph-2614e-67d7467f6-786nc"
Jul 07 17:07:54.387994 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:54.387971    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-2614e-67d7467f6-786nc"
Jul 07 17:07:54.512073 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:54.512041    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-2614e-67d7467f6-786nc"]
Jul 07 17:07:54.514886 ip-10-0-136-217 kubenswrapper[2578]: W0707 17:07:54.514860    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e9f1188_b89b_445b_ab8c_e21ccb68cafb.slice/crio-9104a761e5e0e656f6a4adf838a5e0b6018d149553de19c3ed41034e72af5958 WatchSource:0}: Error finding container 9104a761e5e0e656f6a4adf838a5e0b6018d149553de19c3ed41034e72af5958: Status 404 returned error can't find the container with id 9104a761e5e0e656f6a4adf838a5e0b6018d149553de19c3ed41034e72af5958
Jul 07 17:07:54.670407 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:54.670320    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-2614e-67d7467f6-786nc" event={"ID":"0e9f1188-b89b-445b-ab8c-e21ccb68cafb","Type":"ContainerStarted","Data":"6edc1c574d3de68e3a55f889c800c1a1da4fa507a009a1c9dc31a3538c6abead"}
Jul 07 17:07:54.670407 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:54.670374    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-2614e-67d7467f6-786nc" event={"ID":"0e9f1188-b89b-445b-ab8c-e21ccb68cafb","Type":"ContainerStarted","Data":"9104a761e5e0e656f6a4adf838a5e0b6018d149553de19c3ed41034e72af5958"}
Jul 07 17:07:54.670598 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:54.670431    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-2614e-67d7467f6-786nc"
Jul 07 17:07:54.687415 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:54.687373    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-2614e-67d7467f6-786nc" podStartSLOduration=0.687358641 podStartE2EDuration="687.358641ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:07:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:07:54.686176264 +0000 UTC m=+1211.462490473" watchObservedRunningTime="2026-07-07 17:07:54.687358641 +0000 UTC m=+1211.463672880"
Jul 07 17:07:57.611586 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:57.611550    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-23477-5bdcf4b799-f8gp5" podUID="0257ed48-2835-4a88-9564-8728d5196d1a" containerName="sequence-graph-23477" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:07:58.593682 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:58.593645    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8411f-predictor-c4654dd58-qkkp6" podUID="179eb22e-f62d-4e1c-9377-0fa1ce2f634f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 07 17:07:59.692058 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:59.692032    2578 generic.go:356] "Generic (PLEG): container finished" podID="0257ed48-2835-4a88-9564-8728d5196d1a" containerID="2483a280ea316b0f4840099cd1395ef96db46a349cad8bf953c84ac077eadf74" exitCode=0
Jul 07 17:07:59.692345 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:59.692100    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-23477-5bdcf4b799-f8gp5" event={"ID":"0257ed48-2835-4a88-9564-8728d5196d1a","Type":"ContainerDied","Data":"2483a280ea316b0f4840099cd1395ef96db46a349cad8bf953c84ac077eadf74"}
Jul 07 17:07:59.817256 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:59.817233    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-23477-5bdcf4b799-f8gp5"
Jul 07 17:07:59.888816 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:59.888781    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0257ed48-2835-4a88-9564-8728d5196d1a-proxy-tls\") pod \"0257ed48-2835-4a88-9564-8728d5196d1a\" (UID: \"0257ed48-2835-4a88-9564-8728d5196d1a\") "
Jul 07 17:07:59.888980 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:59.888877    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0257ed48-2835-4a88-9564-8728d5196d1a-openshift-service-ca-bundle\") pod \"0257ed48-2835-4a88-9564-8728d5196d1a\" (UID: \"0257ed48-2835-4a88-9564-8728d5196d1a\") "
Jul 07 17:07:59.889234 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:59.889216    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0257ed48-2835-4a88-9564-8728d5196d1a-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "0257ed48-2835-4a88-9564-8728d5196d1a" (UID: "0257ed48-2835-4a88-9564-8728d5196d1a"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:07:59.890842 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:59.890821    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0257ed48-2835-4a88-9564-8728d5196d1a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0257ed48-2835-4a88-9564-8728d5196d1a" (UID: "0257ed48-2835-4a88-9564-8728d5196d1a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 17:07:59.990148 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:59.990066    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0257ed48-2835-4a88-9564-8728d5196d1a-openshift-service-ca-bundle\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 17:07:59.990148 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:07:59.990093    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0257ed48-2835-4a88-9564-8728d5196d1a-proxy-tls\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 17:08:00.680474 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:00.680447    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-2614e-67d7467f6-786nc"
Jul 07 17:08:00.698054 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:00.698030    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-23477-5bdcf4b799-f8gp5"
Jul 07 17:08:00.698432 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:00.698072    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-23477-5bdcf4b799-f8gp5" event={"ID":"0257ed48-2835-4a88-9564-8728d5196d1a","Type":"ContainerDied","Data":"cc98df596b8fb4f81b3828c84a02e7f9a3af6e9409910b76ceabd9fc8eec509e"}
Jul 07 17:08:00.698432 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:00.698125    2578 scope.go:117] "RemoveContainer" containerID="2483a280ea316b0f4840099cd1395ef96db46a349cad8bf953c84ac077eadf74"
Jul 07 17:08:00.724766 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:00.724728    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-23477-5bdcf4b799-f8gp5"]
Jul 07 17:08:00.729098 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:00.729076    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-23477-5bdcf4b799-f8gp5"]
Jul 07 17:08:01.724691 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:01.724658    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0257ed48-2835-4a88-9564-8728d5196d1a" path="/var/lib/kubelet/pods/0257ed48-2835-4a88-9564-8728d5196d1a/volumes"
Jul 07 17:08:04.161223 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:04.161146    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-2614e-67d7467f6-786nc"]
Jul 07 17:08:04.161606 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:04.161335    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-2614e-67d7467f6-786nc" podUID="0e9f1188-b89b-445b-ab8c-e21ccb68cafb" containerName="ensemble-graph-2614e" containerID="cri-o://6edc1c574d3de68e3a55f889c800c1a1da4fa507a009a1c9dc31a3538c6abead" gracePeriod=30
Jul 07 17:08:04.242620 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:04.242591    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2614e-predictor-d84c5d694-86rrx"]
Jul 07 17:08:04.242859 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:04.242835    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-2614e-predictor-d84c5d694-86rrx" podUID="287a25a6-c71c-4528-9176-a0e3e5bad741" containerName="kserve-container" containerID="cri-o://2d313822db60f6bb20ad77b2af06b1a0a02723b9209a2b9e8d0d256d72f2a8a7" gracePeriod=30
Jul 07 17:08:04.242931 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:04.242881    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-2614e-predictor-d84c5d694-86rrx" podUID="287a25a6-c71c-4528-9176-a0e3e5bad741" containerName="kube-rbac-proxy" containerID="cri-o://c76d3837d97d0d42d009dc99b885502952a83e0a74459b61630a22ad36c775f8" gracePeriod=30
Jul 07 17:08:04.405976 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:04.405943    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f8ada-predictor-7c47cd4646-qc5fc"]
Jul 07 17:08:04.406264 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:04.406252    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0257ed48-2835-4a88-9564-8728d5196d1a" containerName="sequence-graph-23477"
Jul 07 17:08:04.406306 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:04.406265    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0257ed48-2835-4a88-9564-8728d5196d1a" containerName="sequence-graph-23477"
Jul 07 17:08:04.406345 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:04.406331    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0257ed48-2835-4a88-9564-8728d5196d1a" containerName="sequence-graph-23477"
Jul 07 17:08:04.409533 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:04.409519    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f8ada-predictor-7c47cd4646-qc5fc"
Jul 07 17:08:04.412116 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:04.412055    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-f8ada-predictor-serving-cert\""
Jul 07 17:08:04.412251 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:04.412202    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-f8ada-kube-rbac-proxy-sar-config\""
Jul 07 17:08:04.420120 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:04.420099    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f8ada-predictor-7c47cd4646-qc5fc"]
Jul 07 17:08:04.426186 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:04.426166    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drcs8\" (UniqueName: \"kubernetes.io/projected/4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d-kube-api-access-drcs8\") pod \"error-404-isvc-f8ada-predictor-7c47cd4646-qc5fc\" (UID: \"4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d\") " pod="kserve-ci-e2e-test/error-404-isvc-f8ada-predictor-7c47cd4646-qc5fc"
Jul 07 17:08:04.426276 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:04.426196    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d-proxy-tls\") pod \"error-404-isvc-f8ada-predictor-7c47cd4646-qc5fc\" (UID: \"4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d\") " pod="kserve-ci-e2e-test/error-404-isvc-f8ada-predictor-7c47cd4646-qc5fc"
Jul 07 17:08:04.426276 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:04.426263    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f8ada-predictor-7c47cd4646-qc5fc\" (UID: \"4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d\") " pod="kserve-ci-e2e-test/error-404-isvc-f8ada-predictor-7c47cd4646-qc5fc"
Jul 07 17:08:04.527509 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:04.527465    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drcs8\" (UniqueName: \"kubernetes.io/projected/4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d-kube-api-access-drcs8\") pod \"error-404-isvc-f8ada-predictor-7c47cd4646-qc5fc\" (UID: \"4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d\") " pod="kserve-ci-e2e-test/error-404-isvc-f8ada-predictor-7c47cd4646-qc5fc"
Jul 07 17:08:04.527710 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:04.527520    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d-proxy-tls\") pod \"error-404-isvc-f8ada-predictor-7c47cd4646-qc5fc\" (UID: \"4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d\") " pod="kserve-ci-e2e-test/error-404-isvc-f8ada-predictor-7c47cd4646-qc5fc"
Jul 07 17:08:04.527710 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:04.527617    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f8ada-predictor-7c47cd4646-qc5fc\" (UID: \"4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d\") " pod="kserve-ci-e2e-test/error-404-isvc-f8ada-predictor-7c47cd4646-qc5fc"
Jul 07 17:08:04.527710 ip-10-0-136-217 kubenswrapper[2578]: E0707 17:08:04.527686    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-f8ada-predictor-serving-cert: secret "error-404-isvc-f8ada-predictor-serving-cert" not found
Jul 07 17:08:04.527943 ip-10-0-136-217 kubenswrapper[2578]: E0707 17:08:04.527779    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d-proxy-tls podName:4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d nodeName:}" failed. No retries permitted until 2026-07-07 17:08:05.027757136 +0000 UTC m=+1221.804071335 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d-proxy-tls") pod "error-404-isvc-f8ada-predictor-7c47cd4646-qc5fc" (UID: "4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d") : secret "error-404-isvc-f8ada-predictor-serving-cert" not found
Jul 07 17:08:04.528304 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:04.528283    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f8ada-predictor-7c47cd4646-qc5fc\" (UID: \"4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d\") " pod="kserve-ci-e2e-test/error-404-isvc-f8ada-predictor-7c47cd4646-qc5fc"
Jul 07 17:08:04.538564 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:04.538531    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drcs8\" (UniqueName: \"kubernetes.io/projected/4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d-kube-api-access-drcs8\") pod \"error-404-isvc-f8ada-predictor-7c47cd4646-qc5fc\" (UID: \"4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d\") " pod="kserve-ci-e2e-test/error-404-isvc-f8ada-predictor-7c47cd4646-qc5fc"
Jul 07 17:08:04.715913 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:04.715827    2578 generic.go:356] "Generic (PLEG): container finished" podID="287a25a6-c71c-4528-9176-a0e3e5bad741" containerID="c76d3837d97d0d42d009dc99b885502952a83e0a74459b61630a22ad36c775f8" exitCode=2
Jul 07 17:08:04.715913 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:04.715873    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2614e-predictor-d84c5d694-86rrx" event={"ID":"287a25a6-c71c-4528-9176-a0e3e5bad741","Type":"ContainerDied","Data":"c76d3837d97d0d42d009dc99b885502952a83e0a74459b61630a22ad36c775f8"}
Jul 07 17:08:05.033248 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:05.033149    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d-proxy-tls\") pod \"error-404-isvc-f8ada-predictor-7c47cd4646-qc5fc\" (UID: \"4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d\") " pod="kserve-ci-e2e-test/error-404-isvc-f8ada-predictor-7c47cd4646-qc5fc"
Jul 07 17:08:05.035675 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:05.035633    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d-proxy-tls\") pod \"error-404-isvc-f8ada-predictor-7c47cd4646-qc5fc\" (UID: \"4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d\") " pod="kserve-ci-e2e-test/error-404-isvc-f8ada-predictor-7c47cd4646-qc5fc"
Jul 07 17:08:05.319827 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:05.319790    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f8ada-predictor-7c47cd4646-qc5fc"
Jul 07 17:08:05.446625 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:05.446598    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f8ada-predictor-7c47cd4646-qc5fc"]
Jul 07 17:08:05.449076 ip-10-0-136-217 kubenswrapper[2578]: W0707 17:08:05.449047    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4be9ad02_f0f9_42bf_a0d1_95e129d7bf9d.slice/crio-5f1a405616e9a08d6022cb9a383f51ffbd395f136e88ff5875e7c065cbbe0c62 WatchSource:0}: Error finding container 5f1a405616e9a08d6022cb9a383f51ffbd395f136e88ff5875e7c065cbbe0c62: Status 404 returned error can't find the container with id 5f1a405616e9a08d6022cb9a383f51ffbd395f136e88ff5875e7c065cbbe0c62
Jul 07 17:08:05.678597 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:05.678561    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-2614e-67d7467f6-786nc" podUID="0e9f1188-b89b-445b-ab8c-e21ccb68cafb" containerName="ensemble-graph-2614e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:08:05.724520 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:05.724476    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f8ada-predictor-7c47cd4646-qc5fc" podUID="4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 07 17:08:05.732156 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:05.732122    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f8ada-predictor-7c47cd4646-qc5fc" event={"ID":"4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d","Type":"ContainerStarted","Data":"275c69b0ea66e38186e786d91aef1a96fba173974749ddb9f05f40388147bec2"}
Jul 07 17:08:05.732325 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:05.732191    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f8ada-predictor-7c47cd4646-qc5fc" event={"ID":"4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d","Type":"ContainerStarted","Data":"9193e4fe4b7d69ab86d4190d3f7bbad378cc5229a630f77abfb9e1e2fa4ca521"}
Jul 07 17:08:05.732325 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:05.732230    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-f8ada-predictor-7c47cd4646-qc5fc"
Jul 07 17:08:05.732325 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:05.732260    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-f8ada-predictor-7c47cd4646-qc5fc"
Jul 07 17:08:05.732325 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:05.732291    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f8ada-predictor-7c47cd4646-qc5fc" event={"ID":"4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d","Type":"ContainerStarted","Data":"5f1a405616e9a08d6022cb9a383f51ffbd395f136e88ff5875e7c065cbbe0c62"}
Jul 07 17:08:05.740753 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:05.740683    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-f8ada-predictor-7c47cd4646-qc5fc" podStartSLOduration=1.740667845 podStartE2EDuration="1.740667845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:08:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:08:05.739100501 +0000 UTC m=+1222.515414726" watchObservedRunningTime="2026-07-07 17:08:05.740667845 +0000 UTC m=+1222.516982051"
Jul 07 17:08:06.410572 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:06.410525    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2614e-predictor-d84c5d694-86rrx" podUID="287a25a6-c71c-4528-9176-a0e3e5bad741" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 07 17:08:06.726290 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:06.726200    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f8ada-predictor-7c47cd4646-qc5fc" podUID="4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 07 17:08:07.488614 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:07.488585    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2614e-predictor-d84c5d694-86rrx"
Jul 07 17:08:07.556151 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:07.556120    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2szvf\" (UniqueName: \"kubernetes.io/projected/287a25a6-c71c-4528-9176-a0e3e5bad741-kube-api-access-2szvf\") pod \"287a25a6-c71c-4528-9176-a0e3e5bad741\" (UID: \"287a25a6-c71c-4528-9176-a0e3e5bad741\") "
Jul 07 17:08:07.556305 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:07.556162    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/287a25a6-c71c-4528-9176-a0e3e5bad741-proxy-tls\") pod \"287a25a6-c71c-4528-9176-a0e3e5bad741\" (UID: \"287a25a6-c71c-4528-9176-a0e3e5bad741\") "
Jul 07 17:08:07.556305 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:07.556203    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/287a25a6-c71c-4528-9176-a0e3e5bad741-kube-rbac-proxy-sar-config\") pod \"287a25a6-c71c-4528-9176-a0e3e5bad741\" (UID: \"287a25a6-c71c-4528-9176-a0e3e5bad741\") "
Jul 07 17:08:07.556606 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:07.556582    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/287a25a6-c71c-4528-9176-a0e3e5bad741-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "287a25a6-c71c-4528-9176-a0e3e5bad741" (UID: "287a25a6-c71c-4528-9176-a0e3e5bad741"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:08:07.558265 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:07.558241    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/287a25a6-c71c-4528-9176-a0e3e5bad741-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "287a25a6-c71c-4528-9176-a0e3e5bad741" (UID: "287a25a6-c71c-4528-9176-a0e3e5bad741"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 17:08:07.558338 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:07.558250    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/287a25a6-c71c-4528-9176-a0e3e5bad741-kube-api-access-2szvf" (OuterVolumeSpecName: "kube-api-access-2szvf") pod "287a25a6-c71c-4528-9176-a0e3e5bad741" (UID: "287a25a6-c71c-4528-9176-a0e3e5bad741"). InnerVolumeSpecName "kube-api-access-2szvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:08:07.657164 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:07.657085    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2szvf\" (UniqueName: \"kubernetes.io/projected/287a25a6-c71c-4528-9176-a0e3e5bad741-kube-api-access-2szvf\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 17:08:07.657164 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:07.657111    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/287a25a6-c71c-4528-9176-a0e3e5bad741-proxy-tls\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 17:08:07.657164 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:07.657120    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/287a25a6-c71c-4528-9176-a0e3e5bad741-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 17:08:07.733276 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:07.733023    2578 generic.go:356] "Generic (PLEG): container finished" podID="287a25a6-c71c-4528-9176-a0e3e5bad741" containerID="2d313822db60f6bb20ad77b2af06b1a0a02723b9209a2b9e8d0d256d72f2a8a7" exitCode=0
Jul 07 17:08:07.733276 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:07.733085    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2614e-predictor-d84c5d694-86rrx"
Jul 07 17:08:07.733276 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:07.733213    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2614e-predictor-d84c5d694-86rrx" event={"ID":"287a25a6-c71c-4528-9176-a0e3e5bad741","Type":"ContainerDied","Data":"2d313822db60f6bb20ad77b2af06b1a0a02723b9209a2b9e8d0d256d72f2a8a7"}
Jul 07 17:08:07.733549 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:07.733313    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2614e-predictor-d84c5d694-86rrx" event={"ID":"287a25a6-c71c-4528-9176-a0e3e5bad741","Type":"ContainerDied","Data":"4692a2b2a4d48e0a4f5be0d9ad1fbd9137eebaeffe53c47079c0b50063bdd13f"}
Jul 07 17:08:07.733549 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:07.733384    2578 scope.go:117] "RemoveContainer" containerID="c76d3837d97d0d42d009dc99b885502952a83e0a74459b61630a22ad36c775f8"
Jul 07 17:08:07.742030 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:07.742014    2578 scope.go:117] "RemoveContainer" containerID="2d313822db60f6bb20ad77b2af06b1a0a02723b9209a2b9e8d0d256d72f2a8a7"
Jul 07 17:08:07.749585 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:07.749565    2578 scope.go:117] "RemoveContainer" containerID="c76d3837d97d0d42d009dc99b885502952a83e0a74459b61630a22ad36c775f8"
Jul 07 17:08:07.749657 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:07.749601    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2614e-predictor-d84c5d694-86rrx"]
Jul 07 17:08:07.749849 ip-10-0-136-217 kubenswrapper[2578]: E0707 17:08:07.749830    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c76d3837d97d0d42d009dc99b885502952a83e0a74459b61630a22ad36c775f8\": container with ID starting with c76d3837d97d0d42d009dc99b885502952a83e0a74459b61630a22ad36c775f8 not found: ID does not exist" containerID="c76d3837d97d0d42d009dc99b885502952a83e0a74459b61630a22ad36c775f8"
Jul 07 17:08:07.749897 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:07.749857    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c76d3837d97d0d42d009dc99b885502952a83e0a74459b61630a22ad36c775f8"} err="failed to get container status \"c76d3837d97d0d42d009dc99b885502952a83e0a74459b61630a22ad36c775f8\": rpc error: code = NotFound desc = could not find container \"c76d3837d97d0d42d009dc99b885502952a83e0a74459b61630a22ad36c775f8\": container with ID starting with c76d3837d97d0d42d009dc99b885502952a83e0a74459b61630a22ad36c775f8 not found: ID does not exist"
Jul 07 17:08:07.749897 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:07.749875    2578 scope.go:117] "RemoveContainer" containerID="2d313822db60f6bb20ad77b2af06b1a0a02723b9209a2b9e8d0d256d72f2a8a7"
Jul 07 17:08:07.750094 ip-10-0-136-217 kubenswrapper[2578]: E0707 17:08:07.750079    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d313822db60f6bb20ad77b2af06b1a0a02723b9209a2b9e8d0d256d72f2a8a7\": container with ID starting with 2d313822db60f6bb20ad77b2af06b1a0a02723b9209a2b9e8d0d256d72f2a8a7 not found: ID does not exist" containerID="2d313822db60f6bb20ad77b2af06b1a0a02723b9209a2b9e8d0d256d72f2a8a7"
Jul 07 17:08:07.750142 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:07.750098    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d313822db60f6bb20ad77b2af06b1a0a02723b9209a2b9e8d0d256d72f2a8a7"} err="failed to get container status \"2d313822db60f6bb20ad77b2af06b1a0a02723b9209a2b9e8d0d256d72f2a8a7\": rpc error: code = NotFound desc = could not find container \"2d313822db60f6bb20ad77b2af06b1a0a02723b9209a2b9e8d0d256d72f2a8a7\": container with ID starting with 2d313822db60f6bb20ad77b2af06b1a0a02723b9209a2b9e8d0d256d72f2a8a7 not found: ID does not exist"
Jul 07 17:08:07.753580 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:07.753559    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2614e-predictor-d84c5d694-86rrx"]
Jul 07 17:08:08.593435 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:08.593403    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8411f-predictor-c4654dd58-qkkp6" podUID="179eb22e-f62d-4e1c-9377-0fa1ce2f634f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 07 17:08:09.724692 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:09.724663    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="287a25a6-c71c-4528-9176-a0e3e5bad741" path="/var/lib/kubelet/pods/287a25a6-c71c-4528-9176-a0e3e5bad741/volumes"
Jul 07 17:08:10.678451 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:10.678415    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-2614e-67d7467f6-786nc" podUID="0e9f1188-b89b-445b-ab8c-e21ccb68cafb" containerName="ensemble-graph-2614e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:08:11.731947 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:11.731916    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-f8ada-predictor-7c47cd4646-qc5fc"
Jul 07 17:08:11.732620 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:11.732585    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f8ada-predictor-7c47cd4646-qc5fc" podUID="4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 07 17:08:15.678353 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:15.678317    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-2614e-67d7467f6-786nc" podUID="0e9f1188-b89b-445b-ab8c-e21ccb68cafb" containerName="ensemble-graph-2614e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:08:15.678728 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:15.678423    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-2614e-67d7467f6-786nc"
Jul 07 17:08:18.593902 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:18.593868    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-8411f-predictor-c4654dd58-qkkp6"
Jul 07 17:08:20.677766 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:20.677710    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-2614e-67d7467f6-786nc" podUID="0e9f1188-b89b-445b-ab8c-e21ccb68cafb" containerName="ensemble-graph-2614e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:08:21.732663 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:21.732626    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f8ada-predictor-7c47cd4646-qc5fc" podUID="4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 07 17:08:25.678085 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:25.678041    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-2614e-67d7467f6-786nc" podUID="0e9f1188-b89b-445b-ab8c-e21ccb68cafb" containerName="ensemble-graph-2614e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:08:29.851872 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:29.851844    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-8411f-f58b567b8-llfwp"]
Jul 07 17:08:29.852244 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:29.852181    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="287a25a6-c71c-4528-9176-a0e3e5bad741" containerName="kube-rbac-proxy"
Jul 07 17:08:29.852244 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:29.852195    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="287a25a6-c71c-4528-9176-a0e3e5bad741" containerName="kube-rbac-proxy"
Jul 07 17:08:29.852244 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:29.852207    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="287a25a6-c71c-4528-9176-a0e3e5bad741" containerName="kserve-container"
Jul 07 17:08:29.852244 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:29.852212    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="287a25a6-c71c-4528-9176-a0e3e5bad741" containerName="kserve-container"
Jul 07 17:08:29.852379 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:29.852259    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="287a25a6-c71c-4528-9176-a0e3e5bad741" containerName="kube-rbac-proxy"
Jul 07 17:08:29.852379 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:29.852268    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="287a25a6-c71c-4528-9176-a0e3e5bad741" containerName="kserve-container"
Jul 07 17:08:29.855174 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:29.855158    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-8411f-f58b567b8-llfwp"
Jul 07 17:08:29.857622 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:29.857604    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-8411f-serving-cert\""
Jul 07 17:08:29.857720 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:29.857684    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-8411f-kube-rbac-proxy-sar-config\""
Jul 07 17:08:29.864852 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:29.864835    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-8411f-f58b567b8-llfwp"]
Jul 07 17:08:29.947754 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:29.947705    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12669d4a-108e-499b-954f-55cdeda16f68-openshift-service-ca-bundle\") pod \"sequence-graph-8411f-f58b567b8-llfwp\" (UID: \"12669d4a-108e-499b-954f-55cdeda16f68\") " pod="kserve-ci-e2e-test/sequence-graph-8411f-f58b567b8-llfwp"
Jul 07 17:08:29.947905 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:29.947828    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12669d4a-108e-499b-954f-55cdeda16f68-proxy-tls\") pod \"sequence-graph-8411f-f58b567b8-llfwp\" (UID: \"12669d4a-108e-499b-954f-55cdeda16f68\") " pod="kserve-ci-e2e-test/sequence-graph-8411f-f58b567b8-llfwp"
Jul 07 17:08:30.048912 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:30.048876    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12669d4a-108e-499b-954f-55cdeda16f68-proxy-tls\") pod \"sequence-graph-8411f-f58b567b8-llfwp\" (UID: \"12669d4a-108e-499b-954f-55cdeda16f68\") " pod="kserve-ci-e2e-test/sequence-graph-8411f-f58b567b8-llfwp"
Jul 07 17:08:30.049071 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:30.048937    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12669d4a-108e-499b-954f-55cdeda16f68-openshift-service-ca-bundle\") pod \"sequence-graph-8411f-f58b567b8-llfwp\" (UID: \"12669d4a-108e-499b-954f-55cdeda16f68\") " pod="kserve-ci-e2e-test/sequence-graph-8411f-f58b567b8-llfwp"
Jul 07 17:08:30.049071 ip-10-0-136-217 kubenswrapper[2578]: E0707 17:08:30.049042    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-8411f-serving-cert: secret "sequence-graph-8411f-serving-cert" not found
Jul 07 17:08:30.049140 ip-10-0-136-217 kubenswrapper[2578]: E0707 17:08:30.049129    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12669d4a-108e-499b-954f-55cdeda16f68-proxy-tls podName:12669d4a-108e-499b-954f-55cdeda16f68 nodeName:}" failed. No retries permitted until 2026-07-07 17:08:30.54910874 +0000 UTC m=+1247.325422937 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/12669d4a-108e-499b-954f-55cdeda16f68-proxy-tls") pod "sequence-graph-8411f-f58b567b8-llfwp" (UID: "12669d4a-108e-499b-954f-55cdeda16f68") : secret "sequence-graph-8411f-serving-cert" not found
Jul 07 17:08:30.049617 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:30.049599    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12669d4a-108e-499b-954f-55cdeda16f68-openshift-service-ca-bundle\") pod \"sequence-graph-8411f-f58b567b8-llfwp\" (UID: \"12669d4a-108e-499b-954f-55cdeda16f68\") " pod="kserve-ci-e2e-test/sequence-graph-8411f-f58b567b8-llfwp"
Jul 07 17:08:30.552688 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:30.552640    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12669d4a-108e-499b-954f-55cdeda16f68-proxy-tls\") pod \"sequence-graph-8411f-f58b567b8-llfwp\" (UID: \"12669d4a-108e-499b-954f-55cdeda16f68\") " pod="kserve-ci-e2e-test/sequence-graph-8411f-f58b567b8-llfwp"
Jul 07 17:08:30.555044 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:30.555025    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12669d4a-108e-499b-954f-55cdeda16f68-proxy-tls\") pod \"sequence-graph-8411f-f58b567b8-llfwp\" (UID: \"12669d4a-108e-499b-954f-55cdeda16f68\") " pod="kserve-ci-e2e-test/sequence-graph-8411f-f58b567b8-llfwp"
Jul 07 17:08:30.677745 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:30.677692    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-2614e-67d7467f6-786nc" podUID="0e9f1188-b89b-445b-ab8c-e21ccb68cafb" containerName="ensemble-graph-2614e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:08:30.765986 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:30.765956    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-8411f-f58b567b8-llfwp"
Jul 07 17:08:30.897086 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:30.897060    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-8411f-f58b567b8-llfwp"]
Jul 07 17:08:30.899645 ip-10-0-136-217 kubenswrapper[2578]: W0707 17:08:30.899621    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12669d4a_108e_499b_954f_55cdeda16f68.slice/crio-3048dead96ee4f02ea165d7668a630f4420a49c78e96cc8fb8e114f6fabdc1bc WatchSource:0}: Error finding container 3048dead96ee4f02ea165d7668a630f4420a49c78e96cc8fb8e114f6fabdc1bc: Status 404 returned error can't find the container with id 3048dead96ee4f02ea165d7668a630f4420a49c78e96cc8fb8e114f6fabdc1bc
Jul 07 17:08:31.732839 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:31.732798    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f8ada-predictor-7c47cd4646-qc5fc" podUID="4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 07 17:08:31.835425 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:31.835391    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-8411f-f58b567b8-llfwp" event={"ID":"12669d4a-108e-499b-954f-55cdeda16f68","Type":"ContainerStarted","Data":"537efea55f46dbd6964c3fc9bac4d34c203766f861b27685ca9c57447c5b2d63"}
Jul 07 17:08:31.835614 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:31.835450    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-8411f-f58b567b8-llfwp" event={"ID":"12669d4a-108e-499b-954f-55cdeda16f68","Type":"ContainerStarted","Data":"3048dead96ee4f02ea165d7668a630f4420a49c78e96cc8fb8e114f6fabdc1bc"}
Jul 07 17:08:31.835614 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:31.835515    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-8411f-f58b567b8-llfwp"
Jul 07 17:08:31.852129 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:31.852082    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-8411f-f58b567b8-llfwp" podStartSLOduration=2.852068397 podStartE2EDuration="2.852068397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:08:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:08:31.850627981 +0000 UTC m=+1248.626942209" watchObservedRunningTime="2026-07-07 17:08:31.852068397 +0000 UTC m=+1248.628382648"
Jul 07 17:08:34.305922 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:34.305894    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-2614e-67d7467f6-786nc"
Jul 07 17:08:34.385698 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:34.385664    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e9f1188-b89b-445b-ab8c-e21ccb68cafb-openshift-service-ca-bundle\") pod \"0e9f1188-b89b-445b-ab8c-e21ccb68cafb\" (UID: \"0e9f1188-b89b-445b-ab8c-e21ccb68cafb\") "
Jul 07 17:08:34.385889 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:34.385721    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e9f1188-b89b-445b-ab8c-e21ccb68cafb-proxy-tls\") pod \"0e9f1188-b89b-445b-ab8c-e21ccb68cafb\" (UID: \"0e9f1188-b89b-445b-ab8c-e21ccb68cafb\") "
Jul 07 17:08:34.386046 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:34.386024    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e9f1188-b89b-445b-ab8c-e21ccb68cafb-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "0e9f1188-b89b-445b-ab8c-e21ccb68cafb" (UID: "0e9f1188-b89b-445b-ab8c-e21ccb68cafb"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:08:34.387795 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:34.387774    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e9f1188-b89b-445b-ab8c-e21ccb68cafb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0e9f1188-b89b-445b-ab8c-e21ccb68cafb" (UID: "0e9f1188-b89b-445b-ab8c-e21ccb68cafb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 17:08:34.486693 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:34.486614    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e9f1188-b89b-445b-ab8c-e21ccb68cafb-proxy-tls\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 17:08:34.486693 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:34.486638    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e9f1188-b89b-445b-ab8c-e21ccb68cafb-openshift-service-ca-bundle\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 17:08:34.849203 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:34.849169    2578 generic.go:356] "Generic (PLEG): container finished" podID="0e9f1188-b89b-445b-ab8c-e21ccb68cafb" containerID="6edc1c574d3de68e3a55f889c800c1a1da4fa507a009a1c9dc31a3538c6abead" exitCode=0
Jul 07 17:08:34.849359 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:34.849217    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-2614e-67d7467f6-786nc"
Jul 07 17:08:34.849359 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:34.849250    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-2614e-67d7467f6-786nc" event={"ID":"0e9f1188-b89b-445b-ab8c-e21ccb68cafb","Type":"ContainerDied","Data":"6edc1c574d3de68e3a55f889c800c1a1da4fa507a009a1c9dc31a3538c6abead"}
Jul 07 17:08:34.849359 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:34.849301    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-2614e-67d7467f6-786nc" event={"ID":"0e9f1188-b89b-445b-ab8c-e21ccb68cafb","Type":"ContainerDied","Data":"9104a761e5e0e656f6a4adf838a5e0b6018d149553de19c3ed41034e72af5958"}
Jul 07 17:08:34.849359 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:34.849323    2578 scope.go:117] "RemoveContainer" containerID="6edc1c574d3de68e3a55f889c800c1a1da4fa507a009a1c9dc31a3538c6abead"
Jul 07 17:08:34.857659 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:34.857643    2578 scope.go:117] "RemoveContainer" containerID="6edc1c574d3de68e3a55f889c800c1a1da4fa507a009a1c9dc31a3538c6abead"
Jul 07 17:08:34.857946 ip-10-0-136-217 kubenswrapper[2578]: E0707 17:08:34.857929    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6edc1c574d3de68e3a55f889c800c1a1da4fa507a009a1c9dc31a3538c6abead\": container with ID starting with 6edc1c574d3de68e3a55f889c800c1a1da4fa507a009a1c9dc31a3538c6abead not found: ID does not exist" containerID="6edc1c574d3de68e3a55f889c800c1a1da4fa507a009a1c9dc31a3538c6abead"
Jul 07 17:08:34.858031 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:34.857954    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6edc1c574d3de68e3a55f889c800c1a1da4fa507a009a1c9dc31a3538c6abead"} err="failed to get container status \"6edc1c574d3de68e3a55f889c800c1a1da4fa507a009a1c9dc31a3538c6abead\": rpc error: code = NotFound desc = could not find container \"6edc1c574d3de68e3a55f889c800c1a1da4fa507a009a1c9dc31a3538c6abead\": container with ID starting with 6edc1c574d3de68e3a55f889c800c1a1da4fa507a009a1c9dc31a3538c6abead not found: ID does not exist"
Jul 07 17:08:34.871711 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:34.871691    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-2614e-67d7467f6-786nc"]
Jul 07 17:08:34.875129 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:34.875112    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-2614e-67d7467f6-786nc"]
Jul 07 17:08:35.725432 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:35.725390    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e9f1188-b89b-445b-ab8c-e21ccb68cafb" path="/var/lib/kubelet/pods/0e9f1188-b89b-445b-ab8c-e21ccb68cafb/volumes"
Jul 07 17:08:37.845316 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:37.845288    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-8411f-f58b567b8-llfwp"
Jul 07 17:08:39.938648 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:39.938616    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-8411f-f58b567b8-llfwp"]
Jul 07 17:08:39.939108 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:39.938829    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-8411f-f58b567b8-llfwp" podUID="12669d4a-108e-499b-954f-55cdeda16f68" containerName="sequence-graph-8411f" containerID="cri-o://537efea55f46dbd6964c3fc9bac4d34c203766f861b27685ca9c57447c5b2d63" gracePeriod=30
Jul 07 17:08:40.143065 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:40.143036    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8411f-predictor-c4654dd58-qkkp6"]
Jul 07 17:08:40.143311 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:40.143261    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-8411f-predictor-c4654dd58-qkkp6" podUID="179eb22e-f62d-4e1c-9377-0fa1ce2f634f" containerName="kserve-container" containerID="cri-o://eb92591c2949c7374d4b80806befe06522f8e41fb5ffe63ad50083f68f6c550b" gracePeriod=30
Jul 07 17:08:40.143398 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:40.143341    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-8411f-predictor-c4654dd58-qkkp6" podUID="179eb22e-f62d-4e1c-9377-0fa1ce2f634f" containerName="kube-rbac-proxy" containerID="cri-o://e3af5f93dd02579a481d27cf215898490d14b3fc903681e7d0b88e3f444000de" gracePeriod=30
Jul 07 17:08:40.187229 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:40.187201    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3653b-predictor-799d8b94f6-dj9g8"]
Jul 07 17:08:40.187563 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:40.187549    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e9f1188-b89b-445b-ab8c-e21ccb68cafb" containerName="ensemble-graph-2614e"
Jul 07 17:08:40.187658 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:40.187566    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e9f1188-b89b-445b-ab8c-e21ccb68cafb" containerName="ensemble-graph-2614e"
Jul 07 17:08:40.187719 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:40.187663    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e9f1188-b89b-445b-ab8c-e21ccb68cafb" containerName="ensemble-graph-2614e"
Jul 07 17:08:40.192152 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:40.192102    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3653b-predictor-799d8b94f6-dj9g8"
Jul 07 17:08:40.194403 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:40.194387    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-3653b-kube-rbac-proxy-sar-config\""
Jul 07 17:08:40.194500 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:40.194437    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-3653b-predictor-serving-cert\""
Jul 07 17:08:40.201631 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:40.201608    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3653b-predictor-799d8b94f6-dj9g8"]
Jul 07 17:08:40.337715 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:40.337676    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebca297e-b62c-4edb-add5-c6a2001be637-proxy-tls\") pod \"error-404-isvc-3653b-predictor-799d8b94f6-dj9g8\" (UID: \"ebca297e-b62c-4edb-add5-c6a2001be637\") " pod="kserve-ci-e2e-test/error-404-isvc-3653b-predictor-799d8b94f6-dj9g8"
Jul 07 17:08:40.337870 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:40.337790    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ebca297e-b62c-4edb-add5-c6a2001be637-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-3653b-predictor-799d8b94f6-dj9g8\" (UID: \"ebca297e-b62c-4edb-add5-c6a2001be637\") " pod="kserve-ci-e2e-test/error-404-isvc-3653b-predictor-799d8b94f6-dj9g8"
Jul 07 17:08:40.337870 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:40.337818    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvjx7\" (UniqueName: \"kubernetes.io/projected/ebca297e-b62c-4edb-add5-c6a2001be637-kube-api-access-pvjx7\") pod \"error-404-isvc-3653b-predictor-799d8b94f6-dj9g8\" (UID: \"ebca297e-b62c-4edb-add5-c6a2001be637\") " pod="kserve-ci-e2e-test/error-404-isvc-3653b-predictor-799d8b94f6-dj9g8"
Jul 07 17:08:40.438546 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:40.438497    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ebca297e-b62c-4edb-add5-c6a2001be637-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-3653b-predictor-799d8b94f6-dj9g8\" (UID: \"ebca297e-b62c-4edb-add5-c6a2001be637\") " pod="kserve-ci-e2e-test/error-404-isvc-3653b-predictor-799d8b94f6-dj9g8"
Jul 07 17:08:40.438546 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:40.438556    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvjx7\" (UniqueName: \"kubernetes.io/projected/ebca297e-b62c-4edb-add5-c6a2001be637-kube-api-access-pvjx7\") pod \"error-404-isvc-3653b-predictor-799d8b94f6-dj9g8\" (UID: \"ebca297e-b62c-4edb-add5-c6a2001be637\") " pod="kserve-ci-e2e-test/error-404-isvc-3653b-predictor-799d8b94f6-dj9g8"
Jul 07 17:08:40.438779 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:40.438632    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebca297e-b62c-4edb-add5-c6a2001be637-proxy-tls\") pod \"error-404-isvc-3653b-predictor-799d8b94f6-dj9g8\" (UID: \"ebca297e-b62c-4edb-add5-c6a2001be637\") " pod="kserve-ci-e2e-test/error-404-isvc-3653b-predictor-799d8b94f6-dj9g8"
Jul 07 17:08:40.439275 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:40.439256    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ebca297e-b62c-4edb-add5-c6a2001be637-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-3653b-predictor-799d8b94f6-dj9g8\" (UID: \"ebca297e-b62c-4edb-add5-c6a2001be637\") " pod="kserve-ci-e2e-test/error-404-isvc-3653b-predictor-799d8b94f6-dj9g8"
Jul 07 17:08:40.441061 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:40.441044    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebca297e-b62c-4edb-add5-c6a2001be637-proxy-tls\") pod \"error-404-isvc-3653b-predictor-799d8b94f6-dj9g8\" (UID: \"ebca297e-b62c-4edb-add5-c6a2001be637\") " pod="kserve-ci-e2e-test/error-404-isvc-3653b-predictor-799d8b94f6-dj9g8"
Jul 07 17:08:40.446965 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:40.446906    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvjx7\" (UniqueName: \"kubernetes.io/projected/ebca297e-b62c-4edb-add5-c6a2001be637-kube-api-access-pvjx7\") pod \"error-404-isvc-3653b-predictor-799d8b94f6-dj9g8\" (UID: \"ebca297e-b62c-4edb-add5-c6a2001be637\") " pod="kserve-ci-e2e-test/error-404-isvc-3653b-predictor-799d8b94f6-dj9g8"
Jul 07 17:08:40.502536 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:40.502504    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3653b-predictor-799d8b94f6-dj9g8"
Jul 07 17:08:40.631747 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:40.631668    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3653b-predictor-799d8b94f6-dj9g8"]
Jul 07 17:08:40.634630 ip-10-0-136-217 kubenswrapper[2578]: W0707 17:08:40.634601    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebca297e_b62c_4edb_add5_c6a2001be637.slice/crio-e633bab9c537640b2e8fa67932efba2d91b2fdb8b116c3df49d69c6df462f80c WatchSource:0}: Error finding container e633bab9c537640b2e8fa67932efba2d91b2fdb8b116c3df49d69c6df462f80c: Status 404 returned error can't find the container with id e633bab9c537640b2e8fa67932efba2d91b2fdb8b116c3df49d69c6df462f80c
Jul 07 17:08:40.878965 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:40.878932    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3653b-predictor-799d8b94f6-dj9g8" event={"ID":"ebca297e-b62c-4edb-add5-c6a2001be637","Type":"ContainerStarted","Data":"0f67a12269b363f2e00effb9f4f9f150f2ffbd181fb638cb08cdff6e16504859"}
Jul 07 17:08:40.879150 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:40.878987    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3653b-predictor-799d8b94f6-dj9g8" event={"ID":"ebca297e-b62c-4edb-add5-c6a2001be637","Type":"ContainerStarted","Data":"4177d3dead1323c82fd58436a3b3873dbeefd8a57274d5f275576ddfa598a29b"}
Jul 07 17:08:40.879150 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:40.879007    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3653b-predictor-799d8b94f6-dj9g8" event={"ID":"ebca297e-b62c-4edb-add5-c6a2001be637","Type":"ContainerStarted","Data":"e633bab9c537640b2e8fa67932efba2d91b2fdb8b116c3df49d69c6df462f80c"}
Jul 07 17:08:40.879472 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:40.879454    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-3653b-predictor-799d8b94f6-dj9g8"
Jul 07 17:08:40.879561 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:40.879503    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-3653b-predictor-799d8b94f6-dj9g8"
Jul 07 17:08:40.881176 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:40.881063    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3653b-predictor-799d8b94f6-dj9g8" podUID="ebca297e-b62c-4edb-add5-c6a2001be637" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 07 17:08:40.881794 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:40.881773    2578 generic.go:356] "Generic (PLEG): container finished" podID="179eb22e-f62d-4e1c-9377-0fa1ce2f634f" containerID="e3af5f93dd02579a481d27cf215898490d14b3fc903681e7d0b88e3f444000de" exitCode=2
Jul 07 17:08:40.881889 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:40.881836    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8411f-predictor-c4654dd58-qkkp6" event={"ID":"179eb22e-f62d-4e1c-9377-0fa1ce2f634f","Type":"ContainerDied","Data":"e3af5f93dd02579a481d27cf215898490d14b3fc903681e7d0b88e3f444000de"}
Jul 07 17:08:40.896785 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:40.896704    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-3653b-predictor-799d8b94f6-dj9g8" podStartSLOduration=0.896690624 podStartE2EDuration="896.690624ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:08:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:08:40.895749792 +0000 UTC m=+1257.672064019" watchObservedRunningTime="2026-07-07 17:08:40.896690624 +0000 UTC m=+1257.673004830"
Jul 07 17:08:41.732713 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:41.732670    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f8ada-predictor-7c47cd4646-qc5fc" podUID="4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 07 17:08:41.885547 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:41.885501    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3653b-predictor-799d8b94f6-dj9g8" podUID="ebca297e-b62c-4edb-add5-c6a2001be637" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 07 17:08:42.844329 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:42.844295    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-8411f-f58b567b8-llfwp" podUID="12669d4a-108e-499b-954f-55cdeda16f68" containerName="sequence-graph-8411f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:08:42.889288 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:42.889252    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3653b-predictor-799d8b94f6-dj9g8" podUID="ebca297e-b62c-4edb-add5-c6a2001be637" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 07 17:08:43.489772 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:43.489750    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8411f-predictor-c4654dd58-qkkp6"
Jul 07 17:08:43.668172 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:43.668092    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/179eb22e-f62d-4e1c-9377-0fa1ce2f634f-proxy-tls\") pod \"179eb22e-f62d-4e1c-9377-0fa1ce2f634f\" (UID: \"179eb22e-f62d-4e1c-9377-0fa1ce2f634f\") "
Jul 07 17:08:43.668172 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:43.668151    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/179eb22e-f62d-4e1c-9377-0fa1ce2f634f-kube-rbac-proxy-sar-config\") pod \"179eb22e-f62d-4e1c-9377-0fa1ce2f634f\" (UID: \"179eb22e-f62d-4e1c-9377-0fa1ce2f634f\") "
Jul 07 17:08:43.668172 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:43.668169    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-52zzn\" (UniqueName: \"kubernetes.io/projected/179eb22e-f62d-4e1c-9377-0fa1ce2f634f-kube-api-access-52zzn\") pod \"179eb22e-f62d-4e1c-9377-0fa1ce2f634f\" (UID: \"179eb22e-f62d-4e1c-9377-0fa1ce2f634f\") "
Jul 07 17:08:43.668533 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:43.668506    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/179eb22e-f62d-4e1c-9377-0fa1ce2f634f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "179eb22e-f62d-4e1c-9377-0fa1ce2f634f" (UID: "179eb22e-f62d-4e1c-9377-0fa1ce2f634f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:08:43.670112 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:43.670089    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/179eb22e-f62d-4e1c-9377-0fa1ce2f634f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "179eb22e-f62d-4e1c-9377-0fa1ce2f634f" (UID: "179eb22e-f62d-4e1c-9377-0fa1ce2f634f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 17:08:43.670242 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:43.670222    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/179eb22e-f62d-4e1c-9377-0fa1ce2f634f-kube-api-access-52zzn" (OuterVolumeSpecName: "kube-api-access-52zzn") pod "179eb22e-f62d-4e1c-9377-0fa1ce2f634f" (UID: "179eb22e-f62d-4e1c-9377-0fa1ce2f634f"). InnerVolumeSpecName "kube-api-access-52zzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:08:43.769324 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:43.769299    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/179eb22e-f62d-4e1c-9377-0fa1ce2f634f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 17:08:43.769324 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:43.769324    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52zzn\" (UniqueName: \"kubernetes.io/projected/179eb22e-f62d-4e1c-9377-0fa1ce2f634f-kube-api-access-52zzn\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 17:08:43.769466 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:43.769335    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/179eb22e-f62d-4e1c-9377-0fa1ce2f634f-proxy-tls\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 17:08:43.894485 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:43.894455    2578 generic.go:356] "Generic (PLEG): container finished" podID="179eb22e-f62d-4e1c-9377-0fa1ce2f634f" containerID="eb92591c2949c7374d4b80806befe06522f8e41fb5ffe63ad50083f68f6c550b" exitCode=0
Jul 07 17:08:43.894889 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:43.894508    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8411f-predictor-c4654dd58-qkkp6"
Jul 07 17:08:43.894889 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:43.894526    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8411f-predictor-c4654dd58-qkkp6" event={"ID":"179eb22e-f62d-4e1c-9377-0fa1ce2f634f","Type":"ContainerDied","Data":"eb92591c2949c7374d4b80806befe06522f8e41fb5ffe63ad50083f68f6c550b"}
Jul 07 17:08:43.894889 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:43.894572    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8411f-predictor-c4654dd58-qkkp6" event={"ID":"179eb22e-f62d-4e1c-9377-0fa1ce2f634f","Type":"ContainerDied","Data":"8a95d697d1b013b313b07c9afa623d9aa3fa6a51a601f4845d6a0e6c24e88c04"}
Jul 07 17:08:43.894889 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:43.894598    2578 scope.go:117] "RemoveContainer" containerID="e3af5f93dd02579a481d27cf215898490d14b3fc903681e7d0b88e3f444000de"
Jul 07 17:08:43.902429 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:43.902413    2578 scope.go:117] "RemoveContainer" containerID="eb92591c2949c7374d4b80806befe06522f8e41fb5ffe63ad50083f68f6c550b"
Jul 07 17:08:43.909338 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:43.909323    2578 scope.go:117] "RemoveContainer" containerID="e3af5f93dd02579a481d27cf215898490d14b3fc903681e7d0b88e3f444000de"
Jul 07 17:08:43.909630 ip-10-0-136-217 kubenswrapper[2578]: E0707 17:08:43.909613    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3af5f93dd02579a481d27cf215898490d14b3fc903681e7d0b88e3f444000de\": container with ID starting with e3af5f93dd02579a481d27cf215898490d14b3fc903681e7d0b88e3f444000de not found: ID does not exist" containerID="e3af5f93dd02579a481d27cf215898490d14b3fc903681e7d0b88e3f444000de"
Jul 07 17:08:43.909677 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:43.909638    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3af5f93dd02579a481d27cf215898490d14b3fc903681e7d0b88e3f444000de"} err="failed to get container status \"e3af5f93dd02579a481d27cf215898490d14b3fc903681e7d0b88e3f444000de\": rpc error: code = NotFound desc = could not find container \"e3af5f93dd02579a481d27cf215898490d14b3fc903681e7d0b88e3f444000de\": container with ID starting with e3af5f93dd02579a481d27cf215898490d14b3fc903681e7d0b88e3f444000de not found: ID does not exist"
Jul 07 17:08:43.909677 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:43.909656    2578 scope.go:117] "RemoveContainer" containerID="eb92591c2949c7374d4b80806befe06522f8e41fb5ffe63ad50083f68f6c550b"
Jul 07 17:08:43.909910 ip-10-0-136-217 kubenswrapper[2578]: E0707 17:08:43.909894    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb92591c2949c7374d4b80806befe06522f8e41fb5ffe63ad50083f68f6c550b\": container with ID starting with eb92591c2949c7374d4b80806befe06522f8e41fb5ffe63ad50083f68f6c550b not found: ID does not exist" containerID="eb92591c2949c7374d4b80806befe06522f8e41fb5ffe63ad50083f68f6c550b"
Jul 07 17:08:43.909976 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:43.909915    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb92591c2949c7374d4b80806befe06522f8e41fb5ffe63ad50083f68f6c550b"} err="failed to get container status \"eb92591c2949c7374d4b80806befe06522f8e41fb5ffe63ad50083f68f6c550b\": rpc error: code = NotFound desc = could not find container \"eb92591c2949c7374d4b80806befe06522f8e41fb5ffe63ad50083f68f6c550b\": container with ID starting with eb92591c2949c7374d4b80806befe06522f8e41fb5ffe63ad50083f68f6c550b not found: ID does not exist"
Jul 07 17:08:43.910532 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:43.910516    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8411f-predictor-c4654dd58-qkkp6"]
Jul 07 17:08:43.916277 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:43.916256    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8411f-predictor-c4654dd58-qkkp6"]
Jul 07 17:08:45.725879 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:45.725841    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="179eb22e-f62d-4e1c-9377-0fa1ce2f634f" path="/var/lib/kubelet/pods/179eb22e-f62d-4e1c-9377-0fa1ce2f634f/volumes"
Jul 07 17:08:47.843464 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:47.843422    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-8411f-f58b567b8-llfwp" podUID="12669d4a-108e-499b-954f-55cdeda16f68" containerName="sequence-graph-8411f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:08:47.894290 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:47.894257    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-3653b-predictor-799d8b94f6-dj9g8"
Jul 07 17:08:47.894708 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:47.894681    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3653b-predictor-799d8b94f6-dj9g8" podUID="ebca297e-b62c-4edb-add5-c6a2001be637" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 07 17:08:51.733889 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:51.733861    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-f8ada-predictor-7c47cd4646-qc5fc"
Jul 07 17:08:52.843446 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:52.843411    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-8411f-f58b567b8-llfwp" podUID="12669d4a-108e-499b-954f-55cdeda16f68" containerName="sequence-graph-8411f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:08:52.843834 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:52.843543    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-8411f-f58b567b8-llfwp"
Jul 07 17:08:57.844154 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:57.844120    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-8411f-f58b567b8-llfwp" podUID="12669d4a-108e-499b-954f-55cdeda16f68" containerName="sequence-graph-8411f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:08:57.894869 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:08:57.894831    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3653b-predictor-799d8b94f6-dj9g8" podUID="ebca297e-b62c-4edb-add5-c6a2001be637" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 07 17:09:02.843552 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:02.843512    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-8411f-f58b567b8-llfwp" podUID="12669d4a-108e-499b-954f-55cdeda16f68" containerName="sequence-graph-8411f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:09:04.381290 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:04.381255    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-f8ada-7445689dbf-slsqt"]
Jul 07 17:09:04.381689 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:04.381560    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="179eb22e-f62d-4e1c-9377-0fa1ce2f634f" containerName="kserve-container"
Jul 07 17:09:04.381689 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:04.381571    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="179eb22e-f62d-4e1c-9377-0fa1ce2f634f" containerName="kserve-container"
Jul 07 17:09:04.381689 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:04.381600    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="179eb22e-f62d-4e1c-9377-0fa1ce2f634f" containerName="kube-rbac-proxy"
Jul 07 17:09:04.381689 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:04.381607    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="179eb22e-f62d-4e1c-9377-0fa1ce2f634f" containerName="kube-rbac-proxy"
Jul 07 17:09:04.381689 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:04.381669    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="179eb22e-f62d-4e1c-9377-0fa1ce2f634f" containerName="kserve-container"
Jul 07 17:09:04.381689 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:04.381676    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="179eb22e-f62d-4e1c-9377-0fa1ce2f634f" containerName="kube-rbac-proxy"
Jul 07 17:09:04.386097 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:04.386078    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-f8ada-7445689dbf-slsqt"
Jul 07 17:09:04.388573 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:04.388545    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-f8ada-serving-cert\""
Jul 07 17:09:04.388672 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:04.388575    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-f8ada-kube-rbac-proxy-sar-config\""
Jul 07 17:09:04.393891 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:04.393871    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-f8ada-7445689dbf-slsqt"]
Jul 07 17:09:04.445158 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:04.445131    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c75328a-bda4-4e8b-8c6c-3517d2f815a2-proxy-tls\") pod \"ensemble-graph-f8ada-7445689dbf-slsqt\" (UID: \"7c75328a-bda4-4e8b-8c6c-3517d2f815a2\") " pod="kserve-ci-e2e-test/ensemble-graph-f8ada-7445689dbf-slsqt"
Jul 07 17:09:04.445281 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:04.445207    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c75328a-bda4-4e8b-8c6c-3517d2f815a2-openshift-service-ca-bundle\") pod \"ensemble-graph-f8ada-7445689dbf-slsqt\" (UID: \"7c75328a-bda4-4e8b-8c6c-3517d2f815a2\") " pod="kserve-ci-e2e-test/ensemble-graph-f8ada-7445689dbf-slsqt"
Jul 07 17:09:04.545605 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:04.545576    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c75328a-bda4-4e8b-8c6c-3517d2f815a2-openshift-service-ca-bundle\") pod \"ensemble-graph-f8ada-7445689dbf-slsqt\" (UID: \"7c75328a-bda4-4e8b-8c6c-3517d2f815a2\") " pod="kserve-ci-e2e-test/ensemble-graph-f8ada-7445689dbf-slsqt"
Jul 07 17:09:04.545795 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:04.545619    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c75328a-bda4-4e8b-8c6c-3517d2f815a2-proxy-tls\") pod \"ensemble-graph-f8ada-7445689dbf-slsqt\" (UID: \"7c75328a-bda4-4e8b-8c6c-3517d2f815a2\") " pod="kserve-ci-e2e-test/ensemble-graph-f8ada-7445689dbf-slsqt"
Jul 07 17:09:04.545795 ip-10-0-136-217 kubenswrapper[2578]: E0707 17:09:04.545770    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/ensemble-graph-f8ada-serving-cert: secret "ensemble-graph-f8ada-serving-cert" not found
Jul 07 17:09:04.545921 ip-10-0-136-217 kubenswrapper[2578]: E0707 17:09:04.545828    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c75328a-bda4-4e8b-8c6c-3517d2f815a2-proxy-tls podName:7c75328a-bda4-4e8b-8c6c-3517d2f815a2 nodeName:}" failed. No retries permitted until 2026-07-07 17:09:05.045809456 +0000 UTC m=+1281.822123643 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7c75328a-bda4-4e8b-8c6c-3517d2f815a2-proxy-tls") pod "ensemble-graph-f8ada-7445689dbf-slsqt" (UID: "7c75328a-bda4-4e8b-8c6c-3517d2f815a2") : secret "ensemble-graph-f8ada-serving-cert" not found
Jul 07 17:09:04.546296 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:04.546272    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c75328a-bda4-4e8b-8c6c-3517d2f815a2-openshift-service-ca-bundle\") pod \"ensemble-graph-f8ada-7445689dbf-slsqt\" (UID: \"7c75328a-bda4-4e8b-8c6c-3517d2f815a2\") " pod="kserve-ci-e2e-test/ensemble-graph-f8ada-7445689dbf-slsqt"
Jul 07 17:09:05.049548 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:05.049519    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c75328a-bda4-4e8b-8c6c-3517d2f815a2-proxy-tls\") pod \"ensemble-graph-f8ada-7445689dbf-slsqt\" (UID: \"7c75328a-bda4-4e8b-8c6c-3517d2f815a2\") " pod="kserve-ci-e2e-test/ensemble-graph-f8ada-7445689dbf-slsqt"
Jul 07 17:09:05.051939 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:05.051915    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c75328a-bda4-4e8b-8c6c-3517d2f815a2-proxy-tls\") pod \"ensemble-graph-f8ada-7445689dbf-slsqt\" (UID: \"7c75328a-bda4-4e8b-8c6c-3517d2f815a2\") " pod="kserve-ci-e2e-test/ensemble-graph-f8ada-7445689dbf-slsqt"
Jul 07 17:09:05.296547 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:05.296503    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-f8ada-7445689dbf-slsqt"
Jul 07 17:09:05.425184 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:05.425158    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-f8ada-7445689dbf-slsqt"]
Jul 07 17:09:05.427886 ip-10-0-136-217 kubenswrapper[2578]: W0707 17:09:05.427856    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c75328a_bda4_4e8b_8c6c_3517d2f815a2.slice/crio-85cf1f5e24f1bd75273d4b033aababaa40af03fc8843607a1e256b8ce5ea2414 WatchSource:0}: Error finding container 85cf1f5e24f1bd75273d4b033aababaa40af03fc8843607a1e256b8ce5ea2414: Status 404 returned error can't find the container with id 85cf1f5e24f1bd75273d4b033aababaa40af03fc8843607a1e256b8ce5ea2414
Jul 07 17:09:05.987238 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:05.987202    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-f8ada-7445689dbf-slsqt" event={"ID":"7c75328a-bda4-4e8b-8c6c-3517d2f815a2","Type":"ContainerStarted","Data":"5eb9bb233d2839aed69859d555c38d7d78f4259d2a79e3df9ddc6bc7e4f59afe"}
Jul 07 17:09:05.987406 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:05.987262    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-f8ada-7445689dbf-slsqt" event={"ID":"7c75328a-bda4-4e8b-8c6c-3517d2f815a2","Type":"ContainerStarted","Data":"85cf1f5e24f1bd75273d4b033aababaa40af03fc8843607a1e256b8ce5ea2414"}
Jul 07 17:09:05.987406 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:05.987288    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-f8ada-7445689dbf-slsqt"
Jul 07 17:09:06.005083 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:06.005035    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-f8ada-7445689dbf-slsqt" podStartSLOduration=2.005020596 podStartE2EDuration="2.005020596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:09:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:09:06.004171156 +0000 UTC m=+1282.780485362" watchObservedRunningTime="2026-07-07 17:09:06.005020596 +0000 UTC m=+1282.781334801"
Jul 07 17:09:07.843656 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:07.843610    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-8411f-f58b567b8-llfwp" podUID="12669d4a-108e-499b-954f-55cdeda16f68" containerName="sequence-graph-8411f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:09:07.895135 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:07.895097    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3653b-predictor-799d8b94f6-dj9g8" podUID="ebca297e-b62c-4edb-add5-c6a2001be637" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 07 17:09:10.003684 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:10.003652    2578 generic.go:356] "Generic (PLEG): container finished" podID="12669d4a-108e-499b-954f-55cdeda16f68" containerID="537efea55f46dbd6964c3fc9bac4d34c203766f861b27685ca9c57447c5b2d63" exitCode=0
Jul 07 17:09:10.004066 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:10.003719    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-8411f-f58b567b8-llfwp" event={"ID":"12669d4a-108e-499b-954f-55cdeda16f68","Type":"ContainerDied","Data":"537efea55f46dbd6964c3fc9bac4d34c203766f861b27685ca9c57447c5b2d63"}
Jul 07 17:09:10.084516 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:10.084493    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-8411f-f58b567b8-llfwp"
Jul 07 17:09:10.194460 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:10.194374    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12669d4a-108e-499b-954f-55cdeda16f68-proxy-tls\") pod \"12669d4a-108e-499b-954f-55cdeda16f68\" (UID: \"12669d4a-108e-499b-954f-55cdeda16f68\") "
Jul 07 17:09:10.194460 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:10.194412    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12669d4a-108e-499b-954f-55cdeda16f68-openshift-service-ca-bundle\") pod \"12669d4a-108e-499b-954f-55cdeda16f68\" (UID: \"12669d4a-108e-499b-954f-55cdeda16f68\") "
Jul 07 17:09:10.194832 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:10.194806    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12669d4a-108e-499b-954f-55cdeda16f68-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "12669d4a-108e-499b-954f-55cdeda16f68" (UID: "12669d4a-108e-499b-954f-55cdeda16f68"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:09:10.196466 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:10.196441    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12669d4a-108e-499b-954f-55cdeda16f68-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "12669d4a-108e-499b-954f-55cdeda16f68" (UID: "12669d4a-108e-499b-954f-55cdeda16f68"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 17:09:10.295043 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:10.295011    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12669d4a-108e-499b-954f-55cdeda16f68-proxy-tls\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 17:09:10.295043 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:10.295040    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12669d4a-108e-499b-954f-55cdeda16f68-openshift-service-ca-bundle\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 17:09:11.008767 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:11.008726    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-8411f-f58b567b8-llfwp"
Jul 07 17:09:11.009261 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:11.008718    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-8411f-f58b567b8-llfwp" event={"ID":"12669d4a-108e-499b-954f-55cdeda16f68","Type":"ContainerDied","Data":"3048dead96ee4f02ea165d7668a630f4420a49c78e96cc8fb8e114f6fabdc1bc"}
Jul 07 17:09:11.009261 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:11.008886    2578 scope.go:117] "RemoveContainer" containerID="537efea55f46dbd6964c3fc9bac4d34c203766f861b27685ca9c57447c5b2d63"
Jul 07 17:09:11.028635 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:11.028614    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-8411f-f58b567b8-llfwp"]
Jul 07 17:09:11.033139 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:11.033115    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-8411f-f58b567b8-llfwp"]
Jul 07 17:09:11.731777 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:11.731716    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12669d4a-108e-499b-954f-55cdeda16f68" path="/var/lib/kubelet/pods/12669d4a-108e-499b-954f-55cdeda16f68/volumes"
Jul 07 17:09:11.996836 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:11.996757    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-f8ada-7445689dbf-slsqt"
Jul 07 17:09:17.895627 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:17.895595    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3653b-predictor-799d8b94f6-dj9g8" podUID="ebca297e-b62c-4edb-add5-c6a2001be637" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 07 17:09:27.895319 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:27.895293    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-3653b-predictor-799d8b94f6-dj9g8"
Jul 07 17:09:40.156606 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:40.156573    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-3653b-c47d47c8-ff2k7"]
Jul 07 17:09:40.157064 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:40.156937    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12669d4a-108e-499b-954f-55cdeda16f68" containerName="sequence-graph-8411f"
Jul 07 17:09:40.157064 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:40.156949    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="12669d4a-108e-499b-954f-55cdeda16f68" containerName="sequence-graph-8411f"
Jul 07 17:09:40.157064 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:40.156994    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="12669d4a-108e-499b-954f-55cdeda16f68" containerName="sequence-graph-8411f"
Jul 07 17:09:40.159904 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:40.159889    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-3653b-c47d47c8-ff2k7"
Jul 07 17:09:40.162378 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:40.162358    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-3653b-kube-rbac-proxy-sar-config\""
Jul 07 17:09:40.162568 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:40.162553    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-3653b-serving-cert\""
Jul 07 17:09:40.169204 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:40.169180    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-3653b-c47d47c8-ff2k7"]
Jul 07 17:09:40.254776 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:40.254750    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2499088-f7a4-4598-bbc1-70b3dbe8a156-proxy-tls\") pod \"sequence-graph-3653b-c47d47c8-ff2k7\" (UID: \"b2499088-f7a4-4598-bbc1-70b3dbe8a156\") " pod="kserve-ci-e2e-test/sequence-graph-3653b-c47d47c8-ff2k7"
Jul 07 17:09:40.254927 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:40.254782    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2499088-f7a4-4598-bbc1-70b3dbe8a156-openshift-service-ca-bundle\") pod \"sequence-graph-3653b-c47d47c8-ff2k7\" (UID: \"b2499088-f7a4-4598-bbc1-70b3dbe8a156\") " pod="kserve-ci-e2e-test/sequence-graph-3653b-c47d47c8-ff2k7"
Jul 07 17:09:40.355681 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:40.355644    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2499088-f7a4-4598-bbc1-70b3dbe8a156-proxy-tls\") pod \"sequence-graph-3653b-c47d47c8-ff2k7\" (UID: \"b2499088-f7a4-4598-bbc1-70b3dbe8a156\") " pod="kserve-ci-e2e-test/sequence-graph-3653b-c47d47c8-ff2k7"
Jul 07 17:09:40.355681 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:40.355682    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2499088-f7a4-4598-bbc1-70b3dbe8a156-openshift-service-ca-bundle\") pod \"sequence-graph-3653b-c47d47c8-ff2k7\" (UID: \"b2499088-f7a4-4598-bbc1-70b3dbe8a156\") " pod="kserve-ci-e2e-test/sequence-graph-3653b-c47d47c8-ff2k7"
Jul 07 17:09:40.356276 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:40.356258    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2499088-f7a4-4598-bbc1-70b3dbe8a156-openshift-service-ca-bundle\") pod \"sequence-graph-3653b-c47d47c8-ff2k7\" (UID: \"b2499088-f7a4-4598-bbc1-70b3dbe8a156\") " pod="kserve-ci-e2e-test/sequence-graph-3653b-c47d47c8-ff2k7"
Jul 07 17:09:40.358022 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:40.358003    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2499088-f7a4-4598-bbc1-70b3dbe8a156-proxy-tls\") pod \"sequence-graph-3653b-c47d47c8-ff2k7\" (UID: \"b2499088-f7a4-4598-bbc1-70b3dbe8a156\") " pod="kserve-ci-e2e-test/sequence-graph-3653b-c47d47c8-ff2k7"
Jul 07 17:09:40.471169 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:40.471073    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-3653b-c47d47c8-ff2k7"
Jul 07 17:09:40.596546 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:40.596524    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-3653b-c47d47c8-ff2k7"]
Jul 07 17:09:40.599169 ip-10-0-136-217 kubenswrapper[2578]: W0707 17:09:40.599141    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2499088_f7a4_4598_bbc1_70b3dbe8a156.slice/crio-1e8c5134005ad858e1cddd8a0e6e07ca622f5420400558db6810b65f169552c6 WatchSource:0}: Error finding container 1e8c5134005ad858e1cddd8a0e6e07ca622f5420400558db6810b65f169552c6: Status 404 returned error can't find the container with id 1e8c5134005ad858e1cddd8a0e6e07ca622f5420400558db6810b65f169552c6
Jul 07 17:09:41.131062 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:41.131026    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-3653b-c47d47c8-ff2k7" event={"ID":"b2499088-f7a4-4598-bbc1-70b3dbe8a156","Type":"ContainerStarted","Data":"13f6132e7ad6dbca310498bc927ba3049b82a2541b0f5315221b0f477d92c1d8"}
Jul 07 17:09:41.131266 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:41.131086    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-3653b-c47d47c8-ff2k7" event={"ID":"b2499088-f7a4-4598-bbc1-70b3dbe8a156","Type":"ContainerStarted","Data":"1e8c5134005ad858e1cddd8a0e6e07ca622f5420400558db6810b65f169552c6"}
Jul 07 17:09:41.131266 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:41.131114    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-3653b-c47d47c8-ff2k7"
Jul 07 17:09:41.147816 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:41.147774    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-3653b-c47d47c8-ff2k7" podStartSLOduration=1.147759264 podStartE2EDuration="1.147759264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:09:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:09:41.146912819 +0000 UTC m=+1317.923227038" watchObservedRunningTime="2026-07-07 17:09:41.147759264 +0000 UTC m=+1317.924073507"
Jul 07 17:09:47.140334 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:09:47.140309    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-3653b-c47d47c8-ff2k7"
Jul 07 17:17:19.171182 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:19.171148    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-f8ada-7445689dbf-slsqt"]
Jul 07 17:17:19.171697 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:19.171381    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-f8ada-7445689dbf-slsqt" podUID="7c75328a-bda4-4e8b-8c6c-3517d2f815a2" containerName="ensemble-graph-f8ada" containerID="cri-o://5eb9bb233d2839aed69859d555c38d7d78f4259d2a79e3df9ddc6bc7e4f59afe" gracePeriod=30
Jul 07 17:17:19.489031 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:19.488947    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f587d-predictor-79b9cd4678-mpl9k"]
Jul 07 17:17:19.492852 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:19.492833    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f587d-predictor-79b9cd4678-mpl9k"
Jul 07 17:17:19.495779 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:19.495754    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-f587d-predictor-serving-cert\""
Jul 07 17:17:19.496154 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:19.496128    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-f587d-kube-rbac-proxy-sar-config\""
Jul 07 17:17:19.502406 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:19.502380    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f587d-predictor-79b9cd4678-mpl9k"]
Jul 07 17:17:19.525123 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:19.525096    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f8ada-predictor-7c47cd4646-qc5fc"]
Jul 07 17:17:19.525315 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:19.525295    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-f8ada-predictor-7c47cd4646-qc5fc" podUID="4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d" containerName="kserve-container" containerID="cri-o://9193e4fe4b7d69ab86d4190d3f7bbad378cc5229a630f77abfb9e1e2fa4ca521" gracePeriod=30
Jul 07 17:17:19.525406 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:19.525359    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-f8ada-predictor-7c47cd4646-qc5fc" podUID="4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d" containerName="kube-rbac-proxy" containerID="cri-o://275c69b0ea66e38186e786d91aef1a96fba173974749ddb9f05f40388147bec2" gracePeriod=30
Jul 07 17:17:19.553190 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:19.553156    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58ad52b5-f293-4951-851b-49cff8359d1d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f587d-predictor-79b9cd4678-mpl9k\" (UID: \"58ad52b5-f293-4951-851b-49cff8359d1d\") " pod="kserve-ci-e2e-test/error-404-isvc-f587d-predictor-79b9cd4678-mpl9k"
Jul 07 17:17:19.553347 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:19.553268    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69668\" (UniqueName: \"kubernetes.io/projected/58ad52b5-f293-4951-851b-49cff8359d1d-kube-api-access-69668\") pod \"error-404-isvc-f587d-predictor-79b9cd4678-mpl9k\" (UID: \"58ad52b5-f293-4951-851b-49cff8359d1d\") " pod="kserve-ci-e2e-test/error-404-isvc-f587d-predictor-79b9cd4678-mpl9k"
Jul 07 17:17:19.553347 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:19.553302    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58ad52b5-f293-4951-851b-49cff8359d1d-proxy-tls\") pod \"error-404-isvc-f587d-predictor-79b9cd4678-mpl9k\" (UID: \"58ad52b5-f293-4951-851b-49cff8359d1d\") " pod="kserve-ci-e2e-test/error-404-isvc-f587d-predictor-79b9cd4678-mpl9k"
Jul 07 17:17:19.654233 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:19.654201    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69668\" (UniqueName: \"kubernetes.io/projected/58ad52b5-f293-4951-851b-49cff8359d1d-kube-api-access-69668\") pod \"error-404-isvc-f587d-predictor-79b9cd4678-mpl9k\" (UID: \"58ad52b5-f293-4951-851b-49cff8359d1d\") " pod="kserve-ci-e2e-test/error-404-isvc-f587d-predictor-79b9cd4678-mpl9k"
Jul 07 17:17:19.654380 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:19.654244    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58ad52b5-f293-4951-851b-49cff8359d1d-proxy-tls\") pod \"error-404-isvc-f587d-predictor-79b9cd4678-mpl9k\" (UID: \"58ad52b5-f293-4951-851b-49cff8359d1d\") " pod="kserve-ci-e2e-test/error-404-isvc-f587d-predictor-79b9cd4678-mpl9k"
Jul 07 17:17:19.654380 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:19.654293    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58ad52b5-f293-4951-851b-49cff8359d1d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f587d-predictor-79b9cd4678-mpl9k\" (UID: \"58ad52b5-f293-4951-851b-49cff8359d1d\") " pod="kserve-ci-e2e-test/error-404-isvc-f587d-predictor-79b9cd4678-mpl9k"
Jul 07 17:17:19.655010 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:19.654991    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58ad52b5-f293-4951-851b-49cff8359d1d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f587d-predictor-79b9cd4678-mpl9k\" (UID: \"58ad52b5-f293-4951-851b-49cff8359d1d\") " pod="kserve-ci-e2e-test/error-404-isvc-f587d-predictor-79b9cd4678-mpl9k"
Jul 07 17:17:19.656596 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:19.656576    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58ad52b5-f293-4951-851b-49cff8359d1d-proxy-tls\") pod \"error-404-isvc-f587d-predictor-79b9cd4678-mpl9k\" (UID: \"58ad52b5-f293-4951-851b-49cff8359d1d\") " pod="kserve-ci-e2e-test/error-404-isvc-f587d-predictor-79b9cd4678-mpl9k"
Jul 07 17:17:19.661958 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:19.661937    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69668\" (UniqueName: \"kubernetes.io/projected/58ad52b5-f293-4951-851b-49cff8359d1d-kube-api-access-69668\") pod \"error-404-isvc-f587d-predictor-79b9cd4678-mpl9k\" (UID: \"58ad52b5-f293-4951-851b-49cff8359d1d\") " pod="kserve-ci-e2e-test/error-404-isvc-f587d-predictor-79b9cd4678-mpl9k"
Jul 07 17:17:19.806177 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:19.806143    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f587d-predictor-79b9cd4678-mpl9k"
Jul 07 17:17:19.938715 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:19.938690    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f587d-predictor-79b9cd4678-mpl9k"]
Jul 07 17:17:19.941424 ip-10-0-136-217 kubenswrapper[2578]: W0707 17:17:19.941397    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58ad52b5_f293_4951_851b_49cff8359d1d.slice/crio-182a6946ae92559c6a71287922e3450a713c7e33c4cf81da0bb03fc351c4e702 WatchSource:0}: Error finding container 182a6946ae92559c6a71287922e3450a713c7e33c4cf81da0bb03fc351c4e702: Status 404 returned error can't find the container with id 182a6946ae92559c6a71287922e3450a713c7e33c4cf81da0bb03fc351c4e702
Jul 07 17:17:19.942519 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:19.942497    2578 generic.go:356] "Generic (PLEG): container finished" podID="4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d" containerID="275c69b0ea66e38186e786d91aef1a96fba173974749ddb9f05f40388147bec2" exitCode=2
Jul 07 17:17:19.942603 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:19.942566    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f8ada-predictor-7c47cd4646-qc5fc" event={"ID":"4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d","Type":"ContainerDied","Data":"275c69b0ea66e38186e786d91aef1a96fba173974749ddb9f05f40388147bec2"}
Jul 07 17:17:20.948302 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:20.948268    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f587d-predictor-79b9cd4678-mpl9k" event={"ID":"58ad52b5-f293-4951-851b-49cff8359d1d","Type":"ContainerStarted","Data":"85730c195f8b8d77834655f64a49b49878f0a24196de75ed5c2b6458408c613b"}
Jul 07 17:17:20.948719 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:20.948325    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f587d-predictor-79b9cd4678-mpl9k" event={"ID":"58ad52b5-f293-4951-851b-49cff8359d1d","Type":"ContainerStarted","Data":"7cdc2cf99ab77be158ec6b879eb52d4a6fffe21db774a6c656e9ceae16f032a2"}
Jul 07 17:17:20.948719 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:20.948346    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f587d-predictor-79b9cd4678-mpl9k" event={"ID":"58ad52b5-f293-4951-851b-49cff8359d1d","Type":"ContainerStarted","Data":"182a6946ae92559c6a71287922e3450a713c7e33c4cf81da0bb03fc351c4e702"}
Jul 07 17:17:20.948719 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:20.948424    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-f587d-predictor-79b9cd4678-mpl9k"
Jul 07 17:17:20.967114 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:20.967068    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-f587d-predictor-79b9cd4678-mpl9k" podStartSLOduration=1.9670533369999998 podStartE2EDuration="1.967053337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:17:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:17:20.965549514 +0000 UTC m=+1777.741863723" watchObservedRunningTime="2026-07-07 17:17:20.967053337 +0000 UTC m=+1777.743367543"
Jul 07 17:17:21.726591 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:21.726554    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f8ada-predictor-7c47cd4646-qc5fc" podUID="4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Jul 07 17:17:21.732993 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:21.732958    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f8ada-predictor-7c47cd4646-qc5fc" podUID="4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 07 17:17:21.953432 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:21.953408    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-f587d-predictor-79b9cd4678-mpl9k"
Jul 07 17:17:21.954532 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:21.954506    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f587d-predictor-79b9cd4678-mpl9k" podUID="58ad52b5-f293-4951-851b-49cff8359d1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 07 17:17:21.995255 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:21.995185    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-f8ada-7445689dbf-slsqt" podUID="7c75328a-bda4-4e8b-8c6c-3517d2f815a2" containerName="ensemble-graph-f8ada" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:17:22.871263 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:22.871240    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f8ada-predictor-7c47cd4646-qc5fc"
Jul 07 17:17:22.958325 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:22.958245    2578 generic.go:356] "Generic (PLEG): container finished" podID="4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d" containerID="9193e4fe4b7d69ab86d4190d3f7bbad378cc5229a630f77abfb9e1e2fa4ca521" exitCode=0
Jul 07 17:17:22.958325 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:22.958309    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f8ada-predictor-7c47cd4646-qc5fc"
Jul 07 17:17:22.958786 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:22.958332    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f8ada-predictor-7c47cd4646-qc5fc" event={"ID":"4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d","Type":"ContainerDied","Data":"9193e4fe4b7d69ab86d4190d3f7bbad378cc5229a630f77abfb9e1e2fa4ca521"}
Jul 07 17:17:22.958786 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:22.958425    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f8ada-predictor-7c47cd4646-qc5fc" event={"ID":"4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d","Type":"ContainerDied","Data":"5f1a405616e9a08d6022cb9a383f51ffbd395f136e88ff5875e7c065cbbe0c62"}
Jul 07 17:17:22.958786 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:22.958460    2578 scope.go:117] "RemoveContainer" containerID="275c69b0ea66e38186e786d91aef1a96fba173974749ddb9f05f40388147bec2"
Jul 07 17:17:22.958954 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:22.958784    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f587d-predictor-79b9cd4678-mpl9k" podUID="58ad52b5-f293-4951-851b-49cff8359d1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 07 17:17:22.966341 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:22.966325    2578 scope.go:117] "RemoveContainer" containerID="9193e4fe4b7d69ab86d4190d3f7bbad378cc5229a630f77abfb9e1e2fa4ca521"
Jul 07 17:17:22.973318 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:22.973303    2578 scope.go:117] "RemoveContainer" containerID="275c69b0ea66e38186e786d91aef1a96fba173974749ddb9f05f40388147bec2"
Jul 07 17:17:22.973554 ip-10-0-136-217 kubenswrapper[2578]: E0707 17:17:22.973537    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"275c69b0ea66e38186e786d91aef1a96fba173974749ddb9f05f40388147bec2\": container with ID starting with 275c69b0ea66e38186e786d91aef1a96fba173974749ddb9f05f40388147bec2 not found: ID does not exist" containerID="275c69b0ea66e38186e786d91aef1a96fba173974749ddb9f05f40388147bec2"
Jul 07 17:17:22.973601 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:22.973562    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"275c69b0ea66e38186e786d91aef1a96fba173974749ddb9f05f40388147bec2"} err="failed to get container status \"275c69b0ea66e38186e786d91aef1a96fba173974749ddb9f05f40388147bec2\": rpc error: code = NotFound desc = could not find container \"275c69b0ea66e38186e786d91aef1a96fba173974749ddb9f05f40388147bec2\": container with ID starting with 275c69b0ea66e38186e786d91aef1a96fba173974749ddb9f05f40388147bec2 not found: ID does not exist"
Jul 07 17:17:22.973601 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:22.973578    2578 scope.go:117] "RemoveContainer" containerID="9193e4fe4b7d69ab86d4190d3f7bbad378cc5229a630f77abfb9e1e2fa4ca521"
Jul 07 17:17:22.973818 ip-10-0-136-217 kubenswrapper[2578]: E0707 17:17:22.973802    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9193e4fe4b7d69ab86d4190d3f7bbad378cc5229a630f77abfb9e1e2fa4ca521\": container with ID starting with 9193e4fe4b7d69ab86d4190d3f7bbad378cc5229a630f77abfb9e1e2fa4ca521 not found: ID does not exist" containerID="9193e4fe4b7d69ab86d4190d3f7bbad378cc5229a630f77abfb9e1e2fa4ca521"
Jul 07 17:17:22.973863 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:22.973822    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9193e4fe4b7d69ab86d4190d3f7bbad378cc5229a630f77abfb9e1e2fa4ca521"} err="failed to get container status \"9193e4fe4b7d69ab86d4190d3f7bbad378cc5229a630f77abfb9e1e2fa4ca521\": rpc error: code = NotFound desc = could not find container \"9193e4fe4b7d69ab86d4190d3f7bbad378cc5229a630f77abfb9e1e2fa4ca521\": container with ID starting with 9193e4fe4b7d69ab86d4190d3f7bbad378cc5229a630f77abfb9e1e2fa4ca521 not found: ID does not exist"
Jul 07 17:17:22.983112 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:22.983095    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d-proxy-tls\") pod \"4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d\" (UID: \"4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d\") "
Jul 07 17:17:22.983175 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:22.983132    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drcs8\" (UniqueName: \"kubernetes.io/projected/4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d-kube-api-access-drcs8\") pod \"4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d\" (UID: \"4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d\") "
Jul 07 17:17:22.983208 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:22.983179    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d-kube-rbac-proxy-sar-config\") pod \"4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d\" (UID: \"4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d\") "
Jul 07 17:17:22.983527 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:22.983506    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d" (UID: "4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:17:22.985080 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:22.985062    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d-kube-api-access-drcs8" (OuterVolumeSpecName: "kube-api-access-drcs8") pod "4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d" (UID: "4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d"). InnerVolumeSpecName "kube-api-access-drcs8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:17:22.985151 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:22.985086    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d" (UID: "4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 17:17:23.084503 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:23.084469    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d-proxy-tls\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 17:17:23.084503 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:23.084499    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drcs8\" (UniqueName: \"kubernetes.io/projected/4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d-kube-api-access-drcs8\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 17:17:23.084711 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:23.084513    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 17:17:23.279721 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:23.279694    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f8ada-predictor-7c47cd4646-qc5fc"]
Jul 07 17:17:23.283051 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:23.283026    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f8ada-predictor-7c47cd4646-qc5fc"]
Jul 07 17:17:23.724910 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:23.724880    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d" path="/var/lib/kubelet/pods/4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d/volumes"
Jul 07 17:17:26.995706 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:26.995664    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-f8ada-7445689dbf-slsqt" podUID="7c75328a-bda4-4e8b-8c6c-3517d2f815a2" containerName="ensemble-graph-f8ada" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:17:27.963395 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:27.963370    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-f587d-predictor-79b9cd4678-mpl9k"
Jul 07 17:17:27.963835 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:27.963813    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f587d-predictor-79b9cd4678-mpl9k" podUID="58ad52b5-f293-4951-851b-49cff8359d1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 07 17:17:31.994997 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:31.994943    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-f8ada-7445689dbf-slsqt" podUID="7c75328a-bda4-4e8b-8c6c-3517d2f815a2" containerName="ensemble-graph-f8ada" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:17:31.995455 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:31.995057    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-f8ada-7445689dbf-slsqt"
Jul 07 17:17:36.995556 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:36.995476    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-f8ada-7445689dbf-slsqt" podUID="7c75328a-bda4-4e8b-8c6c-3517d2f815a2" containerName="ensemble-graph-f8ada" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:17:37.964395 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:37.964362    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f587d-predictor-79b9cd4678-mpl9k" podUID="58ad52b5-f293-4951-851b-49cff8359d1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 07 17:17:41.995346 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:41.995307    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-f8ada-7445689dbf-slsqt" podUID="7c75328a-bda4-4e8b-8c6c-3517d2f815a2" containerName="ensemble-graph-f8ada" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:17:46.995158 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:46.995106    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-f8ada-7445689dbf-slsqt" podUID="7c75328a-bda4-4e8b-8c6c-3517d2f815a2" containerName="ensemble-graph-f8ada" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:17:47.963924 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:47.963887    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f587d-predictor-79b9cd4678-mpl9k" podUID="58ad52b5-f293-4951-851b-49cff8359d1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 07 17:17:49.312445 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:49.312423    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-f8ada-7445689dbf-slsqt"
Jul 07 17:17:49.415049 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:49.415014    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c75328a-bda4-4e8b-8c6c-3517d2f815a2-proxy-tls\") pod \"7c75328a-bda4-4e8b-8c6c-3517d2f815a2\" (UID: \"7c75328a-bda4-4e8b-8c6c-3517d2f815a2\") "
Jul 07 17:17:49.415200 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:49.415082    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c75328a-bda4-4e8b-8c6c-3517d2f815a2-openshift-service-ca-bundle\") pod \"7c75328a-bda4-4e8b-8c6c-3517d2f815a2\" (UID: \"7c75328a-bda4-4e8b-8c6c-3517d2f815a2\") "
Jul 07 17:17:49.415445 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:49.415421    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c75328a-bda4-4e8b-8c6c-3517d2f815a2-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "7c75328a-bda4-4e8b-8c6c-3517d2f815a2" (UID: "7c75328a-bda4-4e8b-8c6c-3517d2f815a2"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:17:49.417092 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:49.417069    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c75328a-bda4-4e8b-8c6c-3517d2f815a2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7c75328a-bda4-4e8b-8c6c-3517d2f815a2" (UID: "7c75328a-bda4-4e8b-8c6c-3517d2f815a2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 17:17:49.516370 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:49.516299    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c75328a-bda4-4e8b-8c6c-3517d2f815a2-proxy-tls\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 17:17:49.516370 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:49.516324    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c75328a-bda4-4e8b-8c6c-3517d2f815a2-openshift-service-ca-bundle\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 17:17:50.066661 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:50.066630    2578 generic.go:356] "Generic (PLEG): container finished" podID="7c75328a-bda4-4e8b-8c6c-3517d2f815a2" containerID="5eb9bb233d2839aed69859d555c38d7d78f4259d2a79e3df9ddc6bc7e4f59afe" exitCode=0
Jul 07 17:17:50.066856 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:50.066670    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-f8ada-7445689dbf-slsqt"
Jul 07 17:17:50.066856 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:50.066721    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-f8ada-7445689dbf-slsqt" event={"ID":"7c75328a-bda4-4e8b-8c6c-3517d2f815a2","Type":"ContainerDied","Data":"5eb9bb233d2839aed69859d555c38d7d78f4259d2a79e3df9ddc6bc7e4f59afe"}
Jul 07 17:17:50.066856 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:50.066788    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-f8ada-7445689dbf-slsqt" event={"ID":"7c75328a-bda4-4e8b-8c6c-3517d2f815a2","Type":"ContainerDied","Data":"85cf1f5e24f1bd75273d4b033aababaa40af03fc8843607a1e256b8ce5ea2414"}
Jul 07 17:17:50.066856 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:50.066812    2578 scope.go:117] "RemoveContainer" containerID="5eb9bb233d2839aed69859d555c38d7d78f4259d2a79e3df9ddc6bc7e4f59afe"
Jul 07 17:17:50.074686 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:50.074678    2578 scope.go:117] "RemoveContainer" containerID="5eb9bb233d2839aed69859d555c38d7d78f4259d2a79e3df9ddc6bc7e4f59afe"
Jul 07 17:17:50.074982 ip-10-0-136-217 kubenswrapper[2578]: E0707 17:17:50.074937    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5eb9bb233d2839aed69859d555c38d7d78f4259d2a79e3df9ddc6bc7e4f59afe\": container with ID starting with 5eb9bb233d2839aed69859d555c38d7d78f4259d2a79e3df9ddc6bc7e4f59afe not found: ID does not exist" containerID="5eb9bb233d2839aed69859d555c38d7d78f4259d2a79e3df9ddc6bc7e4f59afe"
Jul 07 17:17:50.074982 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:50.074965    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5eb9bb233d2839aed69859d555c38d7d78f4259d2a79e3df9ddc6bc7e4f59afe"} err="failed to get container status \"5eb9bb233d2839aed69859d555c38d7d78f4259d2a79e3df9ddc6bc7e4f59afe\": rpc error: code = NotFound desc = could not find container \"5eb9bb233d2839aed69859d555c38d7d78f4259d2a79e3df9ddc6bc7e4f59afe\": container with ID starting with 5eb9bb233d2839aed69859d555c38d7d78f4259d2a79e3df9ddc6bc7e4f59afe not found: ID does not exist"
Jul 07 17:17:50.083216 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:50.083198    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-f8ada-7445689dbf-slsqt"]
Jul 07 17:17:50.087639 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:50.087611    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-f8ada-7445689dbf-slsqt"]
Jul 07 17:17:51.725430 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:51.725353    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c75328a-bda4-4e8b-8c6c-3517d2f815a2" path="/var/lib/kubelet/pods/7c75328a-bda4-4e8b-8c6c-3517d2f815a2/volumes"
Jul 07 17:17:54.809330 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:54.809301    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-3653b-c47d47c8-ff2k7"]
Jul 07 17:17:54.809820 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:54.809494    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-3653b-c47d47c8-ff2k7" podUID="b2499088-f7a4-4598-bbc1-70b3dbe8a156" containerName="sequence-graph-3653b" containerID="cri-o://13f6132e7ad6dbca310498bc927ba3049b82a2541b0f5315221b0f477d92c1d8" gracePeriod=30
Jul 07 17:17:55.050701 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:55.050667    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1a68d-predictor-fd8c98474-wxjjp"]
Jul 07 17:17:55.051017 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:55.051004    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d" containerName="kserve-container"
Jul 07 17:17:55.051067 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:55.051019    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d" containerName="kserve-container"
Jul 07 17:17:55.051067 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:55.051027    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c75328a-bda4-4e8b-8c6c-3517d2f815a2" containerName="ensemble-graph-f8ada"
Jul 07 17:17:55.051067 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:55.051032    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c75328a-bda4-4e8b-8c6c-3517d2f815a2" containerName="ensemble-graph-f8ada"
Jul 07 17:17:55.051067 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:55.051047    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d" containerName="kube-rbac-proxy"
Jul 07 17:17:55.051067 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:55.051053    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d" containerName="kube-rbac-proxy"
Jul 07 17:17:55.051221 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:55.051099    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d" containerName="kube-rbac-proxy"
Jul 07 17:17:55.051221 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:55.051109    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4be9ad02-f0f9-42bf-a0d1-95e129d7bf9d" containerName="kserve-container"
Jul 07 17:17:55.051221 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:55.051115    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c75328a-bda4-4e8b-8c6c-3517d2f815a2" containerName="ensemble-graph-f8ada"
Jul 07 17:17:55.054189 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:55.054170    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1a68d-predictor-fd8c98474-wxjjp"
Jul 07 17:17:55.056932 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:55.056904    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-1a68d-kube-rbac-proxy-sar-config\""
Jul 07 17:17:55.056932 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:55.056922    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-1a68d-predictor-serving-cert\""
Jul 07 17:17:55.062584 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:55.062563    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1a68d-predictor-fd8c98474-wxjjp"]
Jul 07 17:17:55.091165 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:55.091139    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3653b-predictor-799d8b94f6-dj9g8"]
Jul 07 17:17:55.091408 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:55.091377    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-3653b-predictor-799d8b94f6-dj9g8" podUID="ebca297e-b62c-4edb-add5-c6a2001be637" containerName="kube-rbac-proxy" containerID="cri-o://0f67a12269b363f2e00effb9f4f9f150f2ffbd181fb638cb08cdff6e16504859" gracePeriod=30
Jul 07 17:17:55.091408 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:55.091356    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-3653b-predictor-799d8b94f6-dj9g8" podUID="ebca297e-b62c-4edb-add5-c6a2001be637" containerName="kserve-container" containerID="cri-o://4177d3dead1323c82fd58436a3b3873dbeefd8a57274d5f275576ddfa598a29b" gracePeriod=30
Jul 07 17:17:55.160339 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:55.160308    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a936a6b-2c12-4ab9-987f-5dbe899ce9ec-proxy-tls\") pod \"error-404-isvc-1a68d-predictor-fd8c98474-wxjjp\" (UID: \"1a936a6b-2c12-4ab9-987f-5dbe899ce9ec\") " pod="kserve-ci-e2e-test/error-404-isvc-1a68d-predictor-fd8c98474-wxjjp"
Jul 07 17:17:55.160500 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:55.160356    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5sms\" (UniqueName: \"kubernetes.io/projected/1a936a6b-2c12-4ab9-987f-5dbe899ce9ec-kube-api-access-q5sms\") pod \"error-404-isvc-1a68d-predictor-fd8c98474-wxjjp\" (UID: \"1a936a6b-2c12-4ab9-987f-5dbe899ce9ec\") " pod="kserve-ci-e2e-test/error-404-isvc-1a68d-predictor-fd8c98474-wxjjp"
Jul 07 17:17:55.160500 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:55.160409    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a936a6b-2c12-4ab9-987f-5dbe899ce9ec-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1a68d-predictor-fd8c98474-wxjjp\" (UID: \"1a936a6b-2c12-4ab9-987f-5dbe899ce9ec\") " pod="kserve-ci-e2e-test/error-404-isvc-1a68d-predictor-fd8c98474-wxjjp"
Jul 07 17:17:55.260815 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:55.260782    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5sms\" (UniqueName: \"kubernetes.io/projected/1a936a6b-2c12-4ab9-987f-5dbe899ce9ec-kube-api-access-q5sms\") pod \"error-404-isvc-1a68d-predictor-fd8c98474-wxjjp\" (UID: \"1a936a6b-2c12-4ab9-987f-5dbe899ce9ec\") " pod="kserve-ci-e2e-test/error-404-isvc-1a68d-predictor-fd8c98474-wxjjp"
Jul 07 17:17:55.260975 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:55.260837    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a936a6b-2c12-4ab9-987f-5dbe899ce9ec-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1a68d-predictor-fd8c98474-wxjjp\" (UID: \"1a936a6b-2c12-4ab9-987f-5dbe899ce9ec\") " pod="kserve-ci-e2e-test/error-404-isvc-1a68d-predictor-fd8c98474-wxjjp"
Jul 07 17:17:55.260975 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:55.260886    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a936a6b-2c12-4ab9-987f-5dbe899ce9ec-proxy-tls\") pod \"error-404-isvc-1a68d-predictor-fd8c98474-wxjjp\" (UID: \"1a936a6b-2c12-4ab9-987f-5dbe899ce9ec\") " pod="kserve-ci-e2e-test/error-404-isvc-1a68d-predictor-fd8c98474-wxjjp"
Jul 07 17:17:55.261060 ip-10-0-136-217 kubenswrapper[2578]: E0707 17:17:55.260994    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-1a68d-predictor-serving-cert: secret "error-404-isvc-1a68d-predictor-serving-cert" not found
Jul 07 17:17:55.261060 ip-10-0-136-217 kubenswrapper[2578]: E0707 17:17:55.261045    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a936a6b-2c12-4ab9-987f-5dbe899ce9ec-proxy-tls podName:1a936a6b-2c12-4ab9-987f-5dbe899ce9ec nodeName:}" failed. No retries permitted until 2026-07-07 17:17:55.76102953 +0000 UTC m=+1812.537343714 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1a936a6b-2c12-4ab9-987f-5dbe899ce9ec-proxy-tls") pod "error-404-isvc-1a68d-predictor-fd8c98474-wxjjp" (UID: "1a936a6b-2c12-4ab9-987f-5dbe899ce9ec") : secret "error-404-isvc-1a68d-predictor-serving-cert" not found
Jul 07 17:17:55.261505 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:55.261485    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a936a6b-2c12-4ab9-987f-5dbe899ce9ec-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1a68d-predictor-fd8c98474-wxjjp\" (UID: \"1a936a6b-2c12-4ab9-987f-5dbe899ce9ec\") " pod="kserve-ci-e2e-test/error-404-isvc-1a68d-predictor-fd8c98474-wxjjp"
Jul 07 17:17:55.270052 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:55.270023    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5sms\" (UniqueName: \"kubernetes.io/projected/1a936a6b-2c12-4ab9-987f-5dbe899ce9ec-kube-api-access-q5sms\") pod \"error-404-isvc-1a68d-predictor-fd8c98474-wxjjp\" (UID: \"1a936a6b-2c12-4ab9-987f-5dbe899ce9ec\") " pod="kserve-ci-e2e-test/error-404-isvc-1a68d-predictor-fd8c98474-wxjjp"
Jul 07 17:17:55.765502 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:55.765470    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a936a6b-2c12-4ab9-987f-5dbe899ce9ec-proxy-tls\") pod \"error-404-isvc-1a68d-predictor-fd8c98474-wxjjp\" (UID: \"1a936a6b-2c12-4ab9-987f-5dbe899ce9ec\") " pod="kserve-ci-e2e-test/error-404-isvc-1a68d-predictor-fd8c98474-wxjjp"
Jul 07 17:17:55.767881 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:55.767859    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a936a6b-2c12-4ab9-987f-5dbe899ce9ec-proxy-tls\") pod \"error-404-isvc-1a68d-predictor-fd8c98474-wxjjp\" (UID: \"1a936a6b-2c12-4ab9-987f-5dbe899ce9ec\") " pod="kserve-ci-e2e-test/error-404-isvc-1a68d-predictor-fd8c98474-wxjjp"
Jul 07 17:17:55.965355 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:55.965324    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1a68d-predictor-fd8c98474-wxjjp"
Jul 07 17:17:56.094235 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:56.094206    2578 generic.go:356] "Generic (PLEG): container finished" podID="ebca297e-b62c-4edb-add5-c6a2001be637" containerID="0f67a12269b363f2e00effb9f4f9f150f2ffbd181fb638cb08cdff6e16504859" exitCode=2
Jul 07 17:17:56.094382 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:56.094261    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3653b-predictor-799d8b94f6-dj9g8" event={"ID":"ebca297e-b62c-4edb-add5-c6a2001be637","Type":"ContainerDied","Data":"0f67a12269b363f2e00effb9f4f9f150f2ffbd181fb638cb08cdff6e16504859"}
Jul 07 17:17:56.097659 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:56.097646    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1a68d-predictor-fd8c98474-wxjjp"]
Jul 07 17:17:56.100305 ip-10-0-136-217 kubenswrapper[2578]: W0707 17:17:56.100282    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a936a6b_2c12_4ab9_987f_5dbe899ce9ec.slice/crio-c1dc0d8bd1b32e273158be64658868bd82e610b1cdd6c3c3c71f22ec00d43b54 WatchSource:0}: Error finding container c1dc0d8bd1b32e273158be64658868bd82e610b1cdd6c3c3c71f22ec00d43b54: Status 404 returned error can't find the container with id c1dc0d8bd1b32e273158be64658868bd82e610b1cdd6c3c3c71f22ec00d43b54
Jul 07 17:17:57.100135 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:57.100098    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1a68d-predictor-fd8c98474-wxjjp" event={"ID":"1a936a6b-2c12-4ab9-987f-5dbe899ce9ec","Type":"ContainerStarted","Data":"2a2ffabd90d1f609a3825dd2540bf5c5343b8365d64b76caf2bff74575b9c741"}
Jul 07 17:17:57.100621 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:57.100164    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1a68d-predictor-fd8c98474-wxjjp" event={"ID":"1a936a6b-2c12-4ab9-987f-5dbe899ce9ec","Type":"ContainerStarted","Data":"4ca7cd93aab2a13195ee842d0401146609eaf811369f99af98e9443d30f076e9"}
Jul 07 17:17:57.100621 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:57.100200    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-1a68d-predictor-fd8c98474-wxjjp"
Jul 07 17:17:57.100621 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:57.100231    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1a68d-predictor-fd8c98474-wxjjp" event={"ID":"1a936a6b-2c12-4ab9-987f-5dbe899ce9ec","Type":"ContainerStarted","Data":"c1dc0d8bd1b32e273158be64658868bd82e610b1cdd6c3c3c71f22ec00d43b54"}
Jul 07 17:17:57.100621 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:57.100277    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-1a68d-predictor-fd8c98474-wxjjp"
Jul 07 17:17:57.101313 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:57.101291    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1a68d-predictor-fd8c98474-wxjjp" podUID="1a936a6b-2c12-4ab9-987f-5dbe899ce9ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 07 17:17:57.117728 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:57.117683    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-1a68d-predictor-fd8c98474-wxjjp" podStartSLOduration=2.117670789 podStartE2EDuration="2.117670789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:17:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:17:57.116811257 +0000 UTC m=+1813.893125485" watchObservedRunningTime="2026-07-07 17:17:57.117670789 +0000 UTC m=+1813.893984996"
Jul 07 17:17:57.138901 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:57.138866    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-3653b-c47d47c8-ff2k7" podUID="b2499088-f7a4-4598-bbc1-70b3dbe8a156" containerName="sequence-graph-3653b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:17:57.889629 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:57.889590    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3653b-predictor-799d8b94f6-dj9g8" podUID="ebca297e-b62c-4edb-add5-c6a2001be637" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Jul 07 17:17:57.894982 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:57.894955    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3653b-predictor-799d8b94f6-dj9g8" podUID="ebca297e-b62c-4edb-add5-c6a2001be637" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 07 17:17:57.964372 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:57.964334    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f587d-predictor-79b9cd4678-mpl9k" podUID="58ad52b5-f293-4951-851b-49cff8359d1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 07 17:17:58.104101 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:58.104067    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1a68d-predictor-fd8c98474-wxjjp" podUID="1a936a6b-2c12-4ab9-987f-5dbe899ce9ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 07 17:17:58.242332 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:58.242297    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3653b-predictor-799d8b94f6-dj9g8"
Jul 07 17:17:58.288073 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:58.288043    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pvjx7\" (UniqueName: \"kubernetes.io/projected/ebca297e-b62c-4edb-add5-c6a2001be637-kube-api-access-pvjx7\") pod \"ebca297e-b62c-4edb-add5-c6a2001be637\" (UID: \"ebca297e-b62c-4edb-add5-c6a2001be637\") "
Jul 07 17:17:58.288225 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:58.288144    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ebca297e-b62c-4edb-add5-c6a2001be637-kube-rbac-proxy-sar-config\") pod \"ebca297e-b62c-4edb-add5-c6a2001be637\" (UID: \"ebca297e-b62c-4edb-add5-c6a2001be637\") "
Jul 07 17:17:58.288225 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:58.288178    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebca297e-b62c-4edb-add5-c6a2001be637-proxy-tls\") pod \"ebca297e-b62c-4edb-add5-c6a2001be637\" (UID: \"ebca297e-b62c-4edb-add5-c6a2001be637\") "
Jul 07 17:17:58.288555 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:58.288532    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebca297e-b62c-4edb-add5-c6a2001be637-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ebca297e-b62c-4edb-add5-c6a2001be637" (UID: "ebca297e-b62c-4edb-add5-c6a2001be637"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:17:58.290086 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:58.290067    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebca297e-b62c-4edb-add5-c6a2001be637-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ebca297e-b62c-4edb-add5-c6a2001be637" (UID: "ebca297e-b62c-4edb-add5-c6a2001be637"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 17:17:58.290257 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:58.290224    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebca297e-b62c-4edb-add5-c6a2001be637-kube-api-access-pvjx7" (OuterVolumeSpecName: "kube-api-access-pvjx7") pod "ebca297e-b62c-4edb-add5-c6a2001be637" (UID: "ebca297e-b62c-4edb-add5-c6a2001be637"). InnerVolumeSpecName "kube-api-access-pvjx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:17:58.389502 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:58.389468    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ebca297e-b62c-4edb-add5-c6a2001be637-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 17:17:58.389502 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:58.389495    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebca297e-b62c-4edb-add5-c6a2001be637-proxy-tls\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 17:17:58.389502 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:58.389505    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvjx7\" (UniqueName: \"kubernetes.io/projected/ebca297e-b62c-4edb-add5-c6a2001be637-kube-api-access-pvjx7\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 17:17:59.109358 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:59.109325    2578 generic.go:356] "Generic (PLEG): container finished" podID="ebca297e-b62c-4edb-add5-c6a2001be637" containerID="4177d3dead1323c82fd58436a3b3873dbeefd8a57274d5f275576ddfa598a29b" exitCode=0
Jul 07 17:17:59.109810 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:59.109377    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3653b-predictor-799d8b94f6-dj9g8"
Jul 07 17:17:59.109810 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:59.109396    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3653b-predictor-799d8b94f6-dj9g8" event={"ID":"ebca297e-b62c-4edb-add5-c6a2001be637","Type":"ContainerDied","Data":"4177d3dead1323c82fd58436a3b3873dbeefd8a57274d5f275576ddfa598a29b"}
Jul 07 17:17:59.109810 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:59.109467    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3653b-predictor-799d8b94f6-dj9g8" event={"ID":"ebca297e-b62c-4edb-add5-c6a2001be637","Type":"ContainerDied","Data":"e633bab9c537640b2e8fa67932efba2d91b2fdb8b116c3df49d69c6df462f80c"}
Jul 07 17:17:59.109810 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:59.109507    2578 scope.go:117] "RemoveContainer" containerID="0f67a12269b363f2e00effb9f4f9f150f2ffbd181fb638cb08cdff6e16504859"
Jul 07 17:17:59.117823 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:59.117804    2578 scope.go:117] "RemoveContainer" containerID="4177d3dead1323c82fd58436a3b3873dbeefd8a57274d5f275576ddfa598a29b"
Jul 07 17:17:59.124592 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:59.124576    2578 scope.go:117] "RemoveContainer" containerID="0f67a12269b363f2e00effb9f4f9f150f2ffbd181fb638cb08cdff6e16504859"
Jul 07 17:17:59.124858 ip-10-0-136-217 kubenswrapper[2578]: E0707 17:17:59.124841    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f67a12269b363f2e00effb9f4f9f150f2ffbd181fb638cb08cdff6e16504859\": container with ID starting with 0f67a12269b363f2e00effb9f4f9f150f2ffbd181fb638cb08cdff6e16504859 not found: ID does not exist" containerID="0f67a12269b363f2e00effb9f4f9f150f2ffbd181fb638cb08cdff6e16504859"
Jul 07 17:17:59.124915 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:59.124866    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f67a12269b363f2e00effb9f4f9f150f2ffbd181fb638cb08cdff6e16504859"} err="failed to get container status \"0f67a12269b363f2e00effb9f4f9f150f2ffbd181fb638cb08cdff6e16504859\": rpc error: code = NotFound desc = could not find container \"0f67a12269b363f2e00effb9f4f9f150f2ffbd181fb638cb08cdff6e16504859\": container with ID starting with 0f67a12269b363f2e00effb9f4f9f150f2ffbd181fb638cb08cdff6e16504859 not found: ID does not exist"
Jul 07 17:17:59.124915 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:59.124884    2578 scope.go:117] "RemoveContainer" containerID="4177d3dead1323c82fd58436a3b3873dbeefd8a57274d5f275576ddfa598a29b"
Jul 07 17:17:59.125097 ip-10-0-136-217 kubenswrapper[2578]: E0707 17:17:59.125080    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4177d3dead1323c82fd58436a3b3873dbeefd8a57274d5f275576ddfa598a29b\": container with ID starting with 4177d3dead1323c82fd58436a3b3873dbeefd8a57274d5f275576ddfa598a29b not found: ID does not exist" containerID="4177d3dead1323c82fd58436a3b3873dbeefd8a57274d5f275576ddfa598a29b"
Jul 07 17:17:59.125152 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:59.125107    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4177d3dead1323c82fd58436a3b3873dbeefd8a57274d5f275576ddfa598a29b"} err="failed to get container status \"4177d3dead1323c82fd58436a3b3873dbeefd8a57274d5f275576ddfa598a29b\": rpc error: code = NotFound desc = could not find container \"4177d3dead1323c82fd58436a3b3873dbeefd8a57274d5f275576ddfa598a29b\": container with ID starting with 4177d3dead1323c82fd58436a3b3873dbeefd8a57274d5f275576ddfa598a29b not found: ID does not exist"
Jul 07 17:17:59.131513 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:59.131490    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3653b-predictor-799d8b94f6-dj9g8"]
Jul 07 17:17:59.136258 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:59.136234    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3653b-predictor-799d8b94f6-dj9g8"]
Jul 07 17:17:59.724576 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:17:59.724547    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebca297e-b62c-4edb-add5-c6a2001be637" path="/var/lib/kubelet/pods/ebca297e-b62c-4edb-add5-c6a2001be637/volumes"
Jul 07 17:18:02.138380 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:02.138346    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-3653b-c47d47c8-ff2k7" podUID="b2499088-f7a4-4598-bbc1-70b3dbe8a156" containerName="sequence-graph-3653b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:18:03.109282 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:03.109256    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-1a68d-predictor-fd8c98474-wxjjp"
Jul 07 17:18:03.109681 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:03.109646    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1a68d-predictor-fd8c98474-wxjjp" podUID="1a936a6b-2c12-4ab9-987f-5dbe899ce9ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 07 17:18:07.138712 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:07.138678    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-3653b-c47d47c8-ff2k7" podUID="b2499088-f7a4-4598-bbc1-70b3dbe8a156" containerName="sequence-graph-3653b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:18:07.139159 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:07.138802    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-3653b-c47d47c8-ff2k7"
Jul 07 17:18:07.965308 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:07.965278    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-f587d-predictor-79b9cd4678-mpl9k"
Jul 07 17:18:12.138725 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:12.138682    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-3653b-c47d47c8-ff2k7" podUID="b2499088-f7a4-4598-bbc1-70b3dbe8a156" containerName="sequence-graph-3653b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:18:13.110055 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:13.110020    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1a68d-predictor-fd8c98474-wxjjp" podUID="1a936a6b-2c12-4ab9-987f-5dbe899ce9ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 07 17:18:17.139185 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:17.139143    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-3653b-c47d47c8-ff2k7" podUID="b2499088-f7a4-4598-bbc1-70b3dbe8a156" containerName="sequence-graph-3653b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:18:19.399069 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:19.399036    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-f587d-68bc64bffc-2zmhp"]
Jul 07 17:18:19.399447 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:19.399390    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebca297e-b62c-4edb-add5-c6a2001be637" containerName="kserve-container"
Jul 07 17:18:19.399447 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:19.399403    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebca297e-b62c-4edb-add5-c6a2001be637" containerName="kserve-container"
Jul 07 17:18:19.399447 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:19.399412    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebca297e-b62c-4edb-add5-c6a2001be637" containerName="kube-rbac-proxy"
Jul 07 17:18:19.399447 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:19.399421    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebca297e-b62c-4edb-add5-c6a2001be637" containerName="kube-rbac-proxy"
Jul 07 17:18:19.399579 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:19.399476    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebca297e-b62c-4edb-add5-c6a2001be637" containerName="kserve-container"
Jul 07 17:18:19.399579 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:19.399486    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebca297e-b62c-4edb-add5-c6a2001be637" containerName="kube-rbac-proxy"
Jul 07 17:18:19.402040 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:19.402023    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-f587d-68bc64bffc-2zmhp"
Jul 07 17:18:19.404568 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:19.404547    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-f587d-serving-cert\""
Jul 07 17:18:19.404819 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:19.404803    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-f587d-kube-rbac-proxy-sar-config\""
Jul 07 17:18:19.411009 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:19.410990    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-f587d-68bc64bffc-2zmhp"]
Jul 07 17:18:19.571989 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:19.571956    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81d084b1-9833-469b-a1b0-dde7cb6f564c-openshift-service-ca-bundle\") pod \"splitter-graph-f587d-68bc64bffc-2zmhp\" (UID: \"81d084b1-9833-469b-a1b0-dde7cb6f564c\") " pod="kserve-ci-e2e-test/splitter-graph-f587d-68bc64bffc-2zmhp"
Jul 07 17:18:19.572160 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:19.572026    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81d084b1-9833-469b-a1b0-dde7cb6f564c-proxy-tls\") pod \"splitter-graph-f587d-68bc64bffc-2zmhp\" (UID: \"81d084b1-9833-469b-a1b0-dde7cb6f564c\") " pod="kserve-ci-e2e-test/splitter-graph-f587d-68bc64bffc-2zmhp"
Jul 07 17:18:19.672482 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:19.672386    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81d084b1-9833-469b-a1b0-dde7cb6f564c-openshift-service-ca-bundle\") pod \"splitter-graph-f587d-68bc64bffc-2zmhp\" (UID: \"81d084b1-9833-469b-a1b0-dde7cb6f564c\") " pod="kserve-ci-e2e-test/splitter-graph-f587d-68bc64bffc-2zmhp"
Jul 07 17:18:19.672482 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:19.672444    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81d084b1-9833-469b-a1b0-dde7cb6f564c-proxy-tls\") pod \"splitter-graph-f587d-68bc64bffc-2zmhp\" (UID: \"81d084b1-9833-469b-a1b0-dde7cb6f564c\") " pod="kserve-ci-e2e-test/splitter-graph-f587d-68bc64bffc-2zmhp"
Jul 07 17:18:19.673111 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:19.673086    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81d084b1-9833-469b-a1b0-dde7cb6f564c-openshift-service-ca-bundle\") pod \"splitter-graph-f587d-68bc64bffc-2zmhp\" (UID: \"81d084b1-9833-469b-a1b0-dde7cb6f564c\") " pod="kserve-ci-e2e-test/splitter-graph-f587d-68bc64bffc-2zmhp"
Jul 07 17:18:19.674797 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:19.674778    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81d084b1-9833-469b-a1b0-dde7cb6f564c-proxy-tls\") pod \"splitter-graph-f587d-68bc64bffc-2zmhp\" (UID: \"81d084b1-9833-469b-a1b0-dde7cb6f564c\") " pod="kserve-ci-e2e-test/splitter-graph-f587d-68bc64bffc-2zmhp"
Jul 07 17:18:19.712931 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:19.712906    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-f587d-68bc64bffc-2zmhp"
Jul 07 17:18:19.842932 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:19.842908    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-f587d-68bc64bffc-2zmhp"]
Jul 07 17:18:19.845610 ip-10-0-136-217 kubenswrapper[2578]: W0707 17:18:19.845582    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81d084b1_9833_469b_a1b0_dde7cb6f564c.slice/crio-ff6ddbf434518c7df4fea59afd1105bf371c9b3231cdc72b07a1e17acc931a4e WatchSource:0}: Error finding container ff6ddbf434518c7df4fea59afd1105bf371c9b3231cdc72b07a1e17acc931a4e: Status 404 returned error can't find the container with id ff6ddbf434518c7df4fea59afd1105bf371c9b3231cdc72b07a1e17acc931a4e
Jul 07 17:18:20.200185 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:20.200153    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-f587d-68bc64bffc-2zmhp" event={"ID":"81d084b1-9833-469b-a1b0-dde7cb6f564c","Type":"ContainerStarted","Data":"5eb9d43af48806c649ab43b29d5cd11cdca5bcd7f9f9a607c1d89dece722975f"}
Jul 07 17:18:20.200366 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:20.200203    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-f587d-68bc64bffc-2zmhp" event={"ID":"81d084b1-9833-469b-a1b0-dde7cb6f564c","Type":"ContainerStarted","Data":"ff6ddbf434518c7df4fea59afd1105bf371c9b3231cdc72b07a1e17acc931a4e"}
Jul 07 17:18:20.200366 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:20.200328    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-f587d-68bc64bffc-2zmhp"
Jul 07 17:18:20.217953 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:20.217903    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-f587d-68bc64bffc-2zmhp" podStartSLOduration=1.217888756 podStartE2EDuration="1.217888756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:18:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:18:20.215321631 +0000 UTC m=+1836.991635840" watchObservedRunningTime="2026-07-07 17:18:20.217888756 +0000 UTC m=+1836.994202961"
Jul 07 17:18:22.138388 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:22.138354    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-3653b-c47d47c8-ff2k7" podUID="b2499088-f7a4-4598-bbc1-70b3dbe8a156" containerName="sequence-graph-3653b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:18:23.110637 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:23.110603    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1a68d-predictor-fd8c98474-wxjjp" podUID="1a936a6b-2c12-4ab9-987f-5dbe899ce9ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 07 17:18:24.992892 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:24.992869    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-3653b-c47d47c8-ff2k7"
Jul 07 17:18:25.118299 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:25.118193    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2499088-f7a4-4598-bbc1-70b3dbe8a156-openshift-service-ca-bundle\") pod \"b2499088-f7a4-4598-bbc1-70b3dbe8a156\" (UID: \"b2499088-f7a4-4598-bbc1-70b3dbe8a156\") "
Jul 07 17:18:25.118299 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:25.118281    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2499088-f7a4-4598-bbc1-70b3dbe8a156-proxy-tls\") pod \"b2499088-f7a4-4598-bbc1-70b3dbe8a156\" (UID: \"b2499088-f7a4-4598-bbc1-70b3dbe8a156\") "
Jul 07 17:18:25.118592 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:25.118567    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2499088-f7a4-4598-bbc1-70b3dbe8a156-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "b2499088-f7a4-4598-bbc1-70b3dbe8a156" (UID: "b2499088-f7a4-4598-bbc1-70b3dbe8a156"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:18:25.120278 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:25.120247    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2499088-f7a4-4598-bbc1-70b3dbe8a156-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b2499088-f7a4-4598-bbc1-70b3dbe8a156" (UID: "b2499088-f7a4-4598-bbc1-70b3dbe8a156"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 17:18:25.219453 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:25.219423    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2499088-f7a4-4598-bbc1-70b3dbe8a156-openshift-service-ca-bundle\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 17:18:25.219453 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:25.219447    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2499088-f7a4-4598-bbc1-70b3dbe8a156-proxy-tls\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 17:18:25.221314 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:25.221285    2578 generic.go:356] "Generic (PLEG): container finished" podID="b2499088-f7a4-4598-bbc1-70b3dbe8a156" containerID="13f6132e7ad6dbca310498bc927ba3049b82a2541b0f5315221b0f477d92c1d8" exitCode=0
Jul 07 17:18:25.221425 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:25.221329    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-3653b-c47d47c8-ff2k7" event={"ID":"b2499088-f7a4-4598-bbc1-70b3dbe8a156","Type":"ContainerDied","Data":"13f6132e7ad6dbca310498bc927ba3049b82a2541b0f5315221b0f477d92c1d8"}
Jul 07 17:18:25.221425 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:25.221335    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-3653b-c47d47c8-ff2k7"
Jul 07 17:18:25.221425 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:25.221411    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-3653b-c47d47c8-ff2k7" event={"ID":"b2499088-f7a4-4598-bbc1-70b3dbe8a156","Type":"ContainerDied","Data":"1e8c5134005ad858e1cddd8a0e6e07ca622f5420400558db6810b65f169552c6"}
Jul 07 17:18:25.221540 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:25.221441    2578 scope.go:117] "RemoveContainer" containerID="13f6132e7ad6dbca310498bc927ba3049b82a2541b0f5315221b0f477d92c1d8"
Jul 07 17:18:25.229598 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:25.229579    2578 scope.go:117] "RemoveContainer" containerID="13f6132e7ad6dbca310498bc927ba3049b82a2541b0f5315221b0f477d92c1d8"
Jul 07 17:18:25.229890 ip-10-0-136-217 kubenswrapper[2578]: E0707 17:18:25.229868    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13f6132e7ad6dbca310498bc927ba3049b82a2541b0f5315221b0f477d92c1d8\": container with ID starting with 13f6132e7ad6dbca310498bc927ba3049b82a2541b0f5315221b0f477d92c1d8 not found: ID does not exist" containerID="13f6132e7ad6dbca310498bc927ba3049b82a2541b0f5315221b0f477d92c1d8"
Jul 07 17:18:25.229941 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:25.229898    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13f6132e7ad6dbca310498bc927ba3049b82a2541b0f5315221b0f477d92c1d8"} err="failed to get container status \"13f6132e7ad6dbca310498bc927ba3049b82a2541b0f5315221b0f477d92c1d8\": rpc error: code = NotFound desc = could not find container \"13f6132e7ad6dbca310498bc927ba3049b82a2541b0f5315221b0f477d92c1d8\": container with ID starting with 13f6132e7ad6dbca310498bc927ba3049b82a2541b0f5315221b0f477d92c1d8 not found: ID does not exist"
Jul 07 17:18:25.241381 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:25.241362    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-3653b-c47d47c8-ff2k7"]
Jul 07 17:18:25.245439 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:25.245420    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-3653b-c47d47c8-ff2k7"]
Jul 07 17:18:25.725275 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:25.725243    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2499088-f7a4-4598-bbc1-70b3dbe8a156" path="/var/lib/kubelet/pods/b2499088-f7a4-4598-bbc1-70b3dbe8a156/volumes"
Jul 07 17:18:26.209680 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:26.209655    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-f587d-68bc64bffc-2zmhp"
Jul 07 17:18:29.484380 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:29.484302    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-f587d-68bc64bffc-2zmhp"]
Jul 07 17:18:29.484910 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:29.484514    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-f587d-68bc64bffc-2zmhp" podUID="81d084b1-9833-469b-a1b0-dde7cb6f564c" containerName="splitter-graph-f587d" containerID="cri-o://5eb9d43af48806c649ab43b29d5cd11cdca5bcd7f9f9a607c1d89dece722975f" gracePeriod=30
Jul 07 17:18:29.568052 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:29.568020    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f587d-predictor-79b9cd4678-mpl9k"]
Jul 07 17:18:29.568640 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:29.568601    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-f587d-predictor-79b9cd4678-mpl9k" podUID="58ad52b5-f293-4951-851b-49cff8359d1d" containerName="kserve-container" containerID="cri-o://7cdc2cf99ab77be158ec6b879eb52d4a6fffe21db774a6c656e9ceae16f032a2" gracePeriod=30
Jul 07 17:18:29.568942 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:29.568890    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-f587d-predictor-79b9cd4678-mpl9k" podUID="58ad52b5-f293-4951-851b-49cff8359d1d" containerName="kube-rbac-proxy" containerID="cri-o://85730c195f8b8d77834655f64a49b49878f0a24196de75ed5c2b6458408c613b" gracePeriod=30
Jul 07 17:18:29.737781 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:29.737694    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f025d-predictor-57f69885b8-p5lrn"]
Jul 07 17:18:29.738079 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:29.738064    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2499088-f7a4-4598-bbc1-70b3dbe8a156" containerName="sequence-graph-3653b"
Jul 07 17:18:29.738118 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:29.738081    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2499088-f7a4-4598-bbc1-70b3dbe8a156" containerName="sequence-graph-3653b"
Jul 07 17:18:29.738152 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:29.738142    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2499088-f7a4-4598-bbc1-70b3dbe8a156" containerName="sequence-graph-3653b"
Jul 07 17:18:29.741026 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:29.741012    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f025d-predictor-57f69885b8-p5lrn"
Jul 07 17:18:29.743609 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:29.743590    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-f025d-predictor-serving-cert\""
Jul 07 17:18:29.743718 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:29.743605    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-f025d-kube-rbac-proxy-sar-config\""
Jul 07 17:18:29.750497 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:29.750474    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f025d-predictor-57f69885b8-p5lrn"]
Jul 07 17:18:29.757346 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:29.757325    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f025d-predictor-57f69885b8-p5lrn\" (UID: \"3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5\") " pod="kserve-ci-e2e-test/error-404-isvc-f025d-predictor-57f69885b8-p5lrn"
Jul 07 17:18:29.757451 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:29.757357    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5snj7\" (UniqueName: \"kubernetes.io/projected/3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5-kube-api-access-5snj7\") pod \"error-404-isvc-f025d-predictor-57f69885b8-p5lrn\" (UID: \"3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5\") " pod="kserve-ci-e2e-test/error-404-isvc-f025d-predictor-57f69885b8-p5lrn"
Jul 07 17:18:29.757521 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:29.757457    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5-proxy-tls\") pod \"error-404-isvc-f025d-predictor-57f69885b8-p5lrn\" (UID: \"3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5\") " pod="kserve-ci-e2e-test/error-404-isvc-f025d-predictor-57f69885b8-p5lrn"
Jul 07 17:18:29.857831 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:29.857803    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5-proxy-tls\") pod \"error-404-isvc-f025d-predictor-57f69885b8-p5lrn\" (UID: \"3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5\") " pod="kserve-ci-e2e-test/error-404-isvc-f025d-predictor-57f69885b8-p5lrn"
Jul 07 17:18:29.857979 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:29.857885    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f025d-predictor-57f69885b8-p5lrn\" (UID: \"3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5\") " pod="kserve-ci-e2e-test/error-404-isvc-f025d-predictor-57f69885b8-p5lrn"
Jul 07 17:18:29.857979 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:29.857912    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5snj7\" (UniqueName: \"kubernetes.io/projected/3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5-kube-api-access-5snj7\") pod \"error-404-isvc-f025d-predictor-57f69885b8-p5lrn\" (UID: \"3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5\") " pod="kserve-ci-e2e-test/error-404-isvc-f025d-predictor-57f69885b8-p5lrn"
Jul 07 17:18:29.857979 ip-10-0-136-217 kubenswrapper[2578]: E0707 17:18:29.857948    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-f025d-predictor-serving-cert: secret "error-404-isvc-f025d-predictor-serving-cert" not found
Jul 07 17:18:29.858129 ip-10-0-136-217 kubenswrapper[2578]: E0707 17:18:29.858005    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5-proxy-tls podName:3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5 nodeName:}" failed. No retries permitted until 2026-07-07 17:18:30.357985492 +0000 UTC m=+1847.134299677 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5-proxy-tls") pod "error-404-isvc-f025d-predictor-57f69885b8-p5lrn" (UID: "3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5") : secret "error-404-isvc-f025d-predictor-serving-cert" not found
Jul 07 17:18:29.858541 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:29.858524    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f025d-predictor-57f69885b8-p5lrn\" (UID: \"3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5\") " pod="kserve-ci-e2e-test/error-404-isvc-f025d-predictor-57f69885b8-p5lrn"
Jul 07 17:18:29.867094 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:29.867073    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5snj7\" (UniqueName: \"kubernetes.io/projected/3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5-kube-api-access-5snj7\") pod \"error-404-isvc-f025d-predictor-57f69885b8-p5lrn\" (UID: \"3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5\") " pod="kserve-ci-e2e-test/error-404-isvc-f025d-predictor-57f69885b8-p5lrn"
Jul 07 17:18:30.244447 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:30.244413    2578 generic.go:356] "Generic (PLEG): container finished" podID="58ad52b5-f293-4951-851b-49cff8359d1d" containerID="85730c195f8b8d77834655f64a49b49878f0a24196de75ed5c2b6458408c613b" exitCode=2
Jul 07 17:18:30.244447 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:30.244425    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f587d-predictor-79b9cd4678-mpl9k" event={"ID":"58ad52b5-f293-4951-851b-49cff8359d1d","Type":"ContainerDied","Data":"85730c195f8b8d77834655f64a49b49878f0a24196de75ed5c2b6458408c613b"}
Jul 07 17:18:30.361896 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:30.361857    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5-proxy-tls\") pod \"error-404-isvc-f025d-predictor-57f69885b8-p5lrn\" (UID: \"3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5\") " pod="kserve-ci-e2e-test/error-404-isvc-f025d-predictor-57f69885b8-p5lrn"
Jul 07 17:18:30.364313 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:30.364291    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5-proxy-tls\") pod \"error-404-isvc-f025d-predictor-57f69885b8-p5lrn\" (UID: \"3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5\") " pod="kserve-ci-e2e-test/error-404-isvc-f025d-predictor-57f69885b8-p5lrn"
Jul 07 17:18:30.650865 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:30.650825    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f025d-predictor-57f69885b8-p5lrn"
Jul 07 17:18:30.776982 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:30.776959    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f025d-predictor-57f69885b8-p5lrn"]
Jul 07 17:18:30.778664 ip-10-0-136-217 kubenswrapper[2578]: W0707 17:18:30.778637    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f29d3cc_e352_4d08_8f37_2cc2ebcb07a5.slice/crio-24338483341c01616d26347e77644bca6e6447b70ee3b92b5a6341b6717377fb WatchSource:0}: Error finding container 24338483341c01616d26347e77644bca6e6447b70ee3b92b5a6341b6717377fb: Status 404 returned error can't find the container with id 24338483341c01616d26347e77644bca6e6447b70ee3b92b5a6341b6717377fb
Jul 07 17:18:31.208111 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:31.208026    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-f587d-68bc64bffc-2zmhp" podUID="81d084b1-9833-469b-a1b0-dde7cb6f564c" containerName="splitter-graph-f587d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:18:31.250388 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:31.250351    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f025d-predictor-57f69885b8-p5lrn" event={"ID":"3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5","Type":"ContainerStarted","Data":"b640c6861831c73668989807fd8b85ae1e6a7ee584f2ba104b666ed9467aabcb"}
Jul 07 17:18:31.250560 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:31.250424    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-f025d-predictor-57f69885b8-p5lrn"
Jul 07 17:18:31.250560 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:31.250451    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f025d-predictor-57f69885b8-p5lrn" event={"ID":"3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5","Type":"ContainerStarted","Data":"76b15b706a31c73c9a218e8a92e11f8fa2ae73d9b44e79a3452d0d2a69a1254f"}
Jul 07 17:18:31.250684 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:31.250566    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-f025d-predictor-57f69885b8-p5lrn"
Jul 07 17:18:31.250684 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:31.250610    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f025d-predictor-57f69885b8-p5lrn" event={"ID":"3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5","Type":"ContainerStarted","Data":"24338483341c01616d26347e77644bca6e6447b70ee3b92b5a6341b6717377fb"}
Jul 07 17:18:31.251647 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:31.251624    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f025d-predictor-57f69885b8-p5lrn" podUID="3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 07 17:18:31.268352 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:31.268305    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-f025d-predictor-57f69885b8-p5lrn" podStartSLOduration=2.268290688 podStartE2EDuration="2.268290688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:18:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:18:31.266814319 +0000 UTC m=+1848.043128545" watchObservedRunningTime="2026-07-07 17:18:31.268290688 +0000 UTC m=+1848.044604894"
Jul 07 17:18:32.254271 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:32.254235    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f025d-predictor-57f69885b8-p5lrn" podUID="3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 07 17:18:32.613459 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:32.613438    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f587d-predictor-79b9cd4678-mpl9k"
Jul 07 17:18:32.679337 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:32.679300    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58ad52b5-f293-4951-851b-49cff8359d1d-kube-rbac-proxy-sar-config\") pod \"58ad52b5-f293-4951-851b-49cff8359d1d\" (UID: \"58ad52b5-f293-4951-851b-49cff8359d1d\") "
Jul 07 17:18:32.679500 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:32.679362    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-69668\" (UniqueName: \"kubernetes.io/projected/58ad52b5-f293-4951-851b-49cff8359d1d-kube-api-access-69668\") pod \"58ad52b5-f293-4951-851b-49cff8359d1d\" (UID: \"58ad52b5-f293-4951-851b-49cff8359d1d\") "
Jul 07 17:18:32.679712 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:32.679694    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58ad52b5-f293-4951-851b-49cff8359d1d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "58ad52b5-f293-4951-851b-49cff8359d1d" (UID: "58ad52b5-f293-4951-851b-49cff8359d1d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:18:32.681400 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:32.681373    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58ad52b5-f293-4951-851b-49cff8359d1d-kube-api-access-69668" (OuterVolumeSpecName: "kube-api-access-69668") pod "58ad52b5-f293-4951-851b-49cff8359d1d" (UID: "58ad52b5-f293-4951-851b-49cff8359d1d"). InnerVolumeSpecName "kube-api-access-69668". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:18:32.780585 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:32.780555    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58ad52b5-f293-4951-851b-49cff8359d1d-proxy-tls\") pod \"58ad52b5-f293-4951-851b-49cff8359d1d\" (UID: \"58ad52b5-f293-4951-851b-49cff8359d1d\") "
Jul 07 17:18:32.780766 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:32.780692    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58ad52b5-f293-4951-851b-49cff8359d1d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 17:18:32.780766 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:32.780702    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69668\" (UniqueName: \"kubernetes.io/projected/58ad52b5-f293-4951-851b-49cff8359d1d-kube-api-access-69668\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 17:18:32.782513 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:32.782493    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58ad52b5-f293-4951-851b-49cff8359d1d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "58ad52b5-f293-4951-851b-49cff8359d1d" (UID: "58ad52b5-f293-4951-851b-49cff8359d1d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 17:18:32.881358 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:32.881323    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58ad52b5-f293-4951-851b-49cff8359d1d-proxy-tls\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 17:18:33.110031 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:33.109999    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1a68d-predictor-fd8c98474-wxjjp" podUID="1a936a6b-2c12-4ab9-987f-5dbe899ce9ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 07 17:18:33.259477 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:33.259445    2578 generic.go:356] "Generic (PLEG): container finished" podID="58ad52b5-f293-4951-851b-49cff8359d1d" containerID="7cdc2cf99ab77be158ec6b879eb52d4a6fffe21db774a6c656e9ceae16f032a2" exitCode=0
Jul 07 17:18:33.259877 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:33.259496    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f587d-predictor-79b9cd4678-mpl9k"
Jul 07 17:18:33.259877 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:33.259533    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f587d-predictor-79b9cd4678-mpl9k" event={"ID":"58ad52b5-f293-4951-851b-49cff8359d1d","Type":"ContainerDied","Data":"7cdc2cf99ab77be158ec6b879eb52d4a6fffe21db774a6c656e9ceae16f032a2"}
Jul 07 17:18:33.259877 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:33.259602    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f587d-predictor-79b9cd4678-mpl9k" event={"ID":"58ad52b5-f293-4951-851b-49cff8359d1d","Type":"ContainerDied","Data":"182a6946ae92559c6a71287922e3450a713c7e33c4cf81da0bb03fc351c4e702"}
Jul 07 17:18:33.259877 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:33.259639    2578 scope.go:117] "RemoveContainer" containerID="85730c195f8b8d77834655f64a49b49878f0a24196de75ed5c2b6458408c613b"
Jul 07 17:18:33.267645 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:33.267629    2578 scope.go:117] "RemoveContainer" containerID="7cdc2cf99ab77be158ec6b879eb52d4a6fffe21db774a6c656e9ceae16f032a2"
Jul 07 17:18:33.274585 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:33.274563    2578 scope.go:117] "RemoveContainer" containerID="85730c195f8b8d77834655f64a49b49878f0a24196de75ed5c2b6458408c613b"
Jul 07 17:18:33.274838 ip-10-0-136-217 kubenswrapper[2578]: E0707 17:18:33.274819    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85730c195f8b8d77834655f64a49b49878f0a24196de75ed5c2b6458408c613b\": container with ID starting with 85730c195f8b8d77834655f64a49b49878f0a24196de75ed5c2b6458408c613b not found: ID does not exist" containerID="85730c195f8b8d77834655f64a49b49878f0a24196de75ed5c2b6458408c613b"
Jul 07 17:18:33.274914 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:33.274847    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85730c195f8b8d77834655f64a49b49878f0a24196de75ed5c2b6458408c613b"} err="failed to get container status \"85730c195f8b8d77834655f64a49b49878f0a24196de75ed5c2b6458408c613b\": rpc error: code = NotFound desc = could not find container \"85730c195f8b8d77834655f64a49b49878f0a24196de75ed5c2b6458408c613b\": container with ID starting with 85730c195f8b8d77834655f64a49b49878f0a24196de75ed5c2b6458408c613b not found: ID does not exist"
Jul 07 17:18:33.274914 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:33.274863    2578 scope.go:117] "RemoveContainer" containerID="7cdc2cf99ab77be158ec6b879eb52d4a6fffe21db774a6c656e9ceae16f032a2"
Jul 07 17:18:33.275094 ip-10-0-136-217 kubenswrapper[2578]: E0707 17:18:33.275076    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cdc2cf99ab77be158ec6b879eb52d4a6fffe21db774a6c656e9ceae16f032a2\": container with ID starting with 7cdc2cf99ab77be158ec6b879eb52d4a6fffe21db774a6c656e9ceae16f032a2 not found: ID does not exist" containerID="7cdc2cf99ab77be158ec6b879eb52d4a6fffe21db774a6c656e9ceae16f032a2"
Jul 07 17:18:33.275133 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:33.275100    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cdc2cf99ab77be158ec6b879eb52d4a6fffe21db774a6c656e9ceae16f032a2"} err="failed to get container status \"7cdc2cf99ab77be158ec6b879eb52d4a6fffe21db774a6c656e9ceae16f032a2\": rpc error: code = NotFound desc = could not find container \"7cdc2cf99ab77be158ec6b879eb52d4a6fffe21db774a6c656e9ceae16f032a2\": container with ID starting with 7cdc2cf99ab77be158ec6b879eb52d4a6fffe21db774a6c656e9ceae16f032a2 not found: ID does not exist"
Jul 07 17:18:33.281588 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:33.281568    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f587d-predictor-79b9cd4678-mpl9k"]
Jul 07 17:18:33.284611 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:33.284592    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f587d-predictor-79b9cd4678-mpl9k"]
Jul 07 17:18:33.724880 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:33.724850    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58ad52b5-f293-4951-851b-49cff8359d1d" path="/var/lib/kubelet/pods/58ad52b5-f293-4951-851b-49cff8359d1d/volumes"
Jul 07 17:18:36.208491 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:36.208457    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-f587d-68bc64bffc-2zmhp" podUID="81d084b1-9833-469b-a1b0-dde7cb6f564c" containerName="splitter-graph-f587d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:18:37.258792 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:37.258763    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-f025d-predictor-57f69885b8-p5lrn"
Jul 07 17:18:37.259169 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:37.259146    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f025d-predictor-57f69885b8-p5lrn" podUID="3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 07 17:18:41.208870 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:41.208828    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-f587d-68bc64bffc-2zmhp" podUID="81d084b1-9833-469b-a1b0-dde7cb6f564c" containerName="splitter-graph-f587d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:18:41.209329 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:41.208930    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-f587d-68bc64bffc-2zmhp"
Jul 07 17:18:43.110661 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:43.110634    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-1a68d-predictor-fd8c98474-wxjjp"
Jul 07 17:18:46.212695 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:46.212660    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-f587d-68bc64bffc-2zmhp" podUID="81d084b1-9833-469b-a1b0-dde7cb6f564c" containerName="splitter-graph-f587d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:18:47.259987 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:47.259949    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f025d-predictor-57f69885b8-p5lrn" podUID="3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 07 17:18:51.208549 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:51.208513    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-f587d-68bc64bffc-2zmhp" podUID="81d084b1-9833-469b-a1b0-dde7cb6f564c" containerName="splitter-graph-f587d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:18:55.043449 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:55.043415    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-1a68d-576ff8fc4c-zhbw6"]
Jul 07 17:18:55.043844 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:55.043815    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58ad52b5-f293-4951-851b-49cff8359d1d" containerName="kube-rbac-proxy"
Jul 07 17:18:55.043844 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:55.043828    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="58ad52b5-f293-4951-851b-49cff8359d1d" containerName="kube-rbac-proxy"
Jul 07 17:18:55.043929 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:55.043847    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58ad52b5-f293-4951-851b-49cff8359d1d" containerName="kserve-container"
Jul 07 17:18:55.043929 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:55.043852    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="58ad52b5-f293-4951-851b-49cff8359d1d" containerName="kserve-container"
Jul 07 17:18:55.043929 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:55.043911    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="58ad52b5-f293-4951-851b-49cff8359d1d" containerName="kserve-container"
Jul 07 17:18:55.043929 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:55.043919    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="58ad52b5-f293-4951-851b-49cff8359d1d" containerName="kube-rbac-proxy"
Jul 07 17:18:55.047053 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:55.047038    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-1a68d-576ff8fc4c-zhbw6"
Jul 07 17:18:55.049649 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:55.049623    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-1a68d-serving-cert\""
Jul 07 17:18:55.049783 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:55.049632    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-1a68d-kube-rbac-proxy-sar-config\""
Jul 07 17:18:55.055412 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:55.055392    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-1a68d-576ff8fc4c-zhbw6"]
Jul 07 17:18:55.161483 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:55.161450    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1fc67c4-4810-47c3-a6da-38457d9eb1b3-openshift-service-ca-bundle\") pod \"switch-graph-1a68d-576ff8fc4c-zhbw6\" (UID: \"f1fc67c4-4810-47c3-a6da-38457d9eb1b3\") " pod="kserve-ci-e2e-test/switch-graph-1a68d-576ff8fc4c-zhbw6"
Jul 07 17:18:55.161668 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:55.161490    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1fc67c4-4810-47c3-a6da-38457d9eb1b3-proxy-tls\") pod \"switch-graph-1a68d-576ff8fc4c-zhbw6\" (UID: \"f1fc67c4-4810-47c3-a6da-38457d9eb1b3\") " pod="kserve-ci-e2e-test/switch-graph-1a68d-576ff8fc4c-zhbw6"
Jul 07 17:18:55.262221 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:55.262191    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1fc67c4-4810-47c3-a6da-38457d9eb1b3-openshift-service-ca-bundle\") pod \"switch-graph-1a68d-576ff8fc4c-zhbw6\" (UID: \"f1fc67c4-4810-47c3-a6da-38457d9eb1b3\") " pod="kserve-ci-e2e-test/switch-graph-1a68d-576ff8fc4c-zhbw6"
Jul 07 17:18:55.262361 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:55.262227    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1fc67c4-4810-47c3-a6da-38457d9eb1b3-proxy-tls\") pod \"switch-graph-1a68d-576ff8fc4c-zhbw6\" (UID: \"f1fc67c4-4810-47c3-a6da-38457d9eb1b3\") " pod="kserve-ci-e2e-test/switch-graph-1a68d-576ff8fc4c-zhbw6"
Jul 07 17:18:55.262361 ip-10-0-136-217 kubenswrapper[2578]: E0707 17:18:55.262343    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-1a68d-serving-cert: secret "switch-graph-1a68d-serving-cert" not found
Jul 07 17:18:55.262441 ip-10-0-136-217 kubenswrapper[2578]: E0707 17:18:55.262421    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1fc67c4-4810-47c3-a6da-38457d9eb1b3-proxy-tls podName:f1fc67c4-4810-47c3-a6da-38457d9eb1b3 nodeName:}" failed. No retries permitted until 2026-07-07 17:18:55.762388242 +0000 UTC m=+1872.538702431 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f1fc67c4-4810-47c3-a6da-38457d9eb1b3-proxy-tls") pod "switch-graph-1a68d-576ff8fc4c-zhbw6" (UID: "f1fc67c4-4810-47c3-a6da-38457d9eb1b3") : secret "switch-graph-1a68d-serving-cert" not found
Jul 07 17:18:55.262915 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:55.262898    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1fc67c4-4810-47c3-a6da-38457d9eb1b3-openshift-service-ca-bundle\") pod \"switch-graph-1a68d-576ff8fc4c-zhbw6\" (UID: \"f1fc67c4-4810-47c3-a6da-38457d9eb1b3\") " pod="kserve-ci-e2e-test/switch-graph-1a68d-576ff8fc4c-zhbw6"
Jul 07 17:18:55.766106 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:55.766063    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1fc67c4-4810-47c3-a6da-38457d9eb1b3-proxy-tls\") pod \"switch-graph-1a68d-576ff8fc4c-zhbw6\" (UID: \"f1fc67c4-4810-47c3-a6da-38457d9eb1b3\") " pod="kserve-ci-e2e-test/switch-graph-1a68d-576ff8fc4c-zhbw6"
Jul 07 17:18:55.768378 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:55.768348    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1fc67c4-4810-47c3-a6da-38457d9eb1b3-proxy-tls\") pod \"switch-graph-1a68d-576ff8fc4c-zhbw6\" (UID: \"f1fc67c4-4810-47c3-a6da-38457d9eb1b3\") " pod="kserve-ci-e2e-test/switch-graph-1a68d-576ff8fc4c-zhbw6"
Jul 07 17:18:55.958169 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:55.958138    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-1a68d-576ff8fc4c-zhbw6"
Jul 07 17:18:56.103281 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:56.103259    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-1a68d-576ff8fc4c-zhbw6"]
Jul 07 17:18:56.105355 ip-10-0-136-217 kubenswrapper[2578]: W0707 17:18:56.105328    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1fc67c4_4810_47c3_a6da_38457d9eb1b3.slice/crio-1025ee6870b5107c4f72b4cbdda937dface94420e10e8501577d29912b81d79b WatchSource:0}: Error finding container 1025ee6870b5107c4f72b4cbdda937dface94420e10e8501577d29912b81d79b: Status 404 returned error can't find the container with id 1025ee6870b5107c4f72b4cbdda937dface94420e10e8501577d29912b81d79b
Jul 07 17:18:56.208809 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:56.208777    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-f587d-68bc64bffc-2zmhp" podUID="81d084b1-9833-469b-a1b0-dde7cb6f564c" containerName="splitter-graph-f587d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:18:56.350119 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:56.350039    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-1a68d-576ff8fc4c-zhbw6" event={"ID":"f1fc67c4-4810-47c3-a6da-38457d9eb1b3","Type":"ContainerStarted","Data":"551ea6812d73733271a2e19dd2a0e7868ac3fdb0223fc1aae4457cfa418301e6"}
Jul 07 17:18:56.350119 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:56.350092    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-1a68d-576ff8fc4c-zhbw6" event={"ID":"f1fc67c4-4810-47c3-a6da-38457d9eb1b3","Type":"ContainerStarted","Data":"1025ee6870b5107c4f72b4cbdda937dface94420e10e8501577d29912b81d79b"}
Jul 07 17:18:56.350287 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:56.350123    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-1a68d-576ff8fc4c-zhbw6"
Jul 07 17:18:56.366095 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:56.366049    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-1a68d-576ff8fc4c-zhbw6" podStartSLOduration=1.366035466 podStartE2EDuration="1.366035466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:18:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:18:56.365374538 +0000 UTC m=+1873.141688746" watchObservedRunningTime="2026-07-07 17:18:56.366035466 +0000 UTC m=+1873.142349673"
Jul 07 17:18:57.259595 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:57.259558    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f025d-predictor-57f69885b8-p5lrn" podUID="3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 07 17:18:59.661577 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:59.661554    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-f587d-68bc64bffc-2zmhp"
Jul 07 17:18:59.799814 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:59.799791    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81d084b1-9833-469b-a1b0-dde7cb6f564c-proxy-tls\") pod \"81d084b1-9833-469b-a1b0-dde7cb6f564c\" (UID: \"81d084b1-9833-469b-a1b0-dde7cb6f564c\") "
Jul 07 17:18:59.799989 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:59.799885    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81d084b1-9833-469b-a1b0-dde7cb6f564c-openshift-service-ca-bundle\") pod \"81d084b1-9833-469b-a1b0-dde7cb6f564c\" (UID: \"81d084b1-9833-469b-a1b0-dde7cb6f564c\") "
Jul 07 17:18:59.800229 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:59.800205    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81d084b1-9833-469b-a1b0-dde7cb6f564c-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "81d084b1-9833-469b-a1b0-dde7cb6f564c" (UID: "81d084b1-9833-469b-a1b0-dde7cb6f564c"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:18:59.801856 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:59.801835    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81d084b1-9833-469b-a1b0-dde7cb6f564c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "81d084b1-9833-469b-a1b0-dde7cb6f564c" (UID: "81d084b1-9833-469b-a1b0-dde7cb6f564c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 17:18:59.900404 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:59.900374    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81d084b1-9833-469b-a1b0-dde7cb6f564c-openshift-service-ca-bundle\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 17:18:59.900404 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:18:59.900399    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81d084b1-9833-469b-a1b0-dde7cb6f564c-proxy-tls\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 17:19:00.368336 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:19:00.368297    2578 generic.go:356] "Generic (PLEG): container finished" podID="81d084b1-9833-469b-a1b0-dde7cb6f564c" containerID="5eb9d43af48806c649ab43b29d5cd11cdca5bcd7f9f9a607c1d89dece722975f" exitCode=137
Jul 07 17:19:00.368485 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:19:00.368346    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-f587d-68bc64bffc-2zmhp"
Jul 07 17:19:00.368485 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:19:00.368349    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-f587d-68bc64bffc-2zmhp" event={"ID":"81d084b1-9833-469b-a1b0-dde7cb6f564c","Type":"ContainerDied","Data":"5eb9d43af48806c649ab43b29d5cd11cdca5bcd7f9f9a607c1d89dece722975f"}
Jul 07 17:19:00.368485 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:19:00.368401    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-f587d-68bc64bffc-2zmhp" event={"ID":"81d084b1-9833-469b-a1b0-dde7cb6f564c","Type":"ContainerDied","Data":"ff6ddbf434518c7df4fea59afd1105bf371c9b3231cdc72b07a1e17acc931a4e"}
Jul 07 17:19:00.368485 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:19:00.368430    2578 scope.go:117] "RemoveContainer" containerID="5eb9d43af48806c649ab43b29d5cd11cdca5bcd7f9f9a607c1d89dece722975f"
Jul 07 17:19:00.376622 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:19:00.376601    2578 scope.go:117] "RemoveContainer" containerID="5eb9d43af48806c649ab43b29d5cd11cdca5bcd7f9f9a607c1d89dece722975f"
Jul 07 17:19:00.376906 ip-10-0-136-217 kubenswrapper[2578]: E0707 17:19:00.376883    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5eb9d43af48806c649ab43b29d5cd11cdca5bcd7f9f9a607c1d89dece722975f\": container with ID starting with 5eb9d43af48806c649ab43b29d5cd11cdca5bcd7f9f9a607c1d89dece722975f not found: ID does not exist" containerID="5eb9d43af48806c649ab43b29d5cd11cdca5bcd7f9f9a607c1d89dece722975f"
Jul 07 17:19:00.376985 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:19:00.376915    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5eb9d43af48806c649ab43b29d5cd11cdca5bcd7f9f9a607c1d89dece722975f"} err="failed to get container status \"5eb9d43af48806c649ab43b29d5cd11cdca5bcd7f9f9a607c1d89dece722975f\": rpc error: code = NotFound desc = could not find container \"5eb9d43af48806c649ab43b29d5cd11cdca5bcd7f9f9a607c1d89dece722975f\": container with ID starting with 5eb9d43af48806c649ab43b29d5cd11cdca5bcd7f9f9a607c1d89dece722975f not found: ID does not exist"
Jul 07 17:19:00.388252 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:19:00.388231    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-f587d-68bc64bffc-2zmhp"]
Jul 07 17:19:00.392120 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:19:00.392097    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-f587d-68bc64bffc-2zmhp"]
Jul 07 17:19:01.725247 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:19:01.725214    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81d084b1-9833-469b-a1b0-dde7cb6f564c" path="/var/lib/kubelet/pods/81d084b1-9833-469b-a1b0-dde7cb6f564c/volumes"
Jul 07 17:19:02.359207 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:19:02.359183    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-1a68d-576ff8fc4c-zhbw6"
Jul 07 17:19:07.259597 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:19:07.259519    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f025d-predictor-57f69885b8-p5lrn" podUID="3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 07 17:19:17.259944 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:19:17.259910    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-f025d-predictor-57f69885b8-p5lrn"
Jul 07 17:19:29.719019 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:19:29.718989    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-f025d-5777858b95-9qjkg"]
Jul 07 17:19:29.719380 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:19:29.719325    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81d084b1-9833-469b-a1b0-dde7cb6f564c" containerName="splitter-graph-f587d"
Jul 07 17:19:29.719380 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:19:29.719335    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="81d084b1-9833-469b-a1b0-dde7cb6f564c" containerName="splitter-graph-f587d"
Jul 07 17:19:29.719456 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:19:29.719396    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="81d084b1-9833-469b-a1b0-dde7cb6f564c" containerName="splitter-graph-f587d"
Jul 07 17:19:29.722442 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:19:29.722422    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-f025d-5777858b95-9qjkg"
Jul 07 17:19:29.725276 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:19:29.725246    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-f025d-serving-cert\""
Jul 07 17:19:29.725276 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:19:29.725266    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-f025d-kube-rbac-proxy-sar-config\""
Jul 07 17:19:29.733201 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:19:29.733180    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-f025d-5777858b95-9qjkg"]
Jul 07 17:19:29.747710 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:19:29.747680    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ad88f05-8718-4453-a7f5-8b77195bc497-proxy-tls\") pod \"splitter-graph-f025d-5777858b95-9qjkg\" (UID: \"1ad88f05-8718-4453-a7f5-8b77195bc497\") " pod="kserve-ci-e2e-test/splitter-graph-f025d-5777858b95-9qjkg"
Jul 07 17:19:29.747850 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:19:29.747753    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ad88f05-8718-4453-a7f5-8b77195bc497-openshift-service-ca-bundle\") pod \"splitter-graph-f025d-5777858b95-9qjkg\" (UID: \"1ad88f05-8718-4453-a7f5-8b77195bc497\") " pod="kserve-ci-e2e-test/splitter-graph-f025d-5777858b95-9qjkg"
Jul 07 17:19:29.848292 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:19:29.848265    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ad88f05-8718-4453-a7f5-8b77195bc497-openshift-service-ca-bundle\") pod \"splitter-graph-f025d-5777858b95-9qjkg\" (UID: \"1ad88f05-8718-4453-a7f5-8b77195bc497\") " pod="kserve-ci-e2e-test/splitter-graph-f025d-5777858b95-9qjkg"
Jul 07 17:19:29.848445 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:19:29.848354    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ad88f05-8718-4453-a7f5-8b77195bc497-proxy-tls\") pod \"splitter-graph-f025d-5777858b95-9qjkg\" (UID: \"1ad88f05-8718-4453-a7f5-8b77195bc497\") " pod="kserve-ci-e2e-test/splitter-graph-f025d-5777858b95-9qjkg"
Jul 07 17:19:29.849067 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:19:29.849042    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ad88f05-8718-4453-a7f5-8b77195bc497-openshift-service-ca-bundle\") pod \"splitter-graph-f025d-5777858b95-9qjkg\" (UID: \"1ad88f05-8718-4453-a7f5-8b77195bc497\") " pod="kserve-ci-e2e-test/splitter-graph-f025d-5777858b95-9qjkg"
Jul 07 17:19:29.850725 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:19:29.850693    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ad88f05-8718-4453-a7f5-8b77195bc497-proxy-tls\") pod \"splitter-graph-f025d-5777858b95-9qjkg\" (UID: \"1ad88f05-8718-4453-a7f5-8b77195bc497\") " pod="kserve-ci-e2e-test/splitter-graph-f025d-5777858b95-9qjkg"
Jul 07 17:19:30.033901 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:19:30.033810    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-f025d-5777858b95-9qjkg"
Jul 07 17:19:30.158133 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:19:30.158108    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-f025d-5777858b95-9qjkg"]
Jul 07 17:19:30.160793 ip-10-0-136-217 kubenswrapper[2578]: W0707 17:19:30.160762    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ad88f05_8718_4453_a7f5_8b77195bc497.slice/crio-a29844296a3f7e3970badc2f6367da3e155949476ec48bc2e9bef52712c9cc91 WatchSource:0}: Error finding container a29844296a3f7e3970badc2f6367da3e155949476ec48bc2e9bef52712c9cc91: Status 404 returned error can't find the container with id a29844296a3f7e3970badc2f6367da3e155949476ec48bc2e9bef52712c9cc91
Jul 07 17:19:30.494483 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:19:30.494450    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-f025d-5777858b95-9qjkg" event={"ID":"1ad88f05-8718-4453-a7f5-8b77195bc497","Type":"ContainerStarted","Data":"f9b7223612ad4d90345a856885a76a67cf0f2c7fd4686798c1e55d6a3d59cb17"}
Jul 07 17:19:30.494664 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:19:30.494508    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-f025d-5777858b95-9qjkg" event={"ID":"1ad88f05-8718-4453-a7f5-8b77195bc497","Type":"ContainerStarted","Data":"a29844296a3f7e3970badc2f6367da3e155949476ec48bc2e9bef52712c9cc91"}
Jul 07 17:19:30.494664 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:19:30.494547    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-f025d-5777858b95-9qjkg"
Jul 07 17:19:30.513063 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:19:30.513014    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-f025d-5777858b95-9qjkg" podStartSLOduration=1.5129988060000001 podStartE2EDuration="1.512998806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:19:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:19:30.51188756 +0000 UTC m=+1907.288201787" watchObservedRunningTime="2026-07-07 17:19:30.512998806 +0000 UTC m=+1907.289313056"
Jul 07 17:19:36.505312 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:19:36.505282    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-f025d-5777858b95-9qjkg"
Jul 07 17:27:44.297890 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:27:44.297859    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-f025d-5777858b95-9qjkg"]
Jul 07 17:27:44.300512 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:27:44.298050    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-f025d-5777858b95-9qjkg" podUID="1ad88f05-8718-4453-a7f5-8b77195bc497" containerName="splitter-graph-f025d" containerID="cri-o://f9b7223612ad4d90345a856885a76a67cf0f2c7fd4686798c1e55d6a3d59cb17" gracePeriod=30
Jul 07 17:27:44.456404 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:27:44.456362    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f025d-predictor-57f69885b8-p5lrn"]
Jul 07 17:27:44.456631 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:27:44.456584    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-f025d-predictor-57f69885b8-p5lrn" podUID="3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5" containerName="kserve-container" containerID="cri-o://76b15b706a31c73c9a218e8a92e11f8fa2ae73d9b44e79a3452d0d2a69a1254f" gracePeriod=30
Jul 07 17:27:44.456701 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:27:44.456614    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-f025d-predictor-57f69885b8-p5lrn" podUID="3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5" containerName="kube-rbac-proxy" containerID="cri-o://b640c6861831c73668989807fd8b85ae1e6a7ee584f2ba104b666ed9467aabcb" gracePeriod=30
Jul 07 17:27:45.409498 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:27:45.409465    2578 generic.go:356] "Generic (PLEG): container finished" podID="3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5" containerID="b640c6861831c73668989807fd8b85ae1e6a7ee584f2ba104b666ed9467aabcb" exitCode=2
Jul 07 17:27:45.409925 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:27:45.409544    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f025d-predictor-57f69885b8-p5lrn" event={"ID":"3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5","Type":"ContainerDied","Data":"b640c6861831c73668989807fd8b85ae1e6a7ee584f2ba104b666ed9467aabcb"}
Jul 07 17:27:46.503356 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:27:46.503322    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-f025d-5777858b95-9qjkg" podUID="1ad88f05-8718-4453-a7f5-8b77195bc497" containerName="splitter-graph-f025d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:27:47.255159 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:27:47.255122    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f025d-predictor-57f69885b8-p5lrn" podUID="3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Jul 07 17:27:47.259459 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:27:47.259429    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f025d-predictor-57f69885b8-p5lrn" podUID="3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 07 17:27:47.419287 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:27:47.419251    2578 generic.go:356] "Generic (PLEG): container finished" podID="3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5" containerID="76b15b706a31c73c9a218e8a92e11f8fa2ae73d9b44e79a3452d0d2a69a1254f" exitCode=0
Jul 07 17:27:47.419448 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:27:47.419318    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f025d-predictor-57f69885b8-p5lrn" event={"ID":"3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5","Type":"ContainerDied","Data":"76b15b706a31c73c9a218e8a92e11f8fa2ae73d9b44e79a3452d0d2a69a1254f"}
Jul 07 17:27:47.498626 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:27:47.498603    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f025d-predictor-57f69885b8-p5lrn"
Jul 07 17:27:47.632842 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:27:47.632813    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5-kube-rbac-proxy-sar-config\") pod \"3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5\" (UID: \"3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5\") "
Jul 07 17:27:47.633196 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:27:47.632867    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5snj7\" (UniqueName: \"kubernetes.io/projected/3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5-kube-api-access-5snj7\") pod \"3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5\" (UID: \"3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5\") "
Jul 07 17:27:47.633196 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:27:47.632942    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5-proxy-tls\") pod \"3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5\" (UID: \"3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5\") "
Jul 07 17:27:47.633274 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:27:47.633206    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5" (UID: "3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:27:47.634972 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:27:47.634946    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5" (UID: "3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 17:27:47.635084 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:27:47.634972    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5-kube-api-access-5snj7" (OuterVolumeSpecName: "kube-api-access-5snj7") pod "3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5" (UID: "3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5"). InnerVolumeSpecName "kube-api-access-5snj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:27:47.733892 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:27:47.733872    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 17:27:47.733892 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:27:47.733893    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5snj7\" (UniqueName: \"kubernetes.io/projected/3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5-kube-api-access-5snj7\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 17:27:47.734025 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:27:47.733903    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5-proxy-tls\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 17:27:48.430482 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:27:48.430447    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f025d-predictor-57f69885b8-p5lrn" event={"ID":"3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5","Type":"ContainerDied","Data":"24338483341c01616d26347e77644bca6e6447b70ee3b92b5a6341b6717377fb"}
Jul 07 17:27:48.430654 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:27:48.430530    2578 scope.go:117] "RemoveContainer" containerID="b640c6861831c73668989807fd8b85ae1e6a7ee584f2ba104b666ed9467aabcb"
Jul 07 17:27:48.430654 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:27:48.430459    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f025d-predictor-57f69885b8-p5lrn"
Jul 07 17:27:48.438493 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:27:48.438476    2578 scope.go:117] "RemoveContainer" containerID="76b15b706a31c73c9a218e8a92e11f8fa2ae73d9b44e79a3452d0d2a69a1254f"
Jul 07 17:27:48.450151 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:27:48.450132    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f025d-predictor-57f69885b8-p5lrn"]
Jul 07 17:27:48.453901 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:27:48.453883    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f025d-predictor-57f69885b8-p5lrn"]
Jul 07 17:27:49.725970 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:27:49.725939    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5" path="/var/lib/kubelet/pods/3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5/volumes"
Jul 07 17:27:51.503637 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:27:51.503602    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-f025d-5777858b95-9qjkg" podUID="1ad88f05-8718-4453-a7f5-8b77195bc497" containerName="splitter-graph-f025d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:27:56.503519 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:27:56.503484    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-f025d-5777858b95-9qjkg" podUID="1ad88f05-8718-4453-a7f5-8b77195bc497" containerName="splitter-graph-f025d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:27:56.503912 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:27:56.503581    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-f025d-5777858b95-9qjkg"
Jul 07 17:28:01.503834 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:28:01.503798    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-f025d-5777858b95-9qjkg" podUID="1ad88f05-8718-4453-a7f5-8b77195bc497" containerName="splitter-graph-f025d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:28:06.503822 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:28:06.503722    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-f025d-5777858b95-9qjkg" podUID="1ad88f05-8718-4453-a7f5-8b77195bc497" containerName="splitter-graph-f025d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:28:11.503747 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:28:11.503701    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-f025d-5777858b95-9qjkg" podUID="1ad88f05-8718-4453-a7f5-8b77195bc497" containerName="splitter-graph-f025d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:28:14.447192 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:28:14.447168    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-f025d-5777858b95-9qjkg"
Jul 07 17:28:14.463816 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:28:14.463784    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ad88f05-8718-4453-a7f5-8b77195bc497-openshift-service-ca-bundle\") pod \"1ad88f05-8718-4453-a7f5-8b77195bc497\" (UID: \"1ad88f05-8718-4453-a7f5-8b77195bc497\") "
Jul 07 17:28:14.463987 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:28:14.463931    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ad88f05-8718-4453-a7f5-8b77195bc497-proxy-tls\") pod \"1ad88f05-8718-4453-a7f5-8b77195bc497\" (UID: \"1ad88f05-8718-4453-a7f5-8b77195bc497\") "
Jul 07 17:28:14.464326 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:28:14.464287    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ad88f05-8718-4453-a7f5-8b77195bc497-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "1ad88f05-8718-4453-a7f5-8b77195bc497" (UID: "1ad88f05-8718-4453-a7f5-8b77195bc497"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:28:14.466521 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:28:14.466487    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ad88f05-8718-4453-a7f5-8b77195bc497-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1ad88f05-8718-4453-a7f5-8b77195bc497" (UID: "1ad88f05-8718-4453-a7f5-8b77195bc497"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 17:28:14.530559 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:28:14.530521    2578 generic.go:356] "Generic (PLEG): container finished" podID="1ad88f05-8718-4453-a7f5-8b77195bc497" containerID="f9b7223612ad4d90345a856885a76a67cf0f2c7fd4686798c1e55d6a3d59cb17" exitCode=0
Jul 07 17:28:14.530714 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:28:14.530567    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-f025d-5777858b95-9qjkg"
Jul 07 17:28:14.530714 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:28:14.530610    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-f025d-5777858b95-9qjkg" event={"ID":"1ad88f05-8718-4453-a7f5-8b77195bc497","Type":"ContainerDied","Data":"f9b7223612ad4d90345a856885a76a67cf0f2c7fd4686798c1e55d6a3d59cb17"}
Jul 07 17:28:14.530714 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:28:14.530670    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-f025d-5777858b95-9qjkg" event={"ID":"1ad88f05-8718-4453-a7f5-8b77195bc497","Type":"ContainerDied","Data":"a29844296a3f7e3970badc2f6367da3e155949476ec48bc2e9bef52712c9cc91"}
Jul 07 17:28:14.530714 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:28:14.530701    2578 scope.go:117] "RemoveContainer" containerID="f9b7223612ad4d90345a856885a76a67cf0f2c7fd4686798c1e55d6a3d59cb17"
Jul 07 17:28:14.538875 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:28:14.538836    2578 scope.go:117] "RemoveContainer" containerID="f9b7223612ad4d90345a856885a76a67cf0f2c7fd4686798c1e55d6a3d59cb17"
Jul 07 17:28:14.539102 ip-10-0-136-217 kubenswrapper[2578]: E0707 17:28:14.539084    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9b7223612ad4d90345a856885a76a67cf0f2c7fd4686798c1e55d6a3d59cb17\": container with ID starting with f9b7223612ad4d90345a856885a76a67cf0f2c7fd4686798c1e55d6a3d59cb17 not found: ID does not exist" containerID="f9b7223612ad4d90345a856885a76a67cf0f2c7fd4686798c1e55d6a3d59cb17"
Jul 07 17:28:14.539166 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:28:14.539116    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9b7223612ad4d90345a856885a76a67cf0f2c7fd4686798c1e55d6a3d59cb17"} err="failed to get container status \"f9b7223612ad4d90345a856885a76a67cf0f2c7fd4686798c1e55d6a3d59cb17\": rpc error: code = NotFound desc = could not find container \"f9b7223612ad4d90345a856885a76a67cf0f2c7fd4686798c1e55d6a3d59cb17\": container with ID starting with f9b7223612ad4d90345a856885a76a67cf0f2c7fd4686798c1e55d6a3d59cb17 not found: ID does not exist"
Jul 07 17:28:14.552031 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:28:14.551985    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-f025d-5777858b95-9qjkg"]
Jul 07 17:28:14.557427 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:28:14.557408    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-f025d-5777858b95-9qjkg"]
Jul 07 17:28:14.565466 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:28:14.565451    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ad88f05-8718-4453-a7f5-8b77195bc497-proxy-tls\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 17:28:14.565524 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:28:14.565468    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ad88f05-8718-4453-a7f5-8b77195bc497-openshift-service-ca-bundle\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 17:28:15.725372 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:28:15.725341    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ad88f05-8718-4453-a7f5-8b77195bc497" path="/var/lib/kubelet/pods/1ad88f05-8718-4453-a7f5-8b77195bc497/volumes"
Jul 07 17:35:14.464272 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:14.464240    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-1a68d-576ff8fc4c-zhbw6"]
Jul 07 17:35:14.466856 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:14.464525    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-1a68d-576ff8fc4c-zhbw6" podUID="f1fc67c4-4810-47c3-a6da-38457d9eb1b3" containerName="switch-graph-1a68d" containerID="cri-o://551ea6812d73733271a2e19dd2a0e7868ac3fdb0223fc1aae4457cfa418301e6" gracePeriod=30
Jul 07 17:35:14.671235 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:14.671205    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1a68d-predictor-fd8c98474-wxjjp"]
Jul 07 17:35:14.671478 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:14.671454    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-1a68d-predictor-fd8c98474-wxjjp" podUID="1a936a6b-2c12-4ab9-987f-5dbe899ce9ec" containerName="kserve-container" containerID="cri-o://4ca7cd93aab2a13195ee842d0401146609eaf811369f99af98e9443d30f076e9" gracePeriod=30
Jul 07 17:35:14.671589 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:14.671545    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-1a68d-predictor-fd8c98474-wxjjp" podUID="1a936a6b-2c12-4ab9-987f-5dbe899ce9ec" containerName="kube-rbac-proxy" containerID="cri-o://2a2ffabd90d1f609a3825dd2540bf5c5343b8365d64b76caf2bff74575b9c741" gracePeriod=30
Jul 07 17:35:15.133426 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:15.133396    2578 generic.go:356] "Generic (PLEG): container finished" podID="1a936a6b-2c12-4ab9-987f-5dbe899ce9ec" containerID="2a2ffabd90d1f609a3825dd2540bf5c5343b8365d64b76caf2bff74575b9c741" exitCode=2
Jul 07 17:35:15.133628 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:15.133447    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1a68d-predictor-fd8c98474-wxjjp" event={"ID":"1a936a6b-2c12-4ab9-987f-5dbe899ce9ec","Type":"ContainerDied","Data":"2a2ffabd90d1f609a3825dd2540bf5c5343b8365d64b76caf2bff74575b9c741"}
Jul 07 17:35:15.436865 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:15.436787    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8qlbs/must-gather-zshbv"]
Jul 07 17:35:15.437098 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:15.437086    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5" containerName="kserve-container"
Jul 07 17:35:15.437141 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:15.437099    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5" containerName="kserve-container"
Jul 07 17:35:15.437141 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:15.437112    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ad88f05-8718-4453-a7f5-8b77195bc497" containerName="splitter-graph-f025d"
Jul 07 17:35:15.437141 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:15.437118    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ad88f05-8718-4453-a7f5-8b77195bc497" containerName="splitter-graph-f025d"
Jul 07 17:35:15.437141 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:15.437127    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5" containerName="kube-rbac-proxy"
Jul 07 17:35:15.437141 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:15.437132    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5" containerName="kube-rbac-proxy"
Jul 07 17:35:15.437294 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:15.437184    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5" containerName="kube-rbac-proxy"
Jul 07 17:35:15.437294 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:15.437192    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ad88f05-8718-4453-a7f5-8b77195bc497" containerName="splitter-graph-f025d"
Jul 07 17:35:15.437294 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:15.437201    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f29d3cc-e352-4d08-8f37-2cc2ebcb07a5" containerName="kserve-container"
Jul 07 17:35:15.440066 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:15.440051    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8qlbs/must-gather-zshbv"
Jul 07 17:35:15.443127 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:15.443103    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8qlbs\"/\"kube-root-ca.crt\""
Jul 07 17:35:15.444257 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:15.444239    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8qlbs\"/\"default-dockercfg-x8fb5\""
Jul 07 17:35:15.444391 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:15.444239    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8qlbs\"/\"openshift-service-ca.crt\""
Jul 07 17:35:15.448401 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:15.448379    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8qlbs/must-gather-zshbv"]
Jul 07 17:35:15.510884 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:15.510855    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4z2b\" (UniqueName: \"kubernetes.io/projected/1bc76da8-0297-4933-bf45-5d949ca5570e-kube-api-access-g4z2b\") pod \"must-gather-zshbv\" (UID: \"1bc76da8-0297-4933-bf45-5d949ca5570e\") " pod="openshift-must-gather-8qlbs/must-gather-zshbv"
Jul 07 17:35:15.511233 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:15.510897    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1bc76da8-0297-4933-bf45-5d949ca5570e-must-gather-output\") pod \"must-gather-zshbv\" (UID: \"1bc76da8-0297-4933-bf45-5d949ca5570e\") " pod="openshift-must-gather-8qlbs/must-gather-zshbv"
Jul 07 17:35:15.611881 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:15.611840    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4z2b\" (UniqueName: \"kubernetes.io/projected/1bc76da8-0297-4933-bf45-5d949ca5570e-kube-api-access-g4z2b\") pod \"must-gather-zshbv\" (UID: \"1bc76da8-0297-4933-bf45-5d949ca5570e\") " pod="openshift-must-gather-8qlbs/must-gather-zshbv"
Jul 07 17:35:15.611881 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:15.611890    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1bc76da8-0297-4933-bf45-5d949ca5570e-must-gather-output\") pod \"must-gather-zshbv\" (UID: \"1bc76da8-0297-4933-bf45-5d949ca5570e\") " pod="openshift-must-gather-8qlbs/must-gather-zshbv"
Jul 07 17:35:15.612175 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:15.612159    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1bc76da8-0297-4933-bf45-5d949ca5570e-must-gather-output\") pod \"must-gather-zshbv\" (UID: \"1bc76da8-0297-4933-bf45-5d949ca5570e\") " pod="openshift-must-gather-8qlbs/must-gather-zshbv"
Jul 07 17:35:15.619790 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:15.619762    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4z2b\" (UniqueName: \"kubernetes.io/projected/1bc76da8-0297-4933-bf45-5d949ca5570e-kube-api-access-g4z2b\") pod \"must-gather-zshbv\" (UID: \"1bc76da8-0297-4933-bf45-5d949ca5570e\") " pod="openshift-must-gather-8qlbs/must-gather-zshbv"
Jul 07 17:35:15.760719 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:15.760639    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8qlbs/must-gather-zshbv"
Jul 07 17:35:15.883761 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:15.883716    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8qlbs/must-gather-zshbv"]
Jul 07 17:35:15.885771 ip-10-0-136-217 kubenswrapper[2578]: W0707 17:35:15.885743    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1bc76da8_0297_4933_bf45_5d949ca5570e.slice/crio-7a490b5153f1538c160c750d041494d9cd3a10a060b99d1850926f13468fd8a0 WatchSource:0}: Error finding container 7a490b5153f1538c160c750d041494d9cd3a10a060b99d1850926f13468fd8a0: Status 404 returned error can't find the container with id 7a490b5153f1538c160c750d041494d9cd3a10a060b99d1850926f13468fd8a0
Jul 07 17:35:15.887563 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:15.887546    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 17:35:16.138475 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:16.138442    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8qlbs/must-gather-zshbv" event={"ID":"1bc76da8-0297-4933-bf45-5d949ca5570e","Type":"ContainerStarted","Data":"7a490b5153f1538c160c750d041494d9cd3a10a060b99d1850926f13468fd8a0"}
Jul 07 17:35:17.359481 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:17.359205    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-1a68d-576ff8fc4c-zhbw6" podUID="f1fc67c4-4810-47c3-a6da-38457d9eb1b3" containerName="switch-graph-1a68d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:35:18.104625 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:18.104587    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1a68d-predictor-fd8c98474-wxjjp" podUID="1a936a6b-2c12-4ab9-987f-5dbe899ce9ec" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Jul 07 17:35:20.160360 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:20.160327    2578 generic.go:356] "Generic (PLEG): container finished" podID="1a936a6b-2c12-4ab9-987f-5dbe899ce9ec" containerID="4ca7cd93aab2a13195ee842d0401146609eaf811369f99af98e9443d30f076e9" exitCode=0
Jul 07 17:35:20.160780 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:20.160407    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1a68d-predictor-fd8c98474-wxjjp" event={"ID":"1a936a6b-2c12-4ab9-987f-5dbe899ce9ec","Type":"ContainerDied","Data":"4ca7cd93aab2a13195ee842d0401146609eaf811369f99af98e9443d30f076e9"}
Jul 07 17:35:21.653311 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:21.653282    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1a68d-predictor-fd8c98474-wxjjp"
Jul 07 17:35:21.770182 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:21.770151    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a936a6b-2c12-4ab9-987f-5dbe899ce9ec-kube-rbac-proxy-sar-config\") pod \"1a936a6b-2c12-4ab9-987f-5dbe899ce9ec\" (UID: \"1a936a6b-2c12-4ab9-987f-5dbe899ce9ec\") "
Jul 07 17:35:21.770352 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:21.770204    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q5sms\" (UniqueName: \"kubernetes.io/projected/1a936a6b-2c12-4ab9-987f-5dbe899ce9ec-kube-api-access-q5sms\") pod \"1a936a6b-2c12-4ab9-987f-5dbe899ce9ec\" (UID: \"1a936a6b-2c12-4ab9-987f-5dbe899ce9ec\") "
Jul 07 17:35:21.770352 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:21.770233    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a936a6b-2c12-4ab9-987f-5dbe899ce9ec-proxy-tls\") pod \"1a936a6b-2c12-4ab9-987f-5dbe899ce9ec\" (UID: \"1a936a6b-2c12-4ab9-987f-5dbe899ce9ec\") "
Jul 07 17:35:21.770557 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:21.770525    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a936a6b-2c12-4ab9-987f-5dbe899ce9ec-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1a936a6b-2c12-4ab9-987f-5dbe899ce9ec" (UID: "1a936a6b-2c12-4ab9-987f-5dbe899ce9ec"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:35:21.772595 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:21.772568    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a936a6b-2c12-4ab9-987f-5dbe899ce9ec-kube-api-access-q5sms" (OuterVolumeSpecName: "kube-api-access-q5sms") pod "1a936a6b-2c12-4ab9-987f-5dbe899ce9ec" (UID: "1a936a6b-2c12-4ab9-987f-5dbe899ce9ec"). InnerVolumeSpecName "kube-api-access-q5sms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:35:21.772743 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:21.772592    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a936a6b-2c12-4ab9-987f-5dbe899ce9ec-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1a936a6b-2c12-4ab9-987f-5dbe899ce9ec" (UID: "1a936a6b-2c12-4ab9-987f-5dbe899ce9ec"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 17:35:21.871648 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:21.871618    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5sms\" (UniqueName: \"kubernetes.io/projected/1a936a6b-2c12-4ab9-987f-5dbe899ce9ec-kube-api-access-q5sms\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 17:35:21.871648 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:21.871647    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a936a6b-2c12-4ab9-987f-5dbe899ce9ec-proxy-tls\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 17:35:21.871846 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:21.871658    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a936a6b-2c12-4ab9-987f-5dbe899ce9ec-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 17:35:22.171303 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:22.171219    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1a68d-predictor-fd8c98474-wxjjp"
Jul 07 17:35:22.171303 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:22.171247    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1a68d-predictor-fd8c98474-wxjjp" event={"ID":"1a936a6b-2c12-4ab9-987f-5dbe899ce9ec","Type":"ContainerDied","Data":"c1dc0d8bd1b32e273158be64658868bd82e610b1cdd6c3c3c71f22ec00d43b54"}
Jul 07 17:35:22.171499 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:22.171313    2578 scope.go:117] "RemoveContainer" containerID="2a2ffabd90d1f609a3825dd2540bf5c5343b8365d64b76caf2bff74575b9c741"
Jul 07 17:35:22.173311 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:22.173285    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8qlbs/must-gather-zshbv" event={"ID":"1bc76da8-0297-4933-bf45-5d949ca5570e","Type":"ContainerStarted","Data":"9a6782b23c41191bee12a21ec7e68b6aef80fd33ecae45bf1101e57862d03f95"}
Jul 07 17:35:22.173427 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:22.173324    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8qlbs/must-gather-zshbv" event={"ID":"1bc76da8-0297-4933-bf45-5d949ca5570e","Type":"ContainerStarted","Data":"f9ce9f4c1d4dfc4d306aec92f972ef9bc733e63de97ef1a0c962c4e78f64088c"}
Jul 07 17:35:22.180887 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:22.180839    2578 scope.go:117] "RemoveContainer" containerID="4ca7cd93aab2a13195ee842d0401146609eaf811369f99af98e9443d30f076e9"
Jul 07 17:35:22.202262 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:22.202224    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8qlbs/must-gather-zshbv" podStartSLOduration=1.171356745 podStartE2EDuration="7.202209479s" totalImagesPullingTime="6.030852734s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:35:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:35:22.189697623 +0000 UTC m=+2858.966011831" watchObservedRunningTime="2026-07-07 17:35:22.202209479 +0000 UTC m=+2858.978523684"
Jul 07 17:35:22.203045 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:22.203029    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1a68d-predictor-fd8c98474-wxjjp"]
Jul 07 17:35:22.206530 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:22.206508    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1a68d-predictor-fd8c98474-wxjjp"]
Jul 07 17:35:22.360087 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:22.360047    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-1a68d-576ff8fc4c-zhbw6" podUID="f1fc67c4-4810-47c3-a6da-38457d9eb1b3" containerName="switch-graph-1a68d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:35:23.725468 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:23.725433    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a936a6b-2c12-4ab9-987f-5dbe899ce9ec" path="/var/lib/kubelet/pods/1a936a6b-2c12-4ab9-987f-5dbe899ce9ec/volumes"
Jul 07 17:35:27.358305 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:27.358263    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-1a68d-576ff8fc4c-zhbw6" podUID="f1fc67c4-4810-47c3-a6da-38457d9eb1b3" containerName="switch-graph-1a68d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:35:27.358699 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:27.358376    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-1a68d-576ff8fc4c-zhbw6"
Jul 07 17:35:30.678410 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:30.678366    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-1a68d-576ff8fc4c-zhbw6_f1fc67c4-4810-47c3-a6da-38457d9eb1b3/switch-graph-1a68d/0.log"
Jul 07 17:35:31.467560 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:31.467533    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-1a68d-576ff8fc4c-zhbw6_f1fc67c4-4810-47c3-a6da-38457d9eb1b3/switch-graph-1a68d/0.log"
Jul 07 17:35:32.259534 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:32.259510    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-1a68d-576ff8fc4c-zhbw6_f1fc67c4-4810-47c3-a6da-38457d9eb1b3/switch-graph-1a68d/0.log"
Jul 07 17:35:32.357978 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:32.357941    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-1a68d-576ff8fc4c-zhbw6" podUID="f1fc67c4-4810-47c3-a6da-38457d9eb1b3" containerName="switch-graph-1a68d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:35:33.027452 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:33.027423    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-1a68d-576ff8fc4c-zhbw6_f1fc67c4-4810-47c3-a6da-38457d9eb1b3/switch-graph-1a68d/0.log"
Jul 07 17:35:33.782445 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:33.782420    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-1a68d-576ff8fc4c-zhbw6_f1fc67c4-4810-47c3-a6da-38457d9eb1b3/switch-graph-1a68d/0.log"
Jul 07 17:35:34.569937 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:34.569911    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-1a68d-576ff8fc4c-zhbw6_f1fc67c4-4810-47c3-a6da-38457d9eb1b3/switch-graph-1a68d/0.log"
Jul 07 17:35:35.338623 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:35.338545    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-1a68d-576ff8fc4c-zhbw6_f1fc67c4-4810-47c3-a6da-38457d9eb1b3/switch-graph-1a68d/0.log"
Jul 07 17:35:36.115010 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:36.114984    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-1a68d-576ff8fc4c-zhbw6_f1fc67c4-4810-47c3-a6da-38457d9eb1b3/switch-graph-1a68d/0.log"
Jul 07 17:35:36.881145 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:36.881110    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-1a68d-576ff8fc4c-zhbw6_f1fc67c4-4810-47c3-a6da-38457d9eb1b3/switch-graph-1a68d/0.log"
Jul 07 17:35:37.359435 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:37.359387    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-1a68d-576ff8fc4c-zhbw6" podUID="f1fc67c4-4810-47c3-a6da-38457d9eb1b3" containerName="switch-graph-1a68d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:35:37.652553 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:37.652488    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-1a68d-576ff8fc4c-zhbw6_f1fc67c4-4810-47c3-a6da-38457d9eb1b3/switch-graph-1a68d/0.log"
Jul 07 17:35:38.420343 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:38.420316    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-1a68d-576ff8fc4c-zhbw6_f1fc67c4-4810-47c3-a6da-38457d9eb1b3/switch-graph-1a68d/0.log"
Jul 07 17:35:39.180933 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:39.180904    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-1a68d-576ff8fc4c-zhbw6_f1fc67c4-4810-47c3-a6da-38457d9eb1b3/switch-graph-1a68d/0.log"
Jul 07 17:35:39.954964 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:39.954922    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-1a68d-576ff8fc4c-zhbw6_f1fc67c4-4810-47c3-a6da-38457d9eb1b3/switch-graph-1a68d/0.log"
Jul 07 17:35:40.732341 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:40.732315    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-1a68d-576ff8fc4c-zhbw6_f1fc67c4-4810-47c3-a6da-38457d9eb1b3/switch-graph-1a68d/0.log"
Jul 07 17:35:42.357909 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:42.357874    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-1a68d-576ff8fc4c-zhbw6" podUID="f1fc67c4-4810-47c3-a6da-38457d9eb1b3" containerName="switch-graph-1a68d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:35:44.611571 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:44.611548    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-1a68d-576ff8fc4c-zhbw6"
Jul 07 17:35:44.662686 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:44.662658    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1fc67c4-4810-47c3-a6da-38457d9eb1b3-proxy-tls\") pod \"f1fc67c4-4810-47c3-a6da-38457d9eb1b3\" (UID: \"f1fc67c4-4810-47c3-a6da-38457d9eb1b3\") "
Jul 07 17:35:44.662858 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:44.662748    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1fc67c4-4810-47c3-a6da-38457d9eb1b3-openshift-service-ca-bundle\") pod \"f1fc67c4-4810-47c3-a6da-38457d9eb1b3\" (UID: \"f1fc67c4-4810-47c3-a6da-38457d9eb1b3\") "
Jul 07 17:35:44.663110 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:44.663085    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1fc67c4-4810-47c3-a6da-38457d9eb1b3-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "f1fc67c4-4810-47c3-a6da-38457d9eb1b3" (UID: "f1fc67c4-4810-47c3-a6da-38457d9eb1b3"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:35:44.664657 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:44.664635    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1fc67c4-4810-47c3-a6da-38457d9eb1b3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f1fc67c4-4810-47c3-a6da-38457d9eb1b3" (UID: "f1fc67c4-4810-47c3-a6da-38457d9eb1b3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 17:35:44.763991 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:44.763917    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1fc67c4-4810-47c3-a6da-38457d9eb1b3-proxy-tls\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 17:35:44.763991 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:44.763940    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1fc67c4-4810-47c3-a6da-38457d9eb1b3-openshift-service-ca-bundle\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 17:35:45.274162 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:45.274130    2578 generic.go:356] "Generic (PLEG): container finished" podID="f1fc67c4-4810-47c3-a6da-38457d9eb1b3" containerID="551ea6812d73733271a2e19dd2a0e7868ac3fdb0223fc1aae4457cfa418301e6" exitCode=0
Jul 07 17:35:45.274351 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:45.274180    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-1a68d-576ff8fc4c-zhbw6"
Jul 07 17:35:45.274351 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:45.274213    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-1a68d-576ff8fc4c-zhbw6" event={"ID":"f1fc67c4-4810-47c3-a6da-38457d9eb1b3","Type":"ContainerDied","Data":"551ea6812d73733271a2e19dd2a0e7868ac3fdb0223fc1aae4457cfa418301e6"}
Jul 07 17:35:45.274351 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:45.274263    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-1a68d-576ff8fc4c-zhbw6" event={"ID":"f1fc67c4-4810-47c3-a6da-38457d9eb1b3","Type":"ContainerDied","Data":"1025ee6870b5107c4f72b4cbdda937dface94420e10e8501577d29912b81d79b"}
Jul 07 17:35:45.274351 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:45.274285    2578 scope.go:117] "RemoveContainer" containerID="551ea6812d73733271a2e19dd2a0e7868ac3fdb0223fc1aae4457cfa418301e6"
Jul 07 17:35:45.276097 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:45.276071    2578 generic.go:356] "Generic (PLEG): container finished" podID="1bc76da8-0297-4933-bf45-5d949ca5570e" containerID="f9ce9f4c1d4dfc4d306aec92f972ef9bc733e63de97ef1a0c962c4e78f64088c" exitCode=0
Jul 07 17:35:45.276216 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:45.276120    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8qlbs/must-gather-zshbv" event={"ID":"1bc76da8-0297-4933-bf45-5d949ca5570e","Type":"ContainerDied","Data":"f9ce9f4c1d4dfc4d306aec92f972ef9bc733e63de97ef1a0c962c4e78f64088c"}
Jul 07 17:35:45.276479 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:45.276463    2578 scope.go:117] "RemoveContainer" containerID="f9ce9f4c1d4dfc4d306aec92f972ef9bc733e63de97ef1a0c962c4e78f64088c"
Jul 07 17:35:45.283489 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:45.283475    2578 scope.go:117] "RemoveContainer" containerID="551ea6812d73733271a2e19dd2a0e7868ac3fdb0223fc1aae4457cfa418301e6"
Jul 07 17:35:45.283790 ip-10-0-136-217 kubenswrapper[2578]: E0707 17:35:45.283772    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"551ea6812d73733271a2e19dd2a0e7868ac3fdb0223fc1aae4457cfa418301e6\": container with ID starting with 551ea6812d73733271a2e19dd2a0e7868ac3fdb0223fc1aae4457cfa418301e6 not found: ID does not exist" containerID="551ea6812d73733271a2e19dd2a0e7868ac3fdb0223fc1aae4457cfa418301e6"
Jul 07 17:35:45.283836 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:45.283798    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"551ea6812d73733271a2e19dd2a0e7868ac3fdb0223fc1aae4457cfa418301e6"} err="failed to get container status \"551ea6812d73733271a2e19dd2a0e7868ac3fdb0223fc1aae4457cfa418301e6\": rpc error: code = NotFound desc = could not find container \"551ea6812d73733271a2e19dd2a0e7868ac3fdb0223fc1aae4457cfa418301e6\": container with ID starting with 551ea6812d73733271a2e19dd2a0e7868ac3fdb0223fc1aae4457cfa418301e6 not found: ID does not exist"
Jul 07 17:35:45.303088 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:45.303065    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-1a68d-576ff8fc4c-zhbw6"]
Jul 07 17:35:45.308195 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:45.308176    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-1a68d-576ff8fc4c-zhbw6"]
Jul 07 17:35:45.496147 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:45.496124    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-8qlbs_must-gather-zshbv_1bc76da8-0297-4933-bf45-5d949ca5570e/gather/0.log"
Jul 07 17:35:45.725277 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:45.725250    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1fc67c4-4810-47c3-a6da-38457d9eb1b3" path="/var/lib/kubelet/pods/f1fc67c4-4810-47c3-a6da-38457d9eb1b3/volumes"
Jul 07 17:35:46.173184 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:46.173153    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ppjh7/must-gather-878h9"]
Jul 07 17:35:46.173459 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:46.173445    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a936a6b-2c12-4ab9-987f-5dbe899ce9ec" containerName="kserve-container"
Jul 07 17:35:46.173511 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:46.173463    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a936a6b-2c12-4ab9-987f-5dbe899ce9ec" containerName="kserve-container"
Jul 07 17:35:46.173511 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:46.173479    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1fc67c4-4810-47c3-a6da-38457d9eb1b3" containerName="switch-graph-1a68d"
Jul 07 17:35:46.173511 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:46.173485    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1fc67c4-4810-47c3-a6da-38457d9eb1b3" containerName="switch-graph-1a68d"
Jul 07 17:35:46.173511 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:46.173493    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a936a6b-2c12-4ab9-987f-5dbe899ce9ec" containerName="kube-rbac-proxy"
Jul 07 17:35:46.173511 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:46.173499    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a936a6b-2c12-4ab9-987f-5dbe899ce9ec" containerName="kube-rbac-proxy"
Jul 07 17:35:46.173680 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:46.173561    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1fc67c4-4810-47c3-a6da-38457d9eb1b3" containerName="switch-graph-1a68d"
Jul 07 17:35:46.173680 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:46.173571    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a936a6b-2c12-4ab9-987f-5dbe899ce9ec" containerName="kube-rbac-proxy"
Jul 07 17:35:46.173680 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:46.173578    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a936a6b-2c12-4ab9-987f-5dbe899ce9ec" containerName="kserve-container"
Jul 07 17:35:46.176322 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:46.176306    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ppjh7/must-gather-878h9"
Jul 07 17:35:46.178860 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:46.178842    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ppjh7\"/\"kube-root-ca.crt\""
Jul 07 17:35:46.179104 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:46.179084    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ppjh7\"/\"openshift-service-ca.crt\""
Jul 07 17:35:46.179227 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:46.179121    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-ppjh7\"/\"default-dockercfg-lc92d\""
Jul 07 17:35:46.187635 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:46.187611    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ppjh7/must-gather-878h9"]
Jul 07 17:35:46.274571 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:46.274534    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwvgs\" (UniqueName: \"kubernetes.io/projected/b4e7f006-c13a-40ee-88f0-a75db8025304-kube-api-access-kwvgs\") pod \"must-gather-878h9\" (UID: \"b4e7f006-c13a-40ee-88f0-a75db8025304\") " pod="openshift-must-gather-ppjh7/must-gather-878h9"
Jul 07 17:35:46.274675 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:46.274596    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b4e7f006-c13a-40ee-88f0-a75db8025304-must-gather-output\") pod \"must-gather-878h9\" (UID: \"b4e7f006-c13a-40ee-88f0-a75db8025304\") " pod="openshift-must-gather-ppjh7/must-gather-878h9"
Jul 07 17:35:46.375494 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:46.375471    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b4e7f006-c13a-40ee-88f0-a75db8025304-must-gather-output\") pod \"must-gather-878h9\" (UID: \"b4e7f006-c13a-40ee-88f0-a75db8025304\") " pod="openshift-must-gather-ppjh7/must-gather-878h9"
Jul 07 17:35:46.375594 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:46.375519    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwvgs\" (UniqueName: \"kubernetes.io/projected/b4e7f006-c13a-40ee-88f0-a75db8025304-kube-api-access-kwvgs\") pod \"must-gather-878h9\" (UID: \"b4e7f006-c13a-40ee-88f0-a75db8025304\") " pod="openshift-must-gather-ppjh7/must-gather-878h9"
Jul 07 17:35:46.375884 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:46.375867    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b4e7f006-c13a-40ee-88f0-a75db8025304-must-gather-output\") pod \"must-gather-878h9\" (UID: \"b4e7f006-c13a-40ee-88f0-a75db8025304\") " pod="openshift-must-gather-ppjh7/must-gather-878h9"
Jul 07 17:35:46.383221 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:46.383195    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwvgs\" (UniqueName: \"kubernetes.io/projected/b4e7f006-c13a-40ee-88f0-a75db8025304-kube-api-access-kwvgs\") pod \"must-gather-878h9\" (UID: \"b4e7f006-c13a-40ee-88f0-a75db8025304\") " pod="openshift-must-gather-ppjh7/must-gather-878h9"
Jul 07 17:35:46.486459 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:46.486389    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ppjh7/must-gather-878h9"
Jul 07 17:35:46.611961 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:46.611936    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ppjh7/must-gather-878h9"]
Jul 07 17:35:46.614380 ip-10-0-136-217 kubenswrapper[2578]: W0707 17:35:46.614348    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb4e7f006_c13a_40ee_88f0_a75db8025304.slice/crio-7798839ec079203f41ff205e5b3d6ce35b6b18778003008f84afba58ba87e990 WatchSource:0}: Error finding container 7798839ec079203f41ff205e5b3d6ce35b6b18778003008f84afba58ba87e990: Status 404 returned error can't find the container with id 7798839ec079203f41ff205e5b3d6ce35b6b18778003008f84afba58ba87e990
Jul 07 17:35:47.285657 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:47.285623    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ppjh7/must-gather-878h9" event={"ID":"b4e7f006-c13a-40ee-88f0-a75db8025304","Type":"ContainerStarted","Data":"7798839ec079203f41ff205e5b3d6ce35b6b18778003008f84afba58ba87e990"}
Jul 07 17:35:48.292176 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:48.292141    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ppjh7/must-gather-878h9" event={"ID":"b4e7f006-c13a-40ee-88f0-a75db8025304","Type":"ContainerStarted","Data":"79bd7ec10b77a45313fe3847e4078c8f1371fa89b820843ca3fbc729380ef289"}
Jul 07 17:35:48.292718 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:48.292690    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ppjh7/must-gather-878h9" event={"ID":"b4e7f006-c13a-40ee-88f0-a75db8025304","Type":"ContainerStarted","Data":"e2dd26a7c76968a9b0e16005e040c83d5a03d7b5d3903f4ea062ba831ddaaf53"}
Jul 07 17:35:48.309034 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:48.308991    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ppjh7/must-gather-878h9" podStartSLOduration=1.297380493 podStartE2EDuration="2.308976803s" totalImagesPullingTime="1.01159631s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:35:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:35:48.307046162 +0000 UTC m=+2885.083360369" watchObservedRunningTime="2026-07-07 17:35:48.308976803 +0000 UTC m=+2885.085291008"
Jul 07 17:35:48.945917 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:48.945889    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-h5f7f_816e0df7-c9cf-4303-a920-a90a16ee015c/global-pull-secret-syncer/0.log"
Jul 07 17:35:49.103705 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:49.103672    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-snjp4_b5aa7217-bfcc-4c09-8d6f-51bb83cce667/konnectivity-agent/0.log"
Jul 07 17:35:49.126889 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:49.126859    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-217.ec2.internal_e0242a47c8f42188a150225d6e13e270/haproxy/0.log"
Jul 07 17:35:50.945914 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:50.945862    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-8qlbs/must-gather-zshbv"]
Jul 07 17:35:50.946673 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:50.946641    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-must-gather-8qlbs/must-gather-zshbv" podUID="1bc76da8-0297-4933-bf45-5d949ca5570e" containerName="copy" containerID="cri-o://9a6782b23c41191bee12a21ec7e68b6aef80fd33ecae45bf1101e57862d03f95" gracePeriod=2
Jul 07 17:35:50.949024 ip-10-0-136-217 kubenswrapper[2578]: E0707 17:35:50.948973    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-zshbv\" is forbidden: User \"system:node:ip-10-0-136-217.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-8qlbs\": no relationship found between node 'ip-10-0-136-217.ec2.internal' and this object" podUID="1bc76da8-0297-4933-bf45-5d949ca5570e" pod="openshift-must-gather-8qlbs/must-gather-zshbv"
Jul 07 17:35:50.949254 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:50.949232    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-8qlbs/must-gather-zshbv"]
Jul 07 17:35:51.307706 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:51.307620    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-8qlbs_must-gather-zshbv_1bc76da8-0297-4933-bf45-5d949ca5570e/copy/0.log"
Jul 07 17:35:51.308376 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:51.308296    2578 generic.go:356] "Generic (PLEG): container finished" podID="1bc76da8-0297-4933-bf45-5d949ca5570e" containerID="9a6782b23c41191bee12a21ec7e68b6aef80fd33ecae45bf1101e57862d03f95" exitCode=143
Jul 07 17:35:51.356899 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:51.356635    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-8qlbs_must-gather-zshbv_1bc76da8-0297-4933-bf45-5d949ca5570e/copy/0.log"
Jul 07 17:35:51.362189 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:51.362164    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8qlbs/must-gather-zshbv"
Jul 07 17:35:51.364942 ip-10-0-136-217 kubenswrapper[2578]: E0707 17:35:51.364906    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-zshbv\" is forbidden: User \"system:node:ip-10-0-136-217.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-8qlbs\": no relationship found between node 'ip-10-0-136-217.ec2.internal' and this object" podUID="1bc76da8-0297-4933-bf45-5d949ca5570e" pod="openshift-must-gather-8qlbs/must-gather-zshbv"
Jul 07 17:35:51.440034 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:51.439995    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4z2b\" (UniqueName: \"kubernetes.io/projected/1bc76da8-0297-4933-bf45-5d949ca5570e-kube-api-access-g4z2b\") pod \"1bc76da8-0297-4933-bf45-5d949ca5570e\" (UID: \"1bc76da8-0297-4933-bf45-5d949ca5570e\") "
Jul 07 17:35:51.440222 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:51.440112    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1bc76da8-0297-4933-bf45-5d949ca5570e-must-gather-output\") pod \"1bc76da8-0297-4933-bf45-5d949ca5570e\" (UID: \"1bc76da8-0297-4933-bf45-5d949ca5570e\") "
Jul 07 17:35:51.441897 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:51.441864    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bc76da8-0297-4933-bf45-5d949ca5570e-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "1bc76da8-0297-4933-bf45-5d949ca5570e" (UID: "1bc76da8-0297-4933-bf45-5d949ca5570e"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:35:51.444532 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:51.444501    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bc76da8-0297-4933-bf45-5d949ca5570e-kube-api-access-g4z2b" (OuterVolumeSpecName: "kube-api-access-g4z2b") pod "1bc76da8-0297-4933-bf45-5d949ca5570e" (UID: "1bc76da8-0297-4933-bf45-5d949ca5570e"). InnerVolumeSpecName "kube-api-access-g4z2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:35:51.541552 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:51.541515    2578 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1bc76da8-0297-4933-bf45-5d949ca5570e-must-gather-output\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 17:35:51.541552 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:51.541555    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4z2b\" (UniqueName: \"kubernetes.io/projected/1bc76da8-0297-4933-bf45-5d949ca5570e-kube-api-access-g4z2b\") on node \"ip-10-0-136-217.ec2.internal\" DevicePath \"\""
Jul 07 17:35:51.729170 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:51.728579    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1bc76da8-0297-4933-bf45-5d949ca5570e" path="/var/lib/kubelet/pods/1bc76da8-0297-4933-bf45-5d949ca5570e/volumes"
Jul 07 17:35:52.315770 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:52.315723    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-8qlbs_must-gather-zshbv_1bc76da8-0297-4933-bf45-5d949ca5570e/copy/0.log"
Jul 07 17:35:52.316346 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:52.316212    2578 scope.go:117] "RemoveContainer" containerID="9a6782b23c41191bee12a21ec7e68b6aef80fd33ecae45bf1101e57862d03f95"
Jul 07 17:35:52.316423 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:52.316377    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8qlbs/must-gather-zshbv"
Jul 07 17:35:52.332006 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:52.331713    2578 scope.go:117] "RemoveContainer" containerID="f9ce9f4c1d4dfc4d306aec92f972ef9bc733e63de97ef1a0c962c4e78f64088c"
Jul 07 17:35:53.717982 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:53.717946    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d5653d04-ba26-49b7-9441-35df51e53991/alertmanager/0.log"
Jul 07 17:35:53.745361 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:53.745336    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d5653d04-ba26-49b7-9441-35df51e53991/config-reloader/0.log"
Jul 07 17:35:53.769418 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:53.769308    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d5653d04-ba26-49b7-9441-35df51e53991/kube-rbac-proxy-web/0.log"
Jul 07 17:35:53.795000 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:53.794969    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d5653d04-ba26-49b7-9441-35df51e53991/kube-rbac-proxy/0.log"
Jul 07 17:35:53.820110 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:53.820086    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d5653d04-ba26-49b7-9441-35df51e53991/kube-rbac-proxy-metric/0.log"
Jul 07 17:35:53.845025 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:53.844948    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d5653d04-ba26-49b7-9441-35df51e53991/prom-label-proxy/0.log"
Jul 07 17:35:53.867535 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:53.867505    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d5653d04-ba26-49b7-9441-35df51e53991/init-config-reloader/0.log"
Jul 07 17:35:54.022940 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:54.022913    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7886dfcf5f-r7svd_323eccaa-741e-495a-81a2-6368321ae1db/metrics-server/0.log"
Jul 07 17:35:54.225558 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:54.225474    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nfgmm_b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168/node-exporter/0.log"
Jul 07 17:35:54.249805 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:54.249775    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nfgmm_b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168/kube-rbac-proxy/0.log"
Jul 07 17:35:54.270190 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:54.270168    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nfgmm_b5d5b47b-4f72-4fe3-b59e-b2c3f8ab7168/init-textfile/0.log"
Jul 07 17:35:54.386043 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:54.386015    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_29004849-731a-420f-bb78-7150391d28c3/prometheus/0.log"
Jul 07 17:35:54.402378 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:54.402353    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_29004849-731a-420f-bb78-7150391d28c3/config-reloader/0.log"
Jul 07 17:35:54.423298 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:54.423249    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_29004849-731a-420f-bb78-7150391d28c3/thanos-sidecar/0.log"
Jul 07 17:35:54.447128 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:54.447104    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_29004849-731a-420f-bb78-7150391d28c3/kube-rbac-proxy-web/0.log"
Jul 07 17:35:54.471299 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:54.471276    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_29004849-731a-420f-bb78-7150391d28c3/kube-rbac-proxy/0.log"
Jul 07 17:35:54.497154 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:54.497075    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_29004849-731a-420f-bb78-7150391d28c3/kube-rbac-proxy-thanos/0.log"
Jul 07 17:35:54.520148 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:54.520115    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_29004849-731a-420f-bb78-7150391d28c3/init-config-reloader/0.log"
Jul 07 17:35:54.549618 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:54.549589    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-p7svq_20a7b5cf-65d2-4a85-88ed-d4b104cf5707/prometheus-operator/0.log"
Jul 07 17:35:54.568642 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:54.568599    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-p7svq_20a7b5cf-65d2-4a85-88ed-d4b104cf5707/kube-rbac-proxy/0.log"
Jul 07 17:35:54.599629 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:54.599547    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-d559d66d5-qtmps_c326cb78-d362-4afb-96ef-dde6c98bafc8/prometheus-operator-admission-webhook/0.log"
Jul 07 17:35:54.635253 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:54.635222    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-86d8764fcc-t7vw9_ff8de6df-aa2e-4d3e-bac0-a746f8433868/telemeter-client/0.log"
Jul 07 17:35:54.666199 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:54.666161    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-86d8764fcc-t7vw9_ff8de6df-aa2e-4d3e-bac0-a746f8433868/reload/0.log"
Jul 07 17:35:54.698385 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:54.698361    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-86d8764fcc-t7vw9_ff8de6df-aa2e-4d3e-bac0-a746f8433868/kube-rbac-proxy/0.log"
Jul 07 17:35:56.826493 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:56.826465    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7989c6f6f8-ghf7m_ab62667f-7d8d-476a-b39f-bf86daaa600c/console/0.log"
Jul 07 17:35:57.085181 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:57.085096    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ppjh7/perf-node-gather-daemonset-rfmsd"]
Jul 07 17:35:57.085869 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:57.085844    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bc76da8-0297-4933-bf45-5d949ca5570e" containerName="gather"
Jul 07 17:35:57.085869 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:57.085867    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bc76da8-0297-4933-bf45-5d949ca5570e" containerName="gather"
Jul 07 17:35:57.086395 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:57.086369    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bc76da8-0297-4933-bf45-5d949ca5570e" containerName="copy"
Jul 07 17:35:57.086534 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:57.086524    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bc76da8-0297-4933-bf45-5d949ca5570e" containerName="copy"
Jul 07 17:35:57.086836 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:57.086822    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bc76da8-0297-4933-bf45-5d949ca5570e" containerName="copy"
Jul 07 17:35:57.086953 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:57.086943    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bc76da8-0297-4933-bf45-5d949ca5570e" containerName="gather"
Jul 07 17:35:57.094399 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:57.094357    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ppjh7/perf-node-gather-daemonset-rfmsd"
Jul 07 17:35:57.097517 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:57.097452    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ppjh7/perf-node-gather-daemonset-rfmsd"]
Jul 07 17:35:57.203522 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:57.203478    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9f7ee63f-e70f-4738-b512-e7e33c99b2de-proc\") pod \"perf-node-gather-daemonset-rfmsd\" (UID: \"9f7ee63f-e70f-4738-b512-e7e33c99b2de\") " pod="openshift-must-gather-ppjh7/perf-node-gather-daemonset-rfmsd"
Jul 07 17:35:57.203794 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:57.203777    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9f7ee63f-e70f-4738-b512-e7e33c99b2de-lib-modules\") pod \"perf-node-gather-daemonset-rfmsd\" (UID: \"9f7ee63f-e70f-4738-b512-e7e33c99b2de\") " pod="openshift-must-gather-ppjh7/perf-node-gather-daemonset-rfmsd"
Jul 07 17:35:57.203987 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:57.203974    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c55n4\" (UniqueName: \"kubernetes.io/projected/9f7ee63f-e70f-4738-b512-e7e33c99b2de-kube-api-access-c55n4\") pod \"perf-node-gather-daemonset-rfmsd\" (UID: \"9f7ee63f-e70f-4738-b512-e7e33c99b2de\") " pod="openshift-must-gather-ppjh7/perf-node-gather-daemonset-rfmsd"
Jul 07 17:35:57.204163 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:57.204150    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9f7ee63f-e70f-4738-b512-e7e33c99b2de-podres\") pod \"perf-node-gather-daemonset-rfmsd\" (UID: \"9f7ee63f-e70f-4738-b512-e7e33c99b2de\") " pod="openshift-must-gather-ppjh7/perf-node-gather-daemonset-rfmsd"
Jul 07 17:35:57.204333 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:57.204290    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9f7ee63f-e70f-4738-b512-e7e33c99b2de-sys\") pod \"perf-node-gather-daemonset-rfmsd\" (UID: \"9f7ee63f-e70f-4738-b512-e7e33c99b2de\") " pod="openshift-must-gather-ppjh7/perf-node-gather-daemonset-rfmsd"
Jul 07 17:35:57.305568 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:57.305538    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9f7ee63f-e70f-4738-b512-e7e33c99b2de-podres\") pod \"perf-node-gather-daemonset-rfmsd\" (UID: \"9f7ee63f-e70f-4738-b512-e7e33c99b2de\") " pod="openshift-must-gather-ppjh7/perf-node-gather-daemonset-rfmsd"
Jul 07 17:35:57.305568 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:57.305571    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9f7ee63f-e70f-4738-b512-e7e33c99b2de-sys\") pod \"perf-node-gather-daemonset-rfmsd\" (UID: \"9f7ee63f-e70f-4738-b512-e7e33c99b2de\") " pod="openshift-must-gather-ppjh7/perf-node-gather-daemonset-rfmsd"
Jul 07 17:35:57.305830 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:57.305613    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9f7ee63f-e70f-4738-b512-e7e33c99b2de-proc\") pod \"perf-node-gather-daemonset-rfmsd\" (UID: \"9f7ee63f-e70f-4738-b512-e7e33c99b2de\") " pod="openshift-must-gather-ppjh7/perf-node-gather-daemonset-rfmsd"
Jul 07 17:35:57.305830 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:57.305628    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9f7ee63f-e70f-4738-b512-e7e33c99b2de-lib-modules\") pod \"perf-node-gather-daemonset-rfmsd\" (UID: \"9f7ee63f-e70f-4738-b512-e7e33c99b2de\") " pod="openshift-must-gather-ppjh7/perf-node-gather-daemonset-rfmsd"
Jul 07 17:35:57.305830 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:57.305655    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c55n4\" (UniqueName: \"kubernetes.io/projected/9f7ee63f-e70f-4738-b512-e7e33c99b2de-kube-api-access-c55n4\") pod \"perf-node-gather-daemonset-rfmsd\" (UID: \"9f7ee63f-e70f-4738-b512-e7e33c99b2de\") " pod="openshift-must-gather-ppjh7/perf-node-gather-daemonset-rfmsd"
Jul 07 17:35:57.305830 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:57.305707    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9f7ee63f-e70f-4738-b512-e7e33c99b2de-podres\") pod \"perf-node-gather-daemonset-rfmsd\" (UID: \"9f7ee63f-e70f-4738-b512-e7e33c99b2de\") " pod="openshift-must-gather-ppjh7/perf-node-gather-daemonset-rfmsd"
Jul 07 17:35:57.305830 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:57.305753    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9f7ee63f-e70f-4738-b512-e7e33c99b2de-proc\") pod \"perf-node-gather-daemonset-rfmsd\" (UID: \"9f7ee63f-e70f-4738-b512-e7e33c99b2de\") " pod="openshift-must-gather-ppjh7/perf-node-gather-daemonset-rfmsd"
Jul 07 17:35:57.305830 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:57.305783    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9f7ee63f-e70f-4738-b512-e7e33c99b2de-sys\") pod \"perf-node-gather-daemonset-rfmsd\" (UID: \"9f7ee63f-e70f-4738-b512-e7e33c99b2de\") " pod="openshift-must-gather-ppjh7/perf-node-gather-daemonset-rfmsd"
Jul 07 17:35:57.305830 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:57.305809    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9f7ee63f-e70f-4738-b512-e7e33c99b2de-lib-modules\") pod \"perf-node-gather-daemonset-rfmsd\" (UID: \"9f7ee63f-e70f-4738-b512-e7e33c99b2de\") " pod="openshift-must-gather-ppjh7/perf-node-gather-daemonset-rfmsd"
Jul 07 17:35:57.313712 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:57.313674    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c55n4\" (UniqueName: \"kubernetes.io/projected/9f7ee63f-e70f-4738-b512-e7e33c99b2de-kube-api-access-c55n4\") pod \"perf-node-gather-daemonset-rfmsd\" (UID: \"9f7ee63f-e70f-4738-b512-e7e33c99b2de\") " pod="openshift-must-gather-ppjh7/perf-node-gather-daemonset-rfmsd"
Jul 07 17:35:57.407475 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:57.407166    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ppjh7/perf-node-gather-daemonset-rfmsd"
Jul 07 17:35:57.574326 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:57.574301    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ppjh7/perf-node-gather-daemonset-rfmsd"]
Jul 07 17:35:57.577042 ip-10-0-136-217 kubenswrapper[2578]: W0707 17:35:57.576998    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9f7ee63f_e70f_4738_b512_e7e33c99b2de.slice/crio-143d246dbd600ef632bf96da12a2a516e4c2f2e080d7e16aad4e7510de3bdd41 WatchSource:0}: Error finding container 143d246dbd600ef632bf96da12a2a516e4c2f2e080d7e16aad4e7510de3bdd41: Status 404 returned error can't find the container with id 143d246dbd600ef632bf96da12a2a516e4c2f2e080d7e16aad4e7510de3bdd41
Jul 07 17:35:58.318752 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:58.318711    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hh2zz_7fa61b07-1978-4db6-9e3f-0dd7862906ef/dns/0.log"
Jul 07 17:35:58.339137 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:58.339110    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hh2zz_7fa61b07-1978-4db6-9e3f-0dd7862906ef/kube-rbac-proxy/0.log"
Jul 07 17:35:58.350549 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:58.350519    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ppjh7/perf-node-gather-daemonset-rfmsd" event={"ID":"9f7ee63f-e70f-4738-b512-e7e33c99b2de","Type":"ContainerStarted","Data":"3b18b505576be1e0f8808d7ab2ac62c4295d1953d586f747d319358060d2d128"}
Jul 07 17:35:58.350704 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:58.350576    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ppjh7/perf-node-gather-daemonset-rfmsd" event={"ID":"9f7ee63f-e70f-4738-b512-e7e33c99b2de","Type":"ContainerStarted","Data":"143d246dbd600ef632bf96da12a2a516e4c2f2e080d7e16aad4e7510de3bdd41"}
Jul 07 17:35:58.350704 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:58.350689    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-ppjh7/perf-node-gather-daemonset-rfmsd"
Jul 07 17:35:58.365923 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:58.365872    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ppjh7/perf-node-gather-daemonset-rfmsd" podStartSLOduration=1.365858029 podStartE2EDuration="1.365858029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:35:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:35:58.364498625 +0000 UTC m=+2895.140812832" watchObservedRunningTime="2026-07-07 17:35:58.365858029 +0000 UTC m=+2895.142172235"
Jul 07 17:35:58.382776 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:58.382751    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-f9r9j_1cc86643-1ed6-4849-ad30-923acbf45ab7/dns-node-resolver/0.log"
Jul 07 17:35:58.841991 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:35:58.841964    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dbsvg_60d7869e-bef8-4020-b685-5f597d1031b9/node-ca/0.log"
Jul 07 17:36:00.026755 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:36:00.026707    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lmvlr_06abc595-8bff-454f-99e4-959c9ffcc06b/serve-healthcheck-canary/0.log"
Jul 07 17:36:00.565103 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:36:00.565078    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-t9dk7_0b1d1a91-1f92-4209-af86-f3496494d229/kube-rbac-proxy/0.log"
Jul 07 17:36:00.584680 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:36:00.584660    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-t9dk7_0b1d1a91-1f92-4209-af86-f3496494d229/exporter/0.log"
Jul 07 17:36:00.605205 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:36:00.605182    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-t9dk7_0b1d1a91-1f92-4209-af86-f3496494d229/extractor/0.log"
Jul 07 17:36:02.674650 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:36:02.674625    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-fd65d7d6-8hr2z_a40fe1eb-57b9-4726-a4b4-17353bde6339/server/0.log"
Jul 07 17:36:02.792637 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:36:02.792606    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-rpclz_65414cbc-76b2-4db6-9ceb-df027e3ae9c0/s3-init/0.log"
Jul 07 17:36:02.809530 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:36:02.809506    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-rpclz_65414cbc-76b2-4db6-9ceb-df027e3ae9c0/download-hf-model/0.log"
Jul 07 17:36:04.364077 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:36:04.364046    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-ppjh7/perf-node-gather-daemonset-rfmsd"
Jul 07 17:36:07.795504 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:36:07.795475    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-86jhq_40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca/kube-multus-additional-cni-plugins/0.log"
Jul 07 17:36:07.817359 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:36:07.817332    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-86jhq_40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca/egress-router-binary-copy/0.log"
Jul 07 17:36:07.836557 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:36:07.836535    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-86jhq_40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca/cni-plugins/0.log"
Jul 07 17:36:07.859337 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:36:07.859302    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-86jhq_40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca/bond-cni-plugin/0.log"
Jul 07 17:36:07.878640 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:36:07.878611    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-86jhq_40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca/routeoverride-cni/0.log"
Jul 07 17:36:07.899477 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:36:07.899452    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-86jhq_40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca/whereabouts-cni-bincopy/0.log"
Jul 07 17:36:07.920707 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:36:07.920673    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-86jhq_40a4b335-1bc8-4e56-91bc-2ba0d32bd0ca/whereabouts-cni/0.log"
Jul 07 17:36:08.280569 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:36:08.280541    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-p92bt_f7e6b596-4e4f-47d2-80e6-900c70180ef5/kube-multus/0.log"
Jul 07 17:36:08.370872 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:36:08.370799    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-kdcss_91e42c9a-31b0-42bf-9055-28fd68ab4ba9/network-metrics-daemon/0.log"
Jul 07 17:36:08.389601 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:36:08.389576    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-kdcss_91e42c9a-31b0-42bf-9055-28fd68ab4ba9/kube-rbac-proxy/0.log"
Jul 07 17:36:09.137969 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:36:09.137938    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bjtgg_26ecad8c-d52d-4d74-b5c1-0fdba353728e/ovn-controller/0.log"
Jul 07 17:36:09.153106 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:36:09.153082    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bjtgg_26ecad8c-d52d-4d74-b5c1-0fdba353728e/ovn-acl-logging/0.log"
Jul 07 17:36:09.173009 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:36:09.172989    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bjtgg_26ecad8c-d52d-4d74-b5c1-0fdba353728e/kube-rbac-proxy-node/0.log"
Jul 07 17:36:09.193383 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:36:09.193364    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bjtgg_26ecad8c-d52d-4d74-b5c1-0fdba353728e/kube-rbac-proxy-ovn-metrics/0.log"
Jul 07 17:36:09.215045 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:36:09.215024    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bjtgg_26ecad8c-d52d-4d74-b5c1-0fdba353728e/northd/0.log"
Jul 07 17:36:09.236006 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:36:09.235976    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bjtgg_26ecad8c-d52d-4d74-b5c1-0fdba353728e/nbdb/0.log"
Jul 07 17:36:09.255852 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:36:09.255829    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bjtgg_26ecad8c-d52d-4d74-b5c1-0fdba353728e/sbdb/0.log"
Jul 07 17:36:09.300252 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:36:09.300220    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bjtgg_26ecad8c-d52d-4d74-b5c1-0fdba353728e/ovnkube-controller/0.log"
Jul 07 17:36:10.663987 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:36:10.663963    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-wwb84_57008745-859f-49ec-9269-7671fa8164a2/network-check-target-container/0.log"
Jul 07 17:36:11.505002 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:36:11.504977    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-llvkg_0bb06482-d151-4a1f-8e49-9101ad901e7a/iptables-alerter/0.log"
Jul 07 17:36:12.117644 ip-10-0-136-217 kubenswrapper[2578]: I0707 17:36:12.117619    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-bvj2w_82568637-39b9-4e53-a44f-6c7148d449a0/tuned/0.log"