Jul 08 16:34:47.603952 ip-10-0-142-94 systemd[1]: Starting Kubernetes Kubelet...
Jul 08 16:34:48.019582 ip-10-0-142-94 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 08 16:34:48.019582 ip-10-0-142-94 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 08 16:34:48.019582 ip-10-0-142-94 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 08 16:34:48.019582 ip-10-0-142-94 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 08 16:34:48.019582 ip-10-0-142-94 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 08 16:34:48.020274 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.019588    2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 08 16:34:48.025399 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025375    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 16:34:48.025399 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025394    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 16:34:48.025399 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025399    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 16:34:48.025399 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025403    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 16:34:48.025399 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025407    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 16:34:48.025726 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025413    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 16:34:48.025726 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025418    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 16:34:48.025726 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025423    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 16:34:48.025726 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025426    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 16:34:48.025726 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025430    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 16:34:48.025726 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025433    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 16:34:48.025726 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025436    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 16:34:48.025726 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025439    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 16:34:48.025726 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025443    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 16:34:48.025726 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025447    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 16:34:48.025726 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025450    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 16:34:48.025726 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025456    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 16:34:48.025726 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025460    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 16:34:48.025726 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025465    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 16:34:48.025726 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025468    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 16:34:48.025726 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025472    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 16:34:48.025726 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025475    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 16:34:48.025726 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025479    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 16:34:48.025726 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025483    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 16:34:48.026465 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025486    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 16:34:48.026465 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025490    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 16:34:48.026465 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025495    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 16:34:48.026465 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025499    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 16:34:48.026465 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025502    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 16:34:48.026465 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025506    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 16:34:48.026465 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025510    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 16:34:48.026465 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025513    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 16:34:48.026465 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025517    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 16:34:48.026465 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025521    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 16:34:48.026465 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025525    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 16:34:48.026465 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025528    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 16:34:48.026465 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025532    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 16:34:48.026465 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025536    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 16:34:48.026465 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025541    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 16:34:48.026465 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025546    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 16:34:48.026465 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025550    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 16:34:48.026465 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025554    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 16:34:48.026465 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025558    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 16:34:48.026465 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025562    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 16:34:48.027347 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025566    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 16:34:48.027347 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025570    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 16:34:48.027347 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025576    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 16:34:48.027347 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025580    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 16:34:48.027347 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025584    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 16:34:48.027347 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025588    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 16:34:48.027347 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025593    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 16:34:48.027347 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025596    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 16:34:48.027347 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025601    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 16:34:48.027347 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025605    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 16:34:48.027347 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025609    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 16:34:48.027347 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025613    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 16:34:48.027347 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025616    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 16:34:48.027347 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025620    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 16:34:48.027347 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025624    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 16:34:48.027347 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025648    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 16:34:48.027347 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025653    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 16:34:48.027347 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025657    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 16:34:48.027347 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025661    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 16:34:48.027347 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025664    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 16:34:48.028152 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025668    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 16:34:48.028152 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025673    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 16:34:48.028152 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025677    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 16:34:48.028152 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025681    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 16:34:48.028152 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025684    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 16:34:48.028152 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025690    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 16:34:48.028152 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025694    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 16:34:48.028152 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025698    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 16:34:48.028152 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025705    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 16:34:48.028152 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025709    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 16:34:48.028152 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025713    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 16:34:48.028152 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025717    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 16:34:48.028152 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025721    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 16:34:48.028152 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025725    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 16:34:48.028152 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025729    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 16:34:48.028152 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025732    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 16:34:48.028152 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025739    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 16:34:48.028152 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025746    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 16:34:48.028152 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025750    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 16:34:48.028985 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025756    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 16:34:48.028985 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025761    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 16:34:48.028985 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025766    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 16:34:48.028985 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025771    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 16:34:48.028985 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025775    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 16:34:48.028985 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025780    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 16:34:48.028985 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025784    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 16:34:48.028985 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025787    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 16:34:48.028985 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025791    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 16:34:48.028985 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025796    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 16:34:48.028985 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025800    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 16:34:48.028985 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025805    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 16:34:48.028985 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025808    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 16:34:48.028985 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025813    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 16:34:48.028985 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025816    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 16:34:48.028985 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.025821    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 16:34:48.028985 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027001    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 16:34:48.028985 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027012    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 16:34:48.028985 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027017    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 16:34:48.028985 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027021    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 16:34:48.029824 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027026    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 16:34:48.029824 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027030    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 16:34:48.029824 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027034    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 16:34:48.029824 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027038    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 16:34:48.029824 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027041    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 16:34:48.029824 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027046    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 16:34:48.029824 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027053    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 16:34:48.029824 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027059    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 16:34:48.029824 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027064    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 16:34:48.029824 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027069    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 16:34:48.029824 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027073    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 16:34:48.029824 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027077    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 16:34:48.029824 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027081    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 16:34:48.029824 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027086    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 16:34:48.029824 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027090    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 16:34:48.029824 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027094    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 16:34:48.029824 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027098    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 16:34:48.029824 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027102    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 16:34:48.029824 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027106    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 16:34:48.029824 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027110    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 16:34:48.030607 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027115    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 16:34:48.030607 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027118    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 16:34:48.030607 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027122    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 16:34:48.030607 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027126    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 16:34:48.030607 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027130    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 16:34:48.030607 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027134    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 16:34:48.030607 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027139    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 16:34:48.030607 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027145    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 16:34:48.030607 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027149    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 16:34:48.030607 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027154    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 16:34:48.030607 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027160    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 16:34:48.030607 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027165    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 16:34:48.030607 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027170    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 16:34:48.030607 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027174    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 16:34:48.030607 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027178    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 16:34:48.030607 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027182    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 16:34:48.030607 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027186    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 16:34:48.030607 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027190    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 16:34:48.030607 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027194    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 16:34:48.031290 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027197    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 16:34:48.031290 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027202    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 16:34:48.031290 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027206    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 16:34:48.031290 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027210    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 16:34:48.031290 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027213    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 16:34:48.031290 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027218    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 16:34:48.031290 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027222    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 16:34:48.031290 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027227    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 16:34:48.031290 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027231    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 16:34:48.031290 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027235    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 16:34:48.031290 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027240    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 16:34:48.031290 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027244    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 16:34:48.031290 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027248    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 16:34:48.031290 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027251    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 16:34:48.031290 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027255    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 16:34:48.031290 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027259    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 16:34:48.031290 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027263    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 16:34:48.031290 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027267    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 16:34:48.031290 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027270    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 16:34:48.031290 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027274    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 16:34:48.031837 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027278    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 16:34:48.031837 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027286    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 16:34:48.031837 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027290    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 16:34:48.031837 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027295    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 16:34:48.031837 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027299    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 16:34:48.031837 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027303    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 16:34:48.031837 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027307    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 16:34:48.031837 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027312    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 16:34:48.031837 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027316    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 16:34:48.031837 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027320    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 16:34:48.031837 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027326    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 16:34:48.031837 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027331    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 16:34:48.031837 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027336    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 16:34:48.031837 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027341    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 16:34:48.031837 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027345    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 16:34:48.031837 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027349    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 16:34:48.031837 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027353    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 16:34:48.031837 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027358    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 16:34:48.031837 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027361    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 16:34:48.032370 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027365    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 16:34:48.032370 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027371    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 16:34:48.032370 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027376    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 16:34:48.032370 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027381    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 16:34:48.032370 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027384    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 16:34:48.032370 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027389    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 16:34:48.032370 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027393    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 16:34:48.032370 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027397    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 16:34:48.032370 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027401    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 16:34:48.032370 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027405    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 16:34:48.032370 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027409    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 16:34:48.032370 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027413    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 16:34:48.032370 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027417    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 16:34:48.032370 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027421    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 16:34:48.032370 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027425    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 16:34:48.032370 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027429    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 16:34:48.032370 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.027433    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 16:34:48.032370 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028070    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 16:34:48.032370 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028080    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 16:34:48.032370 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028084    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 16:34:48.032866 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028089    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 16:34:48.032866 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028093    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 16:34:48.032866 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028097    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 16:34:48.032866 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028102    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 16:34:48.032866 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028106    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 16:34:48.032866 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028110    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 16:34:48.032866 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028113    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 16:34:48.032866 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028117    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 16:34:48.032866 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028121    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 16:34:48.032866 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028125    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 16:34:48.032866 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028130    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 16:34:48.032866 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028134    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 16:34:48.032866 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028138    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 16:34:48.032866 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028144    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 16:34:48.032866 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028149    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 16:34:48.032866 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028153    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 16:34:48.032866 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028156    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 16:34:48.032866 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028160    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 16:34:48.032866 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028163    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 16:34:48.032866 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028168    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 16:34:48.033373 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028172    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 16:34:48.033373 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028178    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 16:34:48.033373 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028184    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 16:34:48.033373 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028189    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 16:34:48.033373 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028193    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 16:34:48.033373 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028198    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 16:34:48.033373 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028203    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 16:34:48.033373 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028207    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 16:34:48.033373 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028213    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 16:34:48.033373 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028218    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 16:34:48.033373 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028223    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 16:34:48.033373 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028227    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 16:34:48.033373 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028232    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 16:34:48.033373 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028236    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 16:34:48.033373 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028241    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 16:34:48.033373 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028245    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 16:34:48.033373 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028249    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 16:34:48.033373 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028253    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 16:34:48.033373 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028258    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 16:34:48.033837 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028262    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 16:34:48.033837 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028267    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 16:34:48.033837 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028271    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 16:34:48.033837 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028275    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 16:34:48.033837 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028279    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 16:34:48.033837 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028284    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 16:34:48.033837 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028288    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 16:34:48.033837 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028294    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 16:34:48.033837 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028299    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 16:34:48.033837 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028302    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 16:34:48.033837 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028306    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 16:34:48.033837 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028310    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 16:34:48.033837 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028314    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 16:34:48.033837 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028320    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 16:34:48.033837 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028325    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 16:34:48.033837 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028329    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 16:34:48.033837 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028333    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 16:34:48.033837 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028337    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 16:34:48.033837 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028341    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 16:34:48.034273 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028345    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 16:34:48.034273 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028349    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 16:34:48.034273 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028353    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 16:34:48.034273 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028357    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 16:34:48.034273 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028361    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 16:34:48.034273 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028365    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 16:34:48.034273 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028369    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 16:34:48.034273 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028372    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 16:34:48.034273 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028377    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 16:34:48.034273 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028381    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 16:34:48.034273 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028385    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 16:34:48.034273 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028389    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 16:34:48.034273 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028395    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 16:34:48.034273 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028402    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 16:34:48.034273 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028406    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 16:34:48.034273 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028410    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 16:34:48.034273 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028413    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 16:34:48.034273 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028417    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 16:34:48.034273 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028421    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 16:34:48.034273 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028425    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 16:34:48.034756 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028429    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 16:34:48.034756 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028434    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 16:34:48.034756 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028438    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 16:34:48.034756 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028442    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 16:34:48.034756 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028446    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 16:34:48.034756 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028450    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 16:34:48.034756 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028454    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 16:34:48.034756 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028458    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 16:34:48.034756 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028462    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 16:34:48.034756 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028466    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 16:34:48.034756 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028470    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 16:34:48.034756 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028474    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 16:34:48.034756 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028478    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 16:34:48.034756 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028482    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 16:34:48.034756 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028486    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 16:34:48.034756 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028490    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 16:34:48.034756 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028493    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 16:34:48.034756 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.028498    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 16:34:48.034756 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028594    2578 flags.go:64] FLAG: --address="0.0.0.0"
Jul 08 16:34:48.034756 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028605    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 08 16:34:48.034756 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028615    2578 flags.go:64] FLAG: --anonymous-auth="true"
Jul 08 16:34:48.035309 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028621    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 08 16:34:48.035309 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028647    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 08 16:34:48.035309 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028653    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 08 16:34:48.035309 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028661    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 08 16:34:48.035309 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028668    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 08 16:34:48.035309 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028673    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 08 16:34:48.035309 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028677    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 08 16:34:48.035309 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028683    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 08 16:34:48.035309 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028688    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 08 16:34:48.035309 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028692    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 08 16:34:48.035309 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028697    2578 flags.go:64] FLAG: --cgroup-root=""
Jul 08 16:34:48.035309 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028701    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 08 16:34:48.035309 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028705    2578 flags.go:64] FLAG: --client-ca-file=""
Jul 08 16:34:48.035309 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028710    2578 flags.go:64] FLAG: --cloud-provider="external"
Jul 08 16:34:48.035309 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028715    2578 flags.go:64] FLAG: --cluster-dns="[]"
Jul 08 16:34:48.035309 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028722    2578 flags.go:64] FLAG: --cluster-domain=""
Jul 08 16:34:48.035309 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028726    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 08 16:34:48.035309 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028732    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 08 16:34:48.035309 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028737    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 08 16:34:48.035309 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028742    2578 flags.go:64] FLAG: --container-log-max-files="5"
Jul 08 16:34:48.035309 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028747    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 08 16:34:48.035309 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028752    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 08 16:34:48.035309 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028757    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 08 16:34:48.035870 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028762    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 08 16:34:48.035870 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028767    2578 flags.go:64] FLAG: --contention-profiling="false"
Jul 08 16:34:48.035870 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028772    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 08 16:34:48.035870 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028776    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 08 16:34:48.035870 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028781    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 08 16:34:48.035870 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028786    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 08 16:34:48.035870 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028792    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 08 16:34:48.035870 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028797    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 08 16:34:48.035870 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028801    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 08 16:34:48.035870 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028806    2578 flags.go:64] FLAG: --enable-load-reader="false"
Jul 08 16:34:48.035870 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028810    2578 flags.go:64] FLAG: --enable-server="true"
Jul 08 16:34:48.035870 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028814    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 08 16:34:48.035870 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028822    2578 flags.go:64] FLAG: --event-burst="100"
Jul 08 16:34:48.035870 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028826    2578 flags.go:64] FLAG: --event-qps="50"
Jul 08 16:34:48.035870 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028831    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 08 16:34:48.035870 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028835    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 08 16:34:48.035870 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028840    2578 flags.go:64] FLAG: --eviction-hard=""
Jul 08 16:34:48.035870 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028846    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 08 16:34:48.035870 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028850    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 08 16:34:48.035870 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028854    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 08 16:34:48.035870 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028859    2578 flags.go:64] FLAG: --eviction-soft=""
Jul 08 16:34:48.035870 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028864    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 08 16:34:48.035870 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028868    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 08 16:34:48.035870 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028874    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 08 16:34:48.035870 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028879    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 08 16:34:48.036504 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028883    2578 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 08 16:34:48.036504 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028889    2578 flags.go:64] FLAG: --fail-swap-on="true"
Jul 08 16:34:48.036504 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028894    2578 flags.go:64] FLAG: --feature-gates=""
Jul 08 16:34:48.036504 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028899    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 08 16:34:48.036504 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028904    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 08 16:34:48.036504 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028908    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 08 16:34:48.036504 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028913    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 08 16:34:48.036504 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028918    2578 flags.go:64] FLAG: --healthz-port="10248"
Jul 08 16:34:48.036504 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028923    2578 flags.go:64] FLAG: --help="false"
Jul 08 16:34:48.036504 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028927    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-142-94.ec2.internal"
Jul 08 16:34:48.036504 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028932    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 08 16:34:48.036504 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028936    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 08 16:34:48.036504 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028941    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 08 16:34:48.036504 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028946    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 08 16:34:48.036504 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028951    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 08 16:34:48.036504 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028955    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 08 16:34:48.036504 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028960    2578 flags.go:64] FLAG: --image-service-endpoint=""
Jul 08 16:34:48.036504 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028964    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 08 16:34:48.036504 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028968    2578 flags.go:64] FLAG: --kube-api-burst="100"
Jul 08 16:34:48.036504 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028974    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 08 16:34:48.036504 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028979    2578 flags.go:64] FLAG: --kube-api-qps="50"
Jul 08 16:34:48.036504 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028983    2578 flags.go:64] FLAG: --kube-reserved=""
Jul 08 16:34:48.036504 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028987    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 08 16:34:48.036504 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028991    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 08 16:34:48.037085 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.028996    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 08 16:34:48.037085 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029000    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 08 16:34:48.037085 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029004    2578 flags.go:64] FLAG: --lock-file=""
Jul 08 16:34:48.037085 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029008    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 08 16:34:48.037085 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029012    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 08 16:34:48.037085 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029017    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 08 16:34:48.037085 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029026    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 08 16:34:48.037085 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029031    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 08 16:34:48.037085 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029035    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 08 16:34:48.037085 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029041    2578 flags.go:64] FLAG: --logging-format="text"
Jul 08 16:34:48.037085 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029045    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 08 16:34:48.037085 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029050    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 08 16:34:48.037085 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029054    2578 flags.go:64] FLAG: --manifest-url=""
Jul 08 16:34:48.037085 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029059    2578 flags.go:64] FLAG: --manifest-url-header=""
Jul 08 16:34:48.037085 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029066    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 08 16:34:48.037085 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029070    2578 flags.go:64] FLAG: --max-open-files="1000000"
Jul 08 16:34:48.037085 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029076    2578 flags.go:64] FLAG: --max-pods="110"
Jul 08 16:34:48.037085 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029081    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 08 16:34:48.037085 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029085    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 08 16:34:48.037085 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029089    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 08 16:34:48.037085 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029094    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 08 16:34:48.037085 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029099    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 08 16:34:48.037085 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029103    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 08 16:34:48.037085 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029107    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 08 16:34:48.037085 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029119    2578 flags.go:64] FLAG: --node-status-max-images="50"
Jul 08 16:34:48.037701 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029123    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 08 16:34:48.037701 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029128    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 08 16:34:48.037701 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029132    2578 flags.go:64] FLAG: --pod-cidr=""
Jul 08 16:34:48.037701 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029136    2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5"
Jul 08 16:34:48.037701 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029145    2578 flags.go:64] FLAG: --pod-manifest-path=""
Jul 08 16:34:48.037701 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029150    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 08 16:34:48.037701 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029154    2578 flags.go:64] FLAG: --pods-per-core="0"
Jul 08 16:34:48.037701 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029159    2578 flags.go:64] FLAG: --port="10250"
Jul 08 16:34:48.037701 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029163    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 08 16:34:48.037701 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029167    2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0069ff8d36d9bf383"
Jul 08 16:34:48.037701 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029172    2578 flags.go:64] FLAG: --qos-reserved=""
Jul 08 16:34:48.037701 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029177    2578 flags.go:64] FLAG: --read-only-port="10255"
Jul 08 16:34:48.037701 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029181    2578 flags.go:64] FLAG: --register-node="true"
Jul 08 16:34:48.037701 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029186    2578 flags.go:64] FLAG: --register-with-taints=""
Jul 08 16:34:48.037701 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029191    2578 flags.go:64] FLAG: --registry-burst="10"
Jul 08 16:34:48.037701 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029197    2578 flags.go:64] FLAG: --registry-qps="5"
Jul 08 16:34:48.037701 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029201    2578 flags.go:64] FLAG: --reserved-cpus=""
Jul 08 16:34:48.037701 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029207    2578 flags.go:64] FLAG: --reserved-memory=""
Jul 08 16:34:48.037701 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029212    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 08 16:34:48.037701 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029217    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 08 16:34:48.037701 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029222    2578 flags.go:64] FLAG: --rotate-certificates="false"
Jul 08 16:34:48.037701 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029226    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 08 16:34:48.037701 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029231    2578 flags.go:64] FLAG: --runonce="false"
Jul 08 16:34:48.037701 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029235    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 08 16:34:48.037701 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029240    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 08 16:34:48.038292 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029244    2578 flags.go:64] FLAG: --seccomp-default="false"
Jul 08 16:34:48.038292 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029248    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 08 16:34:48.038292 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029253    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 08 16:34:48.038292 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029258    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 08 16:34:48.038292 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029262    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 08 16:34:48.038292 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029266    2578 flags.go:64] FLAG: --storage-driver-password="root"
Jul 08 16:34:48.038292 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029271    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 08 16:34:48.038292 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029275    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 08 16:34:48.038292 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029279    2578 flags.go:64] FLAG: --storage-driver-user="root"
Jul 08 16:34:48.038292 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029284    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 08 16:34:48.038292 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029289    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 08 16:34:48.038292 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029293    2578 flags.go:64] FLAG: --system-cgroups=""
Jul 08 16:34:48.038292 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029297    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 08 16:34:48.038292 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029305    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 08 16:34:48.038292 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029310    2578 flags.go:64] FLAG: --tls-cert-file=""
Jul 08 16:34:48.038292 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029314    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 08 16:34:48.038292 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029320    2578 flags.go:64] FLAG: --tls-min-version=""
Jul 08 16:34:48.038292 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029325    2578 flags.go:64] FLAG: --tls-private-key-file=""
Jul 08 16:34:48.038292 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029329    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 08 16:34:48.038292 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029333    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 08 16:34:48.038292 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029338    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 08 16:34:48.038292 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029342    2578 flags.go:64] FLAG: --v="2"
Jul 08 16:34:48.038292 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029348    2578 flags.go:64] FLAG: --version="false"
Jul 08 16:34:48.038292 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029355    2578 flags.go:64] FLAG: --vmodule=""
Jul 08 16:34:48.038292 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029362    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 08 16:34:48.038911 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.029367    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 08 16:34:48.038911 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029542    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 16:34:48.038911 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029550    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 16:34:48.038911 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029555    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 16:34:48.038911 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029560    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 16:34:48.038911 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029565    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 16:34:48.038911 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029569    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 16:34:48.038911 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029574    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 16:34:48.038911 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029578    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 16:34:48.038911 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029581    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 16:34:48.038911 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029586    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 16:34:48.038911 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029590    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 16:34:48.038911 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029593    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 16:34:48.038911 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029597    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 16:34:48.038911 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029601    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 16:34:48.038911 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029605    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 16:34:48.038911 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029609    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 16:34:48.038911 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029614    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 16:34:48.038911 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029618    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 16:34:48.038911 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029622    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 16:34:48.039383 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029626    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 16:34:48.039383 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029651    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 16:34:48.039383 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029654    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 16:34:48.039383 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029658    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 16:34:48.039383 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029662    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 16:34:48.039383 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029666    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 16:34:48.039383 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029669    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 16:34:48.039383 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029674    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 16:34:48.039383 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029678    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 16:34:48.039383 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029682    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 16:34:48.039383 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029687    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 16:34:48.039383 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029694    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 16:34:48.039383 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029698    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 16:34:48.039383 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029702    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 16:34:48.039383 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029706    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 16:34:48.039383 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029710    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 16:34:48.039383 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029716    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 16:34:48.039383 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029721    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 16:34:48.039383 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029727    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 16:34:48.039872 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029731    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 16:34:48.039872 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029736    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 16:34:48.039872 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029741    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 16:34:48.039872 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029745    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 16:34:48.039872 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029750    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 16:34:48.039872 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029754    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 16:34:48.039872 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029760    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 16:34:48.039872 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029765    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 16:34:48.039872 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029770    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 16:34:48.039872 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029774    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 16:34:48.039872 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029778    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 16:34:48.039872 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029782    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 16:34:48.039872 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029786    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 16:34:48.039872 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029789    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 16:34:48.039872 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029793    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 16:34:48.039872 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029798    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 16:34:48.039872 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029802    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 16:34:48.039872 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029806    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 16:34:48.039872 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029810    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 16:34:48.040346 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029814    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 16:34:48.040346 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029818    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 16:34:48.040346 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029822    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 16:34:48.040346 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029826    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 16:34:48.040346 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029829    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 16:34:48.040346 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029834    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 16:34:48.040346 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029847    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 16:34:48.040346 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029851    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 16:34:48.040346 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029855    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 16:34:48.040346 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029859    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 16:34:48.040346 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029863    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 16:34:48.040346 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029867    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 16:34:48.040346 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029871    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 16:34:48.040346 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029875    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 16:34:48.040346 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029880    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 16:34:48.040346 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029884    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 16:34:48.040346 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029888    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 16:34:48.040346 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029891    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 16:34:48.040346 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029896    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 16:34:48.040346 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029900    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 16:34:48.040879 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029903    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 16:34:48.040879 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029908    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 16:34:48.040879 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029913    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 16:34:48.040879 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029917    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 16:34:48.040879 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029921    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 16:34:48.040879 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029925    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 16:34:48.040879 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029931    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 16:34:48.040879 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029936    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 16:34:48.040879 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029941    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 16:34:48.040879 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029946    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 16:34:48.040879 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029950    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 16:34:48.040879 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029955    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 16:34:48.040879 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029959    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 16:34:48.040879 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029963    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 16:34:48.040879 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029967    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 16:34:48.040879 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029972    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 16:34:48.040879 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029977    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 16:34:48.040879 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029983    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 16:34:48.040879 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029988    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 16:34:48.041335 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029994    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 16:34:48.041335 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.029998    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 16:34:48.041335 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.030002    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 16:34:48.041335 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.030011    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 08 16:34:48.041335 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.037049    2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 08 16:34:48.041335 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.037066    2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 08 16:34:48.041335 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.037085    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 08 16:34:48.041335 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.037091    2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 08 16:34:48.041335 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037154    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 16:34:48.041335 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037159    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 16:34:48.041335 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037163    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 16:34:48.041335 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037166    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 16:34:48.041335 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037169    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 16:34:48.041335 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037171    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 16:34:48.041335 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037174    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 16:34:48.041719 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037176    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 16:34:48.041719 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037179    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 16:34:48.041719 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037182    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 16:34:48.041719 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037184    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 16:34:48.041719 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037186    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 16:34:48.041719 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037189    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 16:34:48.041719 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037191    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 16:34:48.041719 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037194    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 16:34:48.041719 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037196    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 16:34:48.041719 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037199    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 16:34:48.041719 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037201    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 16:34:48.041719 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037203    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 16:34:48.041719 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037206    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 16:34:48.041719 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037208    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 16:34:48.041719 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037210    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 16:34:48.041719 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037213    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 16:34:48.041719 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037216    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 16:34:48.041719 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037219    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 16:34:48.041719 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037221    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 16:34:48.041719 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037224    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 16:34:48.041719 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037226    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 16:34:48.042217 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037229    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 16:34:48.042217 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037233    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 16:34:48.042217 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037236    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 16:34:48.042217 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037239    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 16:34:48.042217 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037241    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 16:34:48.042217 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037244    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 16:34:48.042217 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037246    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 16:34:48.042217 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037249    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 16:34:48.042217 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037251    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 16:34:48.042217 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037253    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 16:34:48.042217 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037256    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 16:34:48.042217 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037260    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 16:34:48.042217 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037263    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 16:34:48.042217 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037266    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 16:34:48.042217 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037268    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 16:34:48.042217 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037271    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 16:34:48.042217 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037274    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 16:34:48.042217 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037276    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 16:34:48.042217 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037278    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 16:34:48.042217 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037280    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 16:34:48.042710 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037283    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 16:34:48.042710 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037285    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 16:34:48.042710 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037287    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 16:34:48.042710 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037290    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 16:34:48.042710 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037293    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 16:34:48.042710 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037295    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 16:34:48.042710 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037298    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 16:34:48.042710 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037300    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 16:34:48.042710 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037302    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 16:34:48.042710 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037307    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 16:34:48.042710 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037310    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 16:34:48.042710 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037313    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 16:34:48.042710 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037316    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 16:34:48.042710 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037320    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 16:34:48.042710 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037323    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 16:34:48.042710 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037326    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 16:34:48.042710 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037328    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 16:34:48.042710 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037331    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 16:34:48.042710 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037333    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 16:34:48.043174 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037336    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 16:34:48.043174 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037338    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 16:34:48.043174 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037341    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 16:34:48.043174 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037343    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 16:34:48.043174 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037346    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 16:34:48.043174 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037348    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 16:34:48.043174 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037350    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 16:34:48.043174 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037353    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 16:34:48.043174 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037355    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 16:34:48.043174 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037358    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 16:34:48.043174 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037360    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 16:34:48.043174 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037363    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 16:34:48.043174 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037366    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 16:34:48.043174 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037369    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 16:34:48.043174 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037372    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 16:34:48.043174 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037375    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 16:34:48.043174 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037378    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 16:34:48.043174 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037380    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 16:34:48.043174 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037383    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 16:34:48.043606 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037386    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 16:34:48.043606 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037389    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 16:34:48.043606 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037391    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 16:34:48.043606 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037394    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 16:34:48.043606 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037397    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 16:34:48.043606 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037400    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 16:34:48.043606 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037402    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 16:34:48.043606 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037406    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 16:34:48.043606 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037409    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 16:34:48.043606 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037411    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 16:34:48.043606 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037413    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 16:34:48.043606 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037415    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 16:34:48.043606 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037418    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 16:34:48.043606 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.037422    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 08 16:34:48.043606 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037584    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 16:34:48.044045 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037591    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 16:34:48.044045 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037594    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 16:34:48.044045 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037597    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 16:34:48.044045 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037599    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 16:34:48.044045 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037602    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 16:34:48.044045 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037605    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 16:34:48.044045 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037607    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 16:34:48.044045 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037609    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 16:34:48.044045 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037612    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 16:34:48.044045 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037614    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 16:34:48.044045 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037617    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 16:34:48.044045 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037619    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 16:34:48.044045 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037622    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 16:34:48.044045 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037625    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 16:34:48.044045 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037646    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 16:34:48.044045 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037651    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 16:34:48.044045 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037655    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 16:34:48.044045 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037657    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 16:34:48.044045 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037660    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 16:34:48.044484 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037663    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 16:34:48.044484 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037666    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 16:34:48.044484 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037668    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 16:34:48.044484 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037671    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 16:34:48.044484 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037674    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 16:34:48.044484 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037676    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 16:34:48.044484 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037679    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 16:34:48.044484 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037681    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 16:34:48.044484 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037684    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 16:34:48.044484 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037686    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 16:34:48.044484 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037688    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 16:34:48.044484 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037691    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 16:34:48.044484 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037693    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 16:34:48.044484 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037695    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 16:34:48.044484 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037697    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 16:34:48.044484 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037700    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 16:34:48.044484 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037702    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 16:34:48.044484 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037705    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 16:34:48.044484 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037707    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 16:34:48.044484 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037710    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 16:34:48.044967 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037712    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 16:34:48.044967 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037715    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 16:34:48.044967 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037718    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 16:34:48.044967 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037720    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 16:34:48.044967 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037722    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 16:34:48.044967 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037725    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 16:34:48.044967 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037727    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 16:34:48.044967 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037729    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 16:34:48.044967 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037732    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 16:34:48.044967 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037734    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 16:34:48.044967 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037737    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 16:34:48.044967 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037739    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 16:34:48.044967 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037741    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 16:34:48.044967 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037744    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 16:34:48.044967 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037746    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 16:34:48.044967 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037748    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 16:34:48.044967 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037758    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 16:34:48.044967 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037761    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 16:34:48.044967 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037764    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 16:34:48.044967 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037766    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 16:34:48.045431 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037769    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 16:34:48.045431 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037771    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 16:34:48.045431 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037773    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 16:34:48.045431 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037776    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 16:34:48.045431 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037778    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 16:34:48.045431 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037780    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 16:34:48.045431 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037782    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 16:34:48.045431 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037785    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 16:34:48.045431 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037787    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 16:34:48.045431 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037790    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 16:34:48.045431 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037793    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 16:34:48.045431 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037796    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 16:34:48.045431 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037798    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 16:34:48.045431 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037800    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 16:34:48.045431 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037803    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 16:34:48.045431 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037805    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 16:34:48.045431 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037807    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 16:34:48.045431 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037810    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 16:34:48.045431 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037812    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 16:34:48.045939 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037815    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 16:34:48.045939 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037817    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 16:34:48.045939 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037819    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 16:34:48.045939 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037821    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 16:34:48.045939 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037824    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 16:34:48.045939 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037826    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 16:34:48.045939 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037828    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 16:34:48.045939 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037831    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 16:34:48.045939 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037833    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 16:34:48.045939 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037835    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 16:34:48.045939 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037841    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 16:34:48.045939 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037844    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 16:34:48.045939 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037846    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 16:34:48.045939 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037849    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 16:34:48.045939 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037851    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 16:34:48.045939 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037853    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 16:34:48.045939 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037856    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 16:34:48.045939 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037858    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 16:34:48.045939 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037860    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 16:34:48.045939 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:48.037863    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 16:34:48.046702 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.037867    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 08 16:34:48.046702 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.037974    2578 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 08 16:34:48.046702 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.039968    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 08 16:34:48.046702 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.041762    2578 server.go:1019] "Starting client certificate rotation"
Jul 08 16:34:48.046702 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.041893    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 08 16:34:48.046702 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.041947    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 08 16:34:48.046702 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.044372    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 08 16:34:48.046702 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.046065    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 08 16:34:48.059671 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.059647    2578 log.go:25] "Validated CRI v1 runtime API"
Jul 08 16:34:48.061099 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.061071    2578 log.go:25] "Validated CRI v1 image API"
Jul 08 16:34:48.062656 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.062618    2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 08 16:34:48.065288 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.065268    2578 fs.go:135] Filesystem UUIDs: map[01db7493-02dc-41ce-a530-67a48af4ea0d:/dev/nvme0n1p3 0c941c02-2160-42af-8722-0675065b3246:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Jul 08 16:34:48.065349 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.065288    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 08 16:34:48.070803 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.070784    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 08 16:34:48.071579 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.071472    2578 manager.go:216] Machine: {Timestamp:2026-07-08 16:34:48.069211155 +0000 UTC m=+0.340660687 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3182852 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f8d79ef921584146ac83df788966b SystemUUID:ec2f8d79-ef92-1584-146a-c83df788966b BootID:1edc94c7-e087-4da0-bb76-d7cdbac3cf02 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:61:ae:7c:5d:4f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:61:ae:7c:5d:4f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:de:2b:7b:2b:c5:84 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 08 16:34:48.071671 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.071582    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 08 16:34:48.071703 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.071687    2578 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 08 16:34:48.072824 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.072799    2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 08 16:34:48.072963 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.072827    2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-94.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 08 16:34:48.075115 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.075102    2578 topology_manager.go:138] "Creating topology manager with none policy"
Jul 08 16:34:48.075163 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.075116    2578 container_manager_linux.go:309] "Creating device plugin manager"
Jul 08 16:34:48.075163 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.075125    2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 08 16:34:48.075730 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.075718    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 08 16:34:48.077061 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.077050    2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 08 16:34:48.080090 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.080076    2578 state_mem.go:36] "Initialized new in-memory state store"
Jul 08 16:34:48.080218 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.080208    2578 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 08 16:34:48.080260 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.080244    2578 kubelet.go:486] "Attempting to sync node with API server"
Jul 08 16:34:48.080293 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.080261    2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 08 16:34:48.080293 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.080274    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 08 16:34:48.080293 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.080283    2578 kubelet.go:398] "Adding apiserver pod source"
Jul 08 16:34:48.080409 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.080303    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 08 16:34:48.080789 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.080774    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 08 16:34:48.080862 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.080798    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 08 16:34:48.083028 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.083010    2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 08 16:34:48.084799 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.084780    2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 08 16:34:48.084868 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.084814    2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 08 16:34:48.085971 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.085960    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 08 16:34:48.086024 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.085975    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 08 16:34:48.086024 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.085981    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 08 16:34:48.086024 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.085987    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 08 16:34:48.086024 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.085996    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 08 16:34:48.086024 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.086001    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 08 16:34:48.086024 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.086010    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 08 16:34:48.086024 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.086015    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 08 16:34:48.086939 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.086805    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 08 16:34:48.086939 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.086899    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 08 16:34:48.086939 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.086919    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 08 16:34:48.086939 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.086930    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 08 16:34:48.088566 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.088228    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 08 16:34:48.088566 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.088251    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 08 16:34:48.091471 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:48.091181    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 08 16:34:48.091471 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:48.091181    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-94.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 08 16:34:48.092003 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.091988    2578 server.go:1268] "Started kubelet"
Jul 08 16:34:48.092092 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.092052    2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 08 16:34:48.092165 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.092115    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 08 16:34:48.092255 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.092179    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 08 16:34:48.092924 ip-10-0-142-94 systemd[1]: Started Kubernetes Kubelet.
Jul 08 16:34:48.094239 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.093824    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-t72mf"
Jul 08 16:34:48.094239 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.094115    2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 08 16:34:48.094349 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.094282    2578 server.go:310] "Adding debug handlers to kubelet server"
Jul 08 16:34:48.098689 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:48.098669    2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 08 16:34:48.099487 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.099474    2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 08 16:34:48.099556 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.099533    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 08 16:34:48.099608 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:48.099562    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-94.ec2.internal\" not found"
Jul 08 16:34:48.100228 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.100212    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 08 16:34:48.100343 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.100331    2578 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 08 16:34:48.100431 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.100422    2578 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 08 16:34:48.100685 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.100667    2578 factory.go:55] Registering systemd factory
Jul 08 16:34:48.100755 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.100689    2578 factory.go:223] Registration of the systemd container factory successfully
Jul 08 16:34:48.100986 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.100969    2578 reconstruct.go:97] "Volume reconstruction finished"
Jul 08 16:34:48.101043 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.100989    2578 reconciler.go:29] "Reconciler: start to sync state"
Jul 08 16:34:48.101157 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.101146    2578 factory.go:153] Registering CRI-O factory
Jul 08 16:34:48.101206 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.101162    2578 factory.go:223] Registration of the crio container factory successfully
Jul 08 16:34:48.101282 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.101259    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 08 16:34:48.101330 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.101298    2578 factory.go:103] Registering Raw factory
Jul 08 16:34:48.101330 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.101315    2578 manager.go:1196] Started watching for new ooms in manager
Jul 08 16:34:48.102024 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.102010    2578 manager.go:349] Starting recovery of all containers
Jul 08 16:34:48.102654 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.102607    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-t72mf"
Jul 08 16:34:48.103090 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.103070    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-94.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 08 16:34:48.103210 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:48.103189    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-94.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 08 16:34:48.103918 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:48.103893    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 08 16:34:48.104194 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:48.103271    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-94.ec2.internal.18c05db5db82b05d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-94.ec2.internal,UID:ip-10-0-142-94.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-94.ec2.internal,},FirstTimestamp:2026-07-08 16:34:48.091955293 +0000 UTC m=+0.363404824,LastTimestamp:2026-07-08 16:34:48.091955293 +0000 UTC m=+0.363404824,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-94.ec2.internal,}"
Jul 08 16:34:48.112211 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.112096    2578 manager.go:354] Recovery completed
Jul 08 16:34:48.118000 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.117984    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 16:34:48.119952 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.119933    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-94.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 16:34:48.120038 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.119962    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-94.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 16:34:48.120038 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.119972    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-94.ec2.internal" event="NodeHasSufficientPID"
Jul 08 16:34:48.120500 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.120486    2578 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 08 16:34:48.120554 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.120501    2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 08 16:34:48.120554 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.120504    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 16:34:48.120554 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.120518    2578 state_mem.go:36] "Initialized new in-memory state store"
Jul 08 16:34:48.121386 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.121365    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-94.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 16:34:48.121420 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.121403    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-94.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 16:34:48.121420 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.121417    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-94.ec2.internal" event="NodeHasSufficientPID"
Jul 08 16:34:48.121863 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.121852    2578 policy_none.go:49] "None policy: Start"
Jul 08 16:34:48.121900 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.121867    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 08 16:34:48.121900 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.121877    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 08 16:34:48.123299 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.123279    2578 policy_none.go:47] "Start"
Jul 08 16:34:48.158170 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.158151    2578 manager.go:337] "Starting Device Plugin manager"
Jul 08 16:34:48.158255 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:48.158191    2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 08 16:34:48.158255 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.158204    2578 server.go:87] "Starting device plugin registration server"
Jul 08 16:34:48.158470 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.158457    2578 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 08 16:34:48.158505 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.158475    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 08 16:34:48.158605 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.158584    2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 08 16:34:48.158722 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.158688    2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 08 16:34:48.158722 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.158698    2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 08 16:34:48.159291 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:48.159270    2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 08 16:34:48.159354 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:48.159334    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-94.ec2.internal\" not found"
Jul 08 16:34:48.259274 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.259234    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 16:34:48.260444 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.260293    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-94.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 16:34:48.260444 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.260327    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-94.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 16:34:48.260444 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.260340    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-94.ec2.internal" event="NodeHasSufficientPID"
Jul 08 16:34:48.260444 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.260368    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-94.ec2.internal"
Jul 08 16:34:48.270433 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.270358    2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-94.ec2.internal"
Jul 08 16:34:48.270433 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:48.270393    2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-94.ec2.internal\": node \"ip-10-0-142-94.ec2.internal\" not found"
Jul 08 16:34:48.278394 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.278364    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 08 16:34:48.279670 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.279652    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 08 16:34:48.279739 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.279699    2578 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 08 16:34:48.279739 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.279728    2578 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 08 16:34:48.279803 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:48.279767    2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 08 16:34:48.282078 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.282052    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 08 16:34:48.296075 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:48.296050    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-94.ec2.internal\" not found"
Jul 08 16:34:48.380908 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.380844    2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-94.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-94.ec2.internal"]
Jul 08 16:34:48.380988 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.380955    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 16:34:48.381927 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.381911    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-94.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 16:34:48.382012 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.381941    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-94.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 16:34:48.382012 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.381950    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-94.ec2.internal" event="NodeHasSufficientPID"
Jul 08 16:34:48.383163 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.383150    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 16:34:48.383283 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.383268    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-94.ec2.internal"
Jul 08 16:34:48.383338 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.383303    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 16:34:48.383904 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.383888    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-94.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 16:34:48.383998 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.383912    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-94.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 16:34:48.383998 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.383891    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-94.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 16:34:48.383998 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.383953    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-94.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 16:34:48.383998 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.383921    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-94.ec2.internal" event="NodeHasSufficientPID"
Jul 08 16:34:48.383998 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.383984    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-94.ec2.internal" event="NodeHasSufficientPID"
Jul 08 16:34:48.385071 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.385055    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-94.ec2.internal"
Jul 08 16:34:48.385152 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.385087    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 16:34:48.385797 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.385781    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-94.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 16:34:48.385889 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.385809    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-94.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 16:34:48.385889 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.385823    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-94.ec2.internal" event="NodeHasSufficientPID"
Jul 08 16:34:48.396627 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:48.396601    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-94.ec2.internal\" not found"
Jul 08 16:34:48.403232 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.403210    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/17aa1819e2ea358d71f7d6edb68bc268-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-94.ec2.internal\" (UID: \"17aa1819e2ea358d71f7d6edb68bc268\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-94.ec2.internal"
Jul 08 16:34:48.403312 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.403237    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/70024d4eaabe656e42acf5ac910ecc55-config\") pod \"kube-apiserver-proxy-ip-10-0-142-94.ec2.internal\" (UID: \"70024d4eaabe656e42acf5ac910ecc55\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-94.ec2.internal"
Jul 08 16:34:48.403312 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.403253    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/17aa1819e2ea358d71f7d6edb68bc268-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-94.ec2.internal\" (UID: \"17aa1819e2ea358d71f7d6edb68bc268\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-94.ec2.internal"
Jul 08 16:34:48.404285 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:48.404270    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-94.ec2.internal\" not found" node="ip-10-0-142-94.ec2.internal"
Jul 08 16:34:48.408801 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:48.408785    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-94.ec2.internal\" not found" node="ip-10-0-142-94.ec2.internal"
Jul 08 16:34:48.497487 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:48.497455    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-94.ec2.internal\" not found"
Jul 08 16:34:48.503841 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.503810    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/70024d4eaabe656e42acf5ac910ecc55-config\") pod \"kube-apiserver-proxy-ip-10-0-142-94.ec2.internal\" (UID: \"70024d4eaabe656e42acf5ac910ecc55\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-94.ec2.internal"
Jul 08 16:34:48.503909 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.503847    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/17aa1819e2ea358d71f7d6edb68bc268-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-94.ec2.internal\" (UID: \"17aa1819e2ea358d71f7d6edb68bc268\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-94.ec2.internal"
Jul 08 16:34:48.503909 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.503865    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/17aa1819e2ea358d71f7d6edb68bc268-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-94.ec2.internal\" (UID: \"17aa1819e2ea358d71f7d6edb68bc268\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-94.ec2.internal"
Jul 08 16:34:48.503909 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.503905    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/17aa1819e2ea358d71f7d6edb68bc268-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-94.ec2.internal\" (UID: \"17aa1819e2ea358d71f7d6edb68bc268\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-94.ec2.internal"
Jul 08 16:34:48.504012 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.503906    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/70024d4eaabe656e42acf5ac910ecc55-config\") pod \"kube-apiserver-proxy-ip-10-0-142-94.ec2.internal\" (UID: \"70024d4eaabe656e42acf5ac910ecc55\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-94.ec2.internal"
Jul 08 16:34:48.504012 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.503936    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/17aa1819e2ea358d71f7d6edb68bc268-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-94.ec2.internal\" (UID: \"17aa1819e2ea358d71f7d6edb68bc268\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-94.ec2.internal"
Jul 08 16:34:48.598553 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:48.598491    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-94.ec2.internal\" not found"
Jul 08 16:34:48.699179 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:48.699141    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-94.ec2.internal\" not found"
Jul 08 16:34:48.706514 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.706488    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-94.ec2.internal"
Jul 08 16:34:48.712294 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:48.712274    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-94.ec2.internal"
Jul 08 16:34:48.800079 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:48.800049    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-94.ec2.internal\" not found"
Jul 08 16:34:48.906676 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:48.906595    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-94.ec2.internal\" not found"
Jul 08 16:34:49.001698 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.001666    2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 08 16:34:49.041290 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.041232    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 08 16:34:49.041892 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.041418    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 08 16:34:49.041892 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.041470    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 08 16:34:49.081601 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.081569    2578 apiserver.go:52] "Watching apiserver"
Jul 08 16:34:49.097055 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.097028    2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 08 16:34:49.100957 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.099196    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-wflvt","openshift-multus/multus-additional-cni-plugins-g4stg","openshift-network-diagnostics/network-check-target-hzjsx","openshift-image-registry/node-ca-vnjdb","openshift-multus/multus-bg589","openshift-multus/network-metrics-daemon-f5jcr","openshift-network-operator/iptables-alerter-2bv8p","openshift-ovn-kubernetes/ovnkube-node-2shvk","kube-system/global-pull-secret-syncer-d9p5w","kube-system/konnectivity-agent-ht2nh","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q2rs4","openshift-cluster-node-tuning-operator/tuned-wnsct"]
Jul 08 16:34:49.100957 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.100737    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-94.ec2.internal"
Jul 08 16:34:49.100957 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.100751    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 08 16:34:49.103454 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.103436    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wflvt"
Jul 08 16:34:49.104534 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.104513    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-g4stg"
Jul 08 16:34:49.105498 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.105480    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hzjsx"
Jul 08 16:34:49.105585 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:49.105557    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hzjsx" podUID="9bbb98bc-99e1-4401-96ab-ad9aae630c48"
Jul 08 16:34:49.105759 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.105732    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 08 16:34:49.105880 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.105771    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wgbd8\""
Jul 08 16:34:49.105999 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.105986    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 08 16:34:49.106489 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.106473    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vnjdb"
Jul 08 16:34:49.106732 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.106669    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-07 16:29:48 +0000 UTC" deadline="2028-04-02 11:42:32.466006416 +0000 UTC"
Jul 08 16:34:49.106732 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.106730    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15211h7m43.359280371s"
Jul 08 16:34:49.107052 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.107033    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 08 16:34:49.107129 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.107097    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/84d1dbea-4585-46d9-b520-1329dba8df82-hosts-file\") pod \"node-resolver-wflvt\" (UID: \"84d1dbea-4585-46d9-b520-1329dba8df82\") " pod="openshift-dns/node-resolver-wflvt"
Jul 08 16:34:49.107193 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.107132    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84d1dbea-4585-46d9-b520-1329dba8df82-tmp-dir\") pod \"node-resolver-wflvt\" (UID: \"84d1dbea-4585-46d9-b520-1329dba8df82\") " pod="openshift-dns/node-resolver-wflvt"
Jul 08 16:34:49.107193 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.107153    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx5x6\" (UniqueName: \"kubernetes.io/projected/84d1dbea-4585-46d9-b520-1329dba8df82-kube-api-access-jx5x6\") pod \"node-resolver-wflvt\" (UID: \"84d1dbea-4585-46d9-b520-1329dba8df82\") " pod="openshift-dns/node-resolver-wflvt"
Jul 08 16:34:49.107193 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.107186    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 08 16:34:49.107329 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.107233    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 08 16:34:49.107329 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.107281    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 08 16:34:49.107427 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.107327    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 08 16:34:49.107427 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.107339    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-lc84l\""
Jul 08 16:34:49.107563 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.107549    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bg589"
Jul 08 16:34:49.108741 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.108699    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 08 16:34:49.109139 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.108937    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 08 16:34:49.109139 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.109014    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-z4c77\""
Jul 08 16:34:49.109267 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.109187    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 08 16:34:49.109353 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.109261    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f5jcr"
Jul 08 16:34:49.109353 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:49.109336    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f5jcr" podUID="0f2a26b1-8d19-4874-a633-1cd61e29091a"
Jul 08 16:34:49.109756 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.109739    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-7s65t\""
Jul 08 16:34:49.109823 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.109794    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 08 16:34:49.110674 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.110630    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2bv8p"
Jul 08 16:34:49.112012 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.111995    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:34:49.112902 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.112883    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 08 16:34:49.113029 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.113013    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 08 16:34:49.113072 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.113054    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4p7ft\""
Jul 08 16:34:49.113372 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.113356    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 08 16:34:49.113486 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.113473    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d9p5w"
Jul 08 16:34:49.113544 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:49.113527    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d9p5w" podUID="e6f14c39-461f-44d2-811a-87cfb899d92c"
Jul 08 16:34:49.114249 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.114234    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 08 16:34:49.114325 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.114247    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 08 16:34:49.114325 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.114263    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 08 16:34:49.114525 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.114513    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 08 16:34:49.114588 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.114572    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-h9h2l\""
Jul 08 16:34:49.114626 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.114608    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 08 16:34:49.114658 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.114574    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 08 16:34:49.114832 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.114819    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ht2nh"
Jul 08 16:34:49.115710 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.115678    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 08 16:34:49.115803 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.115789    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-94.ec2.internal"
Jul 08 16:34:49.116209 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.116195    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q2rs4"
Jul 08 16:34:49.117157 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.117139    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 08 16:34:49.117255 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.117138    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 08 16:34:49.117255 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.117217    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-9xwsd\""
Jul 08 16:34:49.117825 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.117807    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-94.ec2.internal"]
Jul 08 16:34:49.117902 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.117852    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 08 16:34:49.117902 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.117898    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wnsct"
Jul 08 16:34:49.118221 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.118206    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 08 16:34:49.118460 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.118446    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-cz7xb\""
Jul 08 16:34:49.118549 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.118459    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 08 16:34:49.118656 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.118627    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 08 16:34:49.120091 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.120069    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 08 16:34:49.120215 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.120134    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-dlh8l\""
Jul 08 16:34:49.120408 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.120354    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 08 16:34:49.130254 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.130231    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 08 16:34:49.130390 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.130376    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-142-94.ec2.internal"]
Jul 08 16:34:49.145400 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.145375    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-sm5bk"
Jul 08 16:34:49.154744 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.154724    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-sm5bk"
Jul 08 16:34:49.201560 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.201535    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 08 16:34:49.207402 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.207376    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0-cni-binary-copy\") pod \"multus-bg589\" (UID: \"193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0\") " pod="openshift-multus/multus-bg589"
Jul 08 16:34:49.207534 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.207406    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f08a1b00-78fd-43ef-b2aa-2712d09fb1b3-host-kubelet\") pod \"ovnkube-node-2shvk\" (UID: \"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:34:49.207534 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.207426    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/413800cd-9139-4b8b-b807-78d658dc06ba-socket-dir\") pod \"aws-ebs-csi-driver-node-q2rs4\" (UID: \"413800cd-9139-4b8b-b807-78d658dc06ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q2rs4"
Jul 08 16:34:49.207534 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.207446    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b33e50d5-554b-4d53-aafe-3dc13ae3fa6c-etc-sysctl-d\") pod \"tuned-wnsct\" (UID: \"b33e50d5-554b-4d53-aafe-3dc13ae3fa6c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnsct"
Jul 08 16:34:49.207534 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.207485    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/416e11c0-3ba2-451d-a901-9681ec0d4e3e-serviceca\") pod \"node-ca-vnjdb\" (UID: \"416e11c0-3ba2-451d-a901-9681ec0d4e3e\") " pod="openshift-image-registry/node-ca-vnjdb"
Jul 08 16:34:49.207534 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.207519    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0-multus-socket-dir-parent\") pod \"multus-bg589\" (UID: \"193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0\") " pod="openshift-multus/multus-bg589"
Jul 08 16:34:49.207746 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.207544    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f08a1b00-78fd-43ef-b2aa-2712d09fb1b3-host-slash\") pod \"ovnkube-node-2shvk\" (UID: \"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:34:49.207746 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.207566    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/413800cd-9139-4b8b-b807-78d658dc06ba-etc-selinux\") pod \"aws-ebs-csi-driver-node-q2rs4\" (UID: \"413800cd-9139-4b8b-b807-78d658dc06ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q2rs4"
Jul 08 16:34:49.207746 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.207583    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e6f14c39-461f-44d2-811a-87cfb899d92c-dbus\") pod \"global-pull-secret-syncer-d9p5w\" (UID: \"e6f14c39-461f-44d2-811a-87cfb899d92c\") " pod="kube-system/global-pull-secret-syncer-d9p5w"
Jul 08 16:34:49.207746 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.207599    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f08a1b00-78fd-43ef-b2aa-2712d09fb1b3-run-systemd\") pod \"ovnkube-node-2shvk\" (UID: \"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:34:49.207746 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.207615    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f08a1b00-78fd-43ef-b2aa-2712d09fb1b3-run-openvswitch\") pod \"ovnkube-node-2shvk\" (UID: \"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:34:49.207746 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.207715    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f08a1b00-78fd-43ef-b2aa-2712d09fb1b3-host-run-ovn-kubernetes\") pod \"ovnkube-node-2shvk\" (UID: \"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:34:49.207985 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.207748    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84d1dbea-4585-46d9-b520-1329dba8df82-tmp-dir\") pod \"node-resolver-wflvt\" (UID: \"84d1dbea-4585-46d9-b520-1329dba8df82\") " pod="openshift-dns/node-resolver-wflvt"
Jul 08 16:34:49.207985 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.207798    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/28a74820-f159-4264-af3e-c983d2c68a51-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g4stg\" (UID: \"28a74820-f159-4264-af3e-c983d2c68a51\") " pod="openshift-multus/multus-additional-cni-plugins-g4stg"
Jul 08 16:34:49.207985 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.207836    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b33e50d5-554b-4d53-aafe-3dc13ae3fa6c-run\") pod \"tuned-wnsct\" (UID: \"b33e50d5-554b-4d53-aafe-3dc13ae3fa6c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnsct"
Jul 08 16:34:49.207985 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.207865    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b33e50d5-554b-4d53-aafe-3dc13ae3fa6c-tmp\") pod \"tuned-wnsct\" (UID: \"b33e50d5-554b-4d53-aafe-3dc13ae3fa6c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnsct"
Jul 08 16:34:49.207985 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.207894    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0-host-var-lib-cni-bin\") pod \"multus-bg589\" (UID: \"193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0\") " pod="openshift-multus/multus-bg589"
Jul 08 16:34:49.207985 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.207920    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0-multus-conf-dir\") pod \"multus-bg589\" (UID: \"193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0\") " pod="openshift-multus/multus-bg589"
Jul 08 16:34:49.207985 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.207965    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qcxg\" (UniqueName: \"kubernetes.io/projected/f08a1b00-78fd-43ef-b2aa-2712d09fb1b3-kube-api-access-4qcxg\") pod \"ovnkube-node-2shvk\" (UID: \"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:34:49.208236 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.207991    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/413800cd-9139-4b8b-b807-78d658dc06ba-device-dir\") pod \"aws-ebs-csi-driver-node-q2rs4\" (UID: \"413800cd-9139-4b8b-b807-78d658dc06ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q2rs4"
Jul 08 16:34:49.208236 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.208015    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0-etc-kubernetes\") pod \"multus-bg589\" (UID: \"193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0\") " pod="openshift-multus/multus-bg589"
Jul 08 16:34:49.208236 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.208040    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b33e50d5-554b-4d53-aafe-3dc13ae3fa6c-sys\") pod \"tuned-wnsct\" (UID: \"b33e50d5-554b-4d53-aafe-3dc13ae3fa6c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnsct"
Jul 08 16:34:49.208236 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.208048    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84d1dbea-4585-46d9-b520-1329dba8df82-tmp-dir\") pod \"node-resolver-wflvt\" (UID: \"84d1dbea-4585-46d9-b520-1329dba8df82\") " pod="openshift-dns/node-resolver-wflvt"
Jul 08 16:34:49.208236 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.208065    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f08a1b00-78fd-43ef-b2aa-2712d09fb1b3-etc-openvswitch\") pod \"ovnkube-node-2shvk\" (UID: \"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:34:49.208236 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.208087    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/413800cd-9139-4b8b-b807-78d658dc06ba-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q2rs4\" (UID: \"413800cd-9139-4b8b-b807-78d658dc06ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q2rs4"
Jul 08 16:34:49.208236 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.208121    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7ce643e8-a633-411e-9d39-6f349e7ed426-iptables-alerter-script\") pod \"iptables-alerter-2bv8p\" (UID: \"7ce643e8-a633-411e-9d39-6f349e7ed426\") " pod="openshift-network-operator/iptables-alerter-2bv8p"
Jul 08 16:34:49.208236 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.208144    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrnqs\" (UniqueName: \"kubernetes.io/projected/7ce643e8-a633-411e-9d39-6f349e7ed426-kube-api-access-qrnqs\") pod \"iptables-alerter-2bv8p\" (UID: \"7ce643e8-a633-411e-9d39-6f349e7ed426\") " pod="openshift-network-operator/iptables-alerter-2bv8p"
Jul 08 16:34:49.208236 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.208168    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0-cnibin\") pod \"multus-bg589\" (UID: \"193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0\") " pod="openshift-multus/multus-bg589"
Jul 08 16:34:49.208236 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.208185    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f08a1b00-78fd-43ef-b2aa-2712d09fb1b3-node-log\") pod \"ovnkube-node-2shvk\" (UID: \"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:34:49.208236 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.208200    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b33e50d5-554b-4d53-aafe-3dc13ae3fa6c-etc-kubernetes\") pod \"tuned-wnsct\" (UID: \"b33e50d5-554b-4d53-aafe-3dc13ae3fa6c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnsct"
Jul 08 16:34:49.208236 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.208215    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0-multus-daemon-config\") pod \"multus-bg589\" (UID: \"193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0\") " pod="openshift-multus/multus-bg589"
Jul 08 16:34:49.208236 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.208230    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9pjz\" (UniqueName: \"kubernetes.io/projected/193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0-kube-api-access-f9pjz\") pod \"multus-bg589\" (UID: \"193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0\") " pod="openshift-multus/multus-bg589"
Jul 08 16:34:49.208820 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.208260    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f08a1b00-78fd-43ef-b2aa-2712d09fb1b3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2shvk\" (UID: \"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:34:49.208820 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.208278    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/28a74820-f159-4264-af3e-c983d2c68a51-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g4stg\" (UID: \"28a74820-f159-4264-af3e-c983d2c68a51\") " pod="openshift-multus/multus-additional-cni-plugins-g4stg"
Jul 08 16:34:49.208820 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.208297    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7ce643e8-a633-411e-9d39-6f349e7ed426-host-slash\") pod \"iptables-alerter-2bv8p\" (UID: \"7ce643e8-a633-411e-9d39-6f349e7ed426\") " pod="openshift-network-operator/iptables-alerter-2bv8p"
Jul 08 16:34:49.208820 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.208316    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cfft\" (UniqueName: \"kubernetes.io/projected/b33e50d5-554b-4d53-aafe-3dc13ae3fa6c-kube-api-access-6cfft\") pod \"tuned-wnsct\" (UID: \"b33e50d5-554b-4d53-aafe-3dc13ae3fa6c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnsct"
Jul 08 16:34:49.208820 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.208341    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0-host-run-multus-certs\") pod \"multus-bg589\" (UID: \"193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0\") " pod="openshift-multus/multus-bg589"
Jul 08 16:34:49.208820 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.208381    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f08a1b00-78fd-43ef-b2aa-2712d09fb1b3-run-ovn\") pod \"ovnkube-node-2shvk\" (UID: \"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:34:49.208820 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.208421    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f08a1b00-78fd-43ef-b2aa-2712d09fb1b3-host-cni-netd\") pod \"ovnkube-node-2shvk\" (UID: \"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:34:49.208820 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.208448    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/413800cd-9139-4b8b-b807-78d658dc06ba-sys-fs\") pod \"aws-ebs-csi-driver-node-q2rs4\" (UID: \"413800cd-9139-4b8b-b807-78d658dc06ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q2rs4"
Jul 08 16:34:49.208820 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.208477    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6cnn\" (UniqueName: \"kubernetes.io/projected/413800cd-9139-4b8b-b807-78d658dc06ba-kube-api-access-g6cnn\") pod \"aws-ebs-csi-driver-node-q2rs4\" (UID: \"413800cd-9139-4b8b-b807-78d658dc06ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q2rs4"
Jul 08 16:34:49.208820 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.208507    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx5x6\" (UniqueName: \"kubernetes.io/projected/84d1dbea-4585-46d9-b520-1329dba8df82-kube-api-access-jx5x6\") pod \"node-resolver-wflvt\" (UID: \"84d1dbea-4585-46d9-b520-1329dba8df82\") " pod="openshift-dns/node-resolver-wflvt"
Jul 08 16:34:49.208820 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.208575    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0-system-cni-dir\") pod \"multus-bg589\" (UID: \"193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0\") " pod="openshift-multus/multus-bg589"
Jul 08 16:34:49.208820 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.208601    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f08a1b00-78fd-43ef-b2aa-2712d09fb1b3-var-lib-openvswitch\") pod \"ovnkube-node-2shvk\" (UID: \"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:34:49.208820 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.208624    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/28a74820-f159-4264-af3e-c983d2c68a51-cnibin\") pod \"multus-additional-cni-plugins-g4stg\" (UID: \"28a74820-f159-4264-af3e-c983d2c68a51\") " pod="openshift-multus/multus-additional-cni-plugins-g4stg"
Jul 08 16:34:49.208820 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.208668    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/28a74820-f159-4264-af3e-c983d2c68a51-os-release\") pod \"multus-additional-cni-plugins-g4stg\" (UID: \"28a74820-f159-4264-af3e-c983d2c68a51\") " pod="openshift-multus/multus-additional-cni-plugins-g4stg"
Jul 08 16:34:49.208820 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.208694    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f08a1b00-78fd-43ef-b2aa-2712d09fb1b3-log-socket\") pod \"ovnkube-node-2shvk\" (UID: \"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:34:49.208820 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.208718    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/84d1dbea-4585-46d9-b520-1329dba8df82-hosts-file\") pod \"node-resolver-wflvt\" (UID: \"84d1dbea-4585-46d9-b520-1329dba8df82\") " pod="openshift-dns/node-resolver-wflvt"
Jul 08 16:34:49.209636 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.208759    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxs62\" (UniqueName: \"kubernetes.io/projected/28a74820-f159-4264-af3e-c983d2c68a51-kube-api-access-sxs62\") pod \"multus-additional-cni-plugins-g4stg\" (UID: \"28a74820-f159-4264-af3e-c983d2c68a51\") " pod="openshift-multus/multus-additional-cni-plugins-g4stg"
Jul 08 16:34:49.209636 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.208764    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/84d1dbea-4585-46d9-b520-1329dba8df82-hosts-file\") pod \"node-resolver-wflvt\" (UID: \"84d1dbea-4585-46d9-b520-1329dba8df82\") " pod="openshift-dns/node-resolver-wflvt"
Jul 08 16:34:49.209636 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.208795    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f08a1b00-78fd-43ef-b2aa-2712d09fb1b3-systemd-units\") pod \"ovnkube-node-2shvk\" (UID: \"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:34:49.209636 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.208822    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b33e50d5-554b-4d53-aafe-3dc13ae3fa6c-etc-modprobe-d\") pod \"tuned-wnsct\" (UID: \"b33e50d5-554b-4d53-aafe-3dc13ae3fa6c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnsct"
Jul 08 16:34:49.209636 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.208846    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b33e50d5-554b-4d53-aafe-3dc13ae3fa6c-var-lib-kubelet\") pod \"tuned-wnsct\" (UID: \"b33e50d5-554b-4d53-aafe-3dc13ae3fa6c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnsct"
Jul 08 16:34:49.209636 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.208867    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b33e50d5-554b-4d53-aafe-3dc13ae3fa6c-host\") pod \"tuned-wnsct\" (UID: \"b33e50d5-554b-4d53-aafe-3dc13ae3fa6c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnsct"
Jul 08 16:34:49.209636 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.208897    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0-multus-cni-dir\") pod \"multus-bg589\" (UID: \"193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0\") " pod="openshift-multus/multus-bg589"
Jul 08 16:34:49.209636 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.208927    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0-host-run-k8s-cni-cncf-io\") pod \"multus-bg589\" (UID: \"193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0\") " pod="openshift-multus/multus-bg589"
Jul 08 16:34:49.209636 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.208955    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b33e50d5-554b-4d53-aafe-3dc13ae3fa6c-etc-sysctl-conf\") pod \"tuned-wnsct\" (UID: \"b33e50d5-554b-4d53-aafe-3dc13ae3fa6c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnsct"
Jul 08 16:34:49.209636 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.208982    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhvdc\" (UniqueName: \"kubernetes.io/projected/9bbb98bc-99e1-4401-96ab-ad9aae630c48-kube-api-access-lhvdc\") pod \"network-check-target-hzjsx\" (UID: \"9bbb98bc-99e1-4401-96ab-ad9aae630c48\") " pod="openshift-network-diagnostics/network-check-target-hzjsx"
Jul 08 16:34:49.209636 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.209010    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0-host-run-netns\") pod \"multus-bg589\" (UID: \"193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0\") " pod="openshift-multus/multus-bg589"
Jul 08 16:34:49.209636 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.209038    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0-host-var-lib-cni-multus\") pod \"multus-bg589\" (UID: \"193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0\") " pod="openshift-multus/multus-bg589"
Jul 08 16:34:49.209636 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.209066    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f2a26b1-8d19-4874-a633-1cd61e29091a-metrics-certs\") pod \"network-metrics-daemon-f5jcr\" (UID: \"0f2a26b1-8d19-4874-a633-1cd61e29091a\") " pod="openshift-multus/network-metrics-daemon-f5jcr"
Jul 08 16:34:49.209636 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.209093    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f08a1b00-78fd-43ef-b2aa-2712d09fb1b3-ovnkube-config\") pod \"ovnkube-node-2shvk\" (UID: \"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:34:49.209636 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.209122    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f08a1b00-78fd-43ef-b2aa-2712d09fb1b3-ovn-node-metrics-cert\") pod \"ovnkube-node-2shvk\" (UID: \"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:34:49.209636 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.209148    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/28a74820-f159-4264-af3e-c983d2c68a51-system-cni-dir\") pod \"multus-additional-cni-plugins-g4stg\" (UID: \"28a74820-f159-4264-af3e-c983d2c68a51\") " pod="openshift-multus/multus-additional-cni-plugins-g4stg"
Jul 08 16:34:49.210537 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.209204    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/28a74820-f159-4264-af3e-c983d2c68a51-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g4stg\" (UID: \"28a74820-f159-4264-af3e-c983d2c68a51\") " pod="openshift-multus/multus-additional-cni-plugins-g4stg"
Jul 08 16:34:49.210537 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.209235    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3b861d99-7199-4174-a922-d0aae9cd010a-agent-certs\") pod \"konnectivity-agent-ht2nh\" (UID: \"3b861d99-7199-4174-a922-d0aae9cd010a\") " pod="kube-system/konnectivity-agent-ht2nh"
Jul 08 16:34:49.210537 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.209267    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b33e50d5-554b-4d53-aafe-3dc13ae3fa6c-lib-modules\") pod \"tuned-wnsct\" (UID: \"b33e50d5-554b-4d53-aafe-3dc13ae3fa6c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnsct"
Jul 08 16:34:49.210537 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.209296    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0-hostroot\") pod \"multus-bg589\" (UID: \"193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0\") " pod="openshift-multus/multus-bg589"
Jul 08 16:34:49.210537 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.209324    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f08a1b00-78fd-43ef-b2aa-2712d09fb1b3-env-overrides\") pod \"ovnkube-node-2shvk\" (UID: \"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:34:49.210537 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.209350    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b33e50d5-554b-4d53-aafe-3dc13ae3fa6c-etc-sysconfig\") pod \"tuned-wnsct\" (UID: \"b33e50d5-554b-4d53-aafe-3dc13ae3fa6c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnsct"
Jul 08 16:34:49.210537 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.209405    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b33e50d5-554b-4d53-aafe-3dc13ae3fa6c-etc-systemd\") pod \"tuned-wnsct\" (UID: \"b33e50d5-554b-4d53-aafe-3dc13ae3fa6c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnsct"
Jul 08 16:34:49.210537 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.209434    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/416e11c0-3ba2-451d-a901-9681ec0d4e3e-host\") pod \"node-ca-vnjdb\" (UID: \"416e11c0-3ba2-451d-a901-9681ec0d4e3e\") " pod="openshift-image-registry/node-ca-vnjdb"
Jul 08 16:34:49.210537 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.209461    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0-os-release\") pod \"multus-bg589\" (UID: \"193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0\") " pod="openshift-multus/multus-bg589"
Jul 08 16:34:49.210537 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.209491    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqnxm\" (UniqueName: \"kubernetes.io/projected/0f2a26b1-8d19-4874-a633-1cd61e29091a-kube-api-access-nqnxm\") pod \"network-metrics-daemon-f5jcr\" (UID: \"0f2a26b1-8d19-4874-a633-1cd61e29091a\") " pod="openshift-multus/network-metrics-daemon-f5jcr"
Jul 08 16:34:49.210537 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.209523    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f08a1b00-78fd-43ef-b2aa-2712d09fb1b3-ovnkube-script-lib\") pod \"ovnkube-node-2shvk\" (UID: \"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:34:49.210537 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.209550    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/28a74820-f159-4264-af3e-c983d2c68a51-cni-binary-copy\") pod \"multus-additional-cni-plugins-g4stg\" (UID: \"28a74820-f159-4264-af3e-c983d2c68a51\") " pod="openshift-multus/multus-additional-cni-plugins-g4stg"
Jul 08 16:34:49.210537 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.209575    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3b861d99-7199-4174-a922-d0aae9cd010a-konnectivity-ca\") pod \"konnectivity-agent-ht2nh\" (UID: \"3b861d99-7199-4174-a922-d0aae9cd010a\") " pod="kube-system/konnectivity-agent-ht2nh"
Jul 08 16:34:49.210537 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.209600    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0-host-var-lib-kubelet\") pod \"multus-bg589\" (UID: \"193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0\") " pod="openshift-multus/multus-bg589"
Jul 08 16:34:49.210537 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.209629    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f08a1b00-78fd-43ef-b2aa-2712d09fb1b3-host-run-netns\") pod \"ovnkube-node-2shvk\" (UID: \"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:34:49.210537 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.209654    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f08a1b00-78fd-43ef-b2aa-2712d09fb1b3-host-cni-bin\") pod \"ovnkube-node-2shvk\" (UID: \"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:34:49.211263 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.209688    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/413800cd-9139-4b8b-b807-78d658dc06ba-registration-dir\") pod \"aws-ebs-csi-driver-node-q2rs4\" (UID: \"413800cd-9139-4b8b-b807-78d658dc06ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q2rs4"
Jul 08 16:34:49.211263 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.209712    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e6f14c39-461f-44d2-811a-87cfb899d92c-kubelet-config\") pod \"global-pull-secret-syncer-d9p5w\" (UID: \"e6f14c39-461f-44d2-811a-87cfb899d92c\") " pod="kube-system/global-pull-secret-syncer-d9p5w"
Jul 08 16:34:49.211263 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.209727    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e6f14c39-461f-44d2-811a-87cfb899d92c-original-pull-secret\") pod \"global-pull-secret-syncer-d9p5w\" (UID: \"e6f14c39-461f-44d2-811a-87cfb899d92c\") " pod="kube-system/global-pull-secret-syncer-d9p5w"
Jul 08 16:34:49.211263 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.209743    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b33e50d5-554b-4d53-aafe-3dc13ae3fa6c-etc-tuned\") pod \"tuned-wnsct\" (UID: \"b33e50d5-554b-4d53-aafe-3dc13ae3fa6c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnsct"
Jul 08 16:34:49.211263 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.209760    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snvfl\" (UniqueName: \"kubernetes.io/projected/416e11c0-3ba2-451d-a901-9681ec0d4e3e-kube-api-access-snvfl\") pod \"node-ca-vnjdb\" (UID: \"416e11c0-3ba2-451d-a901-9681ec0d4e3e\") " pod="openshift-image-registry/node-ca-vnjdb"
Jul 08 16:34:49.216028 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.216000    2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 08 16:34:49.219790 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.219767    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx5x6\" (UniqueName: \"kubernetes.io/projected/84d1dbea-4585-46d9-b520-1329dba8df82-kube-api-access-jx5x6\") pod \"node-resolver-wflvt\" (UID: \"84d1dbea-4585-46d9-b520-1329dba8df82\") " pod="openshift-dns/node-resolver-wflvt"
Jul 08 16:34:49.237503 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.237481    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 08 16:34:49.311067 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.311007    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0-system-cni-dir\") pod \"multus-bg589\" (UID: \"193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0\") " pod="openshift-multus/multus-bg589"
Jul 08 16:34:49.311067 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.311064    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f08a1b00-78fd-43ef-b2aa-2712d09fb1b3-var-lib-openvswitch\") pod \"ovnkube-node-2shvk\" (UID: \"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:34:49.311321 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.311093    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/28a74820-f159-4264-af3e-c983d2c68a51-cnibin\") pod \"multus-additional-cni-plugins-g4stg\" (UID: \"28a74820-f159-4264-af3e-c983d2c68a51\") " pod="openshift-multus/multus-additional-cni-plugins-g4stg"
Jul 08 16:34:49.311321 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.311118    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/28a74820-f159-4264-af3e-c983d2c68a51-os-release\") pod \"multus-additional-cni-plugins-g4stg\" (UID: \"28a74820-f159-4264-af3e-c983d2c68a51\") " pod="openshift-multus/multus-additional-cni-plugins-g4stg"
Jul 08 16:34:49.311321 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.311163    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0-system-cni-dir\") pod \"multus-bg589\" (UID: \"193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0\") " pod="openshift-multus/multus-bg589"
Jul 08 16:34:49.311321 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.311182    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/28a74820-f159-4264-af3e-c983d2c68a51-cnibin\") pod \"multus-additional-cni-plugins-g4stg\" (UID: \"28a74820-f159-4264-af3e-c983d2c68a51\") " pod="openshift-multus/multus-additional-cni-plugins-g4stg"
Jul 08 16:34:49.311321 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.311197    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/28a74820-f159-4264-af3e-c983d2c68a51-os-release\") pod \"multus-additional-cni-plugins-g4stg\" (UID: \"28a74820-f159-4264-af3e-c983d2c68a51\") " pod="openshift-multus/multus-additional-cni-plugins-g4stg"
Jul 08 16:34:49.311321 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.311167    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f08a1b00-78fd-43ef-b2aa-2712d09fb1b3-var-lib-openvswitch\") pod \"ovnkube-node-2shvk\" (UID: \"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:34:49.311321 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.311239    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f08a1b00-78fd-43ef-b2aa-2712d09fb1b3-log-socket\") pod \"ovnkube-node-2shvk\" (UID: \"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:34:49.311321 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.311272    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxs62\" (UniqueName: \"kubernetes.io/projected/28a74820-f159-4264-af3e-c983d2c68a51-kube-api-access-sxs62\") pod \"multus-additional-cni-plugins-g4stg\" (UID: \"28a74820-f159-4264-af3e-c983d2c68a51\") " pod="openshift-multus/multus-additional-cni-plugins-g4stg"
Jul 08 16:34:49.311321 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.311301    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f08a1b00-78fd-43ef-b2aa-2712d09fb1b3-log-socket\") pod \"ovnkube-node-2shvk\" (UID: \"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:34:49.311321 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.311300    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f08a1b00-78fd-43ef-b2aa-2712d09fb1b3-systemd-units\") pod \"ovnkube-node-2shvk\" (UID: \"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:34:49.311837 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.311330    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b33e50d5-554b-4d53-aafe-3dc13ae3fa6c-etc-modprobe-d\") pod \"tuned-wnsct\" (UID: \"b33e50d5-554b-4d53-aafe-3dc13ae3fa6c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnsct"
Jul 08 16:34:49.311837 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.311355    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b33e50d5-554b-4d53-aafe-3dc13ae3fa6c-var-lib-kubelet\") pod \"tuned-wnsct\" (UID: \"b33e50d5-554b-4d53-aafe-3dc13ae3fa6c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnsct"
Jul 08 16:34:49.311837 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.311381    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b33e50d5-554b-4d53-aafe-3dc13ae3fa6c-host\") pod \"tuned-wnsct\" (UID: \"b33e50d5-554b-4d53-aafe-3dc13ae3fa6c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnsct"
Jul 08 16:34:49.311837 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.311404    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0-multus-cni-dir\") pod \"multus-bg589\" (UID: \"193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0\") " pod="openshift-multus/multus-bg589"
Jul 08 16:34:49.311837 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.311431    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0-host-run-k8s-cni-cncf-io\") pod \"multus-bg589\" (UID: \"193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0\") " pod="openshift-multus/multus-bg589"
Jul 08 16:34:49.311837 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.311440    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f08a1b00-78fd-43ef-b2aa-2712d09fb1b3-systemd-units\") pod \"ovnkube-node-2shvk\" (UID: \"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:34:49.311837 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.311455    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b33e50d5-554b-4d53-aafe-3dc13ae3fa6c-etc-sysctl-conf\") pod \"tuned-wnsct\" (UID: \"b33e50d5-554b-4d53-aafe-3dc13ae3fa6c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnsct"
Jul 08 16:34:49.311837 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.311471    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b33e50d5-554b-4d53-aafe-3dc13ae3fa6c-var-lib-kubelet\") pod \"tuned-wnsct\" (UID: \"b33e50d5-554b-4d53-aafe-3dc13ae3fa6c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnsct"
Jul 08 16:34:49.311837 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.311482    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhvdc\" (UniqueName: \"kubernetes.io/projected/9bbb98bc-99e1-4401-96ab-ad9aae630c48-kube-api-access-lhvdc\") pod \"network-check-target-hzjsx\" (UID: \"9bbb98bc-99e1-4401-96ab-ad9aae630c48\") " pod="openshift-network-diagnostics/network-check-target-hzjsx"
Jul 08 16:34:49.311837 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.311506    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0-host-run-netns\") pod \"multus-bg589\" (UID: \"193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0\") " pod="openshift-multus/multus-bg589"
Jul 08 16:34:49.311837 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.311509    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b33e50d5-554b-4d53-aafe-3dc13ae3fa6c-etc-modprobe-d\") pod \"tuned-wnsct\" (UID: \"b33e50d5-554b-4d53-aafe-3dc13ae3fa6c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnsct"
Jul 08 16:34:49.311837 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.311523    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b33e50d5-554b-4d53-aafe-3dc13ae3fa6c-host\") pod \"tuned-wnsct\" (UID: \"b33e50d5-554b-4d53-aafe-3dc13ae3fa6c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnsct"
Jul 08 16:34:49.311837 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.311534    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0-host-var-lib-cni-multus\") pod \"multus-bg589\" (UID: \"193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0\") " pod="openshift-multus/multus-bg589"
Jul 08 16:34:49.311837 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.311560    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f2a26b1-8d19-4874-a633-1cd61e29091a-metrics-certs\") pod \"network-metrics-daemon-f5jcr\" (UID: \"0f2a26b1-8d19-4874-a633-1cd61e29091a\") " pod="openshift-multus/network-metrics-daemon-f5jcr"
Jul 08 16:34:49.311837 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.311565    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0-host-run-k8s-cni-cncf-io\") pod \"multus-bg589\" (UID: \"193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0\") " pod="openshift-multus/multus-bg589"
Jul 08 16:34:49.311837 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.311573    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0-multus-cni-dir\") pod \"multus-bg589\" (UID: \"193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0\") " pod="openshift-multus/multus-bg589"
Jul 08 16:34:49.311837 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.311585    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f08a1b00-78fd-43ef-b2aa-2712d09fb1b3-ovnkube-config\") pod \"ovnkube-node-2shvk\" (UID: \"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:34:49.311837 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.311634    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f08a1b00-78fd-43ef-b2aa-2712d09fb1b3-ovn-node-metrics-cert\") pod \"ovnkube-node-2shvk\" (UID: \"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:34:49.312887 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.311638    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0-host-var-lib-cni-multus\") pod \"multus-bg589\" (UID: \"193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0\") " pod="openshift-multus/multus-bg589"
Jul 08 16:34:49.312887 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.311659    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/28a74820-f159-4264-af3e-c983d2c68a51-system-cni-dir\") pod \"multus-additional-cni-plugins-g4stg\" (UID: \"28a74820-f159-4264-af3e-c983d2c68a51\") " pod="openshift-multus/multus-additional-cni-plugins-g4stg"
Jul 08 16:34:49.312887 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.311668    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b33e50d5-554b-4d53-aafe-3dc13ae3fa6c-etc-sysctl-conf\") pod \"tuned-wnsct\" (UID: \"b33e50d5-554b-4d53-aafe-3dc13ae3fa6c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnsct"
Jul 08 16:34:49.312887 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.311607    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0-host-run-netns\") pod \"multus-bg589\" (UID: \"193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0\") " pod="openshift-multus/multus-bg589"
Jul 08 16:34:49.312887 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.311729    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/28a74820-f159-4264-af3e-c983d2c68a51-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g4stg\" (UID: \"28a74820-f159-4264-af3e-c983d2c68a51\") " pod="openshift-multus/multus-additional-cni-plugins-g4stg"
Jul 08 16:34:49.312887 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.311761    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3b861d99-7199-4174-a922-d0aae9cd010a-agent-certs\") pod \"konnectivity-agent-ht2nh\" (UID: \"3b861d99-7199-4174-a922-d0aae9cd010a\") " pod="kube-system/konnectivity-agent-ht2nh"
Jul 08 16:34:49.312887 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.311790    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b33e50d5-554b-4d53-aafe-3dc13ae3fa6c-lib-modules\") pod \"tuned-wnsct\" (UID: \"b33e50d5-554b-4d53-aafe-3dc13ae3fa6c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnsct"
Jul 08 16:34:49.312887 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.311813    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0-hostroot\") pod \"multus-bg589\" (UID: \"193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0\") " pod="openshift-multus/multus-bg589"
Jul 08 16:34:49.312887 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:49.311765    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:49.312887 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.311839    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f08a1b00-78fd-43ef-b2aa-2712d09fb1b3-env-overrides\") pod \"ovnkube-node-2shvk\" (UID: \"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:34:49.312887 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.311865    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b33e50d5-554b-4d53-aafe-3dc13ae3fa6c-etc-sysconfig\") pod \"tuned-wnsct\" (UID: \"b33e50d5-554b-4d53-aafe-3dc13ae3fa6c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnsct"
Jul 08 16:34:49.312887 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.311890    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b33e50d5-554b-4d53-aafe-3dc13ae3fa6c-etc-systemd\") pod \"tuned-wnsct\" (UID: \"b33e50d5-554b-4d53-aafe-3dc13ae3fa6c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnsct"
Jul 08 16:34:49.312887 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.311915    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/416e11c0-3ba2-451d-a901-9681ec0d4e3e-host\") pod \"node-ca-vnjdb\" (UID: \"416e11c0-3ba2-451d-a901-9681ec0d4e3e\") " pod="openshift-image-registry/node-ca-vnjdb"
Jul 08 16:34:49.312887 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:49.311940    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f2a26b1-8d19-4874-a633-1cd61e29091a-metrics-certs podName:0f2a26b1-8d19-4874-a633-1cd61e29091a nodeName:}" failed. No retries permitted until 2026-07-08 16:34:49.811910039 +0000 UTC m=+2.083359594 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0f2a26b1-8d19-4874-a633-1cd61e29091a-metrics-certs") pod "network-metrics-daemon-f5jcr" (UID: "0f2a26b1-8d19-4874-a633-1cd61e29091a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:49.312887 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.311959    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/416e11c0-3ba2-451d-a901-9681ec0d4e3e-host\") pod \"node-ca-vnjdb\" (UID: \"416e11c0-3ba2-451d-a901-9681ec0d4e3e\") " pod="openshift-image-registry/node-ca-vnjdb"
Jul 08 16:34:49.312887 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.311980    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0-os-release\") pod \"multus-bg589\" (UID: \"193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0\") " pod="openshift-multus/multus-bg589"
Jul 08 16:34:49.312887 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.311989    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b33e50d5-554b-4d53-aafe-3dc13ae3fa6c-lib-modules\") pod \"tuned-wnsct\" (UID: \"b33e50d5-554b-4d53-aafe-3dc13ae3fa6c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnsct"
Jul 08 16:34:49.313797 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.312013    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqnxm\" (UniqueName: \"kubernetes.io/projected/0f2a26b1-8d19-4874-a633-1cd61e29091a-kube-api-access-nqnxm\") pod \"network-metrics-daemon-f5jcr\" (UID: \"0f2a26b1-8d19-4874-a633-1cd61e29091a\") " pod="openshift-multus/network-metrics-daemon-f5jcr"
Jul 08 16:34:49.313797 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.312042    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f08a1b00-78fd-43ef-b2aa-2712d09fb1b3-ovnkube-script-lib\") pod \"ovnkube-node-2shvk\" (UID: \"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:34:49.313797 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.312046    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b33e50d5-554b-4d53-aafe-3dc13ae3fa6c-etc-sysconfig\") pod \"tuned-wnsct\" (UID: \"b33e50d5-554b-4d53-aafe-3dc13ae3fa6c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnsct"
Jul 08 16:34:49.313797 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.312072    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/28a74820-f159-4264-af3e-c983d2c68a51-cni-binary-copy\") pod \"multus-additional-cni-plugins-g4stg\" (UID: \"28a74820-f159-4264-af3e-c983d2c68a51\") " pod="openshift-multus/multus-additional-cni-plugins-g4stg"
Jul 08 16:34:49.313797 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.312084    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/28a74820-f159-4264-af3e-c983d2c68a51-system-cni-dir\") pod \"multus-additional-cni-plugins-g4stg\" (UID: \"28a74820-f159-4264-af3e-c983d2c68a51\") " pod="openshift-multus/multus-additional-cni-plugins-g4stg"
Jul 08 16:34:49.313797 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.312100    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3b861d99-7199-4174-a922-d0aae9cd010a-konnectivity-ca\") pod \"konnectivity-agent-ht2nh\" (UID: \"3b861d99-7199-4174-a922-d0aae9cd010a\") " pod="kube-system/konnectivity-agent-ht2nh"
Jul 08 16:34:49.313797 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.312106    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0-hostroot\") pod \"multus-bg589\" (UID: \"193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0\") " pod="openshift-multus/multus-bg589"
Jul 08 16:34:49.313797 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.312133    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0-host-var-lib-kubelet\") pod \"multus-bg589\" (UID: \"193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0\") " pod="openshift-multus/multus-bg589"
Jul 08 16:34:49.313797 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.312148    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0-os-release\") pod \"multus-bg589\" (UID: \"193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0\") " pod="openshift-multus/multus-bg589"
Jul 08 16:34:49.313797 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.312161    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f08a1b00-78fd-43ef-b2aa-2712d09fb1b3-host-run-netns\") pod \"ovnkube-node-2shvk\" (UID: \"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:34:49.313797 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.312200    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f08a1b00-78fd-43ef-b2aa-2712d09fb1b3-host-run-netns\") pod \"ovnkube-node-2shvk\" (UID: \"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:34:49.313797 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.312251    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b33e50d5-554b-4d53-aafe-3dc13ae3fa6c-etc-systemd\") pod \"tuned-wnsct\" (UID: \"b33e50d5-554b-4d53-aafe-3dc13ae3fa6c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnsct"
Jul 08 16:34:49.313797 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.312288    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0-host-var-lib-kubelet\") pod \"multus-bg589\" (UID: \"193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0\") " pod="openshift-multus/multus-bg589"
Jul 08 16:34:49.313797 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.312775    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3b861d99-7199-4174-a922-d0aae9cd010a-konnectivity-ca\") pod \"konnectivity-agent-ht2nh\" (UID: \"3b861d99-7199-4174-a922-d0aae9cd010a\") " pod="kube-system/konnectivity-agent-ht2nh"
Jul 08 16:34:49.313797 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.312886    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f08a1b00-78fd-43ef-b2aa-2712d09fb1b3-ovnkube-script-lib\") pod \"ovnkube-node-2shvk\" (UID: \"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:34:49.313797 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.312938    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f08a1b00-78fd-43ef-b2aa-2712d09fb1b3-host-cni-bin\") pod \"ovnkube-node-2shvk\" (UID: \"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:34:49.313797 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.312967    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/413800cd-9139-4b8b-b807-78d658dc06ba-registration-dir\") pod \"aws-ebs-csi-driver-node-q2rs4\" (UID: \"413800cd-9139-4b8b-b807-78d658dc06ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q2rs4"
Jul 08 16:34:49.314613 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.312996    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e6f14c39-461f-44d2-811a-87cfb899d92c-kubelet-config\") pod \"global-pull-secret-syncer-d9p5w\" (UID: \"e6f14c39-461f-44d2-811a-87cfb899d92c\") " pod="kube-system/global-pull-secret-syncer-d9p5w"
Jul 08 16:34:49.314613 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.313019    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/28a74820-f159-4264-af3e-c983d2c68a51-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g4stg\" (UID: \"28a74820-f159-4264-af3e-c983d2c68a51\") " pod="openshift-multus/multus-additional-cni-plugins-g4stg"
Jul 08 16:34:49.314613 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.313019    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/28a74820-f159-4264-af3e-c983d2c68a51-cni-binary-copy\") pod \"multus-additional-cni-plugins-g4stg\" (UID: \"28a74820-f159-4264-af3e-c983d2c68a51\") " pod="openshift-multus/multus-additional-cni-plugins-g4stg"
Jul 08 16:34:49.314613 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.313023    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e6f14c39-461f-44d2-811a-87cfb899d92c-original-pull-secret\") pod \"global-pull-secret-syncer-d9p5w\" (UID: \"e6f14c39-461f-44d2-811a-87cfb899d92c\") " pod="kube-system/global-pull-secret-syncer-d9p5w"
Jul 08 16:34:49.314613 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.313082    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e6f14c39-461f-44d2-811a-87cfb899d92c-kubelet-config\") pod \"global-pull-secret-syncer-d9p5w\" (UID: \"e6f14c39-461f-44d2-811a-87cfb899d92c\") " pod="kube-system/global-pull-secret-syncer-d9p5w"
Jul 08 16:34:49.314613 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.313087    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b33e50d5-554b-4d53-aafe-3dc13ae3fa6c-etc-tuned\") pod \"tuned-wnsct\" (UID: \"b33e50d5-554b-4d53-aafe-3dc13ae3fa6c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnsct"
Jul 08 16:34:49.314613 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.313096    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/413800cd-9139-4b8b-b807-78d658dc06ba-registration-dir\") pod \"aws-ebs-csi-driver-node-q2rs4\" (UID: \"413800cd-9139-4b8b-b807-78d658dc06ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q2rs4"
Jul 08 16:34:49.314613 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.313120    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snvfl\" (UniqueName: \"kubernetes.io/projected/416e11c0-3ba2-451d-a901-9681ec0d4e3e-kube-api-access-snvfl\") pod \"node-ca-vnjdb\" (UID: \"416e11c0-3ba2-451d-a901-9681ec0d4e3e\") " pod="openshift-image-registry/node-ca-vnjdb"
Jul 08 16:34:49.314613 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.313145    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0-cni-binary-copy\") pod \"multus-bg589\" (UID: \"193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0\") " pod="openshift-multus/multus-bg589"
Jul 08 16:34:49.314613 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.313174    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f08a1b00-78fd-43ef-b2aa-2712d09fb1b3-host-kubelet\") pod \"ovnkube-node-2shvk\" (UID: \"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:34:49.314613 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:49.313182    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:49.314613 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.313201    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/413800cd-9139-4b8b-b807-78d658dc06ba-socket-dir\") pod \"aws-ebs-csi-driver-node-q2rs4\" (UID: \"413800cd-9139-4b8b-b807-78d658dc06ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q2rs4"
Jul 08 16:34:49.314613 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.313229    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b33e50d5-554b-4d53-aafe-3dc13ae3fa6c-etc-sysctl-d\") pod \"tuned-wnsct\" (UID: \"b33e50d5-554b-4d53-aafe-3dc13ae3fa6c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnsct"
Jul 08 16:34:49.314613 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.313082    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f08a1b00-78fd-43ef-b2aa-2712d09fb1b3-host-cni-bin\") pod \"ovnkube-node-2shvk\" (UID: \"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:34:49.314613 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.313253    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/416e11c0-3ba2-451d-a901-9681ec0d4e3e-serviceca\") pod \"node-ca-vnjdb\" (UID: \"416e11c0-3ba2-451d-a901-9681ec0d4e3e\") " pod="openshift-image-registry/node-ca-vnjdb"
Jul 08 16:34:49.314613 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.313279    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0-multus-socket-dir-parent\") pod \"multus-bg589\" (UID: \"193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0\") " pod="openshift-multus/multus-bg589"
Jul 08 16:34:49.314613 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.313307    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f08a1b00-78fd-43ef-b2aa-2712d09fb1b3-host-slash\") pod \"ovnkube-node-2shvk\" (UID: \"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:34:49.314613 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.313340    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/413800cd-9139-4b8b-b807-78d658dc06ba-etc-selinux\") pod \"aws-ebs-csi-driver-node-q2rs4\" (UID: \"413800cd-9139-4b8b-b807-78d658dc06ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q2rs4"
Jul 08 16:34:49.315626 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.313365    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e6f14c39-461f-44d2-811a-87cfb899d92c-dbus\") pod \"global-pull-secret-syncer-d9p5w\" (UID: \"e6f14c39-461f-44d2-811a-87cfb899d92c\") " pod="kube-system/global-pull-secret-syncer-d9p5w"
Jul 08 16:34:49.315626 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.313390    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f08a1b00-78fd-43ef-b2aa-2712d09fb1b3-run-systemd\") pod \"ovnkube-node-2shvk\" (UID: \"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:34:49.315626 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.313415    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f08a1b00-78fd-43ef-b2aa-2712d09fb1b3-run-openvswitch\") pod \"ovnkube-node-2shvk\" (UID: \"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:34:49.315626 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.313448    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f08a1b00-78fd-43ef-b2aa-2712d09fb1b3-host-run-ovn-kubernetes\") pod \"ovnkube-node-2shvk\" (UID: \"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:34:49.315626 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.313455    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f08a1b00-78fd-43ef-b2aa-2712d09fb1b3-host-slash\") pod \"ovnkube-node-2shvk\" (UID: \"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:34:49.315626 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.313455    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f08a1b00-78fd-43ef-b2aa-2712d09fb1b3-host-kubelet\") pod \"ovnkube-node-2shvk\" (UID: \"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:34:49.315626 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:49.313495    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6f14c39-461f-44d2-811a-87cfb899d92c-original-pull-secret podName:e6f14c39-461f-44d2-811a-87cfb899d92c nodeName:}" failed. No retries permitted until 2026-07-08 16:34:49.81346518 +0000 UTC m=+2.084914703 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e6f14c39-461f-44d2-811a-87cfb899d92c-original-pull-secret") pod "global-pull-secret-syncer-d9p5w" (UID: "e6f14c39-461f-44d2-811a-87cfb899d92c") : object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:49.315626 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.313503    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f08a1b00-78fd-43ef-b2aa-2712d09fb1b3-host-run-ovn-kubernetes\") pod \"ovnkube-node-2shvk\" (UID: \"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:34:49.315626 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.313519    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/28a74820-f159-4264-af3e-c983d2c68a51-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g4stg\" (UID: \"28a74820-f159-4264-af3e-c983d2c68a51\") " pod="openshift-multus/multus-additional-cni-plugins-g4stg"
Jul 08 16:34:49.315626 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.313551    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b33e50d5-554b-4d53-aafe-3dc13ae3fa6c-run\") pod \"tuned-wnsct\" (UID: \"b33e50d5-554b-4d53-aafe-3dc13ae3fa6c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnsct"
Jul 08 16:34:49.315626 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.313574    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b33e50d5-554b-4d53-aafe-3dc13ae3fa6c-tmp\") pod \"tuned-wnsct\" (UID: \"b33e50d5-554b-4d53-aafe-3dc13ae3fa6c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnsct"
Jul 08 16:34:49.315626 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.313598    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/413800cd-9139-4b8b-b807-78d658dc06ba-etc-selinux\") pod \"aws-ebs-csi-driver-node-q2rs4\" (UID: \"413800cd-9139-4b8b-b807-78d658dc06ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q2rs4"
Jul 08 16:34:49.315626 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.313598    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0-host-var-lib-cni-bin\") pod \"multus-bg589\" (UID: \"193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0\") " pod="openshift-multus/multus-bg589"
Jul 08 16:34:49.315626 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.313629    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0-host-var-lib-cni-bin\") pod \"multus-bg589\" (UID: \"193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0\") " pod="openshift-multus/multus-bg589"
Jul 08 16:34:49.315626 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.313643    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0-multus-conf-dir\") pod \"multus-bg589\" (UID: \"193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0\") " pod="openshift-multus/multus-bg589"
Jul 08 16:34:49.315626 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.313674    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qcxg\" (UniqueName: \"kubernetes.io/projected/f08a1b00-78fd-43ef-b2aa-2712d09fb1b3-kube-api-access-4qcxg\") pod \"ovnkube-node-2shvk\" (UID: \"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:34:49.316490 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.313701    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/413800cd-9139-4b8b-b807-78d658dc06ba-device-dir\") pod \"aws-ebs-csi-driver-node-q2rs4\" (UID: \"413800cd-9139-4b8b-b807-78d658dc06ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q2rs4"
Jul 08 16:34:49.316490 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.313747    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0-etc-kubernetes\") pod \"multus-bg589\" (UID: \"193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0\") " pod="openshift-multus/multus-bg589"
Jul 08 16:34:49.316490 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.313770    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b33e50d5-554b-4d53-aafe-3dc13ae3fa6c-sys\") pod \"tuned-wnsct\" (UID: \"b33e50d5-554b-4d53-aafe-3dc13ae3fa6c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnsct"
Jul 08 16:34:49.316490 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.313770    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b33e50d5-554b-4d53-aafe-3dc13ae3fa6c-etc-sysctl-d\") pod \"tuned-wnsct\" (UID: \"b33e50d5-554b-4d53-aafe-3dc13ae3fa6c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnsct"
Jul 08 16:34:49.316490 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.313795    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f08a1b00-78fd-43ef-b2aa-2712d09fb1b3-etc-openvswitch\") pod \"ovnkube-node-2shvk\" (UID: \"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:34:49.316490 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.313821    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/413800cd-9139-4b8b-b807-78d658dc06ba-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q2rs4\" (UID: \"413800cd-9139-4b8b-b807-78d658dc06ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q2rs4"
Jul 08 16:34:49.316490 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.313869    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7ce643e8-a633-411e-9d39-6f349e7ed426-iptables-alerter-script\") pod \"iptables-alerter-2bv8p\" (UID: \"7ce643e8-a633-411e-9d39-6f349e7ed426\") " pod="openshift-network-operator/iptables-alerter-2bv8p"
Jul 08 16:34:49.316490 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.313901    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrnqs\" (UniqueName: \"kubernetes.io/projected/7ce643e8-a633-411e-9d39-6f349e7ed426-kube-api-access-qrnqs\") pod \"iptables-alerter-2bv8p\" (UID: \"7ce643e8-a633-411e-9d39-6f349e7ed426\") " pod="openshift-network-operator/iptables-alerter-2bv8p"
Jul 08 16:34:49.316490 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.313929    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0-cnibin\") pod \"multus-bg589\" (UID: \"193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0\") " pod="openshift-multus/multus-bg589"
Jul 08 16:34:49.316490 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.313952    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/416e11c0-3ba2-451d-a901-9681ec0d4e3e-serviceca\") pod \"node-ca-vnjdb\" (UID: \"416e11c0-3ba2-451d-a901-9681ec0d4e3e\") " pod="openshift-image-registry/node-ca-vnjdb"
Jul 08 16:34:49.316490 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.313969    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f08a1b00-78fd-43ef-b2aa-2712d09fb1b3-node-log\") pod \"ovnkube-node-2shvk\" (UID: \"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:34:49.316490 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.313992    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b33e50d5-554b-4d53-aafe-3dc13ae3fa6c-etc-kubernetes\") pod \"tuned-wnsct\" (UID: \"b33e50d5-554b-4d53-aafe-3dc13ae3fa6c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnsct"
Jul 08 16:34:49.316490 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.313999    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0-cni-binary-copy\") pod \"multus-bg589\" (UID: \"193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0\") " pod="openshift-multus/multus-bg589"
Jul 08 16:34:49.316490 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.314024    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0-multus-socket-dir-parent\") pod \"multus-bg589\" (UID: \"193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0\") " pod="openshift-multus/multus-bg589"
Jul 08 16:34:49.316490 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.314055    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0-multus-daemon-config\") pod \"multus-bg589\" (UID: \"193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0\") " pod="openshift-multus/multus-bg589"
Jul 08 16:34:49.316490 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.314086    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9pjz\" (UniqueName: \"kubernetes.io/projected/193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0-kube-api-access-f9pjz\") pod \"multus-bg589\" (UID: \"193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0\") " pod="openshift-multus/multus-bg589"
Jul 08 16:34:49.316490 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.314088    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b33e50d5-554b-4d53-aafe-3dc13ae3fa6c-run\") pod \"tuned-wnsct\" (UID: \"b33e50d5-554b-4d53-aafe-3dc13ae3fa6c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnsct"
Jul 08 16:34:49.316490 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.314112    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f08a1b00-78fd-43ef-b2aa-2712d09fb1b3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2shvk\" (UID: \"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:34:49.317544 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.314129    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f08a1b00-78fd-43ef-b2aa-2712d09fb1b3-run-systemd\") pod \"ovnkube-node-2shvk\" (UID: \"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:34:49.317544 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.314138    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/28a74820-f159-4264-af3e-c983d2c68a51-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g4stg\" (UID: \"28a74820-f159-4264-af3e-c983d2c68a51\") " pod="openshift-multus/multus-additional-cni-plugins-g4stg"
Jul 08 16:34:49.317544 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.313390    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/413800cd-9139-4b8b-b807-78d658dc06ba-socket-dir\") pod \"aws-ebs-csi-driver-node-q2rs4\" (UID: \"413800cd-9139-4b8b-b807-78d658dc06ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q2rs4"
Jul 08 16:34:49.317544 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.314168    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7ce643e8-a633-411e-9d39-6f349e7ed426-host-slash\") pod \"iptables-alerter-2bv8p\" (UID: \"7ce643e8-a633-411e-9d39-6f349e7ed426\") " pod="openshift-network-operator/iptables-alerter-2bv8p"
Jul 08 16:34:49.317544 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.314186    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f08a1b00-78fd-43ef-b2aa-2712d09fb1b3-run-openvswitch\") pod \"ovnkube-node-2shvk\" (UID: \"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:34:49.317544 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.314191    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cfft\" (UniqueName: \"kubernetes.io/projected/b33e50d5-554b-4d53-aafe-3dc13ae3fa6c-kube-api-access-6cfft\") pod \"tuned-wnsct\" (UID: \"b33e50d5-554b-4d53-aafe-3dc13ae3fa6c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnsct"
Jul 08 16:34:49.317544 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.314226    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0-multus-conf-dir\") pod \"multus-bg589\" (UID: \"193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0\") " pod="openshift-multus/multus-bg589"
Jul 08 16:34:49.317544 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.314227    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/28a74820-f159-4264-af3e-c983d2c68a51-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g4stg\" (UID: \"28a74820-f159-4264-af3e-c983d2c68a51\") " pod="openshift-multus/multus-additional-cni-plugins-g4stg"
Jul 08 16:34:49.317544 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.314272    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b33e50d5-554b-4d53-aafe-3dc13ae3fa6c-etc-kubernetes\") pod \"tuned-wnsct\" (UID: \"b33e50d5-554b-4d53-aafe-3dc13ae3fa6c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnsct"
Jul 08 16:34:49.317544 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.314346    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b33e50d5-554b-4d53-aafe-3dc13ae3fa6c-sys\") pod \"tuned-wnsct\" (UID: \"b33e50d5-554b-4d53-aafe-3dc13ae3fa6c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnsct"
Jul 08 16:34:49.317544 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.314390    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f08a1b00-78fd-43ef-b2aa-2712d09fb1b3-etc-openvswitch\") pod \"ovnkube-node-2shvk\" (UID: \"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:34:49.317544 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.314431    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/413800cd-9139-4b8b-b807-78d658dc06ba-device-dir\") pod \"aws-ebs-csi-driver-node-q2rs4\" (UID: \"413800cd-9139-4b8b-b807-78d658dc06ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q2rs4"
Jul 08 16:34:49.317544 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.314447    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/413800cd-9139-4b8b-b807-78d658dc06ba-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q2rs4\" (UID: \"413800cd-9139-4b8b-b807-78d658dc06ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q2rs4"
Jul 08 16:34:49.317544 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.314462    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0-host-run-multus-certs\") pod \"multus-bg589\" (UID: \"193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0\") " pod="openshift-multus/multus-bg589"
Jul 08 16:34:49.317544 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.314494    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f08a1b00-78fd-43ef-b2aa-2712d09fb1b3-run-ovn\") pod \"ovnkube-node-2shvk\" (UID: \"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:34:49.317544 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.314520    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f08a1b00-78fd-43ef-b2aa-2712d09fb1b3-host-cni-netd\") pod \"ovnkube-node-2shvk\" (UID: \"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:34:49.317544 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.314546    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/413800cd-9139-4b8b-b807-78d658dc06ba-sys-fs\") pod \"aws-ebs-csi-driver-node-q2rs4\" (UID: \"413800cd-9139-4b8b-b807-78d658dc06ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q2rs4"
Jul 08 16:34:49.318385 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.314572    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6cnn\" (UniqueName: \"kubernetes.io/projected/413800cd-9139-4b8b-b807-78d658dc06ba-kube-api-access-g6cnn\") pod \"aws-ebs-csi-driver-node-q2rs4\" (UID: \"413800cd-9139-4b8b-b807-78d658dc06ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q2rs4"
Jul 08 16:34:49.318385 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.314868    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0-host-run-multus-certs\") pod \"multus-bg589\" (UID: \"193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0\") " pod="openshift-multus/multus-bg589"
Jul 08 16:34:49.318385 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.314921    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f08a1b00-78fd-43ef-b2aa-2712d09fb1b3-run-ovn\") pod \"ovnkube-node-2shvk\" (UID: \"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:34:49.318385 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.314960    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f08a1b00-78fd-43ef-b2aa-2712d09fb1b3-host-cni-netd\") pod \"ovnkube-node-2shvk\" (UID: \"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:34:49.318385 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.315021    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/413800cd-9139-4b8b-b807-78d658dc06ba-sys-fs\") pod \"aws-ebs-csi-driver-node-q2rs4\" (UID: \"413800cd-9139-4b8b-b807-78d658dc06ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q2rs4"
Jul 08 16:34:49.318385 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.315049    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0-multus-daemon-config\") pod \"multus-bg589\" (UID: \"193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0\") " pod="openshift-multus/multus-bg589"
Jul 08 16:34:49.318385 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.315061    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7ce643e8-a633-411e-9d39-6f349e7ed426-iptables-alerter-script\") pod \"iptables-alerter-2bv8p\" (UID: \"7ce643e8-a633-411e-9d39-6f349e7ed426\") " pod="openshift-network-operator/iptables-alerter-2bv8p"
Jul 08 16:34:49.318385 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.314085    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0-etc-kubernetes\") pod \"multus-bg589\" (UID: \"193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0\") " pod="openshift-multus/multus-bg589"
Jul 08 16:34:49.318385 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.315138    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f08a1b00-78fd-43ef-b2aa-2712d09fb1b3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2shvk\" (UID: \"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:34:49.318385 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.315283    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/28a74820-f159-4264-af3e-c983d2c68a51-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g4stg\" (UID: \"28a74820-f159-4264-af3e-c983d2c68a51\") " pod="openshift-multus/multus-additional-cni-plugins-g4stg"
Jul 08 16:34:49.318385 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.315334    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7ce643e8-a633-411e-9d39-6f349e7ed426-host-slash\") pod \"iptables-alerter-2bv8p\" (UID: \"7ce643e8-a633-411e-9d39-6f349e7ed426\") " pod="openshift-network-operator/iptables-alerter-2bv8p"
Jul 08 16:34:49.318385 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.315336    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f08a1b00-78fd-43ef-b2aa-2712d09fb1b3-node-log\") pod \"ovnkube-node-2shvk\" (UID: \"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:34:49.318385 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.315373    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0-cnibin\") pod \"multus-bg589\" (UID: \"193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0\") " pod="openshift-multus/multus-bg589"
Jul 08 16:34:49.318385 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.315467    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e6f14c39-461f-44d2-811a-87cfb899d92c-dbus\") pod \"global-pull-secret-syncer-d9p5w\" (UID: \"e6f14c39-461f-44d2-811a-87cfb899d92c\") " pod="kube-system/global-pull-secret-syncer-d9p5w"
Jul 08 16:34:49.318385 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.316036    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f08a1b00-78fd-43ef-b2aa-2712d09fb1b3-env-overrides\") pod \"ovnkube-node-2shvk\" (UID: \"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:34:49.318385 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.315878    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f08a1b00-78fd-43ef-b2aa-2712d09fb1b3-ovnkube-config\") pod \"ovnkube-node-2shvk\" (UID: \"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:34:49.318385 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.316839    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b33e50d5-554b-4d53-aafe-3dc13ae3fa6c-tmp\") pod \"tuned-wnsct\" (UID: \"b33e50d5-554b-4d53-aafe-3dc13ae3fa6c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnsct"
Jul 08 16:34:49.318385 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.317516    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f08a1b00-78fd-43ef-b2aa-2712d09fb1b3-ovn-node-metrics-cert\") pod \"ovnkube-node-2shvk\" (UID: \"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:34:49.319051 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.318742    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b33e50d5-554b-4d53-aafe-3dc13ae3fa6c-etc-tuned\") pod \"tuned-wnsct\" (UID: \"b33e50d5-554b-4d53-aafe-3dc13ae3fa6c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnsct"
Jul 08 16:34:49.319051 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.318860    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3b861d99-7199-4174-a922-d0aae9cd010a-agent-certs\") pod \"konnectivity-agent-ht2nh\" (UID: \"3b861d99-7199-4174-a922-d0aae9cd010a\") " pod="kube-system/konnectivity-agent-ht2nh"
Jul 08 16:34:49.321645 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:49.321624    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 16:34:49.321765 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:49.321652    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 16:34:49.321765 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:49.321663    2578 projected.go:196] Error preparing data for projected volume kube-api-access-lhvdc for pod openshift-network-diagnostics/network-check-target-hzjsx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:49.321765 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:49.321754    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9bbb98bc-99e1-4401-96ab-ad9aae630c48-kube-api-access-lhvdc podName:9bbb98bc-99e1-4401-96ab-ad9aae630c48 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:49.821733814 +0000 UTC m=+2.093183335 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lhvdc" (UniqueName: "kubernetes.io/projected/9bbb98bc-99e1-4401-96ab-ad9aae630c48-kube-api-access-lhvdc") pod "network-check-target-hzjsx" (UID: "9bbb98bc-99e1-4401-96ab-ad9aae630c48") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:49.324563 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.324531    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxs62\" (UniqueName: \"kubernetes.io/projected/28a74820-f159-4264-af3e-c983d2c68a51-kube-api-access-sxs62\") pod \"multus-additional-cni-plugins-g4stg\" (UID: \"28a74820-f159-4264-af3e-c983d2c68a51\") " pod="openshift-multus/multus-additional-cni-plugins-g4stg"
Jul 08 16:34:49.325012 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.324988    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqnxm\" (UniqueName: \"kubernetes.io/projected/0f2a26b1-8d19-4874-a633-1cd61e29091a-kube-api-access-nqnxm\") pod \"network-metrics-daemon-f5jcr\" (UID: \"0f2a26b1-8d19-4874-a633-1cd61e29091a\") " pod="openshift-multus/network-metrics-daemon-f5jcr"
Jul 08 16:34:49.325330 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.325311    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrnqs\" (UniqueName: \"kubernetes.io/projected/7ce643e8-a633-411e-9d39-6f349e7ed426-kube-api-access-qrnqs\") pod \"iptables-alerter-2bv8p\" (UID: \"7ce643e8-a633-411e-9d39-6f349e7ed426\") " pod="openshift-network-operator/iptables-alerter-2bv8p"
Jul 08 16:34:49.325963 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.325933    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cfft\" (UniqueName: \"kubernetes.io/projected/b33e50d5-554b-4d53-aafe-3dc13ae3fa6c-kube-api-access-6cfft\") pod \"tuned-wnsct\" (UID: \"b33e50d5-554b-4d53-aafe-3dc13ae3fa6c\") " pod="openshift-cluster-node-tuning-operator/tuned-wnsct"
Jul 08 16:34:49.326064 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.326044    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qcxg\" (UniqueName: \"kubernetes.io/projected/f08a1b00-78fd-43ef-b2aa-2712d09fb1b3-kube-api-access-4qcxg\") pod \"ovnkube-node-2shvk\" (UID: \"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:34:49.327104 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.327081    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9pjz\" (UniqueName: \"kubernetes.io/projected/193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0-kube-api-access-f9pjz\") pod \"multus-bg589\" (UID: \"193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0\") " pod="openshift-multus/multus-bg589"
Jul 08 16:34:49.328413 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.328389    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6cnn\" (UniqueName: \"kubernetes.io/projected/413800cd-9139-4b8b-b807-78d658dc06ba-kube-api-access-g6cnn\") pod \"aws-ebs-csi-driver-node-q2rs4\" (UID: \"413800cd-9139-4b8b-b807-78d658dc06ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q2rs4"
Jul 08 16:34:49.328557 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.328539    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snvfl\" (UniqueName: \"kubernetes.io/projected/416e11c0-3ba2-451d-a901-9681ec0d4e3e-kube-api-access-snvfl\") pod \"node-ca-vnjdb\" (UID: \"416e11c0-3ba2-451d-a901-9681ec0d4e3e\") " pod="openshift-image-registry/node-ca-vnjdb"
Jul 08 16:34:49.333410 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:49.333374    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17aa1819e2ea358d71f7d6edb68bc268.slice/crio-bb23022f59a11db3eff79c6a607be2214472a663155ae9a4ed9e9a351d377efc WatchSource:0}: Error finding container bb23022f59a11db3eff79c6a607be2214472a663155ae9a4ed9e9a351d377efc: Status 404 returned error can't find the container with id bb23022f59a11db3eff79c6a607be2214472a663155ae9a4ed9e9a351d377efc
Jul 08 16:34:49.333689 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:49.333661    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70024d4eaabe656e42acf5ac910ecc55.slice/crio-d1020c40967c2b72aa1a3677b5ee2d0b4840858ae3cc382d5b2b4044449477bb WatchSource:0}: Error finding container d1020c40967c2b72aa1a3677b5ee2d0b4840858ae3cc382d5b2b4044449477bb: Status 404 returned error can't find the container with id d1020c40967c2b72aa1a3677b5ee2d0b4840858ae3cc382d5b2b4044449477bb
Jul 08 16:34:49.334596 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.334575    2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 08 16:34:49.337501 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.337481    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 08 16:34:49.432355 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.432258    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wflvt"
Jul 08 16:34:49.441023 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:49.440997    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84d1dbea_4585_46d9_b520_1329dba8df82.slice/crio-3c2f6aa8e8447e4a916d89143eb3f07e901abfcacd3e0d94620d7aded34e52b0 WatchSource:0}: Error finding container 3c2f6aa8e8447e4a916d89143eb3f07e901abfcacd3e0d94620d7aded34e52b0: Status 404 returned error can't find the container with id 3c2f6aa8e8447e4a916d89143eb3f07e901abfcacd3e0d94620d7aded34e52b0
Jul 08 16:34:49.451841 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.451818    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-g4stg"
Jul 08 16:34:49.459171 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:49.459140    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28a74820_f159_4264_af3e_c983d2c68a51.slice/crio-1c4fb4bf13119770f0ed10199186ece8f7e68bc996c5e648eff4b3eaac620860 WatchSource:0}: Error finding container 1c4fb4bf13119770f0ed10199186ece8f7e68bc996c5e648eff4b3eaac620860: Status 404 returned error can't find the container with id 1c4fb4bf13119770f0ed10199186ece8f7e68bc996c5e648eff4b3eaac620860
Jul 08 16:34:49.472784 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.472760    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vnjdb"
Jul 08 16:34:49.480757 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:49.480731    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod416e11c0_3ba2_451d_a901_9681ec0d4e3e.slice/crio-0f9a12c46893f9cf2475ae420f732f581e3c21aa409d3f05f20b427f244c2635 WatchSource:0}: Error finding container 0f9a12c46893f9cf2475ae420f732f581e3c21aa409d3f05f20b427f244c2635: Status 404 returned error can't find the container with id 0f9a12c46893f9cf2475ae420f732f581e3c21aa409d3f05f20b427f244c2635
Jul 08 16:34:49.485917 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.485895    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bg589"
Jul 08 16:34:49.493568 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:49.493534    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod193fd6c8_8f4f_4f11_9e06_fd7b8ee997c0.slice/crio-ac840adbbe6ad93d9483c576850ce2b74906337dc124de95c6fb6321ec03f56d WatchSource:0}: Error finding container ac840adbbe6ad93d9483c576850ce2b74906337dc124de95c6fb6321ec03f56d: Status 404 returned error can't find the container with id ac840adbbe6ad93d9483c576850ce2b74906337dc124de95c6fb6321ec03f56d
Jul 08 16:34:49.498999 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.498980    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2bv8p"
Jul 08 16:34:49.506563 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:49.506541    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ce643e8_a633_411e_9d39_6f349e7ed426.slice/crio-b3cb40863fa0ee952de456aeae2dd5ed3b9aedb6c3c968057c97f8bf5dc9132e WatchSource:0}: Error finding container b3cb40863fa0ee952de456aeae2dd5ed3b9aedb6c3c968057c97f8bf5dc9132e: Status 404 returned error can't find the container with id b3cb40863fa0ee952de456aeae2dd5ed3b9aedb6c3c968057c97f8bf5dc9132e
Jul 08 16:34:49.517650 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.517626    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:34:49.524796 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:49.524776    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf08a1b00_78fd_43ef_b2aa_2712d09fb1b3.slice/crio-2d005b30447c5402de5c1d2a96b0622aad0d101343af05a1adef19998ea18ad7 WatchSource:0}: Error finding container 2d005b30447c5402de5c1d2a96b0622aad0d101343af05a1adef19998ea18ad7: Status 404 returned error can't find the container with id 2d005b30447c5402de5c1d2a96b0622aad0d101343af05a1adef19998ea18ad7
Jul 08 16:34:49.539449 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.539433    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ht2nh"
Jul 08 16:34:49.546517 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:49.546494    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b861d99_7199_4174_a922_d0aae9cd010a.slice/crio-d857d8379a1da680d2680665e025d8096c270a48417bd440f9eb7ab0639cfbb8 WatchSource:0}: Error finding container d857d8379a1da680d2680665e025d8096c270a48417bd440f9eb7ab0639cfbb8: Status 404 returned error can't find the container with id d857d8379a1da680d2680665e025d8096c270a48417bd440f9eb7ab0639cfbb8
Jul 08 16:34:49.554885 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.554867    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q2rs4"
Jul 08 16:34:49.560990 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.560958    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wnsct"
Jul 08 16:34:49.562266 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:49.562243    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod413800cd_9139_4b8b_b807_78d658dc06ba.slice/crio-4b76071cf225cde6a060632d055711a2b30060427b19688edd5d0f958d23347d WatchSource:0}: Error finding container 4b76071cf225cde6a060632d055711a2b30060427b19688edd5d0f958d23347d: Status 404 returned error can't find the container with id 4b76071cf225cde6a060632d055711a2b30060427b19688edd5d0f958d23347d
Jul 08 16:34:49.569253 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:34:49.569230    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb33e50d5_554b_4d53_aafe_3dc13ae3fa6c.slice/crio-c971bacc0ad762fcf128ac5471fa6995c707e0138a38fa4111c05b370fe8955d WatchSource:0}: Error finding container c971bacc0ad762fcf128ac5471fa6995c707e0138a38fa4111c05b370fe8955d: Status 404 returned error can't find the container with id c971bacc0ad762fcf128ac5471fa6995c707e0138a38fa4111c05b370fe8955d
Jul 08 16:34:49.821320 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.820011    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e6f14c39-461f-44d2-811a-87cfb899d92c-original-pull-secret\") pod \"global-pull-secret-syncer-d9p5w\" (UID: \"e6f14c39-461f-44d2-811a-87cfb899d92c\") " pod="kube-system/global-pull-secret-syncer-d9p5w"
Jul 08 16:34:49.821320 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.820387    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f2a26b1-8d19-4874-a633-1cd61e29091a-metrics-certs\") pod \"network-metrics-daemon-f5jcr\" (UID: \"0f2a26b1-8d19-4874-a633-1cd61e29091a\") " pod="openshift-multus/network-metrics-daemon-f5jcr"
Jul 08 16:34:49.821320 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:49.820569    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:49.821320 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:49.820640    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f2a26b1-8d19-4874-a633-1cd61e29091a-metrics-certs podName:0f2a26b1-8d19-4874-a633-1cd61e29091a nodeName:}" failed. No retries permitted until 2026-07-08 16:34:50.820620731 +0000 UTC m=+3.092070279 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0f2a26b1-8d19-4874-a633-1cd61e29091a-metrics-certs") pod "network-metrics-daemon-f5jcr" (UID: "0f2a26b1-8d19-4874-a633-1cd61e29091a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:49.821320 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:49.820722    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:49.821320 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:49.820759    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6f14c39-461f-44d2-811a-87cfb899d92c-original-pull-secret podName:e6f14c39-461f-44d2-811a-87cfb899d92c nodeName:}" failed. No retries permitted until 2026-07-08 16:34:50.820746775 +0000 UTC m=+3.092196300 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e6f14c39-461f-44d2-811a-87cfb899d92c-original-pull-secret") pod "global-pull-secret-syncer-d9p5w" (UID: "e6f14c39-461f-44d2-811a-87cfb899d92c") : object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:49.921808 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:49.921770    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhvdc\" (UniqueName: \"kubernetes.io/projected/9bbb98bc-99e1-4401-96ab-ad9aae630c48-kube-api-access-lhvdc\") pod \"network-check-target-hzjsx\" (UID: \"9bbb98bc-99e1-4401-96ab-ad9aae630c48\") " pod="openshift-network-diagnostics/network-check-target-hzjsx"
Jul 08 16:34:49.922012 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:49.921990    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 16:34:49.922083 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:49.922015    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 16:34:49.922083 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:49.922029    2578 projected.go:196] Error preparing data for projected volume kube-api-access-lhvdc for pod openshift-network-diagnostics/network-check-target-hzjsx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:49.922222 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:49.922097    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9bbb98bc-99e1-4401-96ab-ad9aae630c48-kube-api-access-lhvdc podName:9bbb98bc-99e1-4401-96ab-ad9aae630c48 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:50.922076641 +0000 UTC m=+3.193526163 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-lhvdc" (UniqueName: "kubernetes.io/projected/9bbb98bc-99e1-4401-96ab-ad9aae630c48-kube-api-access-lhvdc") pod "network-check-target-hzjsx" (UID: "9bbb98bc-99e1-4401-96ab-ad9aae630c48") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:50.156875 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:50.156717    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-07 16:29:50 +0000 UTC" deadline="2027-12-10 03:25:36.121044796 +0000 UTC"
Jul 08 16:34:50.156875 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:50.156757    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12466h50m45.964292935s"
Jul 08 16:34:50.280433 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:50.280396    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hzjsx"
Jul 08 16:34:50.280710 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:50.280551    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hzjsx" podUID="9bbb98bc-99e1-4401-96ab-ad9aae630c48"
Jul 08 16:34:50.306978 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:50.306903    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-94.ec2.internal" event={"ID":"17aa1819e2ea358d71f7d6edb68bc268","Type":"ContainerStarted","Data":"bb23022f59a11db3eff79c6a607be2214472a663155ae9a4ed9e9a351d377efc"}
Jul 08 16:34:50.326483 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:50.326436    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q2rs4" event={"ID":"413800cd-9139-4b8b-b807-78d658dc06ba","Type":"ContainerStarted","Data":"4b76071cf225cde6a060632d055711a2b30060427b19688edd5d0f958d23347d"}
Jul 08 16:34:50.354730 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:50.350843    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ht2nh" event={"ID":"3b861d99-7199-4174-a922-d0aae9cd010a","Type":"ContainerStarted","Data":"d857d8379a1da680d2680665e025d8096c270a48417bd440f9eb7ab0639cfbb8"}
Jul 08 16:34:50.357621 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:50.357582    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2shvk" event={"ID":"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3","Type":"ContainerStarted","Data":"2d005b30447c5402de5c1d2a96b0622aad0d101343af05a1adef19998ea18ad7"}
Jul 08 16:34:50.374352 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:50.374223    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2bv8p" event={"ID":"7ce643e8-a633-411e-9d39-6f349e7ed426","Type":"ContainerStarted","Data":"b3cb40863fa0ee952de456aeae2dd5ed3b9aedb6c3c968057c97f8bf5dc9132e"}
Jul 08 16:34:50.391622 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:50.391493    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vnjdb" event={"ID":"416e11c0-3ba2-451d-a901-9681ec0d4e3e","Type":"ContainerStarted","Data":"0f9a12c46893f9cf2475ae420f732f581e3c21aa409d3f05f20b427f244c2635"}
Jul 08 16:34:50.418399 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:50.417987    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wflvt" event={"ID":"84d1dbea-4585-46d9-b520-1329dba8df82","Type":"ContainerStarted","Data":"3c2f6aa8e8447e4a916d89143eb3f07e901abfcacd3e0d94620d7aded34e52b0"}
Jul 08 16:34:50.443145 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:50.443022    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-94.ec2.internal" event={"ID":"70024d4eaabe656e42acf5ac910ecc55","Type":"ContainerStarted","Data":"d1020c40967c2b72aa1a3677b5ee2d0b4840858ae3cc382d5b2b4044449477bb"}
Jul 08 16:34:50.482850 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:50.482363    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wnsct" event={"ID":"b33e50d5-554b-4d53-aafe-3dc13ae3fa6c","Type":"ContainerStarted","Data":"c971bacc0ad762fcf128ac5471fa6995c707e0138a38fa4111c05b370fe8955d"}
Jul 08 16:34:50.486281 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:50.486180    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bg589" event={"ID":"193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0","Type":"ContainerStarted","Data":"ac840adbbe6ad93d9483c576850ce2b74906337dc124de95c6fb6321ec03f56d"}
Jul 08 16:34:50.496122 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:50.496025    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g4stg" event={"ID":"28a74820-f159-4264-af3e-c983d2c68a51","Type":"ContainerStarted","Data":"1c4fb4bf13119770f0ed10199186ece8f7e68bc996c5e648eff4b3eaac620860"}
Jul 08 16:34:50.591933 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:50.591895    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 08 16:34:50.828897 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:50.828805    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f2a26b1-8d19-4874-a633-1cd61e29091a-metrics-certs\") pod \"network-metrics-daemon-f5jcr\" (UID: \"0f2a26b1-8d19-4874-a633-1cd61e29091a\") " pod="openshift-multus/network-metrics-daemon-f5jcr"
Jul 08 16:34:50.829065 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:50.828905    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e6f14c39-461f-44d2-811a-87cfb899d92c-original-pull-secret\") pod \"global-pull-secret-syncer-d9p5w\" (UID: \"e6f14c39-461f-44d2-811a-87cfb899d92c\") " pod="kube-system/global-pull-secret-syncer-d9p5w"
Jul 08 16:34:50.829065 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:50.828985    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:50.829065 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:50.829002    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:50.829065 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:50.829060    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6f14c39-461f-44d2-811a-87cfb899d92c-original-pull-secret podName:e6f14c39-461f-44d2-811a-87cfb899d92c nodeName:}" failed. No retries permitted until 2026-07-08 16:34:52.829042422 +0000 UTC m=+5.100491960 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e6f14c39-461f-44d2-811a-87cfb899d92c-original-pull-secret") pod "global-pull-secret-syncer-d9p5w" (UID: "e6f14c39-461f-44d2-811a-87cfb899d92c") : object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:50.829286 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:50.829078    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f2a26b1-8d19-4874-a633-1cd61e29091a-metrics-certs podName:0f2a26b1-8d19-4874-a633-1cd61e29091a nodeName:}" failed. No retries permitted until 2026-07-08 16:34:52.829068505 +0000 UTC m=+5.100518025 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0f2a26b1-8d19-4874-a633-1cd61e29091a-metrics-certs") pod "network-metrics-daemon-f5jcr" (UID: "0f2a26b1-8d19-4874-a633-1cd61e29091a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:50.930275 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:50.930230    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhvdc\" (UniqueName: \"kubernetes.io/projected/9bbb98bc-99e1-4401-96ab-ad9aae630c48-kube-api-access-lhvdc\") pod \"network-check-target-hzjsx\" (UID: \"9bbb98bc-99e1-4401-96ab-ad9aae630c48\") " pod="openshift-network-diagnostics/network-check-target-hzjsx"
Jul 08 16:34:50.930473 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:50.930449    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 16:34:50.930473 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:50.930472    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 16:34:50.930582 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:50.930485    2578 projected.go:196] Error preparing data for projected volume kube-api-access-lhvdc for pod openshift-network-diagnostics/network-check-target-hzjsx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:50.930582 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:50.930548    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9bbb98bc-99e1-4401-96ab-ad9aae630c48-kube-api-access-lhvdc podName:9bbb98bc-99e1-4401-96ab-ad9aae630c48 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:52.930529375 +0000 UTC m=+5.201978897 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-lhvdc" (UniqueName: "kubernetes.io/projected/9bbb98bc-99e1-4401-96ab-ad9aae630c48-kube-api-access-lhvdc") pod "network-check-target-hzjsx" (UID: "9bbb98bc-99e1-4401-96ab-ad9aae630c48") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:51.157865 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:51.157700    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-07 16:29:50 +0000 UTC" deadline="2028-01-07 03:51:07.937051796 +0000 UTC"
Jul 08 16:34:51.157865 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:51.157769    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13139h16m16.779288188s"
Jul 08 16:34:51.281109 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:51.280368    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d9p5w"
Jul 08 16:34:51.281109 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:51.280512    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d9p5w" podUID="e6f14c39-461f-44d2-811a-87cfb899d92c"
Jul 08 16:34:51.281109 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:51.280950    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f5jcr"
Jul 08 16:34:51.281109 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:51.281060    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f5jcr" podUID="0f2a26b1-8d19-4874-a633-1cd61e29091a"
Jul 08 16:34:51.435154 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:51.435056    2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 08 16:34:52.283038 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:52.282982    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hzjsx"
Jul 08 16:34:52.283538 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:52.283115    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hzjsx" podUID="9bbb98bc-99e1-4401-96ab-ad9aae630c48"
Jul 08 16:34:52.848566 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:52.848531    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f2a26b1-8d19-4874-a633-1cd61e29091a-metrics-certs\") pod \"network-metrics-daemon-f5jcr\" (UID: \"0f2a26b1-8d19-4874-a633-1cd61e29091a\") " pod="openshift-multus/network-metrics-daemon-f5jcr"
Jul 08 16:34:52.848796 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:52.848586    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e6f14c39-461f-44d2-811a-87cfb899d92c-original-pull-secret\") pod \"global-pull-secret-syncer-d9p5w\" (UID: \"e6f14c39-461f-44d2-811a-87cfb899d92c\") " pod="kube-system/global-pull-secret-syncer-d9p5w"
Jul 08 16:34:52.848796 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:52.848733    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:52.848796 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:52.848791    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6f14c39-461f-44d2-811a-87cfb899d92c-original-pull-secret podName:e6f14c39-461f-44d2-811a-87cfb899d92c nodeName:}" failed. No retries permitted until 2026-07-08 16:34:56.848773998 +0000 UTC m=+9.120223518 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e6f14c39-461f-44d2-811a-87cfb899d92c-original-pull-secret") pod "global-pull-secret-syncer-d9p5w" (UID: "e6f14c39-461f-44d2-811a-87cfb899d92c") : object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:52.849204 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:52.849171    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:52.849313 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:52.849242    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f2a26b1-8d19-4874-a633-1cd61e29091a-metrics-certs podName:0f2a26b1-8d19-4874-a633-1cd61e29091a nodeName:}" failed. No retries permitted until 2026-07-08 16:34:56.849225969 +0000 UTC m=+9.120675506 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0f2a26b1-8d19-4874-a633-1cd61e29091a-metrics-certs") pod "network-metrics-daemon-f5jcr" (UID: "0f2a26b1-8d19-4874-a633-1cd61e29091a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:52.949370 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:52.949288    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhvdc\" (UniqueName: \"kubernetes.io/projected/9bbb98bc-99e1-4401-96ab-ad9aae630c48-kube-api-access-lhvdc\") pod \"network-check-target-hzjsx\" (UID: \"9bbb98bc-99e1-4401-96ab-ad9aae630c48\") " pod="openshift-network-diagnostics/network-check-target-hzjsx"
Jul 08 16:34:52.949560 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:52.949487    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 16:34:52.949560 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:52.949514    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 16:34:52.949560 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:52.949527    2578 projected.go:196] Error preparing data for projected volume kube-api-access-lhvdc for pod openshift-network-diagnostics/network-check-target-hzjsx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:52.949754 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:52.949586    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9bbb98bc-99e1-4401-96ab-ad9aae630c48-kube-api-access-lhvdc podName:9bbb98bc-99e1-4401-96ab-ad9aae630c48 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:56.949564896 +0000 UTC m=+9.221014431 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-lhvdc" (UniqueName: "kubernetes.io/projected/9bbb98bc-99e1-4401-96ab-ad9aae630c48-kube-api-access-lhvdc") pod "network-check-target-hzjsx" (UID: "9bbb98bc-99e1-4401-96ab-ad9aae630c48") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:53.280996 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:53.280906    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d9p5w"
Jul 08 16:34:53.281171 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:53.280924    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f5jcr"
Jul 08 16:34:53.281171 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:53.281066    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d9p5w" podUID="e6f14c39-461f-44d2-811a-87cfb899d92c"
Jul 08 16:34:53.281297 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:53.281162    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f5jcr" podUID="0f2a26b1-8d19-4874-a633-1cd61e29091a"
Jul 08 16:34:54.282430 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:54.281904    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hzjsx"
Jul 08 16:34:54.282430 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:54.282042    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hzjsx" podUID="9bbb98bc-99e1-4401-96ab-ad9aae630c48"
Jul 08 16:34:55.280041 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:55.280005    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d9p5w"
Jul 08 16:34:55.280241 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:55.280163    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d9p5w" podUID="e6f14c39-461f-44d2-811a-87cfb899d92c"
Jul 08 16:34:55.280241 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:55.280216    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f5jcr"
Jul 08 16:34:55.280395 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:55.280360    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f5jcr" podUID="0f2a26b1-8d19-4874-a633-1cd61e29091a"
Jul 08 16:34:56.281360 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:56.280775    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hzjsx"
Jul 08 16:34:56.281360 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:56.280909    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hzjsx" podUID="9bbb98bc-99e1-4401-96ab-ad9aae630c48"
Jul 08 16:34:56.883093 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:56.883052    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f2a26b1-8d19-4874-a633-1cd61e29091a-metrics-certs\") pod \"network-metrics-daemon-f5jcr\" (UID: \"0f2a26b1-8d19-4874-a633-1cd61e29091a\") " pod="openshift-multus/network-metrics-daemon-f5jcr"
Jul 08 16:34:56.883296 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:56.883123    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e6f14c39-461f-44d2-811a-87cfb899d92c-original-pull-secret\") pod \"global-pull-secret-syncer-d9p5w\" (UID: \"e6f14c39-461f-44d2-811a-87cfb899d92c\") " pod="kube-system/global-pull-secret-syncer-d9p5w"
Jul 08 16:34:56.883296 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:56.883254    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:56.883296 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:56.883288    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:56.883438 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:56.883337    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f2a26b1-8d19-4874-a633-1cd61e29091a-metrics-certs podName:0f2a26b1-8d19-4874-a633-1cd61e29091a nodeName:}" failed. No retries permitted until 2026-07-08 16:35:04.883314878 +0000 UTC m=+17.154764401 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0f2a26b1-8d19-4874-a633-1cd61e29091a-metrics-certs") pod "network-metrics-daemon-f5jcr" (UID: "0f2a26b1-8d19-4874-a633-1cd61e29091a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:56.883438 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:56.883385    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6f14c39-461f-44d2-811a-87cfb899d92c-original-pull-secret podName:e6f14c39-461f-44d2-811a-87cfb899d92c nodeName:}" failed. No retries permitted until 2026-07-08 16:35:04.883372557 +0000 UTC m=+17.154822083 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e6f14c39-461f-44d2-811a-87cfb899d92c-original-pull-secret") pod "global-pull-secret-syncer-d9p5w" (UID: "e6f14c39-461f-44d2-811a-87cfb899d92c") : object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:56.984042 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:56.983928    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhvdc\" (UniqueName: \"kubernetes.io/projected/9bbb98bc-99e1-4401-96ab-ad9aae630c48-kube-api-access-lhvdc\") pod \"network-check-target-hzjsx\" (UID: \"9bbb98bc-99e1-4401-96ab-ad9aae630c48\") " pod="openshift-network-diagnostics/network-check-target-hzjsx"
Jul 08 16:34:56.984204 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:56.984143    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 16:34:56.984204 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:56.984170    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 16:34:56.984204 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:56.984184    2578 projected.go:196] Error preparing data for projected volume kube-api-access-lhvdc for pod openshift-network-diagnostics/network-check-target-hzjsx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:56.984350 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:56.984257    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9bbb98bc-99e1-4401-96ab-ad9aae630c48-kube-api-access-lhvdc podName:9bbb98bc-99e1-4401-96ab-ad9aae630c48 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:04.984235636 +0000 UTC m=+17.255685160 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-lhvdc" (UniqueName: "kubernetes.io/projected/9bbb98bc-99e1-4401-96ab-ad9aae630c48-kube-api-access-lhvdc") pod "network-check-target-hzjsx" (UID: "9bbb98bc-99e1-4401-96ab-ad9aae630c48") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:57.281005 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:57.280907    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f5jcr"
Jul 08 16:34:57.281190 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:57.281079    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f5jcr" podUID="0f2a26b1-8d19-4874-a633-1cd61e29091a"
Jul 08 16:34:57.281262 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:57.281187    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d9p5w"
Jul 08 16:34:57.281323 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:57.281302    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d9p5w" podUID="e6f14c39-461f-44d2-811a-87cfb899d92c"
Jul 08 16:34:58.282113 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:58.281967    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hzjsx"
Jul 08 16:34:58.282113 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:58.282091    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hzjsx" podUID="9bbb98bc-99e1-4401-96ab-ad9aae630c48"
Jul 08 16:34:59.281184 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:59.280195    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f5jcr"
Jul 08 16:34:59.281184 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:34:59.280228    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d9p5w"
Jul 08 16:34:59.281184 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:59.280763    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d9p5w" podUID="e6f14c39-461f-44d2-811a-87cfb899d92c"
Jul 08 16:34:59.281184 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:34:59.280767    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f5jcr" podUID="0f2a26b1-8d19-4874-a633-1cd61e29091a"
Jul 08 16:35:00.280183 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:00.280143    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hzjsx"
Jul 08 16:35:00.280707 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:00.280292    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hzjsx" podUID="9bbb98bc-99e1-4401-96ab-ad9aae630c48"
Jul 08 16:35:01.280198 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:01.280151    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f5jcr"
Jul 08 16:35:01.280198 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:01.280177    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d9p5w"
Jul 08 16:35:01.280712 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:01.280304    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f5jcr" podUID="0f2a26b1-8d19-4874-a633-1cd61e29091a"
Jul 08 16:35:01.280712 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:01.280424    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d9p5w" podUID="e6f14c39-461f-44d2-811a-87cfb899d92c"
Jul 08 16:35:02.283167 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:02.283142    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hzjsx"
Jul 08 16:35:02.283493 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:02.283233    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hzjsx" podUID="9bbb98bc-99e1-4401-96ab-ad9aae630c48"
Jul 08 16:35:03.279975 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:03.279929    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d9p5w"
Jul 08 16:35:03.280169 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:03.279938    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f5jcr"
Jul 08 16:35:03.280169 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:03.280071    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d9p5w" podUID="e6f14c39-461f-44d2-811a-87cfb899d92c"
Jul 08 16:35:03.280169 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:03.280155    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f5jcr" podUID="0f2a26b1-8d19-4874-a633-1cd61e29091a"
Jul 08 16:35:04.280872 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:04.280835    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hzjsx"
Jul 08 16:35:04.281294 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:04.280970    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hzjsx" podUID="9bbb98bc-99e1-4401-96ab-ad9aae630c48"
Jul 08 16:35:04.950186 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:04.950142    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f2a26b1-8d19-4874-a633-1cd61e29091a-metrics-certs\") pod \"network-metrics-daemon-f5jcr\" (UID: \"0f2a26b1-8d19-4874-a633-1cd61e29091a\") " pod="openshift-multus/network-metrics-daemon-f5jcr"
Jul 08 16:35:04.950386 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:04.950207    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e6f14c39-461f-44d2-811a-87cfb899d92c-original-pull-secret\") pod \"global-pull-secret-syncer-d9p5w\" (UID: \"e6f14c39-461f-44d2-811a-87cfb899d92c\") " pod="kube-system/global-pull-secret-syncer-d9p5w"
Jul 08 16:35:04.950386 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:04.950324    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:35:04.950386 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:04.950334    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 16:35:04.950513 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:04.950397    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f2a26b1-8d19-4874-a633-1cd61e29091a-metrics-certs podName:0f2a26b1-8d19-4874-a633-1cd61e29091a nodeName:}" failed. No retries permitted until 2026-07-08 16:35:20.950380544 +0000 UTC m=+33.221830066 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0f2a26b1-8d19-4874-a633-1cd61e29091a-metrics-certs") pod "network-metrics-daemon-f5jcr" (UID: "0f2a26b1-8d19-4874-a633-1cd61e29091a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:35:04.950513 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:04.950413    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6f14c39-461f-44d2-811a-87cfb899d92c-original-pull-secret podName:e6f14c39-461f-44d2-811a-87cfb899d92c nodeName:}" failed. No retries permitted until 2026-07-08 16:35:20.950406848 +0000 UTC m=+33.221856366 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e6f14c39-461f-44d2-811a-87cfb899d92c-original-pull-secret") pod "global-pull-secret-syncer-d9p5w" (UID: "e6f14c39-461f-44d2-811a-87cfb899d92c") : object "kube-system"/"original-pull-secret" not registered
Jul 08 16:35:05.051263 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:05.051231    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhvdc\" (UniqueName: \"kubernetes.io/projected/9bbb98bc-99e1-4401-96ab-ad9aae630c48-kube-api-access-lhvdc\") pod \"network-check-target-hzjsx\" (UID: \"9bbb98bc-99e1-4401-96ab-ad9aae630c48\") " pod="openshift-network-diagnostics/network-check-target-hzjsx"
Jul 08 16:35:05.051465 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:05.051441    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 16:35:05.051514 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:05.051475    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 16:35:05.051514 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:05.051490    2578 projected.go:196] Error preparing data for projected volume kube-api-access-lhvdc for pod openshift-network-diagnostics/network-check-target-hzjsx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:35:05.051607 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:05.051562    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9bbb98bc-99e1-4401-96ab-ad9aae630c48-kube-api-access-lhvdc podName:9bbb98bc-99e1-4401-96ab-ad9aae630c48 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:21.051544164 +0000 UTC m=+33.322993685 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-lhvdc" (UniqueName: "kubernetes.io/projected/9bbb98bc-99e1-4401-96ab-ad9aae630c48-kube-api-access-lhvdc") pod "network-check-target-hzjsx" (UID: "9bbb98bc-99e1-4401-96ab-ad9aae630c48") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:35:05.280279 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:05.280203    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d9p5w"
Jul 08 16:35:05.280503 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:05.280318    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d9p5w" podUID="e6f14c39-461f-44d2-811a-87cfb899d92c"
Jul 08 16:35:05.280503 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:05.280375    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f5jcr"
Jul 08 16:35:05.280654 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:05.280497    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f5jcr" podUID="0f2a26b1-8d19-4874-a633-1cd61e29091a"
Jul 08 16:35:06.280909 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:06.280824    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hzjsx"
Jul 08 16:35:06.281380 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:06.280953    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hzjsx" podUID="9bbb98bc-99e1-4401-96ab-ad9aae630c48"
Jul 08 16:35:07.280771 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:07.280731    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d9p5w"
Jul 08 16:35:07.280954 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:07.280855    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d9p5w" podUID="e6f14c39-461f-44d2-811a-87cfb899d92c"
Jul 08 16:35:07.281335 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:07.281302    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f5jcr"
Jul 08 16:35:07.281453 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:07.281434    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f5jcr" podUID="0f2a26b1-8d19-4874-a633-1cd61e29091a"
Jul 08 16:35:08.281435 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:08.281400    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hzjsx"
Jul 08 16:35:08.281883 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:08.281521    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hzjsx" podUID="9bbb98bc-99e1-4401-96ab-ad9aae630c48"
Jul 08 16:35:09.280851 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:09.280816    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f5jcr"
Jul 08 16:35:09.281049 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:09.280961    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f5jcr" podUID="0f2a26b1-8d19-4874-a633-1cd61e29091a"
Jul 08 16:35:09.281049 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:09.281022    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d9p5w"
Jul 08 16:35:09.281149 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:09.281135    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d9p5w" podUID="e6f14c39-461f-44d2-811a-87cfb899d92c"
Jul 08 16:35:10.280541 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:10.280497    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hzjsx"
Jul 08 16:35:10.281008 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:10.280627    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hzjsx" podUID="9bbb98bc-99e1-4401-96ab-ad9aae630c48"
Jul 08 16:35:11.280866 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:11.280832    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d9p5w"
Jul 08 16:35:11.281365 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:11.280832    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f5jcr"
Jul 08 16:35:11.281365 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:11.280963    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d9p5w" podUID="e6f14c39-461f-44d2-811a-87cfb899d92c"
Jul 08 16:35:11.281365 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:11.281006    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f5jcr" podUID="0f2a26b1-8d19-4874-a633-1cd61e29091a"
Jul 08 16:35:11.556137 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:11.555948    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2shvk_f08a1b00-78fd-43ef-b2aa-2712d09fb1b3/ovn-acl-logging/0.log"
Jul 08 16:35:11.556373 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:11.556345    2578 generic.go:356] "Generic (PLEG): container finished" podID="f08a1b00-78fd-43ef-b2aa-2712d09fb1b3" containerID="a520c597014d7bf55bcb943ae07ec1d4c07f001ba62d80c5ed336072ecb29b59" exitCode=1
Jul 08 16:35:11.556458 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:11.556435    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2shvk" event={"ID":"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3","Type":"ContainerStarted","Data":"f9bd68abbd0cf0f309d7d8f06ff45bc2ff69859f249fcf5a17654b2977907d8f"}
Jul 08 16:35:11.556531 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:11.556514    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2shvk" event={"ID":"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3","Type":"ContainerStarted","Data":"150dec7e0ab05be9f2131b51d751b4dc24423c6f9501de6edafd9fa0a0e40b5b"}
Jul 08 16:35:11.556590 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:11.556572    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2shvk" event={"ID":"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3","Type":"ContainerStarted","Data":"6af36798a1f388d6faf399fb996b52fd858ae389d5a9c79b40ab2efc366fd76d"}
Jul 08 16:35:11.556752 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:11.556619    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2shvk" event={"ID":"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3","Type":"ContainerStarted","Data":"d50c02e6ca4c2d64508505ebd306c136696177b6e88eb847719fcedc1e529eb8"}
Jul 08 16:35:11.556752 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:11.556686    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2shvk" event={"ID":"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3","Type":"ContainerDied","Data":"a520c597014d7bf55bcb943ae07ec1d4c07f001ba62d80c5ed336072ecb29b59"}
Jul 08 16:35:11.556752 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:11.556736    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2shvk" event={"ID":"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3","Type":"ContainerStarted","Data":"8835309918ca0ecda0573bafbae9f8895a84951b13f22615f5cca3ab0af38274"}
Jul 08 16:35:11.561594 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:11.559002    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-94.ec2.internal" event={"ID":"70024d4eaabe656e42acf5ac910ecc55","Type":"ContainerStarted","Data":"145652cf1ad162e11d5a09127914a49a490b92cb0828d18d1ba686f6c582748b"}
Jul 08 16:35:11.562781 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:11.562167    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wnsct" event={"ID":"b33e50d5-554b-4d53-aafe-3dc13ae3fa6c","Type":"ContainerStarted","Data":"3c34d90ee5d298bccaaebc86199736527059600bb8acfbcb6f9db99f860776c5"}
Jul 08 16:35:11.564825 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:11.564788    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bg589" event={"ID":"193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0","Type":"ContainerStarted","Data":"ff3d3f31f11b563907e300257a2cf1a65fc48eb8d38f032f3b38c66a460885ec"}
Jul 08 16:35:11.573705 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:11.573653    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-94.ec2.internal" podStartSLOduration=22.573619815 podStartE2EDuration="22.573619815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:34:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:11.572972406 +0000 UTC m=+23.844421947" watchObservedRunningTime="2026-07-08 16:35:11.573619815 +0000 UTC m=+23.845069355"
Jul 08 16:35:11.613294 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:11.612099    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-wnsct" podStartSLOduration=1.618791576 podStartE2EDuration="22.612085163s" totalImagesPullingTime="20.993293587s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:34:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:11.611822109 +0000 UTC m=+23.883271650" watchObservedRunningTime="2026-07-08 16:35:11.612085163 +0000 UTC m=+23.883534746"
Jul 08 16:35:11.613294 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:11.612543    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-bg589" podStartSLOduration=1.072516646 podStartE2EDuration="22.612533648s" totalImagesPullingTime="21.540017002s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:34:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:11.594849477 +0000 UTC m=+23.866299017" watchObservedRunningTime="2026-07-08 16:35:11.612533648 +0000 UTC m=+23.883983189"
Jul 08 16:35:12.280680 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:12.280622    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hzjsx"
Jul 08 16:35:12.280843 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:12.280787    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hzjsx" podUID="9bbb98bc-99e1-4401-96ab-ad9aae630c48"
Jul 08 16:35:12.567973 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:12.567939    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q2rs4" event={"ID":"413800cd-9139-4b8b-b807-78d658dc06ba","Type":"ContainerStarted","Data":"64bea7652952ddf2d2c0b7cd91ebdb4da965ae2c1f4ec7ab835383ab96741557"}
Jul 08 16:35:12.569385 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:12.569354    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ht2nh" event={"ID":"3b861d99-7199-4174-a922-d0aae9cd010a","Type":"ContainerStarted","Data":"2c6e4da441f68d7de1b5f9353e5b0285e467c97499b6315ef3275cba797a362e"}
Jul 08 16:35:12.570731 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:12.570710    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2bv8p" event={"ID":"7ce643e8-a633-411e-9d39-6f349e7ed426","Type":"ContainerStarted","Data":"0c7f11d191d6640e9bbcda059915c86a8e44fdc366bfaec2b0d10234cb098373"}
Jul 08 16:35:12.572128 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:12.572103    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vnjdb" event={"ID":"416e11c0-3ba2-451d-a901-9681ec0d4e3e","Type":"ContainerStarted","Data":"dc4d56550ae0565115ae1bca637547efb902e3ea6619632c21af3a0bdfd78b8a"}
Jul 08 16:35:12.573495 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:12.573472    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wflvt" event={"ID":"84d1dbea-4585-46d9-b520-1329dba8df82","Type":"ContainerStarted","Data":"f01c1a95666d1d6e538e66bdb391378c05debde789cd4ced6fc80ca638184657"}
Jul 08 16:35:12.574831 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:12.574806    2578 generic.go:356] "Generic (PLEG): container finished" podID="28a74820-f159-4264-af3e-c983d2c68a51" containerID="390b52b5c783fc13f98c4bab97090cb88f8689956ac7e4f9412e123ce7245b84" exitCode=0
Jul 08 16:35:12.574916 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:12.574886    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g4stg" event={"ID":"28a74820-f159-4264-af3e-c983d2c68a51","Type":"ContainerDied","Data":"390b52b5c783fc13f98c4bab97090cb88f8689956ac7e4f9412e123ce7245b84"}
Jul 08 16:35:12.576187 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:12.576168    2578 generic.go:356] "Generic (PLEG): container finished" podID="17aa1819e2ea358d71f7d6edb68bc268" containerID="1633fcb70272428944b0468cdbf88b874515ce63e95fc42b90d8aa0c3f1fac8d" exitCode=0
Jul 08 16:35:12.576271 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:12.576197    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-94.ec2.internal" event={"ID":"17aa1819e2ea358d71f7d6edb68bc268","Type":"ContainerDied","Data":"1633fcb70272428944b0468cdbf88b874515ce63e95fc42b90d8aa0c3f1fac8d"}
Jul 08 16:35:12.590332 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:12.590280    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-ht2nh" podStartSLOduration=2.965509526 podStartE2EDuration="23.590263028s" totalImagesPullingTime="20.624753502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:34:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:12.585271666 +0000 UTC m=+24.856721205" watchObservedRunningTime="2026-07-08 16:35:12.590263028 +0000 UTC m=+24.861712580"
Jul 08 16:35:12.632195 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:12.632129    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-wflvt" podStartSLOduration=2.533302238 podStartE2EDuration="23.632113448s" totalImagesPullingTime="21.09881121s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:34:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:12.631435105 +0000 UTC m=+24.902884646" watchObservedRunningTime="2026-07-08 16:35:12.632113448 +0000 UTC m=+24.903562987"
Jul 08 16:35:12.646007 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:12.645968    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vnjdb" podStartSLOduration=2.955075266 podStartE2EDuration="23.645954505s" totalImagesPullingTime="20.690879239s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:34:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:12.645771059 +0000 UTC m=+24.917220789" watchObservedRunningTime="2026-07-08 16:35:12.645954505 +0000 UTC m=+24.917404045"
Jul 08 16:35:12.819127 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:12.819107    2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 08 16:35:13.171859 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:13.171599    2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-08T16:35:12.819124753Z","UUID":"f3549956-81e7-45b5-83ff-d6564edecd23","Handler":null,"Name":"","Endpoint":""}
Jul 08 16:35:13.173284 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:13.173257    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 08 16:35:13.173413 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:13.173302    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 08 16:35:13.280867 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:13.280833    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f5jcr"
Jul 08 16:35:13.281101 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:13.280882    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d9p5w"
Jul 08 16:35:13.281101 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:13.280971    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f5jcr" podUID="0f2a26b1-8d19-4874-a633-1cd61e29091a"
Jul 08 16:35:13.281101 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:13.281082    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d9p5w" podUID="e6f14c39-461f-44d2-811a-87cfb899d92c"
Jul 08 16:35:13.581095 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:13.581060    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-94.ec2.internal" event={"ID":"17aa1819e2ea358d71f7d6edb68bc268","Type":"ContainerStarted","Data":"403cb51d814f2a8142bd1ea7de4a9db192f2ab5f510cd5e635091e9cdc407406"}
Jul 08 16:35:13.583550 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:13.583522    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q2rs4" event={"ID":"413800cd-9139-4b8b-b807-78d658dc06ba","Type":"ContainerStarted","Data":"a4b2c0ff5424f0587f06f57c743107228c206425dfac082f8aeae77d3f079361"}
Jul 08 16:35:13.597329 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:13.597250    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-94.ec2.internal" podStartSLOduration=24.597235459 podStartE2EDuration="24.597235459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:34:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:13.596730356 +0000 UTC m=+25.868179897" watchObservedRunningTime="2026-07-08 16:35:13.597235459 +0000 UTC m=+25.868684999"
Jul 08 16:35:13.597575 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:13.597521    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2bv8p" podStartSLOduration=3.542848416 podStartE2EDuration="24.597512422s" totalImagesPullingTime="21.054664006s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:34:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:12.660093062 +0000 UTC m=+24.931542608" watchObservedRunningTime="2026-07-08 16:35:13.597512422 +0000 UTC m=+25.868961962"
Jul 08 16:35:13.855538 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:13.855450    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-ht2nh"
Jul 08 16:35:13.856084 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:13.856065    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-ht2nh"
Jul 08 16:35:14.280320 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:14.280232    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hzjsx"
Jul 08 16:35:14.280486 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:14.280367    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hzjsx" podUID="9bbb98bc-99e1-4401-96ab-ad9aae630c48"
Jul 08 16:35:14.588097 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:14.588061    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q2rs4" event={"ID":"413800cd-9139-4b8b-b807-78d658dc06ba","Type":"ContainerStarted","Data":"c56c11146e32d7ae463beb2e1d0080cb493f9ae9fc9960aa8dd5ddf660cf97a0"}
Jul 08 16:35:14.592119 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:14.592095    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2shvk_f08a1b00-78fd-43ef-b2aa-2712d09fb1b3/ovn-acl-logging/0.log"
Jul 08 16:35:14.592554 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:14.592528    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2shvk" event={"ID":"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3","Type":"ContainerStarted","Data":"6d5840822b93d040433547723169b0845f09b9c408dfc02e18d548566f938c26"}
Jul 08 16:35:14.592907 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:14.592884    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-ht2nh"
Jul 08 16:35:14.593457 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:14.593436    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ht2nh"
Jul 08 16:35:14.605817 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:14.605751    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q2rs4" podStartSLOduration=2.758409747 podStartE2EDuration="25.605731175s" totalImagesPullingTime="22.847321428s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:34:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:14.605357469 +0000 UTC m=+26.876807008" watchObservedRunningTime="2026-07-08 16:35:14.605731175 +0000 UTC m=+26.877180716"
Jul 08 16:35:15.280606 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:15.280576    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d9p5w"
Jul 08 16:35:15.280606 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:15.280605    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f5jcr"
Jul 08 16:35:15.280815 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:15.280701    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d9p5w" podUID="e6f14c39-461f-44d2-811a-87cfb899d92c"
Jul 08 16:35:15.280815 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:15.280779    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f5jcr" podUID="0f2a26b1-8d19-4874-a633-1cd61e29091a"
Jul 08 16:35:16.280845 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:16.280815    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hzjsx"
Jul 08 16:35:16.281305 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:16.280949    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hzjsx" podUID="9bbb98bc-99e1-4401-96ab-ad9aae630c48"
Jul 08 16:35:17.280800 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:17.280660    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d9p5w"
Jul 08 16:35:17.280902 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:17.280694    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f5jcr"
Jul 08 16:35:17.280902 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:17.280874    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d9p5w" podUID="e6f14c39-461f-44d2-811a-87cfb899d92c"
Jul 08 16:35:17.281468 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:17.280940    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f5jcr" podUID="0f2a26b1-8d19-4874-a633-1cd61e29091a"
Jul 08 16:35:17.600816 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:17.600785    2578 generic.go:356] "Generic (PLEG): container finished" podID="28a74820-f159-4264-af3e-c983d2c68a51" containerID="325a14e2d4e063df7f44401679e6acc0c2c462450d77de5355bc4f53881c4021" exitCode=0
Jul 08 16:35:17.601000 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:17.600877    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g4stg" event={"ID":"28a74820-f159-4264-af3e-c983d2c68a51","Type":"ContainerDied","Data":"325a14e2d4e063df7f44401679e6acc0c2c462450d77de5355bc4f53881c4021"}
Jul 08 16:35:17.604609 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:17.604591    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2shvk_f08a1b00-78fd-43ef-b2aa-2712d09fb1b3/ovn-acl-logging/0.log"
Jul 08 16:35:17.604944 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:17.604925    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2shvk" event={"ID":"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3","Type":"ContainerStarted","Data":"66d19ad9fd0a5d8293512922112ccc315500780f989a61be07a5125d9c8c669b"}
Jul 08 16:35:17.605217 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:17.605178    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:35:17.605304 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:17.605228    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:35:17.605412 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:17.605394    2578 scope.go:117] "RemoveContainer" containerID="a520c597014d7bf55bcb943ae07ec1d4c07f001ba62d80c5ed336072ecb29b59"
Jul 08 16:35:17.621200 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:17.621179    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:35:17.621881 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:17.621866    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:35:18.281455 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:18.281264    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hzjsx"
Jul 08 16:35:18.281899 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:18.281519    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hzjsx" podUID="9bbb98bc-99e1-4401-96ab-ad9aae630c48"
Jul 08 16:35:18.610243 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:18.610219    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2shvk_f08a1b00-78fd-43ef-b2aa-2712d09fb1b3/ovn-acl-logging/0.log"
Jul 08 16:35:18.610591 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:18.610560    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2shvk" event={"ID":"f08a1b00-78fd-43ef-b2aa-2712d09fb1b3","Type":"ContainerStarted","Data":"d340c0466dab65afa03672fe9ea9df871fcde662bea1d0aa85720041c9408d70"}
Jul 08 16:35:18.610676 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:18.610651    2578 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 08 16:35:18.913501 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:18.913456    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2shvk" podStartSLOduration=8.8291773 podStartE2EDuration="29.913439813s" totalImagesPullingTime="21.084262513s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:34:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:18.642119697 +0000 UTC m=+30.913569238" watchObservedRunningTime="2026-07-08 16:35:18.913439813 +0000 UTC m=+31.184889356"
Jul 08 16:35:18.913697 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:18.913680    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-d9p5w"]
Jul 08 16:35:18.913831 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:18.913819    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d9p5w"
Jul 08 16:35:18.913927 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:18.913910    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d9p5w" podUID="e6f14c39-461f-44d2-811a-87cfb899d92c"
Jul 08 16:35:18.917337 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:18.917287    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hzjsx"]
Jul 08 16:35:18.917443 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:18.917395    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hzjsx"
Jul 08 16:35:18.917504 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:18.917484    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hzjsx" podUID="9bbb98bc-99e1-4401-96ab-ad9aae630c48"
Jul 08 16:35:18.918077 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:18.918052    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-f5jcr"]
Jul 08 16:35:18.918196 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:18.918170    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f5jcr"
Jul 08 16:35:18.918290 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:18.918265    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f5jcr" podUID="0f2a26b1-8d19-4874-a633-1cd61e29091a"
Jul 08 16:35:19.161234 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:19.161155    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:35:19.614809 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:19.614782    2578 generic.go:356] "Generic (PLEG): container finished" podID="28a74820-f159-4264-af3e-c983d2c68a51" containerID="6497a8ebe6088fb505e9e3140af39b178a8c557c71893ef452c8ce968aa2e8e6" exitCode=0
Jul 08 16:35:19.615232 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:19.614865    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g4stg" event={"ID":"28a74820-f159-4264-af3e-c983d2c68a51","Type":"ContainerDied","Data":"6497a8ebe6088fb505e9e3140af39b178a8c557c71893ef452c8ce968aa2e8e6"}
Jul 08 16:35:20.280864 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:20.280788    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hzjsx"
Jul 08 16:35:20.281012 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:20.280789    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f5jcr"
Jul 08 16:35:20.281012 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:20.280921    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hzjsx" podUID="9bbb98bc-99e1-4401-96ab-ad9aae630c48"
Jul 08 16:35:20.281012 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:20.280789    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d9p5w"
Jul 08 16:35:20.281012 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:20.280991    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f5jcr" podUID="0f2a26b1-8d19-4874-a633-1cd61e29091a"
Jul 08 16:35:20.281208 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:20.281105    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d9p5w" podUID="e6f14c39-461f-44d2-811a-87cfb899d92c"
Jul 08 16:35:20.975744 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:20.975663    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f2a26b1-8d19-4874-a633-1cd61e29091a-metrics-certs\") pod \"network-metrics-daemon-f5jcr\" (UID: \"0f2a26b1-8d19-4874-a633-1cd61e29091a\") " pod="openshift-multus/network-metrics-daemon-f5jcr"
Jul 08 16:35:20.975744 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:20.975708    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e6f14c39-461f-44d2-811a-87cfb899d92c-original-pull-secret\") pod \"global-pull-secret-syncer-d9p5w\" (UID: \"e6f14c39-461f-44d2-811a-87cfb899d92c\") " pod="kube-system/global-pull-secret-syncer-d9p5w"
Jul 08 16:35:20.976111 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:20.975808    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 16:35:20.976111 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:20.975808    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:35:20.976111 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:20.975869    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f2a26b1-8d19-4874-a633-1cd61e29091a-metrics-certs podName:0f2a26b1-8d19-4874-a633-1cd61e29091a nodeName:}" failed. No retries permitted until 2026-07-08 16:35:52.975852795 +0000 UTC m=+65.247302333 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0f2a26b1-8d19-4874-a633-1cd61e29091a-metrics-certs") pod "network-metrics-daemon-f5jcr" (UID: "0f2a26b1-8d19-4874-a633-1cd61e29091a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:35:20.976111 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:20.975883    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6f14c39-461f-44d2-811a-87cfb899d92c-original-pull-secret podName:e6f14c39-461f-44d2-811a-87cfb899d92c nodeName:}" failed. No retries permitted until 2026-07-08 16:35:52.975877324 +0000 UTC m=+65.247326842 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e6f14c39-461f-44d2-811a-87cfb899d92c-original-pull-secret") pod "global-pull-secret-syncer-d9p5w" (UID: "e6f14c39-461f-44d2-811a-87cfb899d92c") : object "kube-system"/"original-pull-secret" not registered
Jul 08 16:35:21.076616 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:21.076584    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhvdc\" (UniqueName: \"kubernetes.io/projected/9bbb98bc-99e1-4401-96ab-ad9aae630c48-kube-api-access-lhvdc\") pod \"network-check-target-hzjsx\" (UID: \"9bbb98bc-99e1-4401-96ab-ad9aae630c48\") " pod="openshift-network-diagnostics/network-check-target-hzjsx"
Jul 08 16:35:21.076782 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:21.076765    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 16:35:21.076828 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:21.076784    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 16:35:21.076828 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:21.076795    2578 projected.go:196] Error preparing data for projected volume kube-api-access-lhvdc for pod openshift-network-diagnostics/network-check-target-hzjsx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:35:21.076894 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:21.076848    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9bbb98bc-99e1-4401-96ab-ad9aae630c48-kube-api-access-lhvdc podName:9bbb98bc-99e1-4401-96ab-ad9aae630c48 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:53.076831794 +0000 UTC m=+65.348281315 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-lhvdc" (UniqueName: "kubernetes.io/projected/9bbb98bc-99e1-4401-96ab-ad9aae630c48-kube-api-access-lhvdc") pod "network-check-target-hzjsx" (UID: "9bbb98bc-99e1-4401-96ab-ad9aae630c48") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:35:21.622142 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:21.622108    2578 generic.go:356] "Generic (PLEG): container finished" podID="28a74820-f159-4264-af3e-c983d2c68a51" containerID="e8b0d913ee9a991bfd67643085a3d6fce0f219d706a4dd21750f393b718bd8c8" exitCode=0
Jul 08 16:35:21.622284 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:21.622170    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g4stg" event={"ID":"28a74820-f159-4264-af3e-c983d2c68a51","Type":"ContainerDied","Data":"e8b0d913ee9a991bfd67643085a3d6fce0f219d706a4dd21750f393b718bd8c8"}
Jul 08 16:35:22.280305 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:22.280272    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d9p5w"
Jul 08 16:35:22.280779 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:22.280272    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f5jcr"
Jul 08 16:35:22.280779 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:22.280273    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hzjsx"
Jul 08 16:35:22.280779 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:22.280570    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f5jcr" podUID="0f2a26b1-8d19-4874-a633-1cd61e29091a"
Jul 08 16:35:22.280779 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:22.280387    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d9p5w" podUID="e6f14c39-461f-44d2-811a-87cfb899d92c"
Jul 08 16:35:22.280779 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:22.280662    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hzjsx" podUID="9bbb98bc-99e1-4401-96ab-ad9aae630c48"
Jul 08 16:35:23.472724 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:23.472693    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-94.ec2.internal" event="NodeReady"
Jul 08 16:35:23.473109 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:23.472853    2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 08 16:35:23.538924 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:23.538890    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-kbqcx"]
Jul 08 16:35:23.541150 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:23.541131    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-clfj6"]
Jul 08 16:35:23.541276 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:23.541259    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kbqcx"
Jul 08 16:35:23.542892 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:23.542868    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-clfj6"
Jul 08 16:35:23.547844 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:23.547664    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 08 16:35:23.547944 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:23.547774    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 08 16:35:23.547944 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:23.547793    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 08 16:35:23.548446 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:23.548135    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rvfsx\""
Jul 08 16:35:23.548446 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:23.548185    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-tks2z\""
Jul 08 16:35:23.548446 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:23.548354    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 08 16:35:23.548609 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:23.548585    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 08 16:35:23.564706 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:23.563346    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-clfj6"]
Jul 08 16:35:23.567586 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:23.567569    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kbqcx"]
Jul 08 16:35:23.697567 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:23.697480    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8fad727b-fceb-4ac4-b275-bd741597566e-cert\") pod \"ingress-canary-clfj6\" (UID: \"8fad727b-fceb-4ac4-b275-bd741597566e\") " pod="openshift-ingress-canary/ingress-canary-clfj6"
Jul 08 16:35:23.697567 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:23.697527    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0ee105a4-0d7b-4a50-a212-110cac5e12fb-config-volume\") pod \"dns-default-kbqcx\" (UID: \"0ee105a4-0d7b-4a50-a212-110cac5e12fb\") " pod="openshift-dns/dns-default-kbqcx"
Jul 08 16:35:23.697567 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:23.697555    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ee105a4-0d7b-4a50-a212-110cac5e12fb-tmp-dir\") pod \"dns-default-kbqcx\" (UID: \"0ee105a4-0d7b-4a50-a212-110cac5e12fb\") " pod="openshift-dns/dns-default-kbqcx"
Jul 08 16:35:23.697886 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:23.697595    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc2k9\" (UniqueName: \"kubernetes.io/projected/8fad727b-fceb-4ac4-b275-bd741597566e-kube-api-access-nc2k9\") pod \"ingress-canary-clfj6\" (UID: \"8fad727b-fceb-4ac4-b275-bd741597566e\") " pod="openshift-ingress-canary/ingress-canary-clfj6"
Jul 08 16:35:23.697886 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:23.697618    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42v4t\" (UniqueName: \"kubernetes.io/projected/0ee105a4-0d7b-4a50-a212-110cac5e12fb-kube-api-access-42v4t\") pod \"dns-default-kbqcx\" (UID: \"0ee105a4-0d7b-4a50-a212-110cac5e12fb\") " pod="openshift-dns/dns-default-kbqcx"
Jul 08 16:35:23.697886 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:23.697659    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ee105a4-0d7b-4a50-a212-110cac5e12fb-metrics-tls\") pod \"dns-default-kbqcx\" (UID: \"0ee105a4-0d7b-4a50-a212-110cac5e12fb\") " pod="openshift-dns/dns-default-kbqcx"
Jul 08 16:35:23.798643 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:23.798607    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc2k9\" (UniqueName: \"kubernetes.io/projected/8fad727b-fceb-4ac4-b275-bd741597566e-kube-api-access-nc2k9\") pod \"ingress-canary-clfj6\" (UID: \"8fad727b-fceb-4ac4-b275-bd741597566e\") " pod="openshift-ingress-canary/ingress-canary-clfj6"
Jul 08 16:35:23.798824 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:23.798666    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42v4t\" (UniqueName: \"kubernetes.io/projected/0ee105a4-0d7b-4a50-a212-110cac5e12fb-kube-api-access-42v4t\") pod \"dns-default-kbqcx\" (UID: \"0ee105a4-0d7b-4a50-a212-110cac5e12fb\") " pod="openshift-dns/dns-default-kbqcx"
Jul 08 16:35:23.798824 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:23.798706    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ee105a4-0d7b-4a50-a212-110cac5e12fb-metrics-tls\") pod \"dns-default-kbqcx\" (UID: \"0ee105a4-0d7b-4a50-a212-110cac5e12fb\") " pod="openshift-dns/dns-default-kbqcx"
Jul 08 16:35:23.798824 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:23.798743    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8fad727b-fceb-4ac4-b275-bd741597566e-cert\") pod \"ingress-canary-clfj6\" (UID: \"8fad727b-fceb-4ac4-b275-bd741597566e\") " pod="openshift-ingress-canary/ingress-canary-clfj6"
Jul 08 16:35:23.798824 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:23.798772    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0ee105a4-0d7b-4a50-a212-110cac5e12fb-config-volume\") pod \"dns-default-kbqcx\" (UID: \"0ee105a4-0d7b-4a50-a212-110cac5e12fb\") " pod="openshift-dns/dns-default-kbqcx"
Jul 08 16:35:23.798824 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:23.798796    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ee105a4-0d7b-4a50-a212-110cac5e12fb-tmp-dir\") pod \"dns-default-kbqcx\" (UID: \"0ee105a4-0d7b-4a50-a212-110cac5e12fb\") " pod="openshift-dns/dns-default-kbqcx"
Jul 08 16:35:23.799144 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:23.799126    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ee105a4-0d7b-4a50-a212-110cac5e12fb-tmp-dir\") pod \"dns-default-kbqcx\" (UID: \"0ee105a4-0d7b-4a50-a212-110cac5e12fb\") " pod="openshift-dns/dns-default-kbqcx"
Jul 08 16:35:23.799238 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:23.799223    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 16:35:23.799290 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:23.799281    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ee105a4-0d7b-4a50-a212-110cac5e12fb-metrics-tls podName:0ee105a4-0d7b-4a50-a212-110cac5e12fb nodeName:}" failed. No retries permitted until 2026-07-08 16:35:24.299263816 +0000 UTC m=+36.570713347 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0ee105a4-0d7b-4a50-a212-110cac5e12fb-metrics-tls") pod "dns-default-kbqcx" (UID: "0ee105a4-0d7b-4a50-a212-110cac5e12fb") : secret "dns-default-metrics-tls" not found
Jul 08 16:35:23.799409 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:23.799388    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0ee105a4-0d7b-4a50-a212-110cac5e12fb-config-volume\") pod \"dns-default-kbqcx\" (UID: \"0ee105a4-0d7b-4a50-a212-110cac5e12fb\") " pod="openshift-dns/dns-default-kbqcx"
Jul 08 16:35:23.803611 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:23.803584    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8fad727b-fceb-4ac4-b275-bd741597566e-cert\") pod \"ingress-canary-clfj6\" (UID: \"8fad727b-fceb-4ac4-b275-bd741597566e\") " pod="openshift-ingress-canary/ingress-canary-clfj6"
Jul 08 16:35:23.807901 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:23.807879    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc2k9\" (UniqueName: \"kubernetes.io/projected/8fad727b-fceb-4ac4-b275-bd741597566e-kube-api-access-nc2k9\") pod \"ingress-canary-clfj6\" (UID: \"8fad727b-fceb-4ac4-b275-bd741597566e\") " pod="openshift-ingress-canary/ingress-canary-clfj6"
Jul 08 16:35:23.808364 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:23.808349    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42v4t\" (UniqueName: \"kubernetes.io/projected/0ee105a4-0d7b-4a50-a212-110cac5e12fb-kube-api-access-42v4t\") pod \"dns-default-kbqcx\" (UID: \"0ee105a4-0d7b-4a50-a212-110cac5e12fb\") " pod="openshift-dns/dns-default-kbqcx"
Jul 08 16:35:23.858743 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:23.858711    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-clfj6"
Jul 08 16:35:24.010795 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:24.010762    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-clfj6"]
Jul 08 16:35:24.280780 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:24.280704    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hzjsx"
Jul 08 16:35:24.280935 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:24.280702    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d9p5w"
Jul 08 16:35:24.281361 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:24.280715    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f5jcr"
Jul 08 16:35:24.284462 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:24.284242    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 08 16:35:24.285118 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:24.285087    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6fxjj\""
Jul 08 16:35:24.285733 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:24.285713    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 08 16:35:24.286328 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:24.286308    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 08 16:35:24.286440 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:24.286424    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mshfs\""
Jul 08 16:35:24.286507 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:24.286481    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 08 16:35:24.302785 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:24.302702    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ee105a4-0d7b-4a50-a212-110cac5e12fb-metrics-tls\") pod \"dns-default-kbqcx\" (UID: \"0ee105a4-0d7b-4a50-a212-110cac5e12fb\") " pod="openshift-dns/dns-default-kbqcx"
Jul 08 16:35:24.302916 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:24.302898    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 16:35:24.303016 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:24.302967    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ee105a4-0d7b-4a50-a212-110cac5e12fb-metrics-tls podName:0ee105a4-0d7b-4a50-a212-110cac5e12fb nodeName:}" failed. No retries permitted until 2026-07-08 16:35:25.302948004 +0000 UTC m=+37.574397534 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0ee105a4-0d7b-4a50-a212-110cac5e12fb-metrics-tls") pod "dns-default-kbqcx" (UID: "0ee105a4-0d7b-4a50-a212-110cac5e12fb") : secret "dns-default-metrics-tls" not found
Jul 08 16:35:25.310461 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:25.310426    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ee105a4-0d7b-4a50-a212-110cac5e12fb-metrics-tls\") pod \"dns-default-kbqcx\" (UID: \"0ee105a4-0d7b-4a50-a212-110cac5e12fb\") " pod="openshift-dns/dns-default-kbqcx"
Jul 08 16:35:25.311324 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:25.310605    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 16:35:25.311324 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:25.310704    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ee105a4-0d7b-4a50-a212-110cac5e12fb-metrics-tls podName:0ee105a4-0d7b-4a50-a212-110cac5e12fb nodeName:}" failed. No retries permitted until 2026-07-08 16:35:27.310683887 +0000 UTC m=+39.582133423 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0ee105a4-0d7b-4a50-a212-110cac5e12fb-metrics-tls") pod "dns-default-kbqcx" (UID: "0ee105a4-0d7b-4a50-a212-110cac5e12fb") : secret "dns-default-metrics-tls" not found
Jul 08 16:35:25.463838 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:35:25.463805    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fad727b_fceb_4ac4_b275_bd741597566e.slice/crio-3f0029bce10e7de5bb2591ca75c236e8691cad647ac029d022a1c7bb9927dca5 WatchSource:0}: Error finding container 3f0029bce10e7de5bb2591ca75c236e8691cad647ac029d022a1c7bb9927dca5: Status 404 returned error can't find the container with id 3f0029bce10e7de5bb2591ca75c236e8691cad647ac029d022a1c7bb9927dca5
Jul 08 16:35:25.634504 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:25.634279    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-clfj6" event={"ID":"8fad727b-fceb-4ac4-b275-bd741597566e","Type":"ContainerStarted","Data":"3f0029bce10e7de5bb2591ca75c236e8691cad647ac029d022a1c7bb9927dca5"}
Jul 08 16:35:25.637169 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:25.637137    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g4stg" event={"ID":"28a74820-f159-4264-af3e-c983d2c68a51","Type":"ContainerStarted","Data":"13c542f07460078addd0f3720bc9a7a1af42ac10c73dd717db1b8a5b4ec631c1"}
Jul 08 16:35:26.643116 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:26.643046    2578 generic.go:356] "Generic (PLEG): container finished" podID="28a74820-f159-4264-af3e-c983d2c68a51" containerID="13c542f07460078addd0f3720bc9a7a1af42ac10c73dd717db1b8a5b4ec631c1" exitCode=0
Jul 08 16:35:26.643116 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:26.643094    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g4stg" event={"ID":"28a74820-f159-4264-af3e-c983d2c68a51","Type":"ContainerDied","Data":"13c542f07460078addd0f3720bc9a7a1af42ac10c73dd717db1b8a5b4ec631c1"}
Jul 08 16:35:27.326926 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:27.326887    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ee105a4-0d7b-4a50-a212-110cac5e12fb-metrics-tls\") pod \"dns-default-kbqcx\" (UID: \"0ee105a4-0d7b-4a50-a212-110cac5e12fb\") " pod="openshift-dns/dns-default-kbqcx"
Jul 08 16:35:27.327108 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:27.327068    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 16:35:27.327174 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:27.327162    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ee105a4-0d7b-4a50-a212-110cac5e12fb-metrics-tls podName:0ee105a4-0d7b-4a50-a212-110cac5e12fb nodeName:}" failed. No retries permitted until 2026-07-08 16:35:31.327141636 +0000 UTC m=+43.598591155 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0ee105a4-0d7b-4a50-a212-110cac5e12fb-metrics-tls") pod "dns-default-kbqcx" (UID: "0ee105a4-0d7b-4a50-a212-110cac5e12fb") : secret "dns-default-metrics-tls" not found
Jul 08 16:35:27.649695 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:27.649596    2578 generic.go:356] "Generic (PLEG): container finished" podID="28a74820-f159-4264-af3e-c983d2c68a51" containerID="333bda6f56a7544e6dac6a81f48172a59c0421c8da989a46142a474202d56c51" exitCode=0
Jul 08 16:35:27.649695 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:27.649667    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g4stg" event={"ID":"28a74820-f159-4264-af3e-c983d2c68a51","Type":"ContainerDied","Data":"333bda6f56a7544e6dac6a81f48172a59c0421c8da989a46142a474202d56c51"}
Jul 08 16:35:28.653566 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:28.653521    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-clfj6" event={"ID":"8fad727b-fceb-4ac4-b275-bd741597566e","Type":"ContainerStarted","Data":"a040ce3448f31d594e75b2d6ada93a86765546bfd25d531770e2d15b5dd1517a"}
Jul 08 16:35:28.656782 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:28.656758    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g4stg" event={"ID":"28a74820-f159-4264-af3e-c983d2c68a51","Type":"ContainerStarted","Data":"ae2697a451edea351cb6e9b265d68e103373e49d74bccb3d287d3ddcdfead11c"}
Jul 08 16:35:28.672265 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:28.672221    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-clfj6" podStartSLOduration=2.798596914 podStartE2EDuration="5.672206867s" totalImagesPullingTime="2.873609953s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:35:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:28.67123728 +0000 UTC m=+40.942686819" watchObservedRunningTime="2026-07-08 16:35:28.672206867 +0000 UTC m=+40.943656403"
Jul 08 16:35:28.694719 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:28.694608    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-g4stg" podStartSLOduration=7.950523962 podStartE2EDuration="39.69459155s" totalImagesPullingTime="31.744067588s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:34:49 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:28.693840316 +0000 UTC m=+40.965289856" watchObservedRunningTime="2026-07-08 16:35:28.69459155 +0000 UTC m=+40.966041092"
Jul 08 16:35:31.359216 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:31.359167    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ee105a4-0d7b-4a50-a212-110cac5e12fb-metrics-tls\") pod \"dns-default-kbqcx\" (UID: \"0ee105a4-0d7b-4a50-a212-110cac5e12fb\") " pod="openshift-dns/dns-default-kbqcx"
Jul 08 16:35:31.359594 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:31.359321    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 16:35:31.359594 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:31.359383    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ee105a4-0d7b-4a50-a212-110cac5e12fb-metrics-tls podName:0ee105a4-0d7b-4a50-a212-110cac5e12fb nodeName:}" failed. No retries permitted until 2026-07-08 16:35:39.359367037 +0000 UTC m=+51.630816555 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0ee105a4-0d7b-4a50-a212-110cac5e12fb-metrics-tls") pod "dns-default-kbqcx" (UID: "0ee105a4-0d7b-4a50-a212-110cac5e12fb") : secret "dns-default-metrics-tls" not found
Jul 08 16:35:39.414823 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:39.414782    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ee105a4-0d7b-4a50-a212-110cac5e12fb-metrics-tls\") pod \"dns-default-kbqcx\" (UID: \"0ee105a4-0d7b-4a50-a212-110cac5e12fb\") " pod="openshift-dns/dns-default-kbqcx"
Jul 08 16:35:39.415300 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:39.414904    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 16:35:39.415300 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:39.414954    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ee105a4-0d7b-4a50-a212-110cac5e12fb-metrics-tls podName:0ee105a4-0d7b-4a50-a212-110cac5e12fb nodeName:}" failed. No retries permitted until 2026-07-08 16:35:55.414940764 +0000 UTC m=+67.686390282 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0ee105a4-0d7b-4a50-a212-110cac5e12fb-metrics-tls") pod "dns-default-kbqcx" (UID: "0ee105a4-0d7b-4a50-a212-110cac5e12fb") : secret "dns-default-metrics-tls" not found
Jul 08 16:35:50.640326 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:50.640299    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2shvk"
Jul 08 16:35:53.003535 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:53.003497    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e6f14c39-461f-44d2-811a-87cfb899d92c-original-pull-secret\") pod \"global-pull-secret-syncer-d9p5w\" (UID: \"e6f14c39-461f-44d2-811a-87cfb899d92c\") " pod="kube-system/global-pull-secret-syncer-d9p5w"
Jul 08 16:35:53.003952 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:53.003565    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f2a26b1-8d19-4874-a633-1cd61e29091a-metrics-certs\") pod \"network-metrics-daemon-f5jcr\" (UID: \"0f2a26b1-8d19-4874-a633-1cd61e29091a\") " pod="openshift-multus/network-metrics-daemon-f5jcr"
Jul 08 16:35:53.006053 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:53.006035    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 08 16:35:53.006111 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:53.006032    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 08 16:35:53.014227 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:53.014205    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 08 16:35:53.014277 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:53.014266    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f2a26b1-8d19-4874-a633-1cd61e29091a-metrics-certs podName:0f2a26b1-8d19-4874-a633-1cd61e29091a nodeName:}" failed. No retries permitted until 2026-07-08 16:36:57.014250384 +0000 UTC m=+129.285699902 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0f2a26b1-8d19-4874-a633-1cd61e29091a-metrics-certs") pod "network-metrics-daemon-f5jcr" (UID: "0f2a26b1-8d19-4874-a633-1cd61e29091a") : secret "metrics-daemon-secret" not found
Jul 08 16:35:53.017526 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:53.017510    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e6f14c39-461f-44d2-811a-87cfb899d92c-original-pull-secret\") pod \"global-pull-secret-syncer-d9p5w\" (UID: \"e6f14c39-461f-44d2-811a-87cfb899d92c\") " pod="kube-system/global-pull-secret-syncer-d9p5w"
Jul 08 16:35:53.100312 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:53.100271    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d9p5w"
Jul 08 16:35:53.104400 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:53.104379    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhvdc\" (UniqueName: \"kubernetes.io/projected/9bbb98bc-99e1-4401-96ab-ad9aae630c48-kube-api-access-lhvdc\") pod \"network-check-target-hzjsx\" (UID: \"9bbb98bc-99e1-4401-96ab-ad9aae630c48\") " pod="openshift-network-diagnostics/network-check-target-hzjsx"
Jul 08 16:35:53.107503 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:53.107481    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 08 16:35:53.117570 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:53.117548    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 08 16:35:53.128142 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:53.128115    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhvdc\" (UniqueName: \"kubernetes.io/projected/9bbb98bc-99e1-4401-96ab-ad9aae630c48-kube-api-access-lhvdc\") pod \"network-check-target-hzjsx\" (UID: \"9bbb98bc-99e1-4401-96ab-ad9aae630c48\") " pod="openshift-network-diagnostics/network-check-target-hzjsx"
Jul 08 16:35:53.222048 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:53.222017    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-d9p5w"]
Jul 08 16:35:53.225126 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:35:53.225104    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6f14c39_461f_44d2_811a_87cfb899d92c.slice/crio-47221e1dd0edf65c25d240e2cb75bac0fc4899af5449aeb3ac6aca6e7c83b6c4 WatchSource:0}: Error finding container 47221e1dd0edf65c25d240e2cb75bac0fc4899af5449aeb3ac6aca6e7c83b6c4: Status 404 returned error can't find the container with id 47221e1dd0edf65c25d240e2cb75bac0fc4899af5449aeb3ac6aca6e7c83b6c4
Jul 08 16:35:53.396044 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:53.396013    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6fxjj\""
Jul 08 16:35:53.404110 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:53.404080    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hzjsx"
Jul 08 16:35:53.520553 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:53.520525    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hzjsx"]
Jul 08 16:35:53.523398 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:35:53.523376    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bbb98bc_99e1_4401_96ab_ad9aae630c48.slice/crio-dd201c02aff3b13e4f59d8e42dabeac84e662db3665d8fac1def764f3ea9da28 WatchSource:0}: Error finding container dd201c02aff3b13e4f59d8e42dabeac84e662db3665d8fac1def764f3ea9da28: Status 404 returned error can't find the container with id dd201c02aff3b13e4f59d8e42dabeac84e662db3665d8fac1def764f3ea9da28
Jul 08 16:35:53.712783 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:53.712695    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hzjsx" event={"ID":"9bbb98bc-99e1-4401-96ab-ad9aae630c48","Type":"ContainerStarted","Data":"dd201c02aff3b13e4f59d8e42dabeac84e662db3665d8fac1def764f3ea9da28"}
Jul 08 16:35:53.713796 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:53.713767    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-d9p5w" event={"ID":"e6f14c39-461f-44d2-811a-87cfb899d92c","Type":"ContainerStarted","Data":"47221e1dd0edf65c25d240e2cb75bac0fc4899af5449aeb3ac6aca6e7c83b6c4"}
Jul 08 16:35:55.418860 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:55.418806    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ee105a4-0d7b-4a50-a212-110cac5e12fb-metrics-tls\") pod \"dns-default-kbqcx\" (UID: \"0ee105a4-0d7b-4a50-a212-110cac5e12fb\") " pod="openshift-dns/dns-default-kbqcx"
Jul 08 16:35:55.419311 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:55.418977    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 16:35:55.419311 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:35:55.419061    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ee105a4-0d7b-4a50-a212-110cac5e12fb-metrics-tls podName:0ee105a4-0d7b-4a50-a212-110cac5e12fb nodeName:}" failed. No retries permitted until 2026-07-08 16:36:27.419036184 +0000 UTC m=+99.690485705 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0ee105a4-0d7b-4a50-a212-110cac5e12fb-metrics-tls") pod "dns-default-kbqcx" (UID: "0ee105a4-0d7b-4a50-a212-110cac5e12fb") : secret "dns-default-metrics-tls" not found
Jul 08 16:35:58.728991 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:58.728949    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hzjsx" event={"ID":"9bbb98bc-99e1-4401-96ab-ad9aae630c48","Type":"ContainerStarted","Data":"cb37eda3f2bc9005777075651ad557e39e4349bd3b044a402ae6a29d472c92a6"}
Jul 08 16:35:58.729410 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:58.729067    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hzjsx"
Jul 08 16:35:58.730674 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:58.730651    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-d9p5w" event={"ID":"e6f14c39-461f-44d2-811a-87cfb899d92c","Type":"ContainerStarted","Data":"82a6320639010526375363b346368921bfe1d2db39ff8dcfecb7818db229fb56"}
Jul 08 16:35:58.747153 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:58.747097    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-hzjsx" podStartSLOduration=65.042368392 podStartE2EDuration="1m9.747077235s" totalImagesPullingTime="4.704708843s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:34:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:58.746368737 +0000 UTC m=+71.017818277" watchObservedRunningTime="2026-07-08 16:35:58.747077235 +0000 UTC m=+71.018526775"
Jul 08 16:35:58.765903 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:35:58.765861    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-d9p5w" podStartSLOduration=64.751277309 podStartE2EDuration="1m9.765845757s" totalImagesPullingTime="5.014568448s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:34:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:58.765005775 +0000 UTC m=+71.036455316" watchObservedRunningTime="2026-07-08 16:35:58.765845757 +0000 UTC m=+71.037295297"
Jul 08 16:36:27.445407 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:36:27.445263    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ee105a4-0d7b-4a50-a212-110cac5e12fb-metrics-tls\") pod \"dns-default-kbqcx\" (UID: \"0ee105a4-0d7b-4a50-a212-110cac5e12fb\") " pod="openshift-dns/dns-default-kbqcx"
Jul 08 16:36:27.445407 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:36:27.445403    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 16:36:27.445971 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:36:27.445471    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ee105a4-0d7b-4a50-a212-110cac5e12fb-metrics-tls podName:0ee105a4-0d7b-4a50-a212-110cac5e12fb nodeName:}" failed. No retries permitted until 2026-07-08 16:37:31.445454876 +0000 UTC m=+163.716904394 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0ee105a4-0d7b-4a50-a212-110cac5e12fb-metrics-tls") pod "dns-default-kbqcx" (UID: "0ee105a4-0d7b-4a50-a212-110cac5e12fb") : secret "dns-default-metrics-tls" not found
Jul 08 16:36:29.735812 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:36:29.735782    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-hzjsx"
Jul 08 16:36:57.055961 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:36:57.055916    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f2a26b1-8d19-4874-a633-1cd61e29091a-metrics-certs\") pod \"network-metrics-daemon-f5jcr\" (UID: \"0f2a26b1-8d19-4874-a633-1cd61e29091a\") " pod="openshift-multus/network-metrics-daemon-f5jcr"
Jul 08 16:36:57.056422 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:36:57.056070    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 08 16:36:57.056422 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:36:57.056146    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f2a26b1-8d19-4874-a633-1cd61e29091a-metrics-certs podName:0f2a26b1-8d19-4874-a633-1cd61e29091a nodeName:}" failed. No retries permitted until 2026-07-08 16:38:59.056130024 +0000 UTC m=+251.327579545 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0f2a26b1-8d19-4874-a633-1cd61e29091a-metrics-certs") pod "network-metrics-daemon-f5jcr" (UID: "0f2a26b1-8d19-4874-a633-1cd61e29091a") : secret "metrics-daemon-secret" not found
Jul 08 16:37:26.554498 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:37:26.554451    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-kbqcx" podUID="0ee105a4-0d7b-4a50-a212-110cac5e12fb"
Jul 08 16:37:26.939809 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:37:26.939778    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kbqcx"
Jul 08 16:37:27.305244 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:37:27.305149    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-f5jcr" podUID="0f2a26b1-8d19-4874-a633-1cd61e29091a"
Jul 08 16:37:31.500101 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:37:31.500067    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ee105a4-0d7b-4a50-a212-110cac5e12fb-metrics-tls\") pod \"dns-default-kbqcx\" (UID: \"0ee105a4-0d7b-4a50-a212-110cac5e12fb\") " pod="openshift-dns/dns-default-kbqcx"
Jul 08 16:37:31.500545 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:37:31.500189    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 16:37:31.500545 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:37:31.500249    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ee105a4-0d7b-4a50-a212-110cac5e12fb-metrics-tls podName:0ee105a4-0d7b-4a50-a212-110cac5e12fb nodeName:}" failed. No retries permitted until 2026-07-08 16:39:33.500234789 +0000 UTC m=+285.771684307 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0ee105a4-0d7b-4a50-a212-110cac5e12fb-metrics-tls") pod "dns-default-kbqcx" (UID: "0ee105a4-0d7b-4a50-a212-110cac5e12fb") : secret "dns-default-metrics-tls" not found
Jul 08 16:37:42.280052 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:37:42.279966    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f5jcr"
Jul 08 16:38:53.529077 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.529041    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-kcxv2"]
Jul 08 16:38:53.530700 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.530684    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-kcxv2"
Jul 08 16:38:53.533008 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.532989    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-b5df8\""
Jul 08 16:38:53.536770 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.536741    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-qn2bx"]
Jul 08 16:38:53.538339 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.538322    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-55486dffd7-6ptzh"]
Jul 08 16:38:53.538444 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.538430    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-qn2bx"
Jul 08 16:38:53.539976 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.539960    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55486dffd7-6ptzh"
Jul 08 16:38:53.541100 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.541078    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 08 16:38:53.541358 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.541340    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 08 16:38:53.541985 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.541969    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 08 16:38:53.542350 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.542333    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-srsdb\""
Jul 08 16:38:53.542572 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.542559    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-sghjn\""
Jul 08 16:38:53.542648 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.542576    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 08 16:38:53.543531 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.543473    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 08 16:38:53.544273 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.544247    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 08 16:38:53.544854 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.544833    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 08 16:38:53.551061 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.551037    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-kcxv2"]
Jul 08 16:38:53.552821 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.552797    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 08 16:38:53.553378 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.553357    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 08 16:38:53.554546 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.554520    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-qn2bx"]
Jul 08 16:38:53.556915 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.556895    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-55486dffd7-6ptzh"]
Jul 08 16:38:53.631076 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.631043    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-ctm5m"]
Jul 08 16:38:53.633031 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.633011    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-ctm5m"
Jul 08 16:38:53.633213 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.633187    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dc3f4662-c8c9-42fc-8dfd-acac8b45f74c-image-registry-private-configuration\") pod \"image-registry-55486dffd7-6ptzh\" (UID: \"dc3f4662-c8c9-42fc-8dfd-acac8b45f74c\") " pod="openshift-image-registry/image-registry-55486dffd7-6ptzh"
Jul 08 16:38:53.633382 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.633349    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dc3f4662-c8c9-42fc-8dfd-acac8b45f74c-trusted-ca\") pod \"image-registry-55486dffd7-6ptzh\" (UID: \"dc3f4662-c8c9-42fc-8dfd-acac8b45f74c\") " pod="openshift-image-registry/image-registry-55486dffd7-6ptzh"
Jul 08 16:38:53.633382 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.633378    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfzqj\" (UniqueName: \"kubernetes.io/projected/dc3f4662-c8c9-42fc-8dfd-acac8b45f74c-kube-api-access-tfzqj\") pod \"image-registry-55486dffd7-6ptzh\" (UID: \"dc3f4662-c8c9-42fc-8dfd-acac8b45f74c\") " pod="openshift-image-registry/image-registry-55486dffd7-6ptzh"
Jul 08 16:38:53.633534 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.633398    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dc3f4662-c8c9-42fc-8dfd-acac8b45f74c-ca-trust-extracted\") pod \"image-registry-55486dffd7-6ptzh\" (UID: \"dc3f4662-c8c9-42fc-8dfd-acac8b45f74c\") " pod="openshift-image-registry/image-registry-55486dffd7-6ptzh"
Jul 08 16:38:53.633534 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.633415    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dc3f4662-c8c9-42fc-8dfd-acac8b45f74c-registry-certificates\") pod \"image-registry-55486dffd7-6ptzh\" (UID: \"dc3f4662-c8c9-42fc-8dfd-acac8b45f74c\") " pod="openshift-image-registry/image-registry-55486dffd7-6ptzh"
Jul 08 16:38:53.633534 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.633429    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dc3f4662-c8c9-42fc-8dfd-acac8b45f74c-installation-pull-secrets\") pod \"image-registry-55486dffd7-6ptzh\" (UID: \"dc3f4662-c8c9-42fc-8dfd-acac8b45f74c\") " pod="openshift-image-registry/image-registry-55486dffd7-6ptzh"
Jul 08 16:38:53.633534 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.633453    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4717bdee-2017-4551-b0ec-683ab07f7660-serving-cert\") pod \"console-operator-68b757fb67-qn2bx\" (UID: \"4717bdee-2017-4551-b0ec-683ab07f7660\") " pod="openshift-console-operator/console-operator-68b757fb67-qn2bx"
Jul 08 16:38:53.633534 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.633471    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4717bdee-2017-4551-b0ec-683ab07f7660-trusted-ca\") pod \"console-operator-68b757fb67-qn2bx\" (UID: \"4717bdee-2017-4551-b0ec-683ab07f7660\") " pod="openshift-console-operator/console-operator-68b757fb67-qn2bx"
Jul 08 16:38:53.633534 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.633509    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4717bdee-2017-4551-b0ec-683ab07f7660-config\") pod \"console-operator-68b757fb67-qn2bx\" (UID: \"4717bdee-2017-4551-b0ec-683ab07f7660\") " pod="openshift-console-operator/console-operator-68b757fb67-qn2bx"
Jul 08 16:38:53.633855 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.633560    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hh8h\" (UniqueName: \"kubernetes.io/projected/4717bdee-2017-4551-b0ec-683ab07f7660-kube-api-access-6hh8h\") pod \"console-operator-68b757fb67-qn2bx\" (UID: \"4717bdee-2017-4551-b0ec-683ab07f7660\") " pod="openshift-console-operator/console-operator-68b757fb67-qn2bx"
Jul 08 16:38:53.633855 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.633602    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dc3f4662-c8c9-42fc-8dfd-acac8b45f74c-registry-tls\") pod \"image-registry-55486dffd7-6ptzh\" (UID: \"dc3f4662-c8c9-42fc-8dfd-acac8b45f74c\") " pod="openshift-image-registry/image-registry-55486dffd7-6ptzh"
Jul 08 16:38:53.633855 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.633620    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc9gh\" (UniqueName: \"kubernetes.io/projected/a09322e9-4819-42eb-b98a-cf8ffe9ad70b-kube-api-access-tc9gh\") pod \"network-check-source-989f8b8dc-kcxv2\" (UID: \"a09322e9-4819-42eb-b98a-cf8ffe9ad70b\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-kcxv2"
Jul 08 16:38:53.633855 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.633705    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dc3f4662-c8c9-42fc-8dfd-acac8b45f74c-bound-sa-token\") pod \"image-registry-55486dffd7-6ptzh\" (UID: \"dc3f4662-c8c9-42fc-8dfd-acac8b45f74c\") " pod="openshift-image-registry/image-registry-55486dffd7-6ptzh"
Jul 08 16:38:53.635923 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:38:53.635896    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"insights-operator-5c847dcfc4-ctm5m\" is forbidden: User \"system:node:ip-10-0-142-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-142-94.ec2.internal' and this object" podUID="ec3a3d1a-9e90-40a7-9695-7a1c81f3c41c" pod="openshift-insights/insights-operator-5c847dcfc4-ctm5m"
Jul 08 16:38:53.636147 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:38:53.636127    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"operator-dockercfg-l6s98\" is forbidden: User \"system:node:ip-10-0-142-94.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-142-94.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"operator-dockercfg-l6s98\"" type="*v1.Secret"
Jul 08 16:38:53.636221 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:38:53.636151    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-142-94.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-142-94.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Jul 08 16:38:53.636423 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:38:53.636401    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"openshift-insights-serving-cert\" is forbidden: User \"system:node:ip-10-0-142-94.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-142-94.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" type="*v1.Secret"
Jul 08 16:38:53.636693 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:38:53.636675    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-142-94.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-142-94.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Jul 08 16:38:53.637213 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.637199    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 08 16:38:53.641690 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.641670    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 08 16:38:53.645395 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.645372    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-ctm5m"]
Jul 08 16:38:53.649046 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.649026    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-mn8c2"]
Jul 08 16:38:53.650743 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.650729    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-mn8c2"
Jul 08 16:38:53.652896 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.652874    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-kb2sg\""
Jul 08 16:38:53.653207 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.653188    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 08 16:38:53.653270 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.653252    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 08 16:38:53.661490 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.661468    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-mn8c2"]
Jul 08 16:38:53.734266 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.734228    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hh8h\" (UniqueName: \"kubernetes.io/projected/4717bdee-2017-4551-b0ec-683ab07f7660-kube-api-access-6hh8h\") pod \"console-operator-68b757fb67-qn2bx\" (UID: \"4717bdee-2017-4551-b0ec-683ab07f7660\") " pod="openshift-console-operator/console-operator-68b757fb67-qn2bx"
Jul 08 16:38:53.734458 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.734282    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dc3f4662-c8c9-42fc-8dfd-acac8b45f74c-registry-tls\") pod \"image-registry-55486dffd7-6ptzh\" (UID: \"dc3f4662-c8c9-42fc-8dfd-acac8b45f74c\") " pod="openshift-image-registry/image-registry-55486dffd7-6ptzh"
Jul 08 16:38:53.734458 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.734302    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc9gh\" (UniqueName: \"kubernetes.io/projected/a09322e9-4819-42eb-b98a-cf8ffe9ad70b-kube-api-access-tc9gh\") pod \"network-check-source-989f8b8dc-kcxv2\" (UID: \"a09322e9-4819-42eb-b98a-cf8ffe9ad70b\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-kcxv2"
Jul 08 16:38:53.734458 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.734359    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfrbx\" (UniqueName: \"kubernetes.io/projected/ec3a3d1a-9e90-40a7-9695-7a1c81f3c41c-kube-api-access-pfrbx\") pod \"insights-operator-5c847dcfc4-ctm5m\" (UID: \"ec3a3d1a-9e90-40a7-9695-7a1c81f3c41c\") " pod="openshift-insights/insights-operator-5c847dcfc4-ctm5m"
Jul 08 16:38:53.734458 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.734413    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dc3f4662-c8c9-42fc-8dfd-acac8b45f74c-bound-sa-token\") pod \"image-registry-55486dffd7-6ptzh\" (UID: \"dc3f4662-c8c9-42fc-8dfd-acac8b45f74c\") " pod="openshift-image-registry/image-registry-55486dffd7-6ptzh"
Jul 08 16:38:53.734458 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.734445    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7sg7\" (UniqueName: \"kubernetes.io/projected/c94e9045-53b8-455e-8c97-8a19845535a0-kube-api-access-b7sg7\") pod \"volume-data-source-validator-5d456fdbcd-mn8c2\" (UID: \"c94e9045-53b8-455e-8c97-8a19845535a0\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-mn8c2"
Jul 08 16:38:53.734686 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.734496    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ec3a3d1a-9e90-40a7-9695-7a1c81f3c41c-serving-cert\") pod \"insights-operator-5c847dcfc4-ctm5m\" (UID: \"ec3a3d1a-9e90-40a7-9695-7a1c81f3c41c\") " pod="openshift-insights/insights-operator-5c847dcfc4-ctm5m"
Jul 08 16:38:53.734686 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.734538    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec3a3d1a-9e90-40a7-9695-7a1c81f3c41c-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-ctm5m\" (UID: \"ec3a3d1a-9e90-40a7-9695-7a1c81f3c41c\") " pod="openshift-insights/insights-operator-5c847dcfc4-ctm5m"
Jul 08 16:38:53.734686 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:38:53.734593    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 08 16:38:53.734686 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:38:53.734609    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55486dffd7-6ptzh: secret "image-registry-tls" not found
Jul 08 16:38:53.734686 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.734594    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dc3f4662-c8c9-42fc-8dfd-acac8b45f74c-image-registry-private-configuration\") pod \"image-registry-55486dffd7-6ptzh\" (UID: \"dc3f4662-c8c9-42fc-8dfd-acac8b45f74c\") " pod="openshift-image-registry/image-registry-55486dffd7-6ptzh"
Jul 08 16:38:53.734880 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:38:53.734696    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dc3f4662-c8c9-42fc-8dfd-acac8b45f74c-registry-tls podName:dc3f4662-c8c9-42fc-8dfd-acac8b45f74c nodeName:}" failed. No retries permitted until 2026-07-08 16:38:54.234673708 +0000 UTC m=+246.506123231 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dc3f4662-c8c9-42fc-8dfd-acac8b45f74c-registry-tls") pod "image-registry-55486dffd7-6ptzh" (UID: "dc3f4662-c8c9-42fc-8dfd-acac8b45f74c") : secret "image-registry-tls" not found
Jul 08 16:38:53.734880 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.734721    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dc3f4662-c8c9-42fc-8dfd-acac8b45f74c-trusted-ca\") pod \"image-registry-55486dffd7-6ptzh\" (UID: \"dc3f4662-c8c9-42fc-8dfd-acac8b45f74c\") " pod="openshift-image-registry/image-registry-55486dffd7-6ptzh"
Jul 08 16:38:53.734880 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.734739    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfzqj\" (UniqueName: \"kubernetes.io/projected/dc3f4662-c8c9-42fc-8dfd-acac8b45f74c-kube-api-access-tfzqj\") pod \"image-registry-55486dffd7-6ptzh\" (UID: \"dc3f4662-c8c9-42fc-8dfd-acac8b45f74c\") " pod="openshift-image-registry/image-registry-55486dffd7-6ptzh"
Jul 08 16:38:53.734880 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.734763    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec3a3d1a-9e90-40a7-9695-7a1c81f3c41c-tmp\") pod \"insights-operator-5c847dcfc4-ctm5m\" (UID: \"ec3a3d1a-9e90-40a7-9695-7a1c81f3c41c\") " pod="openshift-insights/insights-operator-5c847dcfc4-ctm5m"
Jul 08 16:38:53.734880 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.734791    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dc3f4662-c8c9-42fc-8dfd-acac8b45f74c-ca-trust-extracted\") pod \"image-registry-55486dffd7-6ptzh\" (UID: \"dc3f4662-c8c9-42fc-8dfd-acac8b45f74c\") " pod="openshift-image-registry/image-registry-55486dffd7-6ptzh"
Jul 08 16:38:53.734880 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.734819    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dc3f4662-c8c9-42fc-8dfd-acac8b45f74c-registry-certificates\") pod \"image-registry-55486dffd7-6ptzh\" (UID: \"dc3f4662-c8c9-42fc-8dfd-acac8b45f74c\") " pod="openshift-image-registry/image-registry-55486dffd7-6ptzh"
Jul 08 16:38:53.734880 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.734850    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dc3f4662-c8c9-42fc-8dfd-acac8b45f74c-installation-pull-secrets\") pod \"image-registry-55486dffd7-6ptzh\" (UID: \"dc3f4662-c8c9-42fc-8dfd-acac8b45f74c\") " pod="openshift-image-registry/image-registry-55486dffd7-6ptzh"
Jul 08 16:38:53.735186 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.734888    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4717bdee-2017-4551-b0ec-683ab07f7660-serving-cert\") pod \"console-operator-68b757fb67-qn2bx\" (UID: \"4717bdee-2017-4551-b0ec-683ab07f7660\") " pod="openshift-console-operator/console-operator-68b757fb67-qn2bx"
Jul 08 16:38:53.735186 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.734914    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4717bdee-2017-4551-b0ec-683ab07f7660-trusted-ca\") pod \"console-operator-68b757fb67-qn2bx\" (UID: \"4717bdee-2017-4551-b0ec-683ab07f7660\") " pod="openshift-console-operator/console-operator-68b757fb67-qn2bx"
Jul 08 16:38:53.735186 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.734934    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ec3a3d1a-9e90-40a7-9695-7a1c81f3c41c-snapshots\") pod \"insights-operator-5c847dcfc4-ctm5m\" (UID: \"ec3a3d1a-9e90-40a7-9695-7a1c81f3c41c\") " pod="openshift-insights/insights-operator-5c847dcfc4-ctm5m"
Jul 08 16:38:53.735186 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.734984    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4717bdee-2017-4551-b0ec-683ab07f7660-config\") pod \"console-operator-68b757fb67-qn2bx\" (UID: \"4717bdee-2017-4551-b0ec-683ab07f7660\") " pod="openshift-console-operator/console-operator-68b757fb67-qn2bx"
Jul 08 16:38:53.735186 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.735015    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec3a3d1a-9e90-40a7-9695-7a1c81f3c41c-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-ctm5m\" (UID: \"ec3a3d1a-9e90-40a7-9695-7a1c81f3c41c\") " pod="openshift-insights/insights-operator-5c847dcfc4-ctm5m"
Jul 08 16:38:53.735392 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.735259    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dc3f4662-c8c9-42fc-8dfd-acac8b45f74c-ca-trust-extracted\") pod \"image-registry-55486dffd7-6ptzh\" (UID: \"dc3f4662-c8c9-42fc-8dfd-acac8b45f74c\") " pod="openshift-image-registry/image-registry-55486dffd7-6ptzh"
Jul 08 16:38:53.736095 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.736073    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dc3f4662-c8c9-42fc-8dfd-acac8b45f74c-registry-certificates\") pod \"image-registry-55486dffd7-6ptzh\" (UID: \"dc3f4662-c8c9-42fc-8dfd-acac8b45f74c\") " pod="openshift-image-registry/image-registry-55486dffd7-6ptzh"
Jul 08 16:38:53.736163 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.736074    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4717bdee-2017-4551-b0ec-683ab07f7660-trusted-ca\") pod \"console-operator-68b757fb67-qn2bx\" (UID: \"4717bdee-2017-4551-b0ec-683ab07f7660\") " pod="openshift-console-operator/console-operator-68b757fb67-qn2bx"
Jul 08 16:38:53.736292 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.736273    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dc3f4662-c8c9-42fc-8dfd-acac8b45f74c-trusted-ca\") pod \"image-registry-55486dffd7-6ptzh\" (UID: \"dc3f4662-c8c9-42fc-8dfd-acac8b45f74c\") " pod="openshift-image-registry/image-registry-55486dffd7-6ptzh"
Jul 08 16:38:53.736331 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.736271    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4717bdee-2017-4551-b0ec-683ab07f7660-config\") pod \"console-operator-68b757fb67-qn2bx\" (UID: \"4717bdee-2017-4551-b0ec-683ab07f7660\") " pod="openshift-console-operator/console-operator-68b757fb67-qn2bx"
Jul 08 16:38:53.737321 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.737302    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4717bdee-2017-4551-b0ec-683ab07f7660-serving-cert\") pod \"console-operator-68b757fb67-qn2bx\" (UID: \"4717bdee-2017-4551-b0ec-683ab07f7660\") " pod="openshift-console-operator/console-operator-68b757fb67-qn2bx"
Jul 08 16:38:53.737729 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.737698    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dc3f4662-c8c9-42fc-8dfd-acac8b45f74c-installation-pull-secrets\") pod \"image-registry-55486dffd7-6ptzh\" (UID: \"dc3f4662-c8c9-42fc-8dfd-acac8b45f74c\") " pod="openshift-image-registry/image-registry-55486dffd7-6ptzh"
Jul 08 16:38:53.737729 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.737707    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dc3f4662-c8c9-42fc-8dfd-acac8b45f74c-image-registry-private-configuration\") pod \"image-registry-55486dffd7-6ptzh\" (UID: \"dc3f4662-c8c9-42fc-8dfd-acac8b45f74c\") " pod="openshift-image-registry/image-registry-55486dffd7-6ptzh"
Jul 08 16:38:53.744054 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.744032    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dc3f4662-c8c9-42fc-8dfd-acac8b45f74c-bound-sa-token\") pod \"image-registry-55486dffd7-6ptzh\" (UID: \"dc3f4662-c8c9-42fc-8dfd-acac8b45f74c\") " pod="openshift-image-registry/image-registry-55486dffd7-6ptzh"
Jul 08 16:38:53.744357 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.744335    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hh8h\" (UniqueName: \"kubernetes.io/projected/4717bdee-2017-4551-b0ec-683ab07f7660-kube-api-access-6hh8h\") pod \"console-operator-68b757fb67-qn2bx\" (UID: \"4717bdee-2017-4551-b0ec-683ab07f7660\") " pod="openshift-console-operator/console-operator-68b757fb67-qn2bx"
Jul 08 16:38:53.744925 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.744906    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfzqj\" (UniqueName: \"kubernetes.io/projected/dc3f4662-c8c9-42fc-8dfd-acac8b45f74c-kube-api-access-tfzqj\") pod \"image-registry-55486dffd7-6ptzh\" (UID: \"dc3f4662-c8c9-42fc-8dfd-acac8b45f74c\") " pod="openshift-image-registry/image-registry-55486dffd7-6ptzh"
Jul 08 16:38:53.745008 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.744979    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc9gh\" (UniqueName: \"kubernetes.io/projected/a09322e9-4819-42eb-b98a-cf8ffe9ad70b-kube-api-access-tc9gh\") pod \"network-check-source-989f8b8dc-kcxv2\" (UID: \"a09322e9-4819-42eb-b98a-cf8ffe9ad70b\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-kcxv2"
Jul 08 16:38:53.835611 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.835574    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ec3a3d1a-9e90-40a7-9695-7a1c81f3c41c-snapshots\") pod \"insights-operator-5c847dcfc4-ctm5m\" (UID: \"ec3a3d1a-9e90-40a7-9695-7a1c81f3c41c\") " pod="openshift-insights/insights-operator-5c847dcfc4-ctm5m"
Jul 08 16:38:53.835831 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.835645    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec3a3d1a-9e90-40a7-9695-7a1c81f3c41c-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-ctm5m\" (UID: \"ec3a3d1a-9e90-40a7-9695-7a1c81f3c41c\") " pod="openshift-insights/insights-operator-5c847dcfc4-ctm5m"
Jul 08 16:38:53.835831 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.835689    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfrbx\" (UniqueName: \"kubernetes.io/projected/ec3a3d1a-9e90-40a7-9695-7a1c81f3c41c-kube-api-access-pfrbx\") pod \"insights-operator-5c847dcfc4-ctm5m\" (UID: \"ec3a3d1a-9e90-40a7-9695-7a1c81f3c41c\") " pod="openshift-insights/insights-operator-5c847dcfc4-ctm5m"
Jul 08 16:38:53.835831 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.835813    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7sg7\" (UniqueName: \"kubernetes.io/projected/c94e9045-53b8-455e-8c97-8a19845535a0-kube-api-access-b7sg7\") pod \"volume-data-source-validator-5d456fdbcd-mn8c2\" (UID: \"c94e9045-53b8-455e-8c97-8a19845535a0\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-mn8c2"
Jul 08 16:38:53.835968 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.835848    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ec3a3d1a-9e90-40a7-9695-7a1c81f3c41c-serving-cert\") pod \"insights-operator-5c847dcfc4-ctm5m\" (UID: \"ec3a3d1a-9e90-40a7-9695-7a1c81f3c41c\") " pod="openshift-insights/insights-operator-5c847dcfc4-ctm5m"
Jul 08 16:38:53.835968 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.835874    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec3a3d1a-9e90-40a7-9695-7a1c81f3c41c-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-ctm5m\" (UID: \"ec3a3d1a-9e90-40a7-9695-7a1c81f3c41c\") " pod="openshift-insights/insights-operator-5c847dcfc4-ctm5m"
Jul 08 16:38:53.835968 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.835912    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec3a3d1a-9e90-40a7-9695-7a1c81f3c41c-tmp\") pod \"insights-operator-5c847dcfc4-ctm5m\" (UID: \"ec3a3d1a-9e90-40a7-9695-7a1c81f3c41c\") " pod="openshift-insights/insights-operator-5c847dcfc4-ctm5m"
Jul 08 16:38:53.836272 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.836252    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec3a3d1a-9e90-40a7-9695-7a1c81f3c41c-tmp\") pod \"insights-operator-5c847dcfc4-ctm5m\" (UID: \"ec3a3d1a-9e90-40a7-9695-7a1c81f3c41c\") " pod="openshift-insights/insights-operator-5c847dcfc4-ctm5m"
Jul 08 16:38:53.836343 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.836309    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ec3a3d1a-9e90-40a7-9695-7a1c81f3c41c-snapshots\") pod \"insights-operator-5c847dcfc4-ctm5m\" (UID: \"ec3a3d1a-9e90-40a7-9695-7a1c81f3c41c\") " pod="openshift-insights/insights-operator-5c847dcfc4-ctm5m"
Jul 08 16:38:53.836383 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.836346    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec3a3d1a-9e90-40a7-9695-7a1c81f3c41c-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-ctm5m\" (UID: \"ec3a3d1a-9e90-40a7-9695-7a1c81f3c41c\") " pod="openshift-insights/insights-operator-5c847dcfc4-ctm5m"
Jul 08 16:38:53.836649 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.836613    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec3a3d1a-9e90-40a7-9695-7a1c81f3c41c-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-ctm5m\" (UID: \"ec3a3d1a-9e90-40a7-9695-7a1c81f3c41c\") " pod="openshift-insights/insights-operator-5c847dcfc4-ctm5m"
Jul 08 16:38:53.839496 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.839480    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-kcxv2"
Jul 08 16:38:53.844050 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.844028    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7sg7\" (UniqueName: \"kubernetes.io/projected/c94e9045-53b8-455e-8c97-8a19845535a0-kube-api-access-b7sg7\") pod \"volume-data-source-validator-5d456fdbcd-mn8c2\" (UID: \"c94e9045-53b8-455e-8c97-8a19845535a0\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-mn8c2"
Jul 08 16:38:53.851932 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.851888    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-qn2bx"
Jul 08 16:38:53.959325 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.959294    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-mn8c2"
Jul 08 16:38:53.975045 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.975016    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-kcxv2"]
Jul 08 16:38:53.979029 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:38:53.979003    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda09322e9_4819_42eb_b98a_cf8ffe9ad70b.slice/crio-658ddea3ff704702f96a4ef25827e2d674e9c5121d171d1802f3edadf40210a6 WatchSource:0}: Error finding container 658ddea3ff704702f96a4ef25827e2d674e9c5121d171d1802f3edadf40210a6: Status 404 returned error can't find the container with id 658ddea3ff704702f96a4ef25827e2d674e9c5121d171d1802f3edadf40210a6
Jul 08 16:38:53.992036 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:53.992011    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-qn2bx"]
Jul 08 16:38:53.995554 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:38:53.995522    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4717bdee_2017_4551_b0ec_683ab07f7660.slice/crio-e76c035f2a532d543812d8c7fb6bdfc785bfe410d1a8e4e96cc686159fb237f4 WatchSource:0}: Error finding container e76c035f2a532d543812d8c7fb6bdfc785bfe410d1a8e4e96cc686159fb237f4: Status 404 returned error can't find the container with id e76c035f2a532d543812d8c7fb6bdfc785bfe410d1a8e4e96cc686159fb237f4
Jul 08 16:38:54.094020 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:54.093928    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-mn8c2"]
Jul 08 16:38:54.098136 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:38:54.098111    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc94e9045_53b8_455e_8c97_8a19845535a0.slice/crio-fc3825ed2d2f03a6486518a79184e5fbb8d20928791cf88b02919efd15a1327b WatchSource:0}: Error finding container fc3825ed2d2f03a6486518a79184e5fbb8d20928791cf88b02919efd15a1327b: Status 404 returned error can't find the container with id fc3825ed2d2f03a6486518a79184e5fbb8d20928791cf88b02919efd15a1327b
Jul 08 16:38:54.145168 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:54.145122    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-qn2bx" event={"ID":"4717bdee-2017-4551-b0ec-683ab07f7660","Type":"ContainerStarted","Data":"e76c035f2a532d543812d8c7fb6bdfc785bfe410d1a8e4e96cc686159fb237f4"}
Jul 08 16:38:54.150140 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:54.146981    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-kcxv2" event={"ID":"a09322e9-4819-42eb-b98a-cf8ffe9ad70b","Type":"ContainerStarted","Data":"53c8513c4936f8050d397b7c91e4bad0f0b252df0461a1e25b5d4622ac3bdf2c"}
Jul 08 16:38:54.150140 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:54.147077    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-kcxv2" event={"ID":"a09322e9-4819-42eb-b98a-cf8ffe9ad70b","Type":"ContainerStarted","Data":"658ddea3ff704702f96a4ef25827e2d674e9c5121d171d1802f3edadf40210a6"}
Jul 08 16:38:54.151914 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:54.151887    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-mn8c2" event={"ID":"c94e9045-53b8-455e-8c97-8a19845535a0","Type":"ContainerStarted","Data":"fc3825ed2d2f03a6486518a79184e5fbb8d20928791cf88b02919efd15a1327b"}
Jul 08 16:38:54.164335 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:54.164288    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-kcxv2" podStartSLOduration=1.164273742 podStartE2EDuration="1.164273742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:38:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:38:54.163977953 +0000 UTC m=+246.435427493" watchObservedRunningTime="2026-07-08 16:38:54.164273742 +0000 UTC m=+246.435723282"
Jul 08 16:38:54.239357 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:54.239308    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dc3f4662-c8c9-42fc-8dfd-acac8b45f74c-registry-tls\") pod \"image-registry-55486dffd7-6ptzh\" (UID: \"dc3f4662-c8c9-42fc-8dfd-acac8b45f74c\") " pod="openshift-image-registry/image-registry-55486dffd7-6ptzh"
Jul 08 16:38:54.239508 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:38:54.239461    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 08 16:38:54.239508 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:38:54.239481    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55486dffd7-6ptzh: secret "image-registry-tls" not found
Jul 08 16:38:54.239594 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:38:54.239569    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dc3f4662-c8c9-42fc-8dfd-acac8b45f74c-registry-tls podName:dc3f4662-c8c9-42fc-8dfd-acac8b45f74c nodeName:}" failed. No retries permitted until 2026-07-08 16:38:55.239547385 +0000 UTC m=+247.510996920 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dc3f4662-c8c9-42fc-8dfd-acac8b45f74c-registry-tls") pod "image-registry-55486dffd7-6ptzh" (UID: "dc3f4662-c8c9-42fc-8dfd-acac8b45f74c") : secret "image-registry-tls" not found
Jul 08 16:38:54.501363 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:54.501270    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 08 16:38:54.503178 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:54.503158    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 08 16:38:54.514588 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:54.514556    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfrbx\" (UniqueName: \"kubernetes.io/projected/ec3a3d1a-9e90-40a7-9695-7a1c81f3c41c-kube-api-access-pfrbx\") pod \"insights-operator-5c847dcfc4-ctm5m\" (UID: \"ec3a3d1a-9e90-40a7-9695-7a1c81f3c41c\") " pod="openshift-insights/insights-operator-5c847dcfc4-ctm5m"
Jul 08 16:38:54.568296 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:54.568255    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-l6s98\""
Jul 08 16:38:54.837132 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:38:54.837095    2578 secret.go:189] Couldn't get secret openshift-insights/openshift-insights-serving-cert: failed to sync secret cache: timed out waiting for the condition
Jul 08 16:38:54.837335 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:38:54.837223    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec3a3d1a-9e90-40a7-9695-7a1c81f3c41c-serving-cert podName:ec3a3d1a-9e90-40a7-9695-7a1c81f3c41c nodeName:}" failed. No retries permitted until 2026-07-08 16:38:55.33719981 +0000 UTC m=+247.608649330 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "serving-cert" (UniqueName: "kubernetes.io/secret/ec3a3d1a-9e90-40a7-9695-7a1c81f3c41c-serving-cert") pod "insights-operator-5c847dcfc4-ctm5m" (UID: "ec3a3d1a-9e90-40a7-9695-7a1c81f3c41c") : failed to sync secret cache: timed out waiting for the condition
Jul 08 16:38:55.217396 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:55.217316    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 08 16:38:55.247596 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:55.247561    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dc3f4662-c8c9-42fc-8dfd-acac8b45f74c-registry-tls\") pod \"image-registry-55486dffd7-6ptzh\" (UID: \"dc3f4662-c8c9-42fc-8dfd-acac8b45f74c\") " pod="openshift-image-registry/image-registry-55486dffd7-6ptzh"
Jul 08 16:38:55.247776 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:38:55.247739    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 08 16:38:55.247776 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:38:55.247754    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55486dffd7-6ptzh: secret "image-registry-tls" not found
Jul 08 16:38:55.247881 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:38:55.247823    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dc3f4662-c8c9-42fc-8dfd-acac8b45f74c-registry-tls podName:dc3f4662-c8c9-42fc-8dfd-acac8b45f74c nodeName:}" failed. No retries permitted until 2026-07-08 16:38:57.247804707 +0000 UTC m=+249.519254242 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dc3f4662-c8c9-42fc-8dfd-acac8b45f74c-registry-tls") pod "image-registry-55486dffd7-6ptzh" (UID: "dc3f4662-c8c9-42fc-8dfd-acac8b45f74c") : secret "image-registry-tls" not found
Jul 08 16:38:55.348509 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:55.348476    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ec3a3d1a-9e90-40a7-9695-7a1c81f3c41c-serving-cert\") pod \"insights-operator-5c847dcfc4-ctm5m\" (UID: \"ec3a3d1a-9e90-40a7-9695-7a1c81f3c41c\") " pod="openshift-insights/insights-operator-5c847dcfc4-ctm5m"
Jul 08 16:38:55.351068 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:55.351038    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ec3a3d1a-9e90-40a7-9695-7a1c81f3c41c-serving-cert\") pod \"insights-operator-5c847dcfc4-ctm5m\" (UID: \"ec3a3d1a-9e90-40a7-9695-7a1c81f3c41c\") " pod="openshift-insights/insights-operator-5c847dcfc4-ctm5m"
Jul 08 16:38:55.442317 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:55.442280    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-ctm5m"
Jul 08 16:38:56.160063 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:56.160025    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-qn2bx" event={"ID":"4717bdee-2017-4551-b0ec-683ab07f7660","Type":"ContainerStarted","Data":"a465c0f5f2ea6903eaf47af49dc481dd097963b51ba8790c22075602fb6a7dbe"}
Jul 08 16:38:56.160485 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:56.160236    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-qn2bx"
Jul 08 16:38:56.161751 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:56.161721    2578 patch_prober.go:28] interesting pod/console-operator-68b757fb67-qn2bx container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.134.0.9:8443/readyz\": dial tcp 10.134.0.9:8443: connect: connection refused" start-of-body=
Jul 08 16:38:56.161902 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:56.161775    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-68b757fb67-qn2bx" podUID="4717bdee-2017-4551-b0ec-683ab07f7660" containerName="console-operator" probeResult="failure" output="Get \"https://10.134.0.9:8443/readyz\": dial tcp 10.134.0.9:8443: connect: connection refused"
Jul 08 16:38:56.161977 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:56.161913    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-mn8c2" event={"ID":"c94e9045-53b8-455e-8c97-8a19845535a0","Type":"ContainerStarted","Data":"b023e52e9ec7b653f4b37a59c34f71dd7f4ac75df071ec5b7b0aa2808d02b059"}
Jul 08 16:38:56.178569 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:56.178331    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-68b757fb67-qn2bx" podStartSLOduration=1.1126094420000001 podStartE2EDuration="3.178314215s" totalImagesPullingTime="2.065704773s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:38:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:38:56.176866114 +0000 UTC m=+248.448315655" watchObservedRunningTime="2026-07-08 16:38:56.178314215 +0000 UTC m=+248.449763754"
Jul 08 16:38:56.190313 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:56.190223    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-ctm5m"]
Jul 08 16:38:56.192838 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:56.192776    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-mn8c2" podStartSLOduration=1.234224741 podStartE2EDuration="3.192754361s" totalImagesPullingTime="1.95852962s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:38:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:38:56.191902947 +0000 UTC m=+248.463352487" watchObservedRunningTime="2026-07-08 16:38:56.192754361 +0000 UTC m=+248.464203901"
Jul 08 16:38:56.195148 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:38:56.195114    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec3a3d1a_9e90_40a7_9695_7a1c81f3c41c.slice/crio-c1e67128f8439045c351bb9f911f0f2b5f13aa31914b9d0eb3c8206c65bc95e0 WatchSource:0}: Error finding container c1e67128f8439045c351bb9f911f0f2b5f13aa31914b9d0eb3c8206c65bc95e0: Status 404 returned error can't find the container with id c1e67128f8439045c351bb9f911f0f2b5f13aa31914b9d0eb3c8206c65bc95e0
Jul 08 16:38:57.167607 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:57.167576    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-qn2bx_4717bdee-2017-4551-b0ec-683ab07f7660/console-operator/0.log"
Jul 08 16:38:57.168098 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:57.167625    2578 generic.go:356] "Generic (PLEG): container finished" podID="4717bdee-2017-4551-b0ec-683ab07f7660" containerID="a465c0f5f2ea6903eaf47af49dc481dd097963b51ba8790c22075602fb6a7dbe" exitCode=255
Jul 08 16:38:57.168098 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:57.167737    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-qn2bx" event={"ID":"4717bdee-2017-4551-b0ec-683ab07f7660","Type":"ContainerDied","Data":"a465c0f5f2ea6903eaf47af49dc481dd097963b51ba8790c22075602fb6a7dbe"}
Jul 08 16:38:57.168098 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:57.168003    2578 scope.go:117] "RemoveContainer" containerID="a465c0f5f2ea6903eaf47af49dc481dd097963b51ba8790c22075602fb6a7dbe"
Jul 08 16:38:57.169357 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:57.169336    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-ctm5m" event={"ID":"ec3a3d1a-9e90-40a7-9695-7a1c81f3c41c","Type":"ContainerStarted","Data":"c1e67128f8439045c351bb9f911f0f2b5f13aa31914b9d0eb3c8206c65bc95e0"}
Jul 08 16:38:57.264442 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:57.264399    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dc3f4662-c8c9-42fc-8dfd-acac8b45f74c-registry-tls\") pod \"image-registry-55486dffd7-6ptzh\" (UID: \"dc3f4662-c8c9-42fc-8dfd-acac8b45f74c\") " pod="openshift-image-registry/image-registry-55486dffd7-6ptzh"
Jul 08 16:38:57.264644 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:38:57.264603    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 08 16:38:57.264721 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:38:57.264653    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55486dffd7-6ptzh: secret "image-registry-tls" not found
Jul 08 16:38:57.264721 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:38:57.264718    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dc3f4662-c8c9-42fc-8dfd-acac8b45f74c-registry-tls podName:dc3f4662-c8c9-42fc-8dfd-acac8b45f74c nodeName:}" failed. No retries permitted until 2026-07-08 16:39:01.264699645 +0000 UTC m=+253.536149163 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dc3f4662-c8c9-42fc-8dfd-acac8b45f74c-registry-tls") pod "image-registry-55486dffd7-6ptzh" (UID: "dc3f4662-c8c9-42fc-8dfd-acac8b45f74c") : secret "image-registry-tls" not found
Jul 08 16:38:58.178287 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:58.178249    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-qn2bx_4717bdee-2017-4551-b0ec-683ab07f7660/console-operator/1.log"
Jul 08 16:38:58.178789 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:58.178715    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-qn2bx_4717bdee-2017-4551-b0ec-683ab07f7660/console-operator/0.log"
Jul 08 16:38:58.178789 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:58.178760    2578 generic.go:356] "Generic (PLEG): container finished" podID="4717bdee-2017-4551-b0ec-683ab07f7660" containerID="f5ae8962cde303ba5b72a08f62f675538a0e34837da715ac26a018013e049aa2" exitCode=255
Jul 08 16:38:58.178894 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:58.178831    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-qn2bx" event={"ID":"4717bdee-2017-4551-b0ec-683ab07f7660","Type":"ContainerDied","Data":"f5ae8962cde303ba5b72a08f62f675538a0e34837da715ac26a018013e049aa2"}
Jul 08 16:38:58.178940 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:58.178913    2578 scope.go:117] "RemoveContainer" containerID="a465c0f5f2ea6903eaf47af49dc481dd097963b51ba8790c22075602fb6a7dbe"
Jul 08 16:38:58.179137 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:58.179112    2578 scope.go:117] "RemoveContainer" containerID="f5ae8962cde303ba5b72a08f62f675538a0e34837da715ac26a018013e049aa2"
Jul 08 16:38:58.179347 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:38:58.179315    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-qn2bx_openshift-console-operator(4717bdee-2017-4551-b0ec-683ab07f7660)\"" pod="openshift-console-operator/console-operator-68b757fb67-qn2bx" podUID="4717bdee-2017-4551-b0ec-683ab07f7660"
Jul 08 16:38:59.078103 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:59.078059    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f2a26b1-8d19-4874-a633-1cd61e29091a-metrics-certs\") pod \"network-metrics-daemon-f5jcr\" (UID: \"0f2a26b1-8d19-4874-a633-1cd61e29091a\") " pod="openshift-multus/network-metrics-daemon-f5jcr"
Jul 08 16:38:59.078312 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:38:59.078195    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 08 16:38:59.078312 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:38:59.078268    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f2a26b1-8d19-4874-a633-1cd61e29091a-metrics-certs podName:0f2a26b1-8d19-4874-a633-1cd61e29091a nodeName:}" failed. No retries permitted until 2026-07-08 16:41:01.078249285 +0000 UTC m=+373.349698809 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0f2a26b1-8d19-4874-a633-1cd61e29091a-metrics-certs") pod "network-metrics-daemon-f5jcr" (UID: "0f2a26b1-8d19-4874-a633-1cd61e29091a") : secret "metrics-daemon-secret" not found
Jul 08 16:38:59.183624 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:59.183594    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-qn2bx_4717bdee-2017-4551-b0ec-683ab07f7660/console-operator/1.log"
Jul 08 16:38:59.184034 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:59.183985    2578 scope.go:117] "RemoveContainer" containerID="f5ae8962cde303ba5b72a08f62f675538a0e34837da715ac26a018013e049aa2"
Jul 08 16:38:59.184172 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:38:59.184152    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-qn2bx_openshift-console-operator(4717bdee-2017-4551-b0ec-683ab07f7660)\"" pod="openshift-console-operator/console-operator-68b757fb67-qn2bx" podUID="4717bdee-2017-4551-b0ec-683ab07f7660"
Jul 08 16:38:59.185325 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:59.185299    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-ctm5m" event={"ID":"ec3a3d1a-9e90-40a7-9695-7a1c81f3c41c","Type":"ContainerStarted","Data":"c082d52676ba4ca60d5e5018efc708415a5c788f27798e1c0c7abd372d32dc4d"}
Jul 08 16:38:59.217610 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:59.217559    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5c847dcfc4-ctm5m" podStartSLOduration=4.179428962 podStartE2EDuration="6.217542619s" totalImagesPullingTime="2.038113657s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:38:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:38:59.217146017 +0000 UTC m=+251.488595560" watchObservedRunningTime="2026-07-08 16:38:59.217542619 +0000 UTC m=+251.488992158"
Jul 08 16:38:59.902219 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:59.902181    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-r6rhh"]
Jul 08 16:38:59.904346 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:59.904329    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-r6rhh"
Jul 08 16:38:59.906842 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:59.906813    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8kpkx\""
Jul 08 16:38:59.906977 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:59.906822    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 08 16:38:59.907493 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:59.907473    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 08 16:38:59.917854 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:59.917825    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-r6rhh"]
Jul 08 16:38:59.985419 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:59.985384    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhr8d\" (UniqueName: \"kubernetes.io/projected/0580073c-aa77-4d66-ab87-156ce96691ba-kube-api-access-mhr8d\") pod \"insights-runtime-extractor-r6rhh\" (UID: \"0580073c-aa77-4d66-ab87-156ce96691ba\") " pod="openshift-insights/insights-runtime-extractor-r6rhh"
Jul 08 16:38:59.985419 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:59.985422    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0580073c-aa77-4d66-ab87-156ce96691ba-crio-socket\") pod \"insights-runtime-extractor-r6rhh\" (UID: \"0580073c-aa77-4d66-ab87-156ce96691ba\") " pod="openshift-insights/insights-runtime-extractor-r6rhh"
Jul 08 16:38:59.985691 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:59.985443    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0580073c-aa77-4d66-ab87-156ce96691ba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r6rhh\" (UID: \"0580073c-aa77-4d66-ab87-156ce96691ba\") " pod="openshift-insights/insights-runtime-extractor-r6rhh"
Jul 08 16:38:59.985691 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:59.985582    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0580073c-aa77-4d66-ab87-156ce96691ba-data-volume\") pod \"insights-runtime-extractor-r6rhh\" (UID: \"0580073c-aa77-4d66-ab87-156ce96691ba\") " pod="openshift-insights/insights-runtime-extractor-r6rhh"
Jul 08 16:38:59.985691 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:38:59.985650    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0580073c-aa77-4d66-ab87-156ce96691ba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r6rhh\" (UID: \"0580073c-aa77-4d66-ab87-156ce96691ba\") " pod="openshift-insights/insights-runtime-extractor-r6rhh"
Jul 08 16:39:00.086098 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:00.086058    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0580073c-aa77-4d66-ab87-156ce96691ba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r6rhh\" (UID: \"0580073c-aa77-4d66-ab87-156ce96691ba\") " pod="openshift-insights/insights-runtime-extractor-r6rhh"
Jul 08 16:39:00.086302 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:00.086110    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhr8d\" (UniqueName: \"kubernetes.io/projected/0580073c-aa77-4d66-ab87-156ce96691ba-kube-api-access-mhr8d\") pod \"insights-runtime-extractor-r6rhh\" (UID: \"0580073c-aa77-4d66-ab87-156ce96691ba\") " pod="openshift-insights/insights-runtime-extractor-r6rhh"
Jul 08 16:39:00.086302 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:00.086127    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0580073c-aa77-4d66-ab87-156ce96691ba-crio-socket\") pod \"insights-runtime-extractor-r6rhh\" (UID: \"0580073c-aa77-4d66-ab87-156ce96691ba\") " pod="openshift-insights/insights-runtime-extractor-r6rhh"
Jul 08 16:39:00.086302 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:00.086142    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0580073c-aa77-4d66-ab87-156ce96691ba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r6rhh\" (UID: \"0580073c-aa77-4d66-ab87-156ce96691ba\") " pod="openshift-insights/insights-runtime-extractor-r6rhh"
Jul 08 16:39:00.086302 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:00.086184    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0580073c-aa77-4d66-ab87-156ce96691ba-data-volume\") pod \"insights-runtime-extractor-r6rhh\" (UID: \"0580073c-aa77-4d66-ab87-156ce96691ba\") " pod="openshift-insights/insights-runtime-extractor-r6rhh"
Jul 08 16:39:00.086302 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:39:00.086241    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 08 16:39:00.086302 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:39:00.086264    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 08 16:39:00.086302 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:00.086245    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0580073c-aa77-4d66-ab87-156ce96691ba-crio-socket\") pod \"insights-runtime-extractor-r6rhh\" (UID: \"0580073c-aa77-4d66-ab87-156ce96691ba\") " pod="openshift-insights/insights-runtime-extractor-r6rhh"
Jul 08 16:39:00.086536 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:39:00.086323    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0580073c-aa77-4d66-ab87-156ce96691ba-kube-rbac-proxy-cm podName:0580073c-aa77-4d66-ab87-156ce96691ba nodeName:}" failed. No retries permitted until 2026-07-08 16:39:00.586302327 +0000 UTC m=+252.857751845 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0580073c-aa77-4d66-ab87-156ce96691ba-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r6rhh" (UID: "0580073c-aa77-4d66-ab87-156ce96691ba") : configmap "kube-rbac-proxy" not found
Jul 08 16:39:00.086536 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:39:00.086362    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0580073c-aa77-4d66-ab87-156ce96691ba-insights-runtime-extractor-tls podName:0580073c-aa77-4d66-ab87-156ce96691ba nodeName:}" failed. No retries permitted until 2026-07-08 16:39:00.586341404 +0000 UTC m=+252.857790937 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0580073c-aa77-4d66-ab87-156ce96691ba-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r6rhh" (UID: "0580073c-aa77-4d66-ab87-156ce96691ba") : secret "insights-runtime-extractor-tls" not found
Jul 08 16:39:00.086536 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:00.086499    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0580073c-aa77-4d66-ab87-156ce96691ba-data-volume\") pod \"insights-runtime-extractor-r6rhh\" (UID: \"0580073c-aa77-4d66-ab87-156ce96691ba\") " pod="openshift-insights/insights-runtime-extractor-r6rhh"
Jul 08 16:39:00.094385 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:00.094347    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhr8d\" (UniqueName: \"kubernetes.io/projected/0580073c-aa77-4d66-ab87-156ce96691ba-kube-api-access-mhr8d\") pod \"insights-runtime-extractor-r6rhh\" (UID: \"0580073c-aa77-4d66-ab87-156ce96691ba\") " pod="openshift-insights/insights-runtime-extractor-r6rhh"
Jul 08 16:39:00.589388 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:00.589339    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0580073c-aa77-4d66-ab87-156ce96691ba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r6rhh\" (UID: \"0580073c-aa77-4d66-ab87-156ce96691ba\") " pod="openshift-insights/insights-runtime-extractor-r6rhh"
Jul 08 16:39:00.589873 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:00.589457    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0580073c-aa77-4d66-ab87-156ce96691ba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r6rhh\" (UID: \"0580073c-aa77-4d66-ab87-156ce96691ba\") " pod="openshift-insights/insights-runtime-extractor-r6rhh"
Jul 08 16:39:00.589873 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:39:00.589540    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 08 16:39:00.589873 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:39:00.589556    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 08 16:39:00.589873 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:39:00.589614    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0580073c-aa77-4d66-ab87-156ce96691ba-insights-runtime-extractor-tls podName:0580073c-aa77-4d66-ab87-156ce96691ba nodeName:}" failed. No retries permitted until 2026-07-08 16:39:01.589596432 +0000 UTC m=+253.861045953 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0580073c-aa77-4d66-ab87-156ce96691ba-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r6rhh" (UID: "0580073c-aa77-4d66-ab87-156ce96691ba") : secret "insights-runtime-extractor-tls" not found
Jul 08 16:39:00.589873 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:39:00.589662    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0580073c-aa77-4d66-ab87-156ce96691ba-kube-rbac-proxy-cm podName:0580073c-aa77-4d66-ab87-156ce96691ba nodeName:}" failed. No retries permitted until 2026-07-08 16:39:01.589623536 +0000 UTC m=+253.861073054 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0580073c-aa77-4d66-ab87-156ce96691ba-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r6rhh" (UID: "0580073c-aa77-4d66-ab87-156ce96691ba") : configmap "kube-rbac-proxy" not found
Jul 08 16:39:00.676825 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:00.676787    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-7bc5n"]
Jul 08 16:39:00.679021 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:00.679003    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-7bc5n"
Jul 08 16:39:00.681465 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:00.681437    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 08 16:39:00.681465 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:00.681454    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 08 16:39:00.682234 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:00.682215    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-mc54k\""
Jul 08 16:39:00.682312 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:00.682262    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 08 16:39:00.682312 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:00.682285    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 08 16:39:00.687787 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:00.687760    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-7bc5n"]
Jul 08 16:39:00.791972 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:00.791930    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3cc361b3-d3e8-4ecc-a074-78136e59c847-signing-key\") pod \"service-ca-758f599c5-7bc5n\" (UID: \"3cc361b3-d3e8-4ecc-a074-78136e59c847\") " pod="openshift-service-ca/service-ca-758f599c5-7bc5n"
Jul 08 16:39:00.791972 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:00.791978    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3cc361b3-d3e8-4ecc-a074-78136e59c847-signing-cabundle\") pod \"service-ca-758f599c5-7bc5n\" (UID: \"3cc361b3-d3e8-4ecc-a074-78136e59c847\") " pod="openshift-service-ca/service-ca-758f599c5-7bc5n"
Jul 08 16:39:00.792184 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:00.791996    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlvrh\" (UniqueName: \"kubernetes.io/projected/3cc361b3-d3e8-4ecc-a074-78136e59c847-kube-api-access-hlvrh\") pod \"service-ca-758f599c5-7bc5n\" (UID: \"3cc361b3-d3e8-4ecc-a074-78136e59c847\") " pod="openshift-service-ca/service-ca-758f599c5-7bc5n"
Jul 08 16:39:00.893553 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:00.893461    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3cc361b3-d3e8-4ecc-a074-78136e59c847-signing-key\") pod \"service-ca-758f599c5-7bc5n\" (UID: \"3cc361b3-d3e8-4ecc-a074-78136e59c847\") " pod="openshift-service-ca/service-ca-758f599c5-7bc5n"
Jul 08 16:39:00.893553 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:00.893498    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3cc361b3-d3e8-4ecc-a074-78136e59c847-signing-cabundle\") pod \"service-ca-758f599c5-7bc5n\" (UID: \"3cc361b3-d3e8-4ecc-a074-78136e59c847\") " pod="openshift-service-ca/service-ca-758f599c5-7bc5n"
Jul 08 16:39:00.893553 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:00.893519    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlvrh\" (UniqueName: \"kubernetes.io/projected/3cc361b3-d3e8-4ecc-a074-78136e59c847-kube-api-access-hlvrh\") pod \"service-ca-758f599c5-7bc5n\" (UID: \"3cc361b3-d3e8-4ecc-a074-78136e59c847\") " pod="openshift-service-ca/service-ca-758f599c5-7bc5n"
Jul 08 16:39:00.894257 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:00.894235    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3cc361b3-d3e8-4ecc-a074-78136e59c847-signing-cabundle\") pod \"service-ca-758f599c5-7bc5n\" (UID: \"3cc361b3-d3e8-4ecc-a074-78136e59c847\") " pod="openshift-service-ca/service-ca-758f599c5-7bc5n"
Jul 08 16:39:00.896090 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:00.896063    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3cc361b3-d3e8-4ecc-a074-78136e59c847-signing-key\") pod \"service-ca-758f599c5-7bc5n\" (UID: \"3cc361b3-d3e8-4ecc-a074-78136e59c847\") " pod="openshift-service-ca/service-ca-758f599c5-7bc5n"
Jul 08 16:39:00.907605 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:00.907571    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlvrh\" (UniqueName: \"kubernetes.io/projected/3cc361b3-d3e8-4ecc-a074-78136e59c847-kube-api-access-hlvrh\") pod \"service-ca-758f599c5-7bc5n\" (UID: \"3cc361b3-d3e8-4ecc-a074-78136e59c847\") " pod="openshift-service-ca/service-ca-758f599c5-7bc5n"
Jul 08 16:39:00.987945 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:00.987910    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-7bc5n"
Jul 08 16:39:01.119752 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:01.119718    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-7bc5n"]
Jul 08 16:39:01.122973 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:39:01.122946    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cc361b3_d3e8_4ecc_a074_78136e59c847.slice/crio-3e71bbeb531b09cd44e079076de0240b6be536a6b2ea18ecf327abd548903509 WatchSource:0}: Error finding container 3e71bbeb531b09cd44e079076de0240b6be536a6b2ea18ecf327abd548903509: Status 404 returned error can't find the container with id 3e71bbeb531b09cd44e079076de0240b6be536a6b2ea18ecf327abd548903509
Jul 08 16:39:01.192128 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:01.192046    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-7bc5n" event={"ID":"3cc361b3-d3e8-4ecc-a074-78136e59c847","Type":"ContainerStarted","Data":"3e71bbeb531b09cd44e079076de0240b6be536a6b2ea18ecf327abd548903509"}
Jul 08 16:39:01.297783 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:01.297749    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dc3f4662-c8c9-42fc-8dfd-acac8b45f74c-registry-tls\") pod \"image-registry-55486dffd7-6ptzh\" (UID: \"dc3f4662-c8c9-42fc-8dfd-acac8b45f74c\") " pod="openshift-image-registry/image-registry-55486dffd7-6ptzh"
Jul 08 16:39:01.297991 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:39:01.297888    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 08 16:39:01.297991 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:39:01.297905    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55486dffd7-6ptzh: secret "image-registry-tls" not found
Jul 08 16:39:01.297991 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:39:01.297978    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dc3f4662-c8c9-42fc-8dfd-acac8b45f74c-registry-tls podName:dc3f4662-c8c9-42fc-8dfd-acac8b45f74c nodeName:}" failed. No retries permitted until 2026-07-08 16:39:09.297959206 +0000 UTC m=+261.569408741 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dc3f4662-c8c9-42fc-8dfd-acac8b45f74c-registry-tls") pod "image-registry-55486dffd7-6ptzh" (UID: "dc3f4662-c8c9-42fc-8dfd-acac8b45f74c") : secret "image-registry-tls" not found
Jul 08 16:39:01.601015 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:01.600970    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0580073c-aa77-4d66-ab87-156ce96691ba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r6rhh\" (UID: \"0580073c-aa77-4d66-ab87-156ce96691ba\") " pod="openshift-insights/insights-runtime-extractor-r6rhh"
Jul 08 16:39:01.601445 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:01.601053    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0580073c-aa77-4d66-ab87-156ce96691ba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r6rhh\" (UID: \"0580073c-aa77-4d66-ab87-156ce96691ba\") " pod="openshift-insights/insights-runtime-extractor-r6rhh"
Jul 08 16:39:01.601445 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:39:01.601116    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 08 16:39:01.601445 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:39:01.601158    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 08 16:39:01.601445 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:39:01.601169    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0580073c-aa77-4d66-ab87-156ce96691ba-kube-rbac-proxy-cm podName:0580073c-aa77-4d66-ab87-156ce96691ba nodeName:}" failed. No retries permitted until 2026-07-08 16:39:03.601156142 +0000 UTC m=+255.872605660 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0580073c-aa77-4d66-ab87-156ce96691ba-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r6rhh" (UID: "0580073c-aa77-4d66-ab87-156ce96691ba") : configmap "kube-rbac-proxy" not found
Jul 08 16:39:01.601445 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:39:01.601250    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0580073c-aa77-4d66-ab87-156ce96691ba-insights-runtime-extractor-tls podName:0580073c-aa77-4d66-ab87-156ce96691ba nodeName:}" failed. No retries permitted until 2026-07-08 16:39:03.601232836 +0000 UTC m=+255.872682358 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0580073c-aa77-4d66-ab87-156ce96691ba-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r6rhh" (UID: "0580073c-aa77-4d66-ab87-156ce96691ba") : secret "insights-runtime-extractor-tls" not found
Jul 08 16:39:02.889993 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:02.889958    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wflvt_84d1dbea-4585-46d9-b520-1329dba8df82/dns-node-resolver/0.log"
Jul 08 16:39:03.617050 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:03.617010    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0580073c-aa77-4d66-ab87-156ce96691ba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r6rhh\" (UID: \"0580073c-aa77-4d66-ab87-156ce96691ba\") " pod="openshift-insights/insights-runtime-extractor-r6rhh"
Jul 08 16:39:03.617224 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:03.617070    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0580073c-aa77-4d66-ab87-156ce96691ba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r6rhh\" (UID: \"0580073c-aa77-4d66-ab87-156ce96691ba\") " pod="openshift-insights/insights-runtime-extractor-r6rhh"
Jul 08 16:39:03.617224 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:39:03.617137    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 08 16:39:03.617224 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:39:03.617187    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 08 16:39:03.617224 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:39:03.617218    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0580073c-aa77-4d66-ab87-156ce96691ba-kube-rbac-proxy-cm podName:0580073c-aa77-4d66-ab87-156ce96691ba nodeName:}" failed. No retries permitted until 2026-07-08 16:39:07.617197366 +0000 UTC m=+259.888646884 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0580073c-aa77-4d66-ab87-156ce96691ba-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r6rhh" (UID: "0580073c-aa77-4d66-ab87-156ce96691ba") : configmap "kube-rbac-proxy" not found
Jul 08 16:39:03.617459 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:39:03.617236    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0580073c-aa77-4d66-ab87-156ce96691ba-insights-runtime-extractor-tls podName:0580073c-aa77-4d66-ab87-156ce96691ba nodeName:}" failed. No retries permitted until 2026-07-08 16:39:07.617227464 +0000 UTC m=+259.888676982 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0580073c-aa77-4d66-ab87-156ce96691ba-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r6rhh" (UID: "0580073c-aa77-4d66-ab87-156ce96691ba") : secret "insights-runtime-extractor-tls" not found
Jul 08 16:39:03.852800 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:03.852753    2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-qn2bx"
Jul 08 16:39:03.853336 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:03.853309    2578 scope.go:117] "RemoveContainer" containerID="f5ae8962cde303ba5b72a08f62f675538a0e34837da715ac26a018013e049aa2"
Jul 08 16:39:03.853545 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:39:03.853523    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-qn2bx_openshift-console-operator(4717bdee-2017-4551-b0ec-683ab07f7660)\"" pod="openshift-console-operator/console-operator-68b757fb67-qn2bx" podUID="4717bdee-2017-4551-b0ec-683ab07f7660"
Jul 08 16:39:03.890148 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:03.890079    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vnjdb_416e11c0-3ba2-451d-a901-9681ec0d4e3e/node-ca/0.log"
Jul 08 16:39:04.201896 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:04.201810    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-7bc5n" event={"ID":"3cc361b3-d3e8-4ecc-a074-78136e59c847","Type":"ContainerStarted","Data":"1d9d5b2c7f47f83b21b46bc74bf23ff7150e78fd3e5423b24e4938a80b99b576"}
Jul 08 16:39:04.689482 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:04.689451    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-clfj6_8fad727b-fceb-4ac4-b275-bd741597566e/serve-healthcheck-canary/0.log"
Jul 08 16:39:06.160773 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:06.160729    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-qn2bx"
Jul 08 16:39:06.161234 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:06.161103    2578 scope.go:117] "RemoveContainer" containerID="f5ae8962cde303ba5b72a08f62f675538a0e34837da715ac26a018013e049aa2"
Jul 08 16:39:06.161288 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:39:06.161249    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-qn2bx_openshift-console-operator(4717bdee-2017-4551-b0ec-683ab07f7660)\"" pod="openshift-console-operator/console-operator-68b757fb67-qn2bx" podUID="4717bdee-2017-4551-b0ec-683ab07f7660"
Jul 08 16:39:07.653913 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:07.653875    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0580073c-aa77-4d66-ab87-156ce96691ba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r6rhh\" (UID: \"0580073c-aa77-4d66-ab87-156ce96691ba\") " pod="openshift-insights/insights-runtime-extractor-r6rhh"
Jul 08 16:39:07.654276 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:07.653953    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0580073c-aa77-4d66-ab87-156ce96691ba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r6rhh\" (UID: \"0580073c-aa77-4d66-ab87-156ce96691ba\") " pod="openshift-insights/insights-runtime-extractor-r6rhh"
Jul 08 16:39:07.654276 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:39:07.654026    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 08 16:39:07.654276 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:39:07.654035    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 08 16:39:07.654276 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:39:07.654078    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0580073c-aa77-4d66-ab87-156ce96691ba-kube-rbac-proxy-cm podName:0580073c-aa77-4d66-ab87-156ce96691ba nodeName:}" failed. No retries permitted until 2026-07-08 16:39:15.654064898 +0000 UTC m=+267.925514416 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0580073c-aa77-4d66-ab87-156ce96691ba-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r6rhh" (UID: "0580073c-aa77-4d66-ab87-156ce96691ba") : configmap "kube-rbac-proxy" not found
Jul 08 16:39:07.654276 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:39:07.654097    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0580073c-aa77-4d66-ab87-156ce96691ba-insights-runtime-extractor-tls podName:0580073c-aa77-4d66-ab87-156ce96691ba nodeName:}" failed. No retries permitted until 2026-07-08 16:39:15.654083927 +0000 UTC m=+267.925533445 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0580073c-aa77-4d66-ab87-156ce96691ba-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r6rhh" (UID: "0580073c-aa77-4d66-ab87-156ce96691ba") : secret "insights-runtime-extractor-tls" not found
Jul 08 16:39:09.368197 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:09.368108    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dc3f4662-c8c9-42fc-8dfd-acac8b45f74c-registry-tls\") pod \"image-registry-55486dffd7-6ptzh\" (UID: \"dc3f4662-c8c9-42fc-8dfd-acac8b45f74c\") " pod="openshift-image-registry/image-registry-55486dffd7-6ptzh"
Jul 08 16:39:09.370839 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:09.370809    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dc3f4662-c8c9-42fc-8dfd-acac8b45f74c-registry-tls\") pod \"image-registry-55486dffd7-6ptzh\" (UID: \"dc3f4662-c8c9-42fc-8dfd-acac8b45f74c\") " pod="openshift-image-registry/image-registry-55486dffd7-6ptzh"
Jul 08 16:39:09.457199 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:09.457166    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55486dffd7-6ptzh"
Jul 08 16:39:09.588791 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:09.588744    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-7bc5n" podStartSLOduration=6.712855296 podStartE2EDuration="9.588726381s" totalImagesPullingTime="2.875871085s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:39:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:39:04.216913957 +0000 UTC m=+256.488363498" watchObservedRunningTime="2026-07-08 16:39:09.588726381 +0000 UTC m=+261.860175966"
Jul 08 16:39:09.589528 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:09.589509    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-55486dffd7-6ptzh"]
Jul 08 16:39:09.592343 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:39:09.592319    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc3f4662_c8c9_42fc_8dfd_acac8b45f74c.slice/crio-8a2f2826a2e7d06f9b0a3c458f53931152180ca44de62a429fc049578aa8a974 WatchSource:0}: Error finding container 8a2f2826a2e7d06f9b0a3c458f53931152180ca44de62a429fc049578aa8a974: Status 404 returned error can't find the container with id 8a2f2826a2e7d06f9b0a3c458f53931152180ca44de62a429fc049578aa8a974
Jul 08 16:39:10.222723 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:10.222688    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55486dffd7-6ptzh" event={"ID":"dc3f4662-c8c9-42fc-8dfd-acac8b45f74c","Type":"ContainerStarted","Data":"407fc8eda2d135b9993ee12261f6da7f09b30b2cb90dfb28fa545ffcc8629062"}
Jul 08 16:39:10.222904 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:10.222738    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55486dffd7-6ptzh" event={"ID":"dc3f4662-c8c9-42fc-8dfd-acac8b45f74c","Type":"ContainerStarted","Data":"8a2f2826a2e7d06f9b0a3c458f53931152180ca44de62a429fc049578aa8a974"}
Jul 08 16:39:10.222904 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:10.222810    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-55486dffd7-6ptzh"
Jul 08 16:39:10.247146 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:10.247091    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-55486dffd7-6ptzh" podStartSLOduration=17.24707013 podStartE2EDuration="17.24707013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:38:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:39:10.245332242 +0000 UTC m=+262.516781783" watchObservedRunningTime="2026-07-08 16:39:10.24707013 +0000 UTC m=+262.518519670"
Jul 08 16:39:15.720102 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:15.720061    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0580073c-aa77-4d66-ab87-156ce96691ba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r6rhh\" (UID: \"0580073c-aa77-4d66-ab87-156ce96691ba\") " pod="openshift-insights/insights-runtime-extractor-r6rhh"
Jul 08 16:39:15.720522 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:15.720161    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0580073c-aa77-4d66-ab87-156ce96691ba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r6rhh\" (UID: \"0580073c-aa77-4d66-ab87-156ce96691ba\") " pod="openshift-insights/insights-runtime-extractor-r6rhh"
Jul 08 16:39:15.720812 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:15.720794    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0580073c-aa77-4d66-ab87-156ce96691ba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r6rhh\" (UID: \"0580073c-aa77-4d66-ab87-156ce96691ba\") " pod="openshift-insights/insights-runtime-extractor-r6rhh"
Jul 08 16:39:15.722609 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:15.722585    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0580073c-aa77-4d66-ab87-156ce96691ba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r6rhh\" (UID: \"0580073c-aa77-4d66-ab87-156ce96691ba\") " pod="openshift-insights/insights-runtime-extractor-r6rhh"
Jul 08 16:39:15.814209 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:15.814165    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-r6rhh"
Jul 08 16:39:15.942557 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:15.942523    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-r6rhh"]
Jul 08 16:39:15.946038 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:39:15.946009    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0580073c_aa77_4d66_ab87_156ce96691ba.slice/crio-b2759aaff2dbee26f360ff6d3d8b16d3849deaa400095384fae0c77796aee724 WatchSource:0}: Error finding container b2759aaff2dbee26f360ff6d3d8b16d3849deaa400095384fae0c77796aee724: Status 404 returned error can't find the container with id b2759aaff2dbee26f360ff6d3d8b16d3849deaa400095384fae0c77796aee724
Jul 08 16:39:16.242302 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:16.242271    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r6rhh" event={"ID":"0580073c-aa77-4d66-ab87-156ce96691ba","Type":"ContainerStarted","Data":"b2759aaff2dbee26f360ff6d3d8b16d3849deaa400095384fae0c77796aee724"}
Jul 08 16:39:18.282211 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:18.282179    2578 scope.go:117] "RemoveContainer" containerID="f5ae8962cde303ba5b72a08f62f675538a0e34837da715ac26a018013e049aa2"
Jul 08 16:39:19.164057 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:19.164016    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-55486dffd7-6ptzh"]
Jul 08 16:39:19.212131 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:19.212100    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-685fc4f98d-99nqw"]
Jul 08 16:39:19.223483 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:19.223458    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-685fc4f98d-99nqw"
Jul 08 16:39:19.234001 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:19.233976    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-685fc4f98d-99nqw"]
Jul 08 16:39:19.250144 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:19.250117    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/654454ed-ca06-46b1-a030-b33bb974dd6d-bound-sa-token\") pod \"image-registry-685fc4f98d-99nqw\" (UID: \"654454ed-ca06-46b1-a030-b33bb974dd6d\") " pod="openshift-image-registry/image-registry-685fc4f98d-99nqw"
Jul 08 16:39:19.250351 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:19.250151    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/654454ed-ca06-46b1-a030-b33bb974dd6d-registry-tls\") pod \"image-registry-685fc4f98d-99nqw\" (UID: \"654454ed-ca06-46b1-a030-b33bb974dd6d\") " pod="openshift-image-registry/image-registry-685fc4f98d-99nqw"
Jul 08 16:39:19.250351 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:19.250166    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/654454ed-ca06-46b1-a030-b33bb974dd6d-trusted-ca\") pod \"image-registry-685fc4f98d-99nqw\" (UID: \"654454ed-ca06-46b1-a030-b33bb974dd6d\") " pod="openshift-image-registry/image-registry-685fc4f98d-99nqw"
Jul 08 16:39:19.250351 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:19.250194    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrlzr\" (UniqueName: \"kubernetes.io/projected/654454ed-ca06-46b1-a030-b33bb974dd6d-kube-api-access-jrlzr\") pod \"image-registry-685fc4f98d-99nqw\" (UID: \"654454ed-ca06-46b1-a030-b33bb974dd6d\") " pod="openshift-image-registry/image-registry-685fc4f98d-99nqw"
Jul 08 16:39:19.250351 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:19.250285    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/654454ed-ca06-46b1-a030-b33bb974dd6d-installation-pull-secrets\") pod \"image-registry-685fc4f98d-99nqw\" (UID: \"654454ed-ca06-46b1-a030-b33bb974dd6d\") " pod="openshift-image-registry/image-registry-685fc4f98d-99nqw"
Jul 08 16:39:19.250351 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:19.250332    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/654454ed-ca06-46b1-a030-b33bb974dd6d-ca-trust-extracted\") pod \"image-registry-685fc4f98d-99nqw\" (UID: \"654454ed-ca06-46b1-a030-b33bb974dd6d\") " pod="openshift-image-registry/image-registry-685fc4f98d-99nqw"
Jul 08 16:39:19.250619 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:19.250360    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/654454ed-ca06-46b1-a030-b33bb974dd6d-registry-certificates\") pod \"image-registry-685fc4f98d-99nqw\" (UID: \"654454ed-ca06-46b1-a030-b33bb974dd6d\") " pod="openshift-image-registry/image-registry-685fc4f98d-99nqw"
Jul 08 16:39:19.250619 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:19.250391    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/654454ed-ca06-46b1-a030-b33bb974dd6d-image-registry-private-configuration\") pod \"image-registry-685fc4f98d-99nqw\" (UID: \"654454ed-ca06-46b1-a030-b33bb974dd6d\") " pod="openshift-image-registry/image-registry-685fc4f98d-99nqw"
Jul 08 16:39:19.351645 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:19.351609    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/654454ed-ca06-46b1-a030-b33bb974dd6d-installation-pull-secrets\") pod \"image-registry-685fc4f98d-99nqw\" (UID: \"654454ed-ca06-46b1-a030-b33bb974dd6d\") " pod="openshift-image-registry/image-registry-685fc4f98d-99nqw"
Jul 08 16:39:19.352093 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:19.351703    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/654454ed-ca06-46b1-a030-b33bb974dd6d-ca-trust-extracted\") pod \"image-registry-685fc4f98d-99nqw\" (UID: \"654454ed-ca06-46b1-a030-b33bb974dd6d\") " pod="openshift-image-registry/image-registry-685fc4f98d-99nqw"
Jul 08 16:39:19.352093 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:19.351732    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/654454ed-ca06-46b1-a030-b33bb974dd6d-registry-certificates\") pod \"image-registry-685fc4f98d-99nqw\" (UID: \"654454ed-ca06-46b1-a030-b33bb974dd6d\") " pod="openshift-image-registry/image-registry-685fc4f98d-99nqw"
Jul 08 16:39:19.352093 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:19.351799    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/654454ed-ca06-46b1-a030-b33bb974dd6d-image-registry-private-configuration\") pod \"image-registry-685fc4f98d-99nqw\" (UID: \"654454ed-ca06-46b1-a030-b33bb974dd6d\") " pod="openshift-image-registry/image-registry-685fc4f98d-99nqw"
Jul 08 16:39:19.352093 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:19.351840    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/654454ed-ca06-46b1-a030-b33bb974dd6d-bound-sa-token\") pod \"image-registry-685fc4f98d-99nqw\" (UID: \"654454ed-ca06-46b1-a030-b33bb974dd6d\") " pod="openshift-image-registry/image-registry-685fc4f98d-99nqw"
Jul 08 16:39:19.352093 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:19.351865    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/654454ed-ca06-46b1-a030-b33bb974dd6d-registry-tls\") pod \"image-registry-685fc4f98d-99nqw\" (UID: \"654454ed-ca06-46b1-a030-b33bb974dd6d\") " pod="openshift-image-registry/image-registry-685fc4f98d-99nqw"
Jul 08 16:39:19.352093 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:19.351889    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/654454ed-ca06-46b1-a030-b33bb974dd6d-trusted-ca\") pod \"image-registry-685fc4f98d-99nqw\" (UID: \"654454ed-ca06-46b1-a030-b33bb974dd6d\") " pod="openshift-image-registry/image-registry-685fc4f98d-99nqw"
Jul 08 16:39:19.352093 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:19.351931    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrlzr\" (UniqueName: \"kubernetes.io/projected/654454ed-ca06-46b1-a030-b33bb974dd6d-kube-api-access-jrlzr\") pod \"image-registry-685fc4f98d-99nqw\" (UID: \"654454ed-ca06-46b1-a030-b33bb974dd6d\") " pod="openshift-image-registry/image-registry-685fc4f98d-99nqw"
Jul 08 16:39:19.352711 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:19.352659    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/654454ed-ca06-46b1-a030-b33bb974dd6d-ca-trust-extracted\") pod \"image-registry-685fc4f98d-99nqw\" (UID: \"654454ed-ca06-46b1-a030-b33bb974dd6d\") " pod="openshift-image-registry/image-registry-685fc4f98d-99nqw"
Jul 08 16:39:19.353210 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:19.353185    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/654454ed-ca06-46b1-a030-b33bb974dd6d-registry-certificates\") pod \"image-registry-685fc4f98d-99nqw\" (UID: \"654454ed-ca06-46b1-a030-b33bb974dd6d\") " pod="openshift-image-registry/image-registry-685fc4f98d-99nqw"
Jul 08 16:39:19.353348 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:19.353283    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/654454ed-ca06-46b1-a030-b33bb974dd6d-trusted-ca\") pod \"image-registry-685fc4f98d-99nqw\" (UID: \"654454ed-ca06-46b1-a030-b33bb974dd6d\") " pod="openshift-image-registry/image-registry-685fc4f98d-99nqw"
Jul 08 16:39:19.354844 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:19.354822    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/654454ed-ca06-46b1-a030-b33bb974dd6d-installation-pull-secrets\") pod \"image-registry-685fc4f98d-99nqw\" (UID: \"654454ed-ca06-46b1-a030-b33bb974dd6d\") " pod="openshift-image-registry/image-registry-685fc4f98d-99nqw"
Jul 08 16:39:19.354934 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:19.354843    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/654454ed-ca06-46b1-a030-b33bb974dd6d-image-registry-private-configuration\") pod \"image-registry-685fc4f98d-99nqw\" (UID: \"654454ed-ca06-46b1-a030-b33bb974dd6d\") " pod="openshift-image-registry/image-registry-685fc4f98d-99nqw"
Jul 08 16:39:19.355012 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:19.354996    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/654454ed-ca06-46b1-a030-b33bb974dd6d-registry-tls\") pod \"image-registry-685fc4f98d-99nqw\" (UID: \"654454ed-ca06-46b1-a030-b33bb974dd6d\") " pod="openshift-image-registry/image-registry-685fc4f98d-99nqw"
Jul 08 16:39:19.361895 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:19.361873    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/654454ed-ca06-46b1-a030-b33bb974dd6d-bound-sa-token\") pod \"image-registry-685fc4f98d-99nqw\" (UID: \"654454ed-ca06-46b1-a030-b33bb974dd6d\") " pod="openshift-image-registry/image-registry-685fc4f98d-99nqw"
Jul 08 16:39:19.364490 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:19.364465    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrlzr\" (UniqueName: \"kubernetes.io/projected/654454ed-ca06-46b1-a030-b33bb974dd6d-kube-api-access-jrlzr\") pod \"image-registry-685fc4f98d-99nqw\" (UID: \"654454ed-ca06-46b1-a030-b33bb974dd6d\") " pod="openshift-image-registry/image-registry-685fc4f98d-99nqw"
Jul 08 16:39:19.532859 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:19.532765    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-685fc4f98d-99nqw"
Jul 08 16:39:19.700862 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:19.700604    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-685fc4f98d-99nqw"]
Jul 08 16:39:19.704151 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:39:19.704121    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod654454ed_ca06_46b1_a030_b33bb974dd6d.slice/crio-1ac8b143689a9ad00809e87fbcab2aa78c2a2b6e221d0fc886939f008c522336 WatchSource:0}: Error finding container 1ac8b143689a9ad00809e87fbcab2aa78c2a2b6e221d0fc886939f008c522336: Status 404 returned error can't find the container with id 1ac8b143689a9ad00809e87fbcab2aa78c2a2b6e221d0fc886939f008c522336
Jul 08 16:39:20.256857 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:20.256820    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r6rhh" event={"ID":"0580073c-aa77-4d66-ab87-156ce96691ba","Type":"ContainerStarted","Data":"a6844d58880be35e13f931a00aabc1eada709785c76a1cf313bb25cfeca9b0fb"}
Jul 08 16:39:20.258347 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:20.258323    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-685fc4f98d-99nqw" event={"ID":"654454ed-ca06-46b1-a030-b33bb974dd6d","Type":"ContainerStarted","Data":"073942d851584611444015006f0852cdc919d9fefc6b35d71502505e12aff712"}
Jul 08 16:39:20.258488 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:20.258364    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-685fc4f98d-99nqw" event={"ID":"654454ed-ca06-46b1-a030-b33bb974dd6d","Type":"ContainerStarted","Data":"1ac8b143689a9ad00809e87fbcab2aa78c2a2b6e221d0fc886939f008c522336"}
Jul 08 16:39:20.258488 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:20.258404    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-685fc4f98d-99nqw"
Jul 08 16:39:20.260311 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:20.260293    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-qn2bx_4717bdee-2017-4551-b0ec-683ab07f7660/console-operator/1.log"
Jul 08 16:39:20.260399 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:20.260333    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-qn2bx" event={"ID":"4717bdee-2017-4551-b0ec-683ab07f7660","Type":"ContainerStarted","Data":"74b4c1a760113dddc4c3974e929d5915024a2408c7b0d2a2ad118d984bc4d0ff"}
Jul 08 16:39:20.260581 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:20.260555    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-qn2bx"
Jul 08 16:39:20.278269 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:20.278119    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-685fc4f98d-99nqw" podStartSLOduration=1.278101065 podStartE2EDuration="1.278101065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:39:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:39:20.277442535 +0000 UTC m=+272.548892076" watchObservedRunningTime="2026-07-08 16:39:20.278101065 +0000 UTC m=+272.549550607"
Jul 08 16:39:20.408276 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:20.408245    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-68b757fb67-qn2bx"
Jul 08 16:39:22.269507 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:22.269468    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r6rhh" event={"ID":"0580073c-aa77-4d66-ab87-156ce96691ba","Type":"ContainerStarted","Data":"f41091bedd20d3ae9170c3cc4a1ae56f65324646225138363b6f89b7cc8fecac"}
Jul 08 16:39:25.281061 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:25.281027    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r6rhh" event={"ID":"0580073c-aa77-4d66-ab87-156ce96691ba","Type":"ContainerStarted","Data":"a61e720bd90abbe3715f9448b3c70adc8e461fe7267d2192ba0448e0ddd31c66"}
Jul 08 16:39:25.309431 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:25.309365    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-r6rhh" podStartSLOduration=18.061410879 podStartE2EDuration="26.309344566s" totalImagesPullingTime="8.247933687s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:38:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:39:25.307684451 +0000 UTC m=+277.579133991" watchObservedRunningTime="2026-07-08 16:39:25.309344566 +0000 UTC m=+277.580794106"
Jul 08 16:39:29.169558 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:29.169518    2578 patch_prober.go:28] interesting pod/image-registry-55486dffd7-6ptzh container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 08 16:39:29.169946 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:29.169571    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-55486dffd7-6ptzh" podUID="dc3f4662-c8c9-42fc-8dfd-acac8b45f74c" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:39:29.940885 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:39:29.940823    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-kbqcx" podUID="0ee105a4-0d7b-4a50-a212-110cac5e12fb"
Jul 08 16:39:30.297364 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:30.297286    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kbqcx"
Jul 08 16:39:31.595626 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:31.595591    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-j6s2h"]
Jul 08 16:39:31.598987 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:31.598971    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-j6s2h"
Jul 08 16:39:31.602509 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:31.602480    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 08 16:39:31.602730 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:31.602714    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 08 16:39:31.602790 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:31.602735    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 08 16:39:31.603608 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:31.603591    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 08 16:39:31.603707 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:31.603610    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-9z8vh\""
Jul 08 16:39:31.603707 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:31.603674    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 08 16:39:31.612788 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:31.612765    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-j6s2h"]
Jul 08 16:39:31.651590 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:31.651554    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc428bbb-0f04-4342-b29e-9f4dfe36fdd1-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-j6s2h\" (UID: \"dc428bbb-0f04-4342-b29e-9f4dfe36fdd1\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-j6s2h"
Jul 08 16:39:31.651791 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:31.651597    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/dc428bbb-0f04-4342-b29e-9f4dfe36fdd1-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-j6s2h\" (UID: \"dc428bbb-0f04-4342-b29e-9f4dfe36fdd1\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-j6s2h"
Jul 08 16:39:31.651791 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:31.651624    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dc428bbb-0f04-4342-b29e-9f4dfe36fdd1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-j6s2h\" (UID: \"dc428bbb-0f04-4342-b29e-9f4dfe36fdd1\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-j6s2h"
Jul 08 16:39:31.651791 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:31.651667    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wppzt\" (UniqueName: \"kubernetes.io/projected/dc428bbb-0f04-4342-b29e-9f4dfe36fdd1-kube-api-access-wppzt\") pod \"prometheus-operator-567f7b8b7c-j6s2h\" (UID: \"dc428bbb-0f04-4342-b29e-9f4dfe36fdd1\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-j6s2h"
Jul 08 16:39:31.752955 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:31.752918    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc428bbb-0f04-4342-b29e-9f4dfe36fdd1-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-j6s2h\" (UID: \"dc428bbb-0f04-4342-b29e-9f4dfe36fdd1\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-j6s2h"
Jul 08 16:39:31.753121 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:31.752970    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/dc428bbb-0f04-4342-b29e-9f4dfe36fdd1-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-j6s2h\" (UID: \"dc428bbb-0f04-4342-b29e-9f4dfe36fdd1\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-j6s2h"
Jul 08 16:39:31.753121 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:31.752995    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dc428bbb-0f04-4342-b29e-9f4dfe36fdd1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-j6s2h\" (UID: \"dc428bbb-0f04-4342-b29e-9f4dfe36fdd1\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-j6s2h"
Jul 08 16:39:31.753121 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:31.753022    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wppzt\" (UniqueName: \"kubernetes.io/projected/dc428bbb-0f04-4342-b29e-9f4dfe36fdd1-kube-api-access-wppzt\") pod \"prometheus-operator-567f7b8b7c-j6s2h\" (UID: \"dc428bbb-0f04-4342-b29e-9f4dfe36fdd1\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-j6s2h"
Jul 08 16:39:31.753239 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:39:31.753115    2578 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 08 16:39:31.753239 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:39:31.753207    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc428bbb-0f04-4342-b29e-9f4dfe36fdd1-prometheus-operator-tls podName:dc428bbb-0f04-4342-b29e-9f4dfe36fdd1 nodeName:}" failed. No retries permitted until 2026-07-08 16:39:32.253187657 +0000 UTC m=+284.524637179 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/dc428bbb-0f04-4342-b29e-9f4dfe36fdd1-prometheus-operator-tls") pod "prometheus-operator-567f7b8b7c-j6s2h" (UID: "dc428bbb-0f04-4342-b29e-9f4dfe36fdd1") : secret "prometheus-operator-tls" not found
Jul 08 16:39:31.753612 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:31.753579    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc428bbb-0f04-4342-b29e-9f4dfe36fdd1-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-j6s2h\" (UID: \"dc428bbb-0f04-4342-b29e-9f4dfe36fdd1\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-j6s2h"
Jul 08 16:39:31.755595 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:31.755572    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dc428bbb-0f04-4342-b29e-9f4dfe36fdd1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-j6s2h\" (UID: \"dc428bbb-0f04-4342-b29e-9f4dfe36fdd1\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-j6s2h"
Jul 08 16:39:31.763583 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:31.763556    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wppzt\" (UniqueName: \"kubernetes.io/projected/dc428bbb-0f04-4342-b29e-9f4dfe36fdd1-kube-api-access-wppzt\") pod \"prometheus-operator-567f7b8b7c-j6s2h\" (UID: \"dc428bbb-0f04-4342-b29e-9f4dfe36fdd1\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-j6s2h"
Jul 08 16:39:32.257027 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:32.256983    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/dc428bbb-0f04-4342-b29e-9f4dfe36fdd1-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-j6s2h\" (UID: \"dc428bbb-0f04-4342-b29e-9f4dfe36fdd1\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-j6s2h"
Jul 08 16:39:32.257219 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:39:32.257165    2578 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 08 16:39:32.257298 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:39:32.257247    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc428bbb-0f04-4342-b29e-9f4dfe36fdd1-prometheus-operator-tls podName:dc428bbb-0f04-4342-b29e-9f4dfe36fdd1 nodeName:}" failed. No retries permitted until 2026-07-08 16:39:33.257229725 +0000 UTC m=+285.528679243 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/dc428bbb-0f04-4342-b29e-9f4dfe36fdd1-prometheus-operator-tls") pod "prometheus-operator-567f7b8b7c-j6s2h" (UID: "dc428bbb-0f04-4342-b29e-9f4dfe36fdd1") : secret "prometheus-operator-tls" not found
Jul 08 16:39:32.301486 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:32.301458    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-66fdd9cf8b-f725b"]
Jul 08 16:39:32.303776 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:32.303756    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66fdd9cf8b-f725b"
Jul 08 16:39:32.306153 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:32.306127    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 08 16:39:32.306153 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:32.306133    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 08 16:39:32.306337 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:32.306158    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 08 16:39:32.307142 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:32.306958    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 08 16:39:32.307142 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:32.307020    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 08 16:39:32.307142 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:32.307036    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 08 16:39:32.307142 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:32.307051    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 08 16:39:32.307142 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:32.307066    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-f2htc\""
Jul 08 16:39:32.312176 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:32.312157    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 08 16:39:32.318285 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:32.318264    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66fdd9cf8b-f725b"]
Jul 08 16:39:32.357709 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:32.357667    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17fd5ffc-5451-4ed5-ae7e-42feeb4c9250-console-serving-cert\") pod \"console-66fdd9cf8b-f725b\" (UID: \"17fd5ffc-5451-4ed5-ae7e-42feeb4c9250\") " pod="openshift-console/console-66fdd9cf8b-f725b"
Jul 08 16:39:32.357877 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:32.357719    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17fd5ffc-5451-4ed5-ae7e-42feeb4c9250-console-config\") pod \"console-66fdd9cf8b-f725b\" (UID: \"17fd5ffc-5451-4ed5-ae7e-42feeb4c9250\") " pod="openshift-console/console-66fdd9cf8b-f725b"
Jul 08 16:39:32.357877 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:32.357783    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17fd5ffc-5451-4ed5-ae7e-42feeb4c9250-console-oauth-config\") pod \"console-66fdd9cf8b-f725b\" (UID: \"17fd5ffc-5451-4ed5-ae7e-42feeb4c9250\") " pod="openshift-console/console-66fdd9cf8b-f725b"
Jul 08 16:39:32.357877 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:32.357805    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17fd5ffc-5451-4ed5-ae7e-42feeb4c9250-trusted-ca-bundle\") pod \"console-66fdd9cf8b-f725b\" (UID: \"17fd5ffc-5451-4ed5-ae7e-42feeb4c9250\") " pod="openshift-console/console-66fdd9cf8b-f725b"
Jul 08 16:39:32.357877 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:32.357838    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17fd5ffc-5451-4ed5-ae7e-42feeb4c9250-oauth-serving-cert\") pod \"console-66fdd9cf8b-f725b\" (UID: \"17fd5ffc-5451-4ed5-ae7e-42feeb4c9250\") " pod="openshift-console/console-66fdd9cf8b-f725b"
Jul 08 16:39:32.358037 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:32.357887    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17fd5ffc-5451-4ed5-ae7e-42feeb4c9250-service-ca\") pod \"console-66fdd9cf8b-f725b\" (UID: \"17fd5ffc-5451-4ed5-ae7e-42feeb4c9250\") " pod="openshift-console/console-66fdd9cf8b-f725b"
Jul 08 16:39:32.358037 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:32.357926    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccs64\" (UniqueName: \"kubernetes.io/projected/17fd5ffc-5451-4ed5-ae7e-42feeb4c9250-kube-api-access-ccs64\") pod \"console-66fdd9cf8b-f725b\" (UID: \"17fd5ffc-5451-4ed5-ae7e-42feeb4c9250\") " pod="openshift-console/console-66fdd9cf8b-f725b"
Jul 08 16:39:32.458853 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:32.458817    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17fd5ffc-5451-4ed5-ae7e-42feeb4c9250-service-ca\") pod \"console-66fdd9cf8b-f725b\" (UID: \"17fd5ffc-5451-4ed5-ae7e-42feeb4c9250\") " pod="openshift-console/console-66fdd9cf8b-f725b"
Jul 08 16:39:32.459020 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:32.458884    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccs64\" (UniqueName: \"kubernetes.io/projected/17fd5ffc-5451-4ed5-ae7e-42feeb4c9250-kube-api-access-ccs64\") pod \"console-66fdd9cf8b-f725b\" (UID: \"17fd5ffc-5451-4ed5-ae7e-42feeb4c9250\") " pod="openshift-console/console-66fdd9cf8b-f725b"
Jul 08 16:39:32.459020 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:32.458938    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17fd5ffc-5451-4ed5-ae7e-42feeb4c9250-console-serving-cert\") pod \"console-66fdd9cf8b-f725b\" (UID: \"17fd5ffc-5451-4ed5-ae7e-42feeb4c9250\") " pod="openshift-console/console-66fdd9cf8b-f725b"
Jul 08 16:39:32.459020 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:32.458961    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17fd5ffc-5451-4ed5-ae7e-42feeb4c9250-console-config\") pod \"console-66fdd9cf8b-f725b\" (UID: \"17fd5ffc-5451-4ed5-ae7e-42feeb4c9250\") " pod="openshift-console/console-66fdd9cf8b-f725b"
Jul 08 16:39:32.459162 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:32.459021    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17fd5ffc-5451-4ed5-ae7e-42feeb4c9250-console-oauth-config\") pod \"console-66fdd9cf8b-f725b\" (UID: \"17fd5ffc-5451-4ed5-ae7e-42feeb4c9250\") " pod="openshift-console/console-66fdd9cf8b-f725b"
Jul 08 16:39:32.459162 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:32.459043    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17fd5ffc-5451-4ed5-ae7e-42feeb4c9250-trusted-ca-bundle\") pod \"console-66fdd9cf8b-f725b\" (UID: \"17fd5ffc-5451-4ed5-ae7e-42feeb4c9250\") " pod="openshift-console/console-66fdd9cf8b-f725b"
Jul 08 16:39:32.459162 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:32.459064    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17fd5ffc-5451-4ed5-ae7e-42feeb4c9250-oauth-serving-cert\") pod \"console-66fdd9cf8b-f725b\" (UID: \"17fd5ffc-5451-4ed5-ae7e-42feeb4c9250\") " pod="openshift-console/console-66fdd9cf8b-f725b"
Jul 08 16:39:32.459693 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:32.459658    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17fd5ffc-5451-4ed5-ae7e-42feeb4c9250-service-ca\") pod \"console-66fdd9cf8b-f725b\" (UID: \"17fd5ffc-5451-4ed5-ae7e-42feeb4c9250\") " pod="openshift-console/console-66fdd9cf8b-f725b"
Jul 08 16:39:32.459975 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:32.459794    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17fd5ffc-5451-4ed5-ae7e-42feeb4c9250-oauth-serving-cert\") pod \"console-66fdd9cf8b-f725b\" (UID: \"17fd5ffc-5451-4ed5-ae7e-42feeb4c9250\") " pod="openshift-console/console-66fdd9cf8b-f725b"
Jul 08 16:39:32.459975 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:32.459805    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17fd5ffc-5451-4ed5-ae7e-42feeb4c9250-console-config\") pod \"console-66fdd9cf8b-f725b\" (UID: \"17fd5ffc-5451-4ed5-ae7e-42feeb4c9250\") " pod="openshift-console/console-66fdd9cf8b-f725b"
Jul 08 16:39:32.459975 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:32.459936    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17fd5ffc-5451-4ed5-ae7e-42feeb4c9250-trusted-ca-bundle\") pod \"console-66fdd9cf8b-f725b\" (UID: \"17fd5ffc-5451-4ed5-ae7e-42feeb4c9250\") " pod="openshift-console/console-66fdd9cf8b-f725b"
Jul 08 16:39:32.461560 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:32.461535    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17fd5ffc-5451-4ed5-ae7e-42feeb4c9250-console-oauth-config\") pod \"console-66fdd9cf8b-f725b\" (UID: \"17fd5ffc-5451-4ed5-ae7e-42feeb4c9250\") " pod="openshift-console/console-66fdd9cf8b-f725b"
Jul 08 16:39:32.461720 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:32.461696    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17fd5ffc-5451-4ed5-ae7e-42feeb4c9250-console-serving-cert\") pod \"console-66fdd9cf8b-f725b\" (UID: \"17fd5ffc-5451-4ed5-ae7e-42feeb4c9250\") " pod="openshift-console/console-66fdd9cf8b-f725b"
Jul 08 16:39:32.468185 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:32.468162    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccs64\" (UniqueName: \"kubernetes.io/projected/17fd5ffc-5451-4ed5-ae7e-42feeb4c9250-kube-api-access-ccs64\") pod \"console-66fdd9cf8b-f725b\" (UID: \"17fd5ffc-5451-4ed5-ae7e-42feeb4c9250\") " pod="openshift-console/console-66fdd9cf8b-f725b"
Jul 08 16:39:32.613916 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:32.613879    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66fdd9cf8b-f725b"
Jul 08 16:39:32.747687 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:32.747622    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66fdd9cf8b-f725b"]
Jul 08 16:39:32.750277 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:39:32.750250    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17fd5ffc_5451_4ed5_ae7e_42feeb4c9250.slice/crio-407f616cc26c1662036a0ca2482bad85e46f462c19995e2db86e627f5850c3fd WatchSource:0}: Error finding container 407f616cc26c1662036a0ca2482bad85e46f462c19995e2db86e627f5850c3fd: Status 404 returned error can't find the container with id 407f616cc26c1662036a0ca2482bad85e46f462c19995e2db86e627f5850c3fd
Jul 08 16:39:33.265298 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:33.265259    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/dc428bbb-0f04-4342-b29e-9f4dfe36fdd1-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-j6s2h\" (UID: \"dc428bbb-0f04-4342-b29e-9f4dfe36fdd1\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-j6s2h"
Jul 08 16:39:33.267868 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:33.267839    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/dc428bbb-0f04-4342-b29e-9f4dfe36fdd1-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-j6s2h\" (UID: \"dc428bbb-0f04-4342-b29e-9f4dfe36fdd1\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-j6s2h"
Jul 08 16:39:33.308373 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:33.308342    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66fdd9cf8b-f725b" event={"ID":"17fd5ffc-5451-4ed5-ae7e-42feeb4c9250","Type":"ContainerStarted","Data":"407f616cc26c1662036a0ca2482bad85e46f462c19995e2db86e627f5850c3fd"}
Jul 08 16:39:33.408164 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:33.408115    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-j6s2h"
Jul 08 16:39:33.533556 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:33.533475    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-j6s2h"]
Jul 08 16:39:33.536061 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:39:33.536037    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc428bbb_0f04_4342_b29e_9f4dfe36fdd1.slice/crio-5c320af61f35b83ad94b6195d363495a0b8968bed32c4e535de45227ba7cd54c WatchSource:0}: Error finding container 5c320af61f35b83ad94b6195d363495a0b8968bed32c4e535de45227ba7cd54c: Status 404 returned error can't find the container with id 5c320af61f35b83ad94b6195d363495a0b8968bed32c4e535de45227ba7cd54c
Jul 08 16:39:33.568806 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:33.568772    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ee105a4-0d7b-4a50-a212-110cac5e12fb-metrics-tls\") pod \"dns-default-kbqcx\" (UID: \"0ee105a4-0d7b-4a50-a212-110cac5e12fb\") " pod="openshift-dns/dns-default-kbqcx"
Jul 08 16:39:33.571147 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:33.571127    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ee105a4-0d7b-4a50-a212-110cac5e12fb-metrics-tls\") pod \"dns-default-kbqcx\" (UID: \"0ee105a4-0d7b-4a50-a212-110cac5e12fb\") " pod="openshift-dns/dns-default-kbqcx"
Jul 08 16:39:33.600524 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:33.600496    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rvfsx\""
Jul 08 16:39:33.608504 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:33.608478    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kbqcx"
Jul 08 16:39:33.734200 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:33.734170    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kbqcx"]
Jul 08 16:39:33.737225 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:39:33.737203    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ee105a4_0d7b_4a50_a212_110cac5e12fb.slice/crio-adab3caf2bd3a716d400283969a05db54fe209fe1e08ecb102d85aba9859cd0d WatchSource:0}: Error finding container adab3caf2bd3a716d400283969a05db54fe209fe1e08ecb102d85aba9859cd0d: Status 404 returned error can't find the container with id adab3caf2bd3a716d400283969a05db54fe209fe1e08ecb102d85aba9859cd0d
Jul 08 16:39:34.313485 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:34.313438    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kbqcx" event={"ID":"0ee105a4-0d7b-4a50-a212-110cac5e12fb","Type":"ContainerStarted","Data":"adab3caf2bd3a716d400283969a05db54fe209fe1e08ecb102d85aba9859cd0d"}
Jul 08 16:39:34.314668 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:34.314616    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-j6s2h" event={"ID":"dc428bbb-0f04-4342-b29e-9f4dfe36fdd1","Type":"ContainerStarted","Data":"5c320af61f35b83ad94b6195d363495a0b8968bed32c4e535de45227ba7cd54c"}
Jul 08 16:39:37.332417 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:37.332337    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66fdd9cf8b-f725b" event={"ID":"17fd5ffc-5451-4ed5-ae7e-42feeb4c9250","Type":"ContainerStarted","Data":"917c9d3c27d7cc1a7157ae64a7ac6cb3154de5b31965e2bb5b5c3dd4472f33d6"}
Jul 08 16:39:37.334695 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:37.334649    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kbqcx" event={"ID":"0ee105a4-0d7b-4a50-a212-110cac5e12fb","Type":"ContainerStarted","Data":"6de708b73969b51968580ca7ef19558ac306f42f4964ec784ac181b6dda55e48"}
Jul 08 16:39:37.334841 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:37.334715    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kbqcx" event={"ID":"0ee105a4-0d7b-4a50-a212-110cac5e12fb","Type":"ContainerStarted","Data":"ebc87ea7551083698f67258370d30e9a3cebbbab72bde82bab2ad91cafeec907"}
Jul 08 16:39:37.334841 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:37.334749    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-kbqcx"
Jul 08 16:39:37.336808 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:37.336788    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-j6s2h" event={"ID":"dc428bbb-0f04-4342-b29e-9f4dfe36fdd1","Type":"ContainerStarted","Data":"91c929f9fa32dfedc2ac60d0327fff92be0b2f533c78c713b49ba91b90add5b5"}
Jul 08 16:39:37.336890 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:37.336827    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-j6s2h" event={"ID":"dc428bbb-0f04-4342-b29e-9f4dfe36fdd1","Type":"ContainerStarted","Data":"ad46aed4596533ee3ad59376033fb84748e6a639da323763f09edc6cd422fe2a"}
Jul 08 16:39:37.364255 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:37.364204    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-66fdd9cf8b-f725b" podStartSLOduration=1.05736954 podStartE2EDuration="5.364186862s" totalImagesPullingTime="4.306817322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:39:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:39:37.36273447 +0000 UTC m=+289.634184004" watchObservedRunningTime="2026-07-08 16:39:37.364186862 +0000 UTC m=+289.635636395"
Jul 08 16:39:37.387617 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:37.387571    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-kbqcx" podStartSLOduration=251.073489183 podStartE2EDuration="4m14.387556341s" totalImagesPullingTime="3.314067158s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:35:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:39:37.386951692 +0000 UTC m=+289.658401234" watchObservedRunningTime="2026-07-08 16:39:37.387556341 +0000 UTC m=+289.659005879"
Jul 08 16:39:37.411488 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:37.411442    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-j6s2h" podStartSLOduration=2.896656845 podStartE2EDuration="6.411422201s" totalImagesPullingTime="3.514765356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:39:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:39:37.410592345 +0000 UTC m=+289.682041884" watchObservedRunningTime="2026-07-08 16:39:37.411422201 +0000 UTC m=+289.682871741"
Jul 08 16:39:39.025413 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:39.025373    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-cl4ch"]
Jul 08 16:39:39.028512 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:39.028492    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cl4ch"
Jul 08 16:39:39.031162 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:39.031137    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 08 16:39:39.031261 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:39.031229    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 08 16:39:39.031335 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:39.031289    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 08 16:39:39.031519 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:39.031504    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-nkmpb\""
Jul 08 16:39:39.116447 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:39.116411    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1988f4a9-d199-4f16-9a8f-b0eaab1b21f7-node-exporter-wtmp\") pod \"node-exporter-cl4ch\" (UID: \"1988f4a9-d199-4f16-9a8f-b0eaab1b21f7\") " pod="openshift-monitoring/node-exporter-cl4ch"
Jul 08 16:39:39.116447 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:39.116449    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1988f4a9-d199-4f16-9a8f-b0eaab1b21f7-root\") pod \"node-exporter-cl4ch\" (UID: \"1988f4a9-d199-4f16-9a8f-b0eaab1b21f7\") " pod="openshift-monitoring/node-exporter-cl4ch"
Jul 08 16:39:39.116738 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:39.116474    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1988f4a9-d199-4f16-9a8f-b0eaab1b21f7-node-exporter-tls\") pod \"node-exporter-cl4ch\" (UID: \"1988f4a9-d199-4f16-9a8f-b0eaab1b21f7\") " pod="openshift-monitoring/node-exporter-cl4ch"
Jul 08 16:39:39.116738 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:39.116528    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1988f4a9-d199-4f16-9a8f-b0eaab1b21f7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cl4ch\" (UID: \"1988f4a9-d199-4f16-9a8f-b0eaab1b21f7\") " pod="openshift-monitoring/node-exporter-cl4ch"
Jul 08 16:39:39.116738 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:39.116566    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1988f4a9-d199-4f16-9a8f-b0eaab1b21f7-sys\") pod \"node-exporter-cl4ch\" (UID: \"1988f4a9-d199-4f16-9a8f-b0eaab1b21f7\") " pod="openshift-monitoring/node-exporter-cl4ch"
Jul 08 16:39:39.116738 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:39.116589    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1988f4a9-d199-4f16-9a8f-b0eaab1b21f7-node-exporter-textfile\") pod \"node-exporter-cl4ch\" (UID: \"1988f4a9-d199-4f16-9a8f-b0eaab1b21f7\") " pod="openshift-monitoring/node-exporter-cl4ch"
Jul 08 16:39:39.116738 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:39.116609    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trbxd\" (UniqueName: \"kubernetes.io/projected/1988f4a9-d199-4f16-9a8f-b0eaab1b21f7-kube-api-access-trbxd\") pod \"node-exporter-cl4ch\" (UID: \"1988f4a9-d199-4f16-9a8f-b0eaab1b21f7\") " pod="openshift-monitoring/node-exporter-cl4ch"
Jul 08 16:39:39.116738 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:39.116682    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1988f4a9-d199-4f16-9a8f-b0eaab1b21f7-metrics-client-ca\") pod \"node-exporter-cl4ch\" (UID: \"1988f4a9-d199-4f16-9a8f-b0eaab1b21f7\") " pod="openshift-monitoring/node-exporter-cl4ch"
Jul 08 16:39:39.116738 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:39.116731    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1988f4a9-d199-4f16-9a8f-b0eaab1b21f7-node-exporter-accelerators-collector-config\") pod \"node-exporter-cl4ch\" (UID: \"1988f4a9-d199-4f16-9a8f-b0eaab1b21f7\") " pod="openshift-monitoring/node-exporter-cl4ch"
Jul 08 16:39:39.169875 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:39.169823    2578 patch_prober.go:28] interesting pod/image-registry-55486dffd7-6ptzh container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 08 16:39:39.170055 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:39.169902    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-55486dffd7-6ptzh" podUID="dc3f4662-c8c9-42fc-8dfd-acac8b45f74c" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:39:39.218319 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:39.218281    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1988f4a9-d199-4f16-9a8f-b0eaab1b21f7-metrics-client-ca\") pod \"node-exporter-cl4ch\" (UID: \"1988f4a9-d199-4f16-9a8f-b0eaab1b21f7\") " pod="openshift-monitoring/node-exporter-cl4ch"
Jul 08 16:39:39.218499 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:39.218361    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1988f4a9-d199-4f16-9a8f-b0eaab1b21f7-node-exporter-accelerators-collector-config\") pod \"node-exporter-cl4ch\" (UID: \"1988f4a9-d199-4f16-9a8f-b0eaab1b21f7\") " pod="openshift-monitoring/node-exporter-cl4ch"
Jul 08 16:39:39.218499 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:39.218392    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1988f4a9-d199-4f16-9a8f-b0eaab1b21f7-node-exporter-wtmp\") pod \"node-exporter-cl4ch\" (UID: \"1988f4a9-d199-4f16-9a8f-b0eaab1b21f7\") " pod="openshift-monitoring/node-exporter-cl4ch"
Jul 08 16:39:39.218499 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:39.218417    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1988f4a9-d199-4f16-9a8f-b0eaab1b21f7-root\") pod \"node-exporter-cl4ch\" (UID: \"1988f4a9-d199-4f16-9a8f-b0eaab1b21f7\") " pod="openshift-monitoring/node-exporter-cl4ch"
Jul 08 16:39:39.218499 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:39.218446    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1988f4a9-d199-4f16-9a8f-b0eaab1b21f7-node-exporter-tls\") pod \"node-exporter-cl4ch\" (UID: \"1988f4a9-d199-4f16-9a8f-b0eaab1b21f7\") " pod="openshift-monitoring/node-exporter-cl4ch"
Jul 08 16:39:39.218499 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:39.218476    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1988f4a9-d199-4f16-9a8f-b0eaab1b21f7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cl4ch\" (UID: \"1988f4a9-d199-4f16-9a8f-b0eaab1b21f7\") " pod="openshift-monitoring/node-exporter-cl4ch"
Jul 08 16:39:39.218797 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:39.218502    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1988f4a9-d199-4f16-9a8f-b0eaab1b21f7-sys\") pod \"node-exporter-cl4ch\" (UID: \"1988f4a9-d199-4f16-9a8f-b0eaab1b21f7\") " pod="openshift-monitoring/node-exporter-cl4ch"
Jul 08 16:39:39.218797 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:39.218501    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1988f4a9-d199-4f16-9a8f-b0eaab1b21f7-root\") pod \"node-exporter-cl4ch\" (UID: \"1988f4a9-d199-4f16-9a8f-b0eaab1b21f7\") " pod="openshift-monitoring/node-exporter-cl4ch"
Jul 08 16:39:39.218797 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:39.218527    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1988f4a9-d199-4f16-9a8f-b0eaab1b21f7-node-exporter-textfile\") pod \"node-exporter-cl4ch\" (UID: \"1988f4a9-d199-4f16-9a8f-b0eaab1b21f7\") " pod="openshift-monitoring/node-exporter-cl4ch"
Jul 08 16:39:39.218797 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:39.218552    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trbxd\" (UniqueName: \"kubernetes.io/projected/1988f4a9-d199-4f16-9a8f-b0eaab1b21f7-kube-api-access-trbxd\") pod \"node-exporter-cl4ch\" (UID: \"1988f4a9-d199-4f16-9a8f-b0eaab1b21f7\") " pod="openshift-monitoring/node-exporter-cl4ch"
Jul 08 16:39:39.218797 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:39.218561    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1988f4a9-d199-4f16-9a8f-b0eaab1b21f7-node-exporter-wtmp\") pod \"node-exporter-cl4ch\" (UID: \"1988f4a9-d199-4f16-9a8f-b0eaab1b21f7\") " pod="openshift-monitoring/node-exporter-cl4ch"
Jul 08 16:39:39.218797 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:39.218570    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1988f4a9-d199-4f16-9a8f-b0eaab1b21f7-sys\") pod \"node-exporter-cl4ch\" (UID: \"1988f4a9-d199-4f16-9a8f-b0eaab1b21f7\") " pod="openshift-monitoring/node-exporter-cl4ch"
Jul 08 16:39:39.219055 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:39.218856    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1988f4a9-d199-4f16-9a8f-b0eaab1b21f7-node-exporter-textfile\") pod \"node-exporter-cl4ch\" (UID: \"1988f4a9-d199-4f16-9a8f-b0eaab1b21f7\") " pod="openshift-monitoring/node-exporter-cl4ch"
Jul 08 16:39:39.219111 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:39.219069    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1988f4a9-d199-4f16-9a8f-b0eaab1b21f7-node-exporter-accelerators-collector-config\") pod \"node-exporter-cl4ch\" (UID: \"1988f4a9-d199-4f16-9a8f-b0eaab1b21f7\") " pod="openshift-monitoring/node-exporter-cl4ch"
Jul 08 16:39:39.219111 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:39.219074    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1988f4a9-d199-4f16-9a8f-b0eaab1b21f7-metrics-client-ca\") pod \"node-exporter-cl4ch\" (UID: \"1988f4a9-d199-4f16-9a8f-b0eaab1b21f7\") " pod="openshift-monitoring/node-exporter-cl4ch"
Jul 08 16:39:39.221266 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:39.221235    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1988f4a9-d199-4f16-9a8f-b0eaab1b21f7-node-exporter-tls\") pod \"node-exporter-cl4ch\" (UID: \"1988f4a9-d199-4f16-9a8f-b0eaab1b21f7\") " pod="openshift-monitoring/node-exporter-cl4ch"
Jul 08 16:39:39.221434 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:39.221414    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1988f4a9-d199-4f16-9a8f-b0eaab1b21f7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cl4ch\" (UID: \"1988f4a9-d199-4f16-9a8f-b0eaab1b21f7\") " pod="openshift-monitoring/node-exporter-cl4ch"
Jul 08 16:39:39.228046 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:39.228017    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trbxd\" (UniqueName: \"kubernetes.io/projected/1988f4a9-d199-4f16-9a8f-b0eaab1b21f7-kube-api-access-trbxd\") pod \"node-exporter-cl4ch\" (UID: \"1988f4a9-d199-4f16-9a8f-b0eaab1b21f7\") " pod="openshift-monitoring/node-exporter-cl4ch"
Jul 08 16:39:39.338334 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:39.338296    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cl4ch"
Jul 08 16:39:39.349136 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:39:39.349106    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1988f4a9_d199_4f16_9a8f_b0eaab1b21f7.slice/crio-0fab15643bfd83da7f7d554f42d3e4bd1722bedb4a6cbf8e81b2471ea5365a9c WatchSource:0}: Error finding container 0fab15643bfd83da7f7d554f42d3e4bd1722bedb4a6cbf8e81b2471ea5365a9c: Status 404 returned error can't find the container with id 0fab15643bfd83da7f7d554f42d3e4bd1722bedb4a6cbf8e81b2471ea5365a9c
Jul 08 16:39:39.537457 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:39.537414    2578 patch_prober.go:28] interesting pod/image-registry-685fc4f98d-99nqw container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 08 16:39:39.537599 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:39.537471    2578 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-685fc4f98d-99nqw" podUID="654454ed-ca06-46b1-a030-b33bb974dd6d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:39:40.348008 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:40.347970    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cl4ch" event={"ID":"1988f4a9-d199-4f16-9a8f-b0eaab1b21f7","Type":"ContainerStarted","Data":"0fab15643bfd83da7f7d554f42d3e4bd1722bedb4a6cbf8e81b2471ea5365a9c"}
Jul 08 16:39:41.268305 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:41.268279    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-685fc4f98d-99nqw"
Jul 08 16:39:42.358553 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:42.358513    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cl4ch" event={"ID":"1988f4a9-d199-4f16-9a8f-b0eaab1b21f7","Type":"ContainerStarted","Data":"ab032862fcde4e8df927a44234183a4eea85a343600b9c7a4f34a42e096e9409"}
Jul 08 16:39:42.614275 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:42.614236    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-66fdd9cf8b-f725b"
Jul 08 16:39:42.614453 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:42.614304    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-66fdd9cf8b-f725b"
Jul 08 16:39:42.618997 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:42.618976    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-66fdd9cf8b-f725b"
Jul 08 16:39:43.363232 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:43.363199    2578 generic.go:356] "Generic (PLEG): container finished" podID="1988f4a9-d199-4f16-9a8f-b0eaab1b21f7" containerID="ab032862fcde4e8df927a44234183a4eea85a343600b9c7a4f34a42e096e9409" exitCode=0
Jul 08 16:39:43.363672 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:43.363294    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cl4ch" event={"ID":"1988f4a9-d199-4f16-9a8f-b0eaab1b21f7","Type":"ContainerDied","Data":"ab032862fcde4e8df927a44234183a4eea85a343600b9c7a4f34a42e096e9409"}
Jul 08 16:39:43.367619 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:43.367580    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-66fdd9cf8b-f725b"
Jul 08 16:39:43.482313 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:43.482280    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5f5d97fccf-hvpl2"]
Jul 08 16:39:43.486467 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:43.486440    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5f5d97fccf-hvpl2"
Jul 08 16:39:43.488744 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:43.488717    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 08 16:39:43.488885 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:43.488838    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 08 16:39:43.489004 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:43.488985    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-f4bp3sepepp0a\""
Jul 08 16:39:43.489077 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:43.488984    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-526bw\""
Jul 08 16:39:43.489122 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:43.489073    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 08 16:39:43.489343 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:43.489329    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 08 16:39:43.495972 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:43.495944    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5f5d97fccf-hvpl2"]
Jul 08 16:39:43.556168 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:43.556131    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/393b64f8-62fc-489e-9717-f63b23485906-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5f5d97fccf-hvpl2\" (UID: \"393b64f8-62fc-489e-9717-f63b23485906\") " pod="openshift-monitoring/metrics-server-5f5d97fccf-hvpl2"
Jul 08 16:39:43.556168 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:43.556174    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt2tr\" (UniqueName: \"kubernetes.io/projected/393b64f8-62fc-489e-9717-f63b23485906-kube-api-access-gt2tr\") pod \"metrics-server-5f5d97fccf-hvpl2\" (UID: \"393b64f8-62fc-489e-9717-f63b23485906\") " pod="openshift-monitoring/metrics-server-5f5d97fccf-hvpl2"
Jul 08 16:39:43.556418 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:43.556199    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/393b64f8-62fc-489e-9717-f63b23485906-metrics-server-audit-profiles\") pod \"metrics-server-5f5d97fccf-hvpl2\" (UID: \"393b64f8-62fc-489e-9717-f63b23485906\") " pod="openshift-monitoring/metrics-server-5f5d97fccf-hvpl2"
Jul 08 16:39:43.556418 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:43.556334    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/393b64f8-62fc-489e-9717-f63b23485906-audit-log\") pod \"metrics-server-5f5d97fccf-hvpl2\" (UID: \"393b64f8-62fc-489e-9717-f63b23485906\") " pod="openshift-monitoring/metrics-server-5f5d97fccf-hvpl2"
Jul 08 16:39:43.556418 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:43.556376    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/393b64f8-62fc-489e-9717-f63b23485906-secret-metrics-server-client-certs\") pod \"metrics-server-5f5d97fccf-hvpl2\" (UID: \"393b64f8-62fc-489e-9717-f63b23485906\") " pod="openshift-monitoring/metrics-server-5f5d97fccf-hvpl2"
Jul 08 16:39:43.556418 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:43.556396    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/393b64f8-62fc-489e-9717-f63b23485906-secret-metrics-server-tls\") pod \"metrics-server-5f5d97fccf-hvpl2\" (UID: \"393b64f8-62fc-489e-9717-f63b23485906\") " pod="openshift-monitoring/metrics-server-5f5d97fccf-hvpl2"
Jul 08 16:39:43.556609 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:43.556436    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/393b64f8-62fc-489e-9717-f63b23485906-client-ca-bundle\") pod \"metrics-server-5f5d97fccf-hvpl2\" (UID: \"393b64f8-62fc-489e-9717-f63b23485906\") " pod="openshift-monitoring/metrics-server-5f5d97fccf-hvpl2"
Jul 08 16:39:43.657541 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:43.657437    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/393b64f8-62fc-489e-9717-f63b23485906-audit-log\") pod \"metrics-server-5f5d97fccf-hvpl2\" (UID: \"393b64f8-62fc-489e-9717-f63b23485906\") " pod="openshift-monitoring/metrics-server-5f5d97fccf-hvpl2"
Jul 08 16:39:43.657541 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:43.657484    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/393b64f8-62fc-489e-9717-f63b23485906-secret-metrics-server-client-certs\") pod \"metrics-server-5f5d97fccf-hvpl2\" (UID: \"393b64f8-62fc-489e-9717-f63b23485906\") " pod="openshift-monitoring/metrics-server-5f5d97fccf-hvpl2"
Jul 08 16:39:43.657541 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:43.657511    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/393b64f8-62fc-489e-9717-f63b23485906-secret-metrics-server-tls\") pod \"metrics-server-5f5d97fccf-hvpl2\" (UID: \"393b64f8-62fc-489e-9717-f63b23485906\") " pod="openshift-monitoring/metrics-server-5f5d97fccf-hvpl2"
Jul 08 16:39:43.657867 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:43.657550    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/393b64f8-62fc-489e-9717-f63b23485906-client-ca-bundle\") pod \"metrics-server-5f5d97fccf-hvpl2\" (UID: \"393b64f8-62fc-489e-9717-f63b23485906\") " pod="openshift-monitoring/metrics-server-5f5d97fccf-hvpl2"
Jul 08 16:39:43.657867 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:43.657598    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/393b64f8-62fc-489e-9717-f63b23485906-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5f5d97fccf-hvpl2\" (UID: \"393b64f8-62fc-489e-9717-f63b23485906\") " pod="openshift-monitoring/metrics-server-5f5d97fccf-hvpl2"
Jul 08 16:39:43.657867 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:43.657826    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt2tr\" (UniqueName: \"kubernetes.io/projected/393b64f8-62fc-489e-9717-f63b23485906-kube-api-access-gt2tr\") pod \"metrics-server-5f5d97fccf-hvpl2\" (UID: \"393b64f8-62fc-489e-9717-f63b23485906\") " pod="openshift-monitoring/metrics-server-5f5d97fccf-hvpl2"
Jul 08 16:39:43.658015 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:43.657884    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/393b64f8-62fc-489e-9717-f63b23485906-metrics-server-audit-profiles\") pod \"metrics-server-5f5d97fccf-hvpl2\" (UID: \"393b64f8-62fc-489e-9717-f63b23485906\") " pod="openshift-monitoring/metrics-server-5f5d97fccf-hvpl2"
Jul 08 16:39:43.658015 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:43.657985    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/393b64f8-62fc-489e-9717-f63b23485906-audit-log\") pod \"metrics-server-5f5d97fccf-hvpl2\" (UID: \"393b64f8-62fc-489e-9717-f63b23485906\") " pod="openshift-monitoring/metrics-server-5f5d97fccf-hvpl2"
Jul 08 16:39:43.658329 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:43.658289    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/393b64f8-62fc-489e-9717-f63b23485906-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5f5d97fccf-hvpl2\" (UID: \"393b64f8-62fc-489e-9717-f63b23485906\") " pod="openshift-monitoring/metrics-server-5f5d97fccf-hvpl2"
Jul 08 16:39:43.658853 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:43.658828    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/393b64f8-62fc-489e-9717-f63b23485906-metrics-server-audit-profiles\") pod \"metrics-server-5f5d97fccf-hvpl2\" (UID: \"393b64f8-62fc-489e-9717-f63b23485906\") " pod="openshift-monitoring/metrics-server-5f5d97fccf-hvpl2"
Jul 08 16:39:43.660475 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:43.660446    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/393b64f8-62fc-489e-9717-f63b23485906-secret-metrics-server-client-certs\") pod \"metrics-server-5f5d97fccf-hvpl2\" (UID: \"393b64f8-62fc-489e-9717-f63b23485906\") " pod="openshift-monitoring/metrics-server-5f5d97fccf-hvpl2"
Jul 08 16:39:43.660576 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:43.660551    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/393b64f8-62fc-489e-9717-f63b23485906-secret-metrics-server-tls\") pod \"metrics-server-5f5d97fccf-hvpl2\" (UID: \"393b64f8-62fc-489e-9717-f63b23485906\") " pod="openshift-monitoring/metrics-server-5f5d97fccf-hvpl2"
Jul 08 16:39:43.660804 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:43.660789    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/393b64f8-62fc-489e-9717-f63b23485906-client-ca-bundle\") pod \"metrics-server-5f5d97fccf-hvpl2\" (UID: \"393b64f8-62fc-489e-9717-f63b23485906\") " pod="openshift-monitoring/metrics-server-5f5d97fccf-hvpl2"
Jul 08 16:39:43.668472 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:43.668417    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt2tr\" (UniqueName: \"kubernetes.io/projected/393b64f8-62fc-489e-9717-f63b23485906-kube-api-access-gt2tr\") pod \"metrics-server-5f5d97fccf-hvpl2\" (UID: \"393b64f8-62fc-489e-9717-f63b23485906\") " pod="openshift-monitoring/metrics-server-5f5d97fccf-hvpl2"
Jul 08 16:39:43.744575 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:43.744539    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-k98gm"]
Jul 08 16:39:43.748928 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:43.748907    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-k98gm"
Jul 08 16:39:43.751539 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:43.751513    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 08 16:39:43.751687 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:43.751587    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-6vqtm\""
Jul 08 16:39:43.759283 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:43.759263    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-k98gm"]
Jul 08 16:39:43.814746 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:43.814709    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5f5d97fccf-hvpl2"
Jul 08 16:39:43.859589 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:43.859559    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vzhn\" (UniqueName: \"kubernetes.io/projected/df1c923d-81c9-4e33-939b-050ede7d6675-kube-api-access-8vzhn\") pod \"monitoring-plugin-d5dcff7cf-k98gm\" (UID: \"df1c923d-81c9-4e33-939b-050ede7d6675\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-k98gm"
Jul 08 16:39:43.859813 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:43.859658    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/df1c923d-81c9-4e33-939b-050ede7d6675-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-k98gm\" (UID: \"df1c923d-81c9-4e33-939b-050ede7d6675\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-k98gm"
Jul 08 16:39:43.945741 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:43.945653    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5f5d97fccf-hvpl2"]
Jul 08 16:39:43.948360 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:39:43.948336    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod393b64f8_62fc_489e_9717_f63b23485906.slice/crio-b63cc6f906d5d547a46ef473ccfca539e7bfaddc62521f2ed404e31fccf48b60 WatchSource:0}: Error finding container b63cc6f906d5d547a46ef473ccfca539e7bfaddc62521f2ed404e31fccf48b60: Status 404 returned error can't find the container with id b63cc6f906d5d547a46ef473ccfca539e7bfaddc62521f2ed404e31fccf48b60
Jul 08 16:39:43.960320 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:43.960290    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/df1c923d-81c9-4e33-939b-050ede7d6675-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-k98gm\" (UID: \"df1c923d-81c9-4e33-939b-050ede7d6675\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-k98gm"
Jul 08 16:39:43.960465 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:43.960343    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vzhn\" (UniqueName: \"kubernetes.io/projected/df1c923d-81c9-4e33-939b-050ede7d6675-kube-api-access-8vzhn\") pod \"monitoring-plugin-d5dcff7cf-k98gm\" (UID: \"df1c923d-81c9-4e33-939b-050ede7d6675\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-k98gm"
Jul 08 16:39:43.962896 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:43.962873    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/df1c923d-81c9-4e33-939b-050ede7d6675-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-k98gm\" (UID: \"df1c923d-81c9-4e33-939b-050ede7d6675\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-k98gm"
Jul 08 16:39:43.967940 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:43.967919    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vzhn\" (UniqueName: \"kubernetes.io/projected/df1c923d-81c9-4e33-939b-050ede7d6675-kube-api-access-8vzhn\") pod \"monitoring-plugin-d5dcff7cf-k98gm\" (UID: \"df1c923d-81c9-4e33-939b-050ede7d6675\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-k98gm"
Jul 08 16:39:44.059096 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:44.059063    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-k98gm"
Jul 08 16:39:44.185555 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:44.185508    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-55486dffd7-6ptzh" podUID="dc3f4662-c8c9-42fc-8dfd-acac8b45f74c" containerName="registry" containerID="cri-o://407fc8eda2d135b9993ee12261f6da7f09b30b2cb90dfb28fa545ffcc8629062" gracePeriod=30
Jul 08 16:39:44.188825 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:44.188796    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-k98gm"]
Jul 08 16:39:44.191626 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:39:44.191601    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf1c923d_81c9_4e33_939b_050ede7d6675.slice/crio-d9e73a2e3dbc3b6d4d939f1c7fe308618dda0e2b41e4eb9e2e8158bbd997ef9b WatchSource:0}: Error finding container d9e73a2e3dbc3b6d4d939f1c7fe308618dda0e2b41e4eb9e2e8158bbd997ef9b: Status 404 returned error can't find the container with id d9e73a2e3dbc3b6d4d939f1c7fe308618dda0e2b41e4eb9e2e8158bbd997ef9b
Jul 08 16:39:44.368825 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:44.368789    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5f5d97fccf-hvpl2" event={"ID":"393b64f8-62fc-489e-9717-f63b23485906","Type":"ContainerStarted","Data":"b63cc6f906d5d547a46ef473ccfca539e7bfaddc62521f2ed404e31fccf48b60"}
Jul 08 16:39:44.370862 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:44.370835    2578 generic.go:356] "Generic (PLEG): container finished" podID="dc3f4662-c8c9-42fc-8dfd-acac8b45f74c" containerID="407fc8eda2d135b9993ee12261f6da7f09b30b2cb90dfb28fa545ffcc8629062" exitCode=0
Jul 08 16:39:44.371152 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:44.371041    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55486dffd7-6ptzh" event={"ID":"dc3f4662-c8c9-42fc-8dfd-acac8b45f74c","Type":"ContainerDied","Data":"407fc8eda2d135b9993ee12261f6da7f09b30b2cb90dfb28fa545ffcc8629062"}
Jul 08 16:39:44.373677 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:44.373648    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cl4ch" event={"ID":"1988f4a9-d199-4f16-9a8f-b0eaab1b21f7","Type":"ContainerStarted","Data":"7a2da65a9f9204016774314cb2d083580a1fe4c591e2348ad35f563efdf2e2c9"}
Jul 08 16:39:44.373804 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:44.373708    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cl4ch" event={"ID":"1988f4a9-d199-4f16-9a8f-b0eaab1b21f7","Type":"ContainerStarted","Data":"a0a89d56fc7b6670472cf594599834406f075206186ce887053eae558d0e9bf2"}
Jul 08 16:39:44.374915 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:44.374892    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-k98gm" event={"ID":"df1c923d-81c9-4e33-939b-050ede7d6675","Type":"ContainerStarted","Data":"d9e73a2e3dbc3b6d4d939f1c7fe308618dda0e2b41e4eb9e2e8158bbd997ef9b"}
Jul 08 16:39:44.393950 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:44.393906    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-cl4ch" podStartSLOduration=3.436246231 podStartE2EDuration="6.393887101s" totalImagesPullingTime="2.885314418s" totalInitContainerRuntime="72.326452ms" isStatefulPod=true podCreationTimestamp="2026-07-08 16:39:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:39:44.393048393 +0000 UTC m=+296.664497933" watchObservedRunningTime="2026-07-08 16:39:44.393887101 +0000 UTC m=+296.665336651"
Jul 08 16:39:44.429533 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:44.429507    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55486dffd7-6ptzh"
Jul 08 16:39:44.566703 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:44.566666    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tfzqj\" (UniqueName: \"kubernetes.io/projected/dc3f4662-c8c9-42fc-8dfd-acac8b45f74c-kube-api-access-tfzqj\") pod \"dc3f4662-c8c9-42fc-8dfd-acac8b45f74c\" (UID: \"dc3f4662-c8c9-42fc-8dfd-acac8b45f74c\") "
Jul 08 16:39:44.566844 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:44.566724    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dc3f4662-c8c9-42fc-8dfd-acac8b45f74c-installation-pull-secrets\") pod \"dc3f4662-c8c9-42fc-8dfd-acac8b45f74c\" (UID: \"dc3f4662-c8c9-42fc-8dfd-acac8b45f74c\") "
Jul 08 16:39:44.566844 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:44.566765    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dc3f4662-c8c9-42fc-8dfd-acac8b45f74c-trusted-ca\") pod \"dc3f4662-c8c9-42fc-8dfd-acac8b45f74c\" (UID: \"dc3f4662-c8c9-42fc-8dfd-acac8b45f74c\") "
Jul 08 16:39:44.566844 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:44.566836    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dc3f4662-c8c9-42fc-8dfd-acac8b45f74c-bound-sa-token\") pod \"dc3f4662-c8c9-42fc-8dfd-acac8b45f74c\" (UID: \"dc3f4662-c8c9-42fc-8dfd-acac8b45f74c\") "
Jul 08 16:39:44.566965 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:44.566870    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dc3f4662-c8c9-42fc-8dfd-acac8b45f74c-registry-certificates\") pod \"dc3f4662-c8c9-42fc-8dfd-acac8b45f74c\" (UID: \"dc3f4662-c8c9-42fc-8dfd-acac8b45f74c\") "
Jul 08 16:39:44.566965 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:44.566904    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dc3f4662-c8c9-42fc-8dfd-acac8b45f74c-ca-trust-extracted\") pod \"dc3f4662-c8c9-42fc-8dfd-acac8b45f74c\" (UID: \"dc3f4662-c8c9-42fc-8dfd-acac8b45f74c\") "
Jul 08 16:39:44.566965 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:44.566956    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dc3f4662-c8c9-42fc-8dfd-acac8b45f74c-image-registry-private-configuration\") pod \"dc3f4662-c8c9-42fc-8dfd-acac8b45f74c\" (UID: \"dc3f4662-c8c9-42fc-8dfd-acac8b45f74c\") "
Jul 08 16:39:44.567096 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:44.566990    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dc3f4662-c8c9-42fc-8dfd-acac8b45f74c-registry-tls\") pod \"dc3f4662-c8c9-42fc-8dfd-acac8b45f74c\" (UID: \"dc3f4662-c8c9-42fc-8dfd-acac8b45f74c\") "
Jul 08 16:39:44.567459 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:44.567347    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc3f4662-c8c9-42fc-8dfd-acac8b45f74c-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "dc3f4662-c8c9-42fc-8dfd-acac8b45f74c" (UID: "dc3f4662-c8c9-42fc-8dfd-acac8b45f74c"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:39:44.567459 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:44.567382    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc3f4662-c8c9-42fc-8dfd-acac8b45f74c-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "dc3f4662-c8c9-42fc-8dfd-acac8b45f74c" (UID: "dc3f4662-c8c9-42fc-8dfd-acac8b45f74c"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:39:44.569251 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:44.569214    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc3f4662-c8c9-42fc-8dfd-acac8b45f74c-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "dc3f4662-c8c9-42fc-8dfd-acac8b45f74c" (UID: "dc3f4662-c8c9-42fc-8dfd-acac8b45f74c"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:39:44.569354 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:44.569283    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc3f4662-c8c9-42fc-8dfd-acac8b45f74c-kube-api-access-tfzqj" (OuterVolumeSpecName: "kube-api-access-tfzqj") pod "dc3f4662-c8c9-42fc-8dfd-acac8b45f74c" (UID: "dc3f4662-c8c9-42fc-8dfd-acac8b45f74c"). InnerVolumeSpecName "kube-api-access-tfzqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:39:44.569463 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:44.569442    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc3f4662-c8c9-42fc-8dfd-acac8b45f74c-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "dc3f4662-c8c9-42fc-8dfd-acac8b45f74c" (UID: "dc3f4662-c8c9-42fc-8dfd-acac8b45f74c"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:39:44.569529 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:44.569452    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc3f4662-c8c9-42fc-8dfd-acac8b45f74c-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "dc3f4662-c8c9-42fc-8dfd-acac8b45f74c" (UID: "dc3f4662-c8c9-42fc-8dfd-acac8b45f74c"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:39:44.569754 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:44.569726    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc3f4662-c8c9-42fc-8dfd-acac8b45f74c-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "dc3f4662-c8c9-42fc-8dfd-acac8b45f74c" (UID: "dc3f4662-c8c9-42fc-8dfd-acac8b45f74c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:39:44.575958 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:44.575933    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc3f4662-c8c9-42fc-8dfd-acac8b45f74c-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "dc3f4662-c8c9-42fc-8dfd-acac8b45f74c" (UID: "dc3f4662-c8c9-42fc-8dfd-acac8b45f74c"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:39:44.667757 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:44.667719    2578 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dc3f4662-c8c9-42fc-8dfd-acac8b45f74c-bound-sa-token\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:39:44.667757 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:44.667749    2578 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dc3f4662-c8c9-42fc-8dfd-acac8b45f74c-registry-certificates\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:39:44.667757 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:44.667759    2578 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dc3f4662-c8c9-42fc-8dfd-acac8b45f74c-ca-trust-extracted\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:39:44.667757 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:44.667769    2578 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dc3f4662-c8c9-42fc-8dfd-acac8b45f74c-image-registry-private-configuration\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:39:44.668015 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:44.667780    2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dc3f4662-c8c9-42fc-8dfd-acac8b45f74c-registry-tls\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:39:44.668015 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:44.667788    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfzqj\" (UniqueName: \"kubernetes.io/projected/dc3f4662-c8c9-42fc-8dfd-acac8b45f74c-kube-api-access-tfzqj\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:39:44.668015 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:44.667795    2578 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dc3f4662-c8c9-42fc-8dfd-acac8b45f74c-installation-pull-secrets\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:39:44.668015 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:44.667805    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dc3f4662-c8c9-42fc-8dfd-acac8b45f74c-trusted-ca\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:39:45.200668 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.199802    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 08 16:39:45.200668 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.200156    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc3f4662-c8c9-42fc-8dfd-acac8b45f74c" containerName="registry"
Jul 08 16:39:45.200668 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.200170    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc3f4662-c8c9-42fc-8dfd-acac8b45f74c" containerName="registry"
Jul 08 16:39:45.200668 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.200255    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc3f4662-c8c9-42fc-8dfd-acac8b45f74c" containerName="registry"
Jul 08 16:39:45.205583 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.203941    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:39:45.206810 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.206783    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-ntzrr\""
Jul 08 16:39:45.206977 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.206956    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 08 16:39:45.207074 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.207059    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 08 16:39:45.207285 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.207263    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 08 16:39:45.207285 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.207272    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 08 16:39:45.207916 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.207894    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 08 16:39:45.208000 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.207935    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 08 16:39:45.208000 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.207956    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 08 16:39:45.208123 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.207956    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 08 16:39:45.208123 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.207960    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-bm1io90ktn00n\""
Jul 08 16:39:45.208302 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.208285    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 08 16:39:45.208371 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.208340    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 08 16:39:45.208429 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.208391    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 08 16:39:45.208429 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.208414    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 08 16:39:45.208523 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.208468    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 08 16:39:45.212439 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.212420    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 08 16:39:45.219988 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.219552    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 08 16:39:45.274309 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.274276    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2ac190a4-147f-4895-90e9-9ecc1c0df60f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:39:45.274506 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.274321    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ac190a4-147f-4895-90e9-9ecc1c0df60f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:39:45.274506 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.274376    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2ac190a4-147f-4895-90e9-9ecc1c0df60f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:39:45.274506 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.274412    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2ac190a4-147f-4895-90e9-9ecc1c0df60f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:39:45.274506 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.274446    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ac190a4-147f-4895-90e9-9ecc1c0df60f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:39:45.274506 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.274482    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2ac190a4-147f-4895-90e9-9ecc1c0df60f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:39:45.274747 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.274539    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2ac190a4-147f-4895-90e9-9ecc1c0df60f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:39:45.274747 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.274568    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2ac190a4-147f-4895-90e9-9ecc1c0df60f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:39:45.274747 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.274585    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2ac190a4-147f-4895-90e9-9ecc1c0df60f-web-config\") pod \"prometheus-k8s-0\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:39:45.274747 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.274612    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2ac190a4-147f-4895-90e9-9ecc1c0df60f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:39:45.274747 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.274693    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ac190a4-147f-4895-90e9-9ecc1c0df60f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:39:45.274969 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.274749    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2ac190a4-147f-4895-90e9-9ecc1c0df60f-config-out\") pod \"prometheus-k8s-0\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:39:45.274969 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.274778    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7gzq\" (UniqueName: \"kubernetes.io/projected/2ac190a4-147f-4895-90e9-9ecc1c0df60f-kube-api-access-c7gzq\") pod \"prometheus-k8s-0\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:39:45.274969 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.274874    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2ac190a4-147f-4895-90e9-9ecc1c0df60f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:39:45.274969 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.274922    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2ac190a4-147f-4895-90e9-9ecc1c0df60f-config\") pod \"prometheus-k8s-0\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:39:45.274969 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.274958    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2ac190a4-147f-4895-90e9-9ecc1c0df60f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:39:45.275168 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.275025    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2ac190a4-147f-4895-90e9-9ecc1c0df60f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:39:45.275168 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.275055    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ac190a4-147f-4895-90e9-9ecc1c0df60f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:39:45.275168 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.275087    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2ac190a4-147f-4895-90e9-9ecc1c0df60f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:39:45.275168 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.275142    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2ac190a4-147f-4895-90e9-9ecc1c0df60f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:39:45.281546 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:39:45.281503    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-f5jcr" podUID="0f2a26b1-8d19-4874-a633-1cd61e29091a"
Jul 08 16:39:45.375769 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.375728    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2ac190a4-147f-4895-90e9-9ecc1c0df60f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:39:45.375769 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.375776    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ac190a4-147f-4895-90e9-9ecc1c0df60f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:39:45.376510 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.375804    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2ac190a4-147f-4895-90e9-9ecc1c0df60f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:39:45.376510 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.375830    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2ac190a4-147f-4895-90e9-9ecc1c0df60f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:39:45.376510 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.375861    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2ac190a4-147f-4895-90e9-9ecc1c0df60f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:39:45.376510 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.375897    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ac190a4-147f-4895-90e9-9ecc1c0df60f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:39:45.376510 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.375931    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2ac190a4-147f-4895-90e9-9ecc1c0df60f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:39:45.376510 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.375963    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2ac190a4-147f-4895-90e9-9ecc1c0df60f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:39:45.376510 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.375993    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ac190a4-147f-4895-90e9-9ecc1c0df60f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:39:45.376510 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.376019    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2ac190a4-147f-4895-90e9-9ecc1c0df60f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:39:45.376510 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.376046    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2ac190a4-147f-4895-90e9-9ecc1c0df60f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:39:45.376510 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.376075    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2ac190a4-147f-4895-90e9-9ecc1c0df60f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:39:45.376510 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.376100    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2ac190a4-147f-4895-90e9-9ecc1c0df60f-web-config\") pod \"prometheus-k8s-0\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:39:45.376510 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.376134    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2ac190a4-147f-4895-90e9-9ecc1c0df60f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:39:45.376510 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.376161    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ac190a4-147f-4895-90e9-9ecc1c0df60f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:39:45.376510 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.376199    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2ac190a4-147f-4895-90e9-9ecc1c0df60f-config-out\") pod \"prometheus-k8s-0\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:39:45.376510 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.376221    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7gzq\" (UniqueName: \"kubernetes.io/projected/2ac190a4-147f-4895-90e9-9ecc1c0df60f-kube-api-access-c7gzq\") pod \"prometheus-k8s-0\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:39:45.376510 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.376274    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2ac190a4-147f-4895-90e9-9ecc1c0df60f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:39:45.376510 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.376314    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2ac190a4-147f-4895-90e9-9ecc1c0df60f-config\") pod \"prometheus-k8s-0\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:39:45.377338 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.376339    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2ac190a4-147f-4895-90e9-9ecc1c0df60f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:39:45.377338 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.376929    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ac190a4-147f-4895-90e9-9ecc1c0df60f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:39:45.377338 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.376963    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ac190a4-147f-4895-90e9-9ecc1c0df60f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:39:45.379501 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.378025    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ac190a4-147f-4895-90e9-9ecc1c0df60f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:39:45.379501 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.378552    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2ac190a4-147f-4895-90e9-9ecc1c0df60f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:39:45.379501 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.378760    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2ac190a4-147f-4895-90e9-9ecc1c0df60f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:39:45.379501 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.379176    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2ac190a4-147f-4895-90e9-9ecc1c0df60f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:39:45.379790 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.379767    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2ac190a4-147f-4895-90e9-9ecc1c0df60f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:39:45.380180 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.379913    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2ac190a4-147f-4895-90e9-9ecc1c0df60f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:39:45.380180 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.378052    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ac190a4-147f-4895-90e9-9ecc1c0df60f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:39:45.380660 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.380591    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2ac190a4-147f-4895-90e9-9ecc1c0df60f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:39:45.382114 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.382079    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2ac190a4-147f-4895-90e9-9ecc1c0df60f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:39:45.383038 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.383020    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2ac190a4-147f-4895-90e9-9ecc1c0df60f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:39:45.383135 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.383086    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2ac190a4-147f-4895-90e9-9ecc1c0df60f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:39:45.383135 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.383084    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/2ac190a4-147f-4895-90e9-9ecc1c0df60f-config\") pod \"prometheus-k8s-0\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:39:45.383553 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.383531    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2ac190a4-147f-4895-90e9-9ecc1c0df60f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:39:45.383672 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.383617    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2ac190a4-147f-4895-90e9-9ecc1c0df60f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:39:45.383778 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.383753    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2ac190a4-147f-4895-90e9-9ecc1c0df60f-config-out\") pod \"prometheus-k8s-0\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:39:45.383778 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.383769    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2ac190a4-147f-4895-90e9-9ecc1c0df60f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:39:45.385509 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.385465    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2ac190a4-147f-4895-90e9-9ecc1c0df60f-web-config\") pod \"prometheus-k8s-0\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:39:45.389602 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.389514    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55486dffd7-6ptzh" event={"ID":"dc3f4662-c8c9-42fc-8dfd-acac8b45f74c","Type":"ContainerDied","Data":"8a2f2826a2e7d06f9b0a3c458f53931152180ca44de62a429fc049578aa8a974"}
Jul 08 16:39:45.389602 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.389543    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55486dffd7-6ptzh"
Jul 08 16:39:45.389602 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.389592    2578 scope.go:117] "RemoveContainer" containerID="407fc8eda2d135b9993ee12261f6da7f09b30b2cb90dfb28fa545ffcc8629062"
Jul 08 16:39:45.390319 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.390250    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7gzq\" (UniqueName: \"kubernetes.io/projected/2ac190a4-147f-4895-90e9-9ecc1c0df60f-kube-api-access-c7gzq\") pod \"prometheus-k8s-0\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:39:45.430026 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.429984    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-55486dffd7-6ptzh"]
Jul 08 16:39:45.432268 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.432234    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-55486dffd7-6ptzh"]
Jul 08 16:39:45.519837 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.518271    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:39:45.828103 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:45.828022    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 08 16:39:45.831244 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:39:45.830940    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ac190a4_147f_4895_90e9_9ecc1c0df60f.slice/crio-aed0b22a444d81c973c540cf97bb03ca915afd9bdb7a57f31bf933f2798d8ed0 WatchSource:0}: Error finding container aed0b22a444d81c973c540cf97bb03ca915afd9bdb7a57f31bf933f2798d8ed0: Status 404 returned error can't find the container with id aed0b22a444d81c973c540cf97bb03ca915afd9bdb7a57f31bf933f2798d8ed0
Jul 08 16:39:46.288203 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:46.288166    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc3f4662-c8c9-42fc-8dfd-acac8b45f74c" path="/var/lib/kubelet/pods/dc3f4662-c8c9-42fc-8dfd-acac8b45f74c/volumes"
Jul 08 16:39:46.397976 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:46.397937    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5f5d97fccf-hvpl2" event={"ID":"393b64f8-62fc-489e-9717-f63b23485906","Type":"ContainerStarted","Data":"816a9de99b0ccf6955180b9c47832f5949262a3014bf5b58e87a8c2ca02950dc"}
Jul 08 16:39:46.399746 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:46.399714    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2ac190a4-147f-4895-90e9-9ecc1c0df60f","Type":"ContainerStarted","Data":"aed0b22a444d81c973c540cf97bb03ca915afd9bdb7a57f31bf933f2798d8ed0"}
Jul 08 16:39:46.417317 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:46.417249    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5f5d97fccf-hvpl2" podStartSLOduration=1.648306472 podStartE2EDuration="3.417229991s" totalImagesPullingTime="1.768923519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:39:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:39:46.415584775 +0000 UTC m=+298.687034315" watchObservedRunningTime="2026-07-08 16:39:46.417229991 +0000 UTC m=+298.688679532"
Jul 08 16:39:47.342276 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:47.342243    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-kbqcx"
Jul 08 16:39:47.405124 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:47.405018    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-k98gm" event={"ID":"df1c923d-81c9-4e33-939b-050ede7d6675","Type":"ContainerStarted","Data":"8cde38f6bf392b8675826dc02d2028593fc7fcca3854e86fc270a44a9caaba49"}
Jul 08 16:39:47.405124 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:47.405085    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-k98gm"
Jul 08 16:39:47.406837 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:47.406812    2578 generic.go:356] "Generic (PLEG): container finished" podID="2ac190a4-147f-4895-90e9-9ecc1c0df60f" containerID="88ab3216165b920c9c4b15b4d30c8aa1a2ca71c8997664bed119fe1aa2865d36" exitCode=0
Jul 08 16:39:47.406977 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:47.406898    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2ac190a4-147f-4895-90e9-9ecc1c0df60f","Type":"ContainerDied","Data":"88ab3216165b920c9c4b15b4d30c8aa1a2ca71c8997664bed119fe1aa2865d36"}
Jul 08 16:39:47.410823 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:47.410796    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-k98gm"
Jul 08 16:39:47.423872 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:47.423826    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-k98gm" podStartSLOduration=2.040686509 podStartE2EDuration="4.423810263s" totalImagesPullingTime="2.383123754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:39:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:39:47.423595343 +0000 UTC m=+299.695044885" watchObservedRunningTime="2026-07-08 16:39:47.423810263 +0000 UTC m=+299.695259802"
Jul 08 16:39:48.123207 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:48.123186    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-qn2bx_4717bdee-2017-4551-b0ec-683ab07f7660/console-operator/1.log"
Jul 08 16:39:48.124362 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:48.124341    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-qn2bx_4717bdee-2017-4551-b0ec-683ab07f7660/console-operator/1.log"
Jul 08 16:39:48.134378 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:48.134335    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2shvk_f08a1b00-78fd-43ef-b2aa-2712d09fb1b3/ovn-acl-logging/0.log"
Jul 08 16:39:48.135814 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:48.135783    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2shvk_f08a1b00-78fd-43ef-b2aa-2712d09fb1b3/ovn-acl-logging/0.log"
Jul 08 16:39:48.142019 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:48.141992    2578 kubelet.go:1648] "Image garbage collection succeeded"
Jul 08 16:39:51.266677 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:51.266653    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 08 16:39:51.427103 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:51.427026    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2ac190a4-147f-4895-90e9-9ecc1c0df60f","Type":"ContainerStarted","Data":"f47703b3d7319ee5ac3cbb855c6933caa47cd1d77c04034eeac1a9ba499cd5cc"}
Jul 08 16:39:51.427247 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:51.427103    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2ac190a4-147f-4895-90e9-9ecc1c0df60f","Type":"ContainerStarted","Data":"532f13eb8e95ab15daa585aad29fe8faefdaff7b69eb8c5b0f20eeecbced013c"}
Jul 08 16:39:54.441358 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:54.441323    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2ac190a4-147f-4895-90e9-9ecc1c0df60f","Type":"ContainerStarted","Data":"1607cc55942ceb3893ac08637da2c4a153cccacd3b923d1455e68ad18f1ab592"}
Jul 08 16:39:54.441886 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:54.441386    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2ac190a4-147f-4895-90e9-9ecc1c0df60f","Type":"ContainerStarted","Data":"dc1c001ce3b12211c906684fb8b9745ecba4003ec2cbb7dcd84cb73d23b6bbad"}
Jul 08 16:39:54.441886 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:54.441410    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2ac190a4-147f-4895-90e9-9ecc1c0df60f","Type":"ContainerStarted","Data":"76e9b89a240d4de422fd5cf0f8ee5fb74813264e7f1916150d0c88679fc20ba1"}
Jul 08 16:39:54.441886 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:54.441434    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2ac190a4-147f-4895-90e9-9ecc1c0df60f","Type":"ContainerStarted","Data":"842190fc6d43e8a898f199c3898234ec3030bcc746cc29768f267650d0a61a70"}
Jul 08 16:39:54.469266 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:54.469208    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=1.884451839 podStartE2EDuration="9.469191808s" totalImagesPullingTime="7.564060563s" totalInitContainerRuntime="20.679406ms" isStatefulPod=false podCreationTimestamp="2026-07-08 16:39:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:39:54.468551736 +0000 UTC m=+306.740001277" watchObservedRunningTime="2026-07-08 16:39:54.469191808 +0000 UTC m=+306.740641350"
Jul 08 16:39:55.519100 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:55.519064    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:39:56.280539 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:39:56.280499    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f5jcr"
Jul 08 16:40:01.450877 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:40:01.450842    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66fdd9cf8b-f725b"]
Jul 08 16:40:03.815161 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:40:03.815127    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5f5d97fccf-hvpl2"
Jul 08 16:40:03.815532 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:40:03.815189    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5f5d97fccf-hvpl2"
Jul 08 16:40:14.515528 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:40:14.515447    2578 generic.go:356] "Generic (PLEG): container finished" podID="ec3a3d1a-9e90-40a7-9695-7a1c81f3c41c" containerID="c082d52676ba4ca60d5e5018efc708415a5c788f27798e1c0c7abd372d32dc4d" exitCode=0
Jul 08 16:40:14.515528 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:40:14.515516    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-ctm5m" event={"ID":"ec3a3d1a-9e90-40a7-9695-7a1c81f3c41c","Type":"ContainerDied","Data":"c082d52676ba4ca60d5e5018efc708415a5c788f27798e1c0c7abd372d32dc4d"}
Jul 08 16:40:14.515947 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:40:14.515871    2578 scope.go:117] "RemoveContainer" containerID="c082d52676ba4ca60d5e5018efc708415a5c788f27798e1c0c7abd372d32dc4d"
Jul 08 16:40:15.521554 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:40:15.521520    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-ctm5m" event={"ID":"ec3a3d1a-9e90-40a7-9695-7a1c81f3c41c","Type":"ContainerStarted","Data":"01d15b2d6ed82ee8c440eb7399cf9e0982396795264abdacf7612c3e628df938"}
Jul 08 16:40:23.820650 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:40:23.820610    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5f5d97fccf-hvpl2"
Jul 08 16:40:23.824435 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:40:23.824411    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5f5d97fccf-hvpl2"
Jul 08 16:40:26.471795 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:40:26.471751    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-66fdd9cf8b-f725b" podUID="17fd5ffc-5451-4ed5-ae7e-42feeb4c9250" containerName="console" containerID="cri-o://917c9d3c27d7cc1a7157ae64a7ac6cb3154de5b31965e2bb5b5c3dd4472f33d6" gracePeriod=15
Jul 08 16:40:26.748943 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:40:26.748913    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66fdd9cf8b-f725b"
Jul 08 16:40:26.862114 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:40:26.862083    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17fd5ffc-5451-4ed5-ae7e-42feeb4c9250-oauth-serving-cert\") pod \"17fd5ffc-5451-4ed5-ae7e-42feeb4c9250\" (UID: \"17fd5ffc-5451-4ed5-ae7e-42feeb4c9250\") "
Jul 08 16:40:26.862299 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:40:26.862128    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17fd5ffc-5451-4ed5-ae7e-42feeb4c9250-service-ca\") pod \"17fd5ffc-5451-4ed5-ae7e-42feeb4c9250\" (UID: \"17fd5ffc-5451-4ed5-ae7e-42feeb4c9250\") "
Jul 08 16:40:26.862299 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:40:26.862213    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17fd5ffc-5451-4ed5-ae7e-42feeb4c9250-console-oauth-config\") pod \"17fd5ffc-5451-4ed5-ae7e-42feeb4c9250\" (UID: \"17fd5ffc-5451-4ed5-ae7e-42feeb4c9250\") "
Jul 08 16:40:26.862299 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:40:26.862243    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ccs64\" (UniqueName: \"kubernetes.io/projected/17fd5ffc-5451-4ed5-ae7e-42feeb4c9250-kube-api-access-ccs64\") pod \"17fd5ffc-5451-4ed5-ae7e-42feeb4c9250\" (UID: \"17fd5ffc-5451-4ed5-ae7e-42feeb4c9250\") "
Jul 08 16:40:26.862299 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:40:26.862282    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17fd5ffc-5451-4ed5-ae7e-42feeb4c9250-trusted-ca-bundle\") pod \"17fd5ffc-5451-4ed5-ae7e-42feeb4c9250\" (UID: \"17fd5ffc-5451-4ed5-ae7e-42feeb4c9250\") "
Jul 08 16:40:26.862299 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:40:26.862303    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17fd5ffc-5451-4ed5-ae7e-42feeb4c9250-console-serving-cert\") pod \"17fd5ffc-5451-4ed5-ae7e-42feeb4c9250\" (UID: \"17fd5ffc-5451-4ed5-ae7e-42feeb4c9250\") "
Jul 08 16:40:26.862569 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:40:26.862326    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17fd5ffc-5451-4ed5-ae7e-42feeb4c9250-console-config\") pod \"17fd5ffc-5451-4ed5-ae7e-42feeb4c9250\" (UID: \"17fd5ffc-5451-4ed5-ae7e-42feeb4c9250\") "
Jul 08 16:40:26.862569 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:40:26.862447    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17fd5ffc-5451-4ed5-ae7e-42feeb4c9250-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "17fd5ffc-5451-4ed5-ae7e-42feeb4c9250" (UID: "17fd5ffc-5451-4ed5-ae7e-42feeb4c9250"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:40:26.862569 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:40:26.862546    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17fd5ffc-5451-4ed5-ae7e-42feeb4c9250-service-ca" (OuterVolumeSpecName: "service-ca") pod "17fd5ffc-5451-4ed5-ae7e-42feeb4c9250" (UID: "17fd5ffc-5451-4ed5-ae7e-42feeb4c9250"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:40:26.862750 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:40:26.862578    2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17fd5ffc-5451-4ed5-ae7e-42feeb4c9250-oauth-serving-cert\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:40:26.862750 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:40:26.862701    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17fd5ffc-5451-4ed5-ae7e-42feeb4c9250-console-config" (OuterVolumeSpecName: "console-config") pod "17fd5ffc-5451-4ed5-ae7e-42feeb4c9250" (UID: "17fd5ffc-5451-4ed5-ae7e-42feeb4c9250"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:40:26.862853 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:40:26.862785    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17fd5ffc-5451-4ed5-ae7e-42feeb4c9250-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "17fd5ffc-5451-4ed5-ae7e-42feeb4c9250" (UID: "17fd5ffc-5451-4ed5-ae7e-42feeb4c9250"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:40:26.864646 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:40:26.864602    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17fd5ffc-5451-4ed5-ae7e-42feeb4c9250-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "17fd5ffc-5451-4ed5-ae7e-42feeb4c9250" (UID: "17fd5ffc-5451-4ed5-ae7e-42feeb4c9250"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:40:26.864914 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:40:26.864887    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17fd5ffc-5451-4ed5-ae7e-42feeb4c9250-kube-api-access-ccs64" (OuterVolumeSpecName: "kube-api-access-ccs64") pod "17fd5ffc-5451-4ed5-ae7e-42feeb4c9250" (UID: "17fd5ffc-5451-4ed5-ae7e-42feeb4c9250"). InnerVolumeSpecName "kube-api-access-ccs64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:40:26.865297 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:40:26.865271    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17fd5ffc-5451-4ed5-ae7e-42feeb4c9250-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "17fd5ffc-5451-4ed5-ae7e-42feeb4c9250" (UID: "17fd5ffc-5451-4ed5-ae7e-42feeb4c9250"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:40:26.963868 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:40:26.963825    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17fd5ffc-5451-4ed5-ae7e-42feeb4c9250-trusted-ca-bundle\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:40:26.963868 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:40:26.963870    2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17fd5ffc-5451-4ed5-ae7e-42feeb4c9250-console-serving-cert\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:40:26.964111 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:40:26.963890    2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17fd5ffc-5451-4ed5-ae7e-42feeb4c9250-console-config\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:40:26.964111 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:40:26.963907    2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17fd5ffc-5451-4ed5-ae7e-42feeb4c9250-service-ca\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:40:26.964111 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:40:26.963921    2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17fd5ffc-5451-4ed5-ae7e-42feeb4c9250-console-oauth-config\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:40:26.964111 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:40:26.963937    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccs64\" (UniqueName: \"kubernetes.io/projected/17fd5ffc-5451-4ed5-ae7e-42feeb4c9250-kube-api-access-ccs64\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:40:27.575739 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:40:27.575712    2578 generic.go:356] "Generic (PLEG): container finished" podID="17fd5ffc-5451-4ed5-ae7e-42feeb4c9250" containerID="917c9d3c27d7cc1a7157ae64a7ac6cb3154de5b31965e2bb5b5c3dd4472f33d6" exitCode=0
Jul 08 16:40:27.576101 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:40:27.575773    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66fdd9cf8b-f725b" event={"ID":"17fd5ffc-5451-4ed5-ae7e-42feeb4c9250","Type":"ContainerDied","Data":"917c9d3c27d7cc1a7157ae64a7ac6cb3154de5b31965e2bb5b5c3dd4472f33d6"}
Jul 08 16:40:27.576101 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:40:27.575778    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66fdd9cf8b-f725b"
Jul 08 16:40:27.576101 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:40:27.575813    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66fdd9cf8b-f725b" event={"ID":"17fd5ffc-5451-4ed5-ae7e-42feeb4c9250","Type":"ContainerDied","Data":"407f616cc26c1662036a0ca2482bad85e46f462c19995e2db86e627f5850c3fd"}
Jul 08 16:40:27.576101 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:40:27.575835    2578 scope.go:117] "RemoveContainer" containerID="917c9d3c27d7cc1a7157ae64a7ac6cb3154de5b31965e2bb5b5c3dd4472f33d6"
Jul 08 16:40:27.591895 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:40:27.591867    2578 scope.go:117] "RemoveContainer" containerID="917c9d3c27d7cc1a7157ae64a7ac6cb3154de5b31965e2bb5b5c3dd4472f33d6"
Jul 08 16:40:27.592235 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:40:27.592201    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"917c9d3c27d7cc1a7157ae64a7ac6cb3154de5b31965e2bb5b5c3dd4472f33d6\": container with ID starting with 917c9d3c27d7cc1a7157ae64a7ac6cb3154de5b31965e2bb5b5c3dd4472f33d6 not found: ID does not exist" containerID="917c9d3c27d7cc1a7157ae64a7ac6cb3154de5b31965e2bb5b5c3dd4472f33d6"
Jul 08 16:40:27.592310 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:40:27.592248    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"917c9d3c27d7cc1a7157ae64a7ac6cb3154de5b31965e2bb5b5c3dd4472f33d6"} err="failed to get container status \"917c9d3c27d7cc1a7157ae64a7ac6cb3154de5b31965e2bb5b5c3dd4472f33d6\": rpc error: code = NotFound desc = could not find container \"917c9d3c27d7cc1a7157ae64a7ac6cb3154de5b31965e2bb5b5c3dd4472f33d6\": container with ID starting with 917c9d3c27d7cc1a7157ae64a7ac6cb3154de5b31965e2bb5b5c3dd4472f33d6 not found: ID does not exist"
Jul 08 16:40:27.612022 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:40:27.611979    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66fdd9cf8b-f725b"]
Jul 08 16:40:27.620800 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:40:27.620777    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-66fdd9cf8b-f725b"]
Jul 08 16:40:28.286267 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:40:28.286237    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17fd5ffc-5451-4ed5-ae7e-42feeb4c9250" path="/var/lib/kubelet/pods/17fd5ffc-5451-4ed5-ae7e-42feeb4c9250/volumes"
Jul 08 16:40:45.519856 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:40:45.519753    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:40:45.539569 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:40:45.539541    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:40:45.660286 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:40:45.660259    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:01.156005 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:01.155969    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f2a26b1-8d19-4874-a633-1cd61e29091a-metrics-certs\") pod \"network-metrics-daemon-f5jcr\" (UID: \"0f2a26b1-8d19-4874-a633-1cd61e29091a\") " pod="openshift-multus/network-metrics-daemon-f5jcr"
Jul 08 16:41:01.158399 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:01.158371    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f2a26b1-8d19-4874-a633-1cd61e29091a-metrics-certs\") pod \"network-metrics-daemon-f5jcr\" (UID: \"0f2a26b1-8d19-4874-a633-1cd61e29091a\") " pod="openshift-multus/network-metrics-daemon-f5jcr"
Jul 08 16:41:01.384161 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:01.384126    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mshfs\""
Jul 08 16:41:01.392044 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:01.392022    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f5jcr"
Jul 08 16:41:01.523978 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:01.523950    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-f5jcr"]
Jul 08 16:41:01.526457 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:41:01.526429    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f2a26b1_8d19_4874_a633_1cd61e29091a.slice/crio-1e58007791d8c4499d1198b92f92749952c9feecdf4e8a7a4e26f25a7f3238e2 WatchSource:0}: Error finding container 1e58007791d8c4499d1198b92f92749952c9feecdf4e8a7a4e26f25a7f3238e2: Status 404 returned error can't find the container with id 1e58007791d8c4499d1198b92f92749952c9feecdf4e8a7a4e26f25a7f3238e2
Jul 08 16:41:01.703917 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:01.703838    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-f5jcr" event={"ID":"0f2a26b1-8d19-4874-a633-1cd61e29091a","Type":"ContainerStarted","Data":"1e58007791d8c4499d1198b92f92749952c9feecdf4e8a7a4e26f25a7f3238e2"}
Jul 08 16:41:03.672287 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:03.672247    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 08 16:41:03.672761 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:03.672573    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2ac190a4-147f-4895-90e9-9ecc1c0df60f" containerName="prometheus" containerID="cri-o://532f13eb8e95ab15daa585aad29fe8faefdaff7b69eb8c5b0f20eeecbced013c" gracePeriod=600
Jul 08 16:41:03.672761 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:03.672593    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2ac190a4-147f-4895-90e9-9ecc1c0df60f" containerName="kube-rbac-proxy" containerID="cri-o://dc1c001ce3b12211c906684fb8b9745ecba4003ec2cbb7dcd84cb73d23b6bbad" gracePeriod=600
Jul 08 16:41:03.672761 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:03.672614    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2ac190a4-147f-4895-90e9-9ecc1c0df60f" containerName="thanos-sidecar" containerID="cri-o://842190fc6d43e8a898f199c3898234ec3030bcc746cc29768f267650d0a61a70" gracePeriod=600
Jul 08 16:41:03.672761 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:03.672668    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2ac190a4-147f-4895-90e9-9ecc1c0df60f" containerName="config-reloader" containerID="cri-o://f47703b3d7319ee5ac3cbb855c6933caa47cd1d77c04034eeac1a9ba499cd5cc" gracePeriod=600
Jul 08 16:41:03.672761 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:03.672674    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2ac190a4-147f-4895-90e9-9ecc1c0df60f" containerName="kube-rbac-proxy-thanos" containerID="cri-o://1607cc55942ceb3893ac08637da2c4a153cccacd3b923d1455e68ad18f1ab592" gracePeriod=600
Jul 08 16:41:03.672761 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:03.672762    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2ac190a4-147f-4895-90e9-9ecc1c0df60f" containerName="kube-rbac-proxy-web" containerID="cri-o://76e9b89a240d4de422fd5cf0f8ee5fb74813264e7f1916150d0c88679fc20ba1" gracePeriod=600
Jul 08 16:41:03.723591 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:03.723549    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-f5jcr" event={"ID":"0f2a26b1-8d19-4874-a633-1cd61e29091a","Type":"ContainerStarted","Data":"b56e29c85822f86af13064686c4a98136284534254c4fac63fc759823ca43430"}
Jul 08 16:41:03.723722 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:03.723692    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-f5jcr" event={"ID":"0f2a26b1-8d19-4874-a633-1cd61e29091a","Type":"ContainerStarted","Data":"37273c2fa61948f536cfff739ec0c8b145567a85ce30f0b68405a34f3f5e364e"}
Jul 08 16:41:03.742107 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:03.742058    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-f5jcr" podStartSLOduration=373.137077073 podStartE2EDuration="6m14.742043152s" totalImagesPullingTime="1.604966079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:34:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:41:03.741502118 +0000 UTC m=+376.012951659" watchObservedRunningTime="2026-07-08 16:41:03.742043152 +0000 UTC m=+376.013492692"
Jul 08 16:41:03.934664 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:03.934611    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:03.980484 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:03.980454    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2ac190a4-147f-4895-90e9-9ecc1c0df60f-prometheus-k8s-db\") pod \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") "
Jul 08 16:41:03.980696 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:03.980495    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ac190a4-147f-4895-90e9-9ecc1c0df60f-configmap-metrics-client-ca\") pod \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") "
Jul 08 16:41:03.980696 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:03.980523    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2ac190a4-147f-4895-90e9-9ecc1c0df60f-prometheus-k8s-rulefiles-2\") pod \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") "
Jul 08 16:41:03.980696 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:03.980576    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ac190a4-147f-4895-90e9-9ecc1c0df60f-configmap-kubelet-serving-ca-bundle\") pod \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") "
Jul 08 16:41:03.980696 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:03.980607    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2ac190a4-147f-4895-90e9-9ecc1c0df60f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") "
Jul 08 16:41:03.980696 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:03.980650    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2ac190a4-147f-4895-90e9-9ecc1c0df60f-secret-prometheus-k8s-tls\") pod \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") "
Jul 08 16:41:03.980696 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:03.980684    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2ac190a4-147f-4895-90e9-9ecc1c0df60f-secret-metrics-client-certs\") pod \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") "
Jul 08 16:41:03.981061 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:03.980721    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ac190a4-147f-4895-90e9-9ecc1c0df60f-configmap-serving-certs-ca-bundle\") pod \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") "
Jul 08 16:41:03.981061 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:03.980748    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ac190a4-147f-4895-90e9-9ecc1c0df60f-prometheus-trusted-ca-bundle\") pod \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") "
Jul 08 16:41:03.981061 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:03.980777    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2ac190a4-147f-4895-90e9-9ecc1c0df60f-thanos-prometheus-http-client-file\") pod \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") "
Jul 08 16:41:03.981061 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:03.980811    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2ac190a4-147f-4895-90e9-9ecc1c0df60f-secret-kube-rbac-proxy\") pod \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") "
Jul 08 16:41:03.981061 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:03.980845    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2ac190a4-147f-4895-90e9-9ecc1c0df60f-web-config\") pod \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") "
Jul 08 16:41:03.981061 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:03.980868    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2ac190a4-147f-4895-90e9-9ecc1c0df60f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") "
Jul 08 16:41:03.981061 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:03.980895    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2ac190a4-147f-4895-90e9-9ecc1c0df60f-config\") pod \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") "
Jul 08 16:41:03.981061 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:03.980927    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c7gzq\" (UniqueName: \"kubernetes.io/projected/2ac190a4-147f-4895-90e9-9ecc1c0df60f-kube-api-access-c7gzq\") pod \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") "
Jul 08 16:41:03.981061 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:03.980952    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2ac190a4-147f-4895-90e9-9ecc1c0df60f-secret-grpc-tls\") pod \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") "
Jul 08 16:41:03.981061 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:03.980984    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2ac190a4-147f-4895-90e9-9ecc1c0df60f-config-out\") pod \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") "
Jul 08 16:41:03.981061 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:03.981026    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2ac190a4-147f-4895-90e9-9ecc1c0df60f-prometheus-k8s-rulefiles-0\") pod \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") "
Jul 08 16:41:03.981061 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:03.981052    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2ac190a4-147f-4895-90e9-9ecc1c0df60f-tls-assets\") pod \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") "
Jul 08 16:41:03.981727 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:03.981085    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2ac190a4-147f-4895-90e9-9ecc1c0df60f-prometheus-k8s-rulefiles-1\") pod \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\" (UID: \"2ac190a4-147f-4895-90e9-9ecc1c0df60f\") "
Jul 08 16:41:03.981727 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:03.981589    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ac190a4-147f-4895-90e9-9ecc1c0df60f-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "2ac190a4-147f-4895-90e9-9ecc1c0df60f" (UID: "2ac190a4-147f-4895-90e9-9ecc1c0df60f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:41:03.981838 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:03.981744    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ac190a4-147f-4895-90e9-9ecc1c0df60f-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "2ac190a4-147f-4895-90e9-9ecc1c0df60f" (UID: "2ac190a4-147f-4895-90e9-9ecc1c0df60f"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:41:03.982046 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:03.981939    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ac190a4-147f-4895-90e9-9ecc1c0df60f-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "2ac190a4-147f-4895-90e9-9ecc1c0df60f" (UID: "2ac190a4-147f-4895-90e9-9ecc1c0df60f"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:41:03.984286 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:03.983436    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ac190a4-147f-4895-90e9-9ecc1c0df60f-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "2ac190a4-147f-4895-90e9-9ecc1c0df60f" (UID: "2ac190a4-147f-4895-90e9-9ecc1c0df60f"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:41:03.984286 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:03.983498    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ac190a4-147f-4895-90e9-9ecc1c0df60f-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "2ac190a4-147f-4895-90e9-9ecc1c0df60f" (UID: "2ac190a4-147f-4895-90e9-9ecc1c0df60f"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:41:03.984286 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:03.983660    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ac190a4-147f-4895-90e9-9ecc1c0df60f-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "2ac190a4-147f-4895-90e9-9ecc1c0df60f" (UID: "2ac190a4-147f-4895-90e9-9ecc1c0df60f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:41:03.984286 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:03.983968    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ac190a4-147f-4895-90e9-9ecc1c0df60f-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "2ac190a4-147f-4895-90e9-9ecc1c0df60f" (UID: "2ac190a4-147f-4895-90e9-9ecc1c0df60f"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:41:03.984286 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:03.984194    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ac190a4-147f-4895-90e9-9ecc1c0df60f-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "2ac190a4-147f-4895-90e9-9ecc1c0df60f" (UID: "2ac190a4-147f-4895-90e9-9ecc1c0df60f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:41:03.986266 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:03.986230    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ac190a4-147f-4895-90e9-9ecc1c0df60f-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "2ac190a4-147f-4895-90e9-9ecc1c0df60f" (UID: "2ac190a4-147f-4895-90e9-9ecc1c0df60f"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:41:03.986368 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:03.986331    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ac190a4-147f-4895-90e9-9ecc1c0df60f-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "2ac190a4-147f-4895-90e9-9ecc1c0df60f" (UID: "2ac190a4-147f-4895-90e9-9ecc1c0df60f"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:41:03.986797 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:03.986767    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ac190a4-147f-4895-90e9-9ecc1c0df60f-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "2ac190a4-147f-4895-90e9-9ecc1c0df60f" (UID: "2ac190a4-147f-4895-90e9-9ecc1c0df60f"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:41:03.988058 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:03.988019    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ac190a4-147f-4895-90e9-9ecc1c0df60f-kube-api-access-c7gzq" (OuterVolumeSpecName: "kube-api-access-c7gzq") pod "2ac190a4-147f-4895-90e9-9ecc1c0df60f" (UID: "2ac190a4-147f-4895-90e9-9ecc1c0df60f"). InnerVolumeSpecName "kube-api-access-c7gzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:41:03.988234 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:03.988206    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ac190a4-147f-4895-90e9-9ecc1c0df60f-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "2ac190a4-147f-4895-90e9-9ecc1c0df60f" (UID: "2ac190a4-147f-4895-90e9-9ecc1c0df60f"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:41:03.990788 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:03.990753    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ac190a4-147f-4895-90e9-9ecc1c0df60f-config-out" (OuterVolumeSpecName: "config-out") pod "2ac190a4-147f-4895-90e9-9ecc1c0df60f" (UID: "2ac190a4-147f-4895-90e9-9ecc1c0df60f"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:41:03.990788 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:03.990772    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ac190a4-147f-4895-90e9-9ecc1c0df60f-config" (OuterVolumeSpecName: "config") pod "2ac190a4-147f-4895-90e9-9ecc1c0df60f" (UID: "2ac190a4-147f-4895-90e9-9ecc1c0df60f"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:41:03.990984 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:03.990963    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ac190a4-147f-4895-90e9-9ecc1c0df60f-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "2ac190a4-147f-4895-90e9-9ecc1c0df60f" (UID: "2ac190a4-147f-4895-90e9-9ecc1c0df60f"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:41:03.991072 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:03.991050    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ac190a4-147f-4895-90e9-9ecc1c0df60f-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "2ac190a4-147f-4895-90e9-9ecc1c0df60f" (UID: "2ac190a4-147f-4895-90e9-9ecc1c0df60f"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:41:03.992286 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:03.992242    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ac190a4-147f-4895-90e9-9ecc1c0df60f-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "2ac190a4-147f-4895-90e9-9ecc1c0df60f" (UID: "2ac190a4-147f-4895-90e9-9ecc1c0df60f"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:41:03.992671 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:03.992646    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ac190a4-147f-4895-90e9-9ecc1c0df60f-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "2ac190a4-147f-4895-90e9-9ecc1c0df60f" (UID: "2ac190a4-147f-4895-90e9-9ecc1c0df60f"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:41:04.002833 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.002803    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ac190a4-147f-4895-90e9-9ecc1c0df60f-web-config" (OuterVolumeSpecName: "web-config") pod "2ac190a4-147f-4895-90e9-9ecc1c0df60f" (UID: "2ac190a4-147f-4895-90e9-9ecc1c0df60f"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:41:04.082624 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.082585    2578 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ac190a4-147f-4895-90e9-9ecc1c0df60f-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:41:04.082624 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.082619    2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2ac190a4-147f-4895-90e9-9ecc1c0df60f-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:41:04.082624 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.082651    2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2ac190a4-147f-4895-90e9-9ecc1c0df60f-secret-prometheus-k8s-tls\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:41:04.082624 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.082661    2578 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2ac190a4-147f-4895-90e9-9ecc1c0df60f-secret-metrics-client-certs\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:41:04.082881 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.082669    2578 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ac190a4-147f-4895-90e9-9ecc1c0df60f-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:41:04.082881 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.082679    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ac190a4-147f-4895-90e9-9ecc1c0df60f-prometheus-trusted-ca-bundle\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:41:04.082881 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.082687    2578 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2ac190a4-147f-4895-90e9-9ecc1c0df60f-thanos-prometheus-http-client-file\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:41:04.082881 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.082695    2578 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2ac190a4-147f-4895-90e9-9ecc1c0df60f-secret-kube-rbac-proxy\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:41:04.082881 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.082703    2578 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2ac190a4-147f-4895-90e9-9ecc1c0df60f-web-config\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:41:04.082881 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.082711    2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2ac190a4-147f-4895-90e9-9ecc1c0df60f-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:41:04.082881 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.082719    2578 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/2ac190a4-147f-4895-90e9-9ecc1c0df60f-config\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:41:04.082881 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.082726    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7gzq\" (UniqueName: \"kubernetes.io/projected/2ac190a4-147f-4895-90e9-9ecc1c0df60f-kube-api-access-c7gzq\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:41:04.082881 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.082733    2578 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2ac190a4-147f-4895-90e9-9ecc1c0df60f-secret-grpc-tls\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:41:04.082881 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.082741    2578 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2ac190a4-147f-4895-90e9-9ecc1c0df60f-config-out\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:41:04.082881 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.082748    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2ac190a4-147f-4895-90e9-9ecc1c0df60f-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:41:04.082881 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.082755    2578 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2ac190a4-147f-4895-90e9-9ecc1c0df60f-tls-assets\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:41:04.082881 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.082763    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2ac190a4-147f-4895-90e9-9ecc1c0df60f-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:41:04.082881 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.082771    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2ac190a4-147f-4895-90e9-9ecc1c0df60f-prometheus-k8s-db\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:41:04.082881 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.082780    2578 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ac190a4-147f-4895-90e9-9ecc1c0df60f-configmap-metrics-client-ca\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:41:04.082881 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.082788    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2ac190a4-147f-4895-90e9-9ecc1c0df60f-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:41:04.730541 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.730505    2578 generic.go:356] "Generic (PLEG): container finished" podID="2ac190a4-147f-4895-90e9-9ecc1c0df60f" containerID="1607cc55942ceb3893ac08637da2c4a153cccacd3b923d1455e68ad18f1ab592" exitCode=0
Jul 08 16:41:04.730541 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.730533    2578 generic.go:356] "Generic (PLEG): container finished" podID="2ac190a4-147f-4895-90e9-9ecc1c0df60f" containerID="dc1c001ce3b12211c906684fb8b9745ecba4003ec2cbb7dcd84cb73d23b6bbad" exitCode=0
Jul 08 16:41:04.730541 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.730539    2578 generic.go:356] "Generic (PLEG): container finished" podID="2ac190a4-147f-4895-90e9-9ecc1c0df60f" containerID="76e9b89a240d4de422fd5cf0f8ee5fb74813264e7f1916150d0c88679fc20ba1" exitCode=0
Jul 08 16:41:04.730541 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.730544    2578 generic.go:356] "Generic (PLEG): container finished" podID="2ac190a4-147f-4895-90e9-9ecc1c0df60f" containerID="842190fc6d43e8a898f199c3898234ec3030bcc746cc29768f267650d0a61a70" exitCode=0
Jul 08 16:41:04.731348 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.730551    2578 generic.go:356] "Generic (PLEG): container finished" podID="2ac190a4-147f-4895-90e9-9ecc1c0df60f" containerID="f47703b3d7319ee5ac3cbb855c6933caa47cd1d77c04034eeac1a9ba499cd5cc" exitCode=0
Jul 08 16:41:04.731348 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.730560    2578 generic.go:356] "Generic (PLEG): container finished" podID="2ac190a4-147f-4895-90e9-9ecc1c0df60f" containerID="532f13eb8e95ab15daa585aad29fe8faefdaff7b69eb8c5b0f20eeecbced013c" exitCode=0
Jul 08 16:41:04.731348 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.730578    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:04.731348 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.730583    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2ac190a4-147f-4895-90e9-9ecc1c0df60f","Type":"ContainerDied","Data":"1607cc55942ceb3893ac08637da2c4a153cccacd3b923d1455e68ad18f1ab592"}
Jul 08 16:41:04.731348 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.730679    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2ac190a4-147f-4895-90e9-9ecc1c0df60f","Type":"ContainerDied","Data":"dc1c001ce3b12211c906684fb8b9745ecba4003ec2cbb7dcd84cb73d23b6bbad"}
Jul 08 16:41:04.731348 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.730714    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2ac190a4-147f-4895-90e9-9ecc1c0df60f","Type":"ContainerDied","Data":"76e9b89a240d4de422fd5cf0f8ee5fb74813264e7f1916150d0c88679fc20ba1"}
Jul 08 16:41:04.731348 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.730743    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2ac190a4-147f-4895-90e9-9ecc1c0df60f","Type":"ContainerDied","Data":"842190fc6d43e8a898f199c3898234ec3030bcc746cc29768f267650d0a61a70"}
Jul 08 16:41:04.731348 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.730754    2578 scope.go:117] "RemoveContainer" containerID="1607cc55942ceb3893ac08637da2c4a153cccacd3b923d1455e68ad18f1ab592"
Jul 08 16:41:04.731348 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.730786    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2ac190a4-147f-4895-90e9-9ecc1c0df60f","Type":"ContainerDied","Data":"f47703b3d7319ee5ac3cbb855c6933caa47cd1d77c04034eeac1a9ba499cd5cc"}
Jul 08 16:41:04.731348 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.730828    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2ac190a4-147f-4895-90e9-9ecc1c0df60f","Type":"ContainerDied","Data":"532f13eb8e95ab15daa585aad29fe8faefdaff7b69eb8c5b0f20eeecbced013c"}
Jul 08 16:41:04.731348 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.730882    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2ac190a4-147f-4895-90e9-9ecc1c0df60f","Type":"ContainerDied","Data":"aed0b22a444d81c973c540cf97bb03ca915afd9bdb7a57f31bf933f2798d8ed0"}
Jul 08 16:41:04.738870 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.738847    2578 scope.go:117] "RemoveContainer" containerID="dc1c001ce3b12211c906684fb8b9745ecba4003ec2cbb7dcd84cb73d23b6bbad"
Jul 08 16:41:04.745930 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.745911    2578 scope.go:117] "RemoveContainer" containerID="76e9b89a240d4de422fd5cf0f8ee5fb74813264e7f1916150d0c88679fc20ba1"
Jul 08 16:41:04.752749 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.752728    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 08 16:41:04.753027 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.753012    2578 scope.go:117] "RemoveContainer" containerID="842190fc6d43e8a898f199c3898234ec3030bcc746cc29768f267650d0a61a70"
Jul 08 16:41:04.758031 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.758009    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 08 16:41:04.760212 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.760195    2578 scope.go:117] "RemoveContainer" containerID="f47703b3d7319ee5ac3cbb855c6933caa47cd1d77c04034eeac1a9ba499cd5cc"
Jul 08 16:41:04.766991 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.766975    2578 scope.go:117] "RemoveContainer" containerID="532f13eb8e95ab15daa585aad29fe8faefdaff7b69eb8c5b0f20eeecbced013c"
Jul 08 16:41:04.774142 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.774123    2578 scope.go:117] "RemoveContainer" containerID="88ab3216165b920c9c4b15b4d30c8aa1a2ca71c8997664bed119fe1aa2865d36"
Jul 08 16:41:04.781495 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.781469    2578 scope.go:117] "RemoveContainer" containerID="1607cc55942ceb3893ac08637da2c4a153cccacd3b923d1455e68ad18f1ab592"
Jul 08 16:41:04.781839 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:41:04.781817    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1607cc55942ceb3893ac08637da2c4a153cccacd3b923d1455e68ad18f1ab592\": container with ID starting with 1607cc55942ceb3893ac08637da2c4a153cccacd3b923d1455e68ad18f1ab592 not found: ID does not exist" containerID="1607cc55942ceb3893ac08637da2c4a153cccacd3b923d1455e68ad18f1ab592"
Jul 08 16:41:04.781931 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.781854    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1607cc55942ceb3893ac08637da2c4a153cccacd3b923d1455e68ad18f1ab592"} err="failed to get container status \"1607cc55942ceb3893ac08637da2c4a153cccacd3b923d1455e68ad18f1ab592\": rpc error: code = NotFound desc = could not find container \"1607cc55942ceb3893ac08637da2c4a153cccacd3b923d1455e68ad18f1ab592\": container with ID starting with 1607cc55942ceb3893ac08637da2c4a153cccacd3b923d1455e68ad18f1ab592 not found: ID does not exist"
Jul 08 16:41:04.781931 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.781886    2578 scope.go:117] "RemoveContainer" containerID="dc1c001ce3b12211c906684fb8b9745ecba4003ec2cbb7dcd84cb73d23b6bbad"
Jul 08 16:41:04.782166 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:41:04.782149    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc1c001ce3b12211c906684fb8b9745ecba4003ec2cbb7dcd84cb73d23b6bbad\": container with ID starting with dc1c001ce3b12211c906684fb8b9745ecba4003ec2cbb7dcd84cb73d23b6bbad not found: ID does not exist" containerID="dc1c001ce3b12211c906684fb8b9745ecba4003ec2cbb7dcd84cb73d23b6bbad"
Jul 08 16:41:04.782208 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.782172    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc1c001ce3b12211c906684fb8b9745ecba4003ec2cbb7dcd84cb73d23b6bbad"} err="failed to get container status \"dc1c001ce3b12211c906684fb8b9745ecba4003ec2cbb7dcd84cb73d23b6bbad\": rpc error: code = NotFound desc = could not find container \"dc1c001ce3b12211c906684fb8b9745ecba4003ec2cbb7dcd84cb73d23b6bbad\": container with ID starting with dc1c001ce3b12211c906684fb8b9745ecba4003ec2cbb7dcd84cb73d23b6bbad not found: ID does not exist"
Jul 08 16:41:04.782208 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.782188    2578 scope.go:117] "RemoveContainer" containerID="76e9b89a240d4de422fd5cf0f8ee5fb74813264e7f1916150d0c88679fc20ba1"
Jul 08 16:41:04.782411 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:41:04.782394    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76e9b89a240d4de422fd5cf0f8ee5fb74813264e7f1916150d0c88679fc20ba1\": container with ID starting with 76e9b89a240d4de422fd5cf0f8ee5fb74813264e7f1916150d0c88679fc20ba1 not found: ID does not exist" containerID="76e9b89a240d4de422fd5cf0f8ee5fb74813264e7f1916150d0c88679fc20ba1"
Jul 08 16:41:04.782480 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.782416    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76e9b89a240d4de422fd5cf0f8ee5fb74813264e7f1916150d0c88679fc20ba1"} err="failed to get container status \"76e9b89a240d4de422fd5cf0f8ee5fb74813264e7f1916150d0c88679fc20ba1\": rpc error: code = NotFound desc = could not find container \"76e9b89a240d4de422fd5cf0f8ee5fb74813264e7f1916150d0c88679fc20ba1\": container with ID starting with 76e9b89a240d4de422fd5cf0f8ee5fb74813264e7f1916150d0c88679fc20ba1 not found: ID does not exist"
Jul 08 16:41:04.782533 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.782481    2578 scope.go:117] "RemoveContainer" containerID="842190fc6d43e8a898f199c3898234ec3030bcc746cc29768f267650d0a61a70"
Jul 08 16:41:04.782767 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:41:04.782738    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"842190fc6d43e8a898f199c3898234ec3030bcc746cc29768f267650d0a61a70\": container with ID starting with 842190fc6d43e8a898f199c3898234ec3030bcc746cc29768f267650d0a61a70 not found: ID does not exist" containerID="842190fc6d43e8a898f199c3898234ec3030bcc746cc29768f267650d0a61a70"
Jul 08 16:41:04.782866 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.782771    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"842190fc6d43e8a898f199c3898234ec3030bcc746cc29768f267650d0a61a70"} err="failed to get container status \"842190fc6d43e8a898f199c3898234ec3030bcc746cc29768f267650d0a61a70\": rpc error: code = NotFound desc = could not find container \"842190fc6d43e8a898f199c3898234ec3030bcc746cc29768f267650d0a61a70\": container with ID starting with 842190fc6d43e8a898f199c3898234ec3030bcc746cc29768f267650d0a61a70 not found: ID does not exist"
Jul 08 16:41:04.782866 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.782794    2578 scope.go:117] "RemoveContainer" containerID="f47703b3d7319ee5ac3cbb855c6933caa47cd1d77c04034eeac1a9ba499cd5cc"
Jul 08 16:41:04.783140 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:41:04.783090    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f47703b3d7319ee5ac3cbb855c6933caa47cd1d77c04034eeac1a9ba499cd5cc\": container with ID starting with f47703b3d7319ee5ac3cbb855c6933caa47cd1d77c04034eeac1a9ba499cd5cc not found: ID does not exist" containerID="f47703b3d7319ee5ac3cbb855c6933caa47cd1d77c04034eeac1a9ba499cd5cc"
Jul 08 16:41:04.783250 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.783119    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f47703b3d7319ee5ac3cbb855c6933caa47cd1d77c04034eeac1a9ba499cd5cc"} err="failed to get container status \"f47703b3d7319ee5ac3cbb855c6933caa47cd1d77c04034eeac1a9ba499cd5cc\": rpc error: code = NotFound desc = could not find container \"f47703b3d7319ee5ac3cbb855c6933caa47cd1d77c04034eeac1a9ba499cd5cc\": container with ID starting with f47703b3d7319ee5ac3cbb855c6933caa47cd1d77c04034eeac1a9ba499cd5cc not found: ID does not exist"
Jul 08 16:41:04.783250 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.783167    2578 scope.go:117] "RemoveContainer" containerID="532f13eb8e95ab15daa585aad29fe8faefdaff7b69eb8c5b0f20eeecbced013c"
Jul 08 16:41:04.783475 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:41:04.783443    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"532f13eb8e95ab15daa585aad29fe8faefdaff7b69eb8c5b0f20eeecbced013c\": container with ID starting with 532f13eb8e95ab15daa585aad29fe8faefdaff7b69eb8c5b0f20eeecbced013c not found: ID does not exist" containerID="532f13eb8e95ab15daa585aad29fe8faefdaff7b69eb8c5b0f20eeecbced013c"
Jul 08 16:41:04.783538 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.783474    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"532f13eb8e95ab15daa585aad29fe8faefdaff7b69eb8c5b0f20eeecbced013c"} err="failed to get container status \"532f13eb8e95ab15daa585aad29fe8faefdaff7b69eb8c5b0f20eeecbced013c\": rpc error: code = NotFound desc = could not find container \"532f13eb8e95ab15daa585aad29fe8faefdaff7b69eb8c5b0f20eeecbced013c\": container with ID starting with 532f13eb8e95ab15daa585aad29fe8faefdaff7b69eb8c5b0f20eeecbced013c not found: ID does not exist"
Jul 08 16:41:04.783538 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.783494    2578 scope.go:117] "RemoveContainer" containerID="88ab3216165b920c9c4b15b4d30c8aa1a2ca71c8997664bed119fe1aa2865d36"
Jul 08 16:41:04.783907 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:41:04.783884    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88ab3216165b920c9c4b15b4d30c8aa1a2ca71c8997664bed119fe1aa2865d36\": container with ID starting with 88ab3216165b920c9c4b15b4d30c8aa1a2ca71c8997664bed119fe1aa2865d36 not found: ID does not exist" containerID="88ab3216165b920c9c4b15b4d30c8aa1a2ca71c8997664bed119fe1aa2865d36"
Jul 08 16:41:04.783988 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.783911    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88ab3216165b920c9c4b15b4d30c8aa1a2ca71c8997664bed119fe1aa2865d36"} err="failed to get container status \"88ab3216165b920c9c4b15b4d30c8aa1a2ca71c8997664bed119fe1aa2865d36\": rpc error: code = NotFound desc = could not find container \"88ab3216165b920c9c4b15b4d30c8aa1a2ca71c8997664bed119fe1aa2865d36\": container with ID starting with 88ab3216165b920c9c4b15b4d30c8aa1a2ca71c8997664bed119fe1aa2865d36 not found: ID does not exist"
Jul 08 16:41:04.783988 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.783925    2578 scope.go:117] "RemoveContainer" containerID="1607cc55942ceb3893ac08637da2c4a153cccacd3b923d1455e68ad18f1ab592"
Jul 08 16:41:04.784187 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.784171    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1607cc55942ceb3893ac08637da2c4a153cccacd3b923d1455e68ad18f1ab592"} err="failed to get container status \"1607cc55942ceb3893ac08637da2c4a153cccacd3b923d1455e68ad18f1ab592\": rpc error: code = NotFound desc = could not find container \"1607cc55942ceb3893ac08637da2c4a153cccacd3b923d1455e68ad18f1ab592\": container with ID starting with 1607cc55942ceb3893ac08637da2c4a153cccacd3b923d1455e68ad18f1ab592 not found: ID does not exist"
Jul 08 16:41:04.784187 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.784186    2578 scope.go:117] "RemoveContainer" containerID="dc1c001ce3b12211c906684fb8b9745ecba4003ec2cbb7dcd84cb73d23b6bbad"
Jul 08 16:41:04.784390 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.784375    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc1c001ce3b12211c906684fb8b9745ecba4003ec2cbb7dcd84cb73d23b6bbad"} err="failed to get container status \"dc1c001ce3b12211c906684fb8b9745ecba4003ec2cbb7dcd84cb73d23b6bbad\": rpc error: code = NotFound desc = could not find container \"dc1c001ce3b12211c906684fb8b9745ecba4003ec2cbb7dcd84cb73d23b6bbad\": container with ID starting with dc1c001ce3b12211c906684fb8b9745ecba4003ec2cbb7dcd84cb73d23b6bbad not found: ID does not exist"
Jul 08 16:41:04.784390 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.784390    2578 scope.go:117] "RemoveContainer" containerID="76e9b89a240d4de422fd5cf0f8ee5fb74813264e7f1916150d0c88679fc20ba1"
Jul 08 16:41:04.784591 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.784576    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 08 16:41:04.784657 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.784610    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76e9b89a240d4de422fd5cf0f8ee5fb74813264e7f1916150d0c88679fc20ba1"} err="failed to get container status \"76e9b89a240d4de422fd5cf0f8ee5fb74813264e7f1916150d0c88679fc20ba1\": rpc error: code = NotFound desc = could not find container \"76e9b89a240d4de422fd5cf0f8ee5fb74813264e7f1916150d0c88679fc20ba1\": container with ID starting with 76e9b89a240d4de422fd5cf0f8ee5fb74813264e7f1916150d0c88679fc20ba1 not found: ID does not exist"
Jul 08 16:41:04.784657 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.784625    2578 scope.go:117] "RemoveContainer" containerID="842190fc6d43e8a898f199c3898234ec3030bcc746cc29768f267650d0a61a70"
Jul 08 16:41:04.784893 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.784873    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"842190fc6d43e8a898f199c3898234ec3030bcc746cc29768f267650d0a61a70"} err="failed to get container status \"842190fc6d43e8a898f199c3898234ec3030bcc746cc29768f267650d0a61a70\": rpc error: code = NotFound desc = could not find container \"842190fc6d43e8a898f199c3898234ec3030bcc746cc29768f267650d0a61a70\": container with ID starting with 842190fc6d43e8a898f199c3898234ec3030bcc746cc29768f267650d0a61a70 not found: ID does not exist"
Jul 08 16:41:04.784893 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.784894    2578 scope.go:117] "RemoveContainer" containerID="f47703b3d7319ee5ac3cbb855c6933caa47cd1d77c04034eeac1a9ba499cd5cc"
Jul 08 16:41:04.785017 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.784897    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17fd5ffc-5451-4ed5-ae7e-42feeb4c9250" containerName="console"
Jul 08 16:41:04.785017 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.784910    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="17fd5ffc-5451-4ed5-ae7e-42feeb4c9250" containerName="console"
Jul 08 16:41:04.785017 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.784918    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ac190a4-147f-4895-90e9-9ecc1c0df60f" containerName="kube-rbac-proxy"
Jul 08 16:41:04.785017 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.784923    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ac190a4-147f-4895-90e9-9ecc1c0df60f" containerName="kube-rbac-proxy"
Jul 08 16:41:04.785017 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.784928    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ac190a4-147f-4895-90e9-9ecc1c0df60f" containerName="kube-rbac-proxy-thanos"
Jul 08 16:41:04.785017 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.784933    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ac190a4-147f-4895-90e9-9ecc1c0df60f" containerName="kube-rbac-proxy-thanos"
Jul 08 16:41:04.785017 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.784941    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ac190a4-147f-4895-90e9-9ecc1c0df60f" containerName="init-config-reloader"
Jul 08 16:41:04.785017 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.784946    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ac190a4-147f-4895-90e9-9ecc1c0df60f" containerName="init-config-reloader"
Jul 08 16:41:04.785017 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.784957    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ac190a4-147f-4895-90e9-9ecc1c0df60f" containerName="config-reloader"
Jul 08 16:41:04.785017 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.784962    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ac190a4-147f-4895-90e9-9ecc1c0df60f" containerName="config-reloader"
Jul 08 16:41:04.785017 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.784970    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ac190a4-147f-4895-90e9-9ecc1c0df60f" containerName="kube-rbac-proxy-web"
Jul 08 16:41:04.785017 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.784975    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ac190a4-147f-4895-90e9-9ecc1c0df60f" containerName="kube-rbac-proxy-web"
Jul 08 16:41:04.785017 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.784985    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ac190a4-147f-4895-90e9-9ecc1c0df60f" containerName="prometheus"
Jul 08 16:41:04.785017 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.784989    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ac190a4-147f-4895-90e9-9ecc1c0df60f" containerName="prometheus"
Jul 08 16:41:04.785017 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.784994    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ac190a4-147f-4895-90e9-9ecc1c0df60f" containerName="thanos-sidecar"
Jul 08 16:41:04.785017 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.785000    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ac190a4-147f-4895-90e9-9ecc1c0df60f" containerName="thanos-sidecar"
Jul 08 16:41:04.785562 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.785049    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ac190a4-147f-4895-90e9-9ecc1c0df60f" containerName="kube-rbac-proxy"
Jul 08 16:41:04.785562 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.785060    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ac190a4-147f-4895-90e9-9ecc1c0df60f" containerName="kube-rbac-proxy-thanos"
Jul 08 16:41:04.785562 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.785070    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ac190a4-147f-4895-90e9-9ecc1c0df60f" containerName="prometheus"
Jul 08 16:41:04.785562 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.785076    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="17fd5ffc-5451-4ed5-ae7e-42feeb4c9250" containerName="console"
Jul 08 16:41:04.785562 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.785081    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ac190a4-147f-4895-90e9-9ecc1c0df60f" containerName="thanos-sidecar"
Jul 08 16:41:04.785562 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.785086    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ac190a4-147f-4895-90e9-9ecc1c0df60f" containerName="kube-rbac-proxy-web"
Jul 08 16:41:04.785562 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.785092    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ac190a4-147f-4895-90e9-9ecc1c0df60f" containerName="config-reloader"
Jul 08 16:41:04.785562 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.785147    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f47703b3d7319ee5ac3cbb855c6933caa47cd1d77c04034eeac1a9ba499cd5cc"} err="failed to get container status \"f47703b3d7319ee5ac3cbb855c6933caa47cd1d77c04034eeac1a9ba499cd5cc\": rpc error: code = NotFound desc = could not find container \"f47703b3d7319ee5ac3cbb855c6933caa47cd1d77c04034eeac1a9ba499cd5cc\": container with ID starting with f47703b3d7319ee5ac3cbb855c6933caa47cd1d77c04034eeac1a9ba499cd5cc not found: ID does not exist"
Jul 08 16:41:04.785562 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.785170    2578 scope.go:117] "RemoveContainer" containerID="532f13eb8e95ab15daa585aad29fe8faefdaff7b69eb8c5b0f20eeecbced013c"
Jul 08 16:41:04.785562 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.785394    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"532f13eb8e95ab15daa585aad29fe8faefdaff7b69eb8c5b0f20eeecbced013c"} err="failed to get container status \"532f13eb8e95ab15daa585aad29fe8faefdaff7b69eb8c5b0f20eeecbced013c\": rpc error: code = NotFound desc = could not find container \"532f13eb8e95ab15daa585aad29fe8faefdaff7b69eb8c5b0f20eeecbced013c\": container with ID starting with 532f13eb8e95ab15daa585aad29fe8faefdaff7b69eb8c5b0f20eeecbced013c not found: ID does not exist"
Jul 08 16:41:04.785562 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.785411    2578 scope.go:117] "RemoveContainer" containerID="88ab3216165b920c9c4b15b4d30c8aa1a2ca71c8997664bed119fe1aa2865d36"
Jul 08 16:41:04.785966 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.785627    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88ab3216165b920c9c4b15b4d30c8aa1a2ca71c8997664bed119fe1aa2865d36"} err="failed to get container status \"88ab3216165b920c9c4b15b4d30c8aa1a2ca71c8997664bed119fe1aa2865d36\": rpc error: code = NotFound desc = could not find container \"88ab3216165b920c9c4b15b4d30c8aa1a2ca71c8997664bed119fe1aa2865d36\": container with ID starting with 88ab3216165b920c9c4b15b4d30c8aa1a2ca71c8997664bed119fe1aa2865d36 not found: ID does not exist"
Jul 08 16:41:04.785966 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.785664    2578 scope.go:117] "RemoveContainer" containerID="1607cc55942ceb3893ac08637da2c4a153cccacd3b923d1455e68ad18f1ab592"
Jul 08 16:41:04.785966 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.785870    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1607cc55942ceb3893ac08637da2c4a153cccacd3b923d1455e68ad18f1ab592"} err="failed to get container status \"1607cc55942ceb3893ac08637da2c4a153cccacd3b923d1455e68ad18f1ab592\": rpc error: code = NotFound desc = could not find container \"1607cc55942ceb3893ac08637da2c4a153cccacd3b923d1455e68ad18f1ab592\": container with ID starting with 1607cc55942ceb3893ac08637da2c4a153cccacd3b923d1455e68ad18f1ab592 not found: ID does not exist"
Jul 08 16:41:04.785966 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.785890    2578 scope.go:117] "RemoveContainer" containerID="dc1c001ce3b12211c906684fb8b9745ecba4003ec2cbb7dcd84cb73d23b6bbad"
Jul 08 16:41:04.786129 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.786099    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc1c001ce3b12211c906684fb8b9745ecba4003ec2cbb7dcd84cb73d23b6bbad"} err="failed to get container status \"dc1c001ce3b12211c906684fb8b9745ecba4003ec2cbb7dcd84cb73d23b6bbad\": rpc error: code = NotFound desc = could not find container \"dc1c001ce3b12211c906684fb8b9745ecba4003ec2cbb7dcd84cb73d23b6bbad\": container with ID starting with dc1c001ce3b12211c906684fb8b9745ecba4003ec2cbb7dcd84cb73d23b6bbad not found: ID does not exist"
Jul 08 16:41:04.786163 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.786131    2578 scope.go:117] "RemoveContainer" containerID="76e9b89a240d4de422fd5cf0f8ee5fb74813264e7f1916150d0c88679fc20ba1"
Jul 08 16:41:04.786393 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.786371    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76e9b89a240d4de422fd5cf0f8ee5fb74813264e7f1916150d0c88679fc20ba1"} err="failed to get container status \"76e9b89a240d4de422fd5cf0f8ee5fb74813264e7f1916150d0c88679fc20ba1\": rpc error: code = NotFound desc = could not find container \"76e9b89a240d4de422fd5cf0f8ee5fb74813264e7f1916150d0c88679fc20ba1\": container with ID starting with 76e9b89a240d4de422fd5cf0f8ee5fb74813264e7f1916150d0c88679fc20ba1 not found: ID does not exist"
Jul 08 16:41:04.786481 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.786395    2578 scope.go:117] "RemoveContainer" containerID="842190fc6d43e8a898f199c3898234ec3030bcc746cc29768f267650d0a61a70"
Jul 08 16:41:04.786703 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.786685    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"842190fc6d43e8a898f199c3898234ec3030bcc746cc29768f267650d0a61a70"} err="failed to get container status \"842190fc6d43e8a898f199c3898234ec3030bcc746cc29768f267650d0a61a70\": rpc error: code = NotFound desc = could not find container \"842190fc6d43e8a898f199c3898234ec3030bcc746cc29768f267650d0a61a70\": container with ID starting with 842190fc6d43e8a898f199c3898234ec3030bcc746cc29768f267650d0a61a70 not found: ID does not exist"
Jul 08 16:41:04.786772 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.786705    2578 scope.go:117] "RemoveContainer" containerID="f47703b3d7319ee5ac3cbb855c6933caa47cd1d77c04034eeac1a9ba499cd5cc"
Jul 08 16:41:04.786927 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.786911    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f47703b3d7319ee5ac3cbb855c6933caa47cd1d77c04034eeac1a9ba499cd5cc"} err="failed to get container status \"f47703b3d7319ee5ac3cbb855c6933caa47cd1d77c04034eeac1a9ba499cd5cc\": rpc error: code = NotFound desc = could not find container \"f47703b3d7319ee5ac3cbb855c6933caa47cd1d77c04034eeac1a9ba499cd5cc\": container with ID starting with f47703b3d7319ee5ac3cbb855c6933caa47cd1d77c04034eeac1a9ba499cd5cc not found: ID does not exist"
Jul 08 16:41:04.786927 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.786927    2578 scope.go:117] "RemoveContainer" containerID="532f13eb8e95ab15daa585aad29fe8faefdaff7b69eb8c5b0f20eeecbced013c"
Jul 08 16:41:04.787169 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.787151    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"532f13eb8e95ab15daa585aad29fe8faefdaff7b69eb8c5b0f20eeecbced013c"} err="failed to get container status \"532f13eb8e95ab15daa585aad29fe8faefdaff7b69eb8c5b0f20eeecbced013c\": rpc error: code = NotFound desc = could not find container \"532f13eb8e95ab15daa585aad29fe8faefdaff7b69eb8c5b0f20eeecbced013c\": container with ID starting with 532f13eb8e95ab15daa585aad29fe8faefdaff7b69eb8c5b0f20eeecbced013c not found: ID does not exist"
Jul 08 16:41:04.787231 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.787170    2578 scope.go:117] "RemoveContainer" containerID="88ab3216165b920c9c4b15b4d30c8aa1a2ca71c8997664bed119fe1aa2865d36"
Jul 08 16:41:04.787442 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.787422    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88ab3216165b920c9c4b15b4d30c8aa1a2ca71c8997664bed119fe1aa2865d36"} err="failed to get container status \"88ab3216165b920c9c4b15b4d30c8aa1a2ca71c8997664bed119fe1aa2865d36\": rpc error: code = NotFound desc = could not find container \"88ab3216165b920c9c4b15b4d30c8aa1a2ca71c8997664bed119fe1aa2865d36\": container with ID starting with 88ab3216165b920c9c4b15b4d30c8aa1a2ca71c8997664bed119fe1aa2865d36 not found: ID does not exist"
Jul 08 16:41:04.787442 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.787440    2578 scope.go:117] "RemoveContainer" containerID="1607cc55942ceb3893ac08637da2c4a153cccacd3b923d1455e68ad18f1ab592"
Jul 08 16:41:04.787678 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.787659    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1607cc55942ceb3893ac08637da2c4a153cccacd3b923d1455e68ad18f1ab592"} err="failed to get container status \"1607cc55942ceb3893ac08637da2c4a153cccacd3b923d1455e68ad18f1ab592\": rpc error: code = NotFound desc = could not find container \"1607cc55942ceb3893ac08637da2c4a153cccacd3b923d1455e68ad18f1ab592\": container with ID starting with 1607cc55942ceb3893ac08637da2c4a153cccacd3b923d1455e68ad18f1ab592 not found: ID does not exist"
Jul 08 16:41:04.787678 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.787679    2578 scope.go:117] "RemoveContainer" containerID="dc1c001ce3b12211c906684fb8b9745ecba4003ec2cbb7dcd84cb73d23b6bbad"
Jul 08 16:41:04.787926 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.787908    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc1c001ce3b12211c906684fb8b9745ecba4003ec2cbb7dcd84cb73d23b6bbad"} err="failed to get container status \"dc1c001ce3b12211c906684fb8b9745ecba4003ec2cbb7dcd84cb73d23b6bbad\": rpc error: code = NotFound desc = could not find container \"dc1c001ce3b12211c906684fb8b9745ecba4003ec2cbb7dcd84cb73d23b6bbad\": container with ID starting with dc1c001ce3b12211c906684fb8b9745ecba4003ec2cbb7dcd84cb73d23b6bbad not found: ID does not exist"
Jul 08 16:41:04.787976 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.787926    2578 scope.go:117] "RemoveContainer" containerID="76e9b89a240d4de422fd5cf0f8ee5fb74813264e7f1916150d0c88679fc20ba1"
Jul 08 16:41:04.788146 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.788129    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76e9b89a240d4de422fd5cf0f8ee5fb74813264e7f1916150d0c88679fc20ba1"} err="failed to get container status \"76e9b89a240d4de422fd5cf0f8ee5fb74813264e7f1916150d0c88679fc20ba1\": rpc error: code = NotFound desc = could not find container \"76e9b89a240d4de422fd5cf0f8ee5fb74813264e7f1916150d0c88679fc20ba1\": container with ID starting with 76e9b89a240d4de422fd5cf0f8ee5fb74813264e7f1916150d0c88679fc20ba1 not found: ID does not exist"
Jul 08 16:41:04.788183 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.788146    2578 scope.go:117] "RemoveContainer" containerID="842190fc6d43e8a898f199c3898234ec3030bcc746cc29768f267650d0a61a70"
Jul 08 16:41:04.788355 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.788337    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"842190fc6d43e8a898f199c3898234ec3030bcc746cc29768f267650d0a61a70"} err="failed to get container status \"842190fc6d43e8a898f199c3898234ec3030bcc746cc29768f267650d0a61a70\": rpc error: code = NotFound desc = could not find container \"842190fc6d43e8a898f199c3898234ec3030bcc746cc29768f267650d0a61a70\": container with ID starting with 842190fc6d43e8a898f199c3898234ec3030bcc746cc29768f267650d0a61a70 not found: ID does not exist"
Jul 08 16:41:04.788412 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.788356    2578 scope.go:117] "RemoveContainer" containerID="f47703b3d7319ee5ac3cbb855c6933caa47cd1d77c04034eeac1a9ba499cd5cc"
Jul 08 16:41:04.788557 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.788539    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f47703b3d7319ee5ac3cbb855c6933caa47cd1d77c04034eeac1a9ba499cd5cc"} err="failed to get container status \"f47703b3d7319ee5ac3cbb855c6933caa47cd1d77c04034eeac1a9ba499cd5cc\": rpc error: code = NotFound desc = could not find container \"f47703b3d7319ee5ac3cbb855c6933caa47cd1d77c04034eeac1a9ba499cd5cc\": container with ID starting with f47703b3d7319ee5ac3cbb855c6933caa47cd1d77c04034eeac1a9ba499cd5cc not found: ID does not exist"
Jul 08 16:41:04.788698 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.788556    2578 scope.go:117] "RemoveContainer" containerID="532f13eb8e95ab15daa585aad29fe8faefdaff7b69eb8c5b0f20eeecbced013c"
Jul 08 16:41:04.788864 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.788820    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"532f13eb8e95ab15daa585aad29fe8faefdaff7b69eb8c5b0f20eeecbced013c"} err="failed to get container status \"532f13eb8e95ab15daa585aad29fe8faefdaff7b69eb8c5b0f20eeecbced013c\": rpc error: code = NotFound desc = could not find container \"532f13eb8e95ab15daa585aad29fe8faefdaff7b69eb8c5b0f20eeecbced013c\": container with ID starting with 532f13eb8e95ab15daa585aad29fe8faefdaff7b69eb8c5b0f20eeecbced013c not found: ID does not exist"
Jul 08 16:41:04.788864 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.788857    2578 scope.go:117] "RemoveContainer" containerID="88ab3216165b920c9c4b15b4d30c8aa1a2ca71c8997664bed119fe1aa2865d36"
Jul 08 16:41:04.789136 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.789118    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88ab3216165b920c9c4b15b4d30c8aa1a2ca71c8997664bed119fe1aa2865d36"} err="failed to get container status \"88ab3216165b920c9c4b15b4d30c8aa1a2ca71c8997664bed119fe1aa2865d36\": rpc error: code = NotFound desc = could not find container \"88ab3216165b920c9c4b15b4d30c8aa1a2ca71c8997664bed119fe1aa2865d36\": container with ID starting with 88ab3216165b920c9c4b15b4d30c8aa1a2ca71c8997664bed119fe1aa2865d36 not found: ID does not exist"
Jul 08 16:41:04.789179 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.789138    2578 scope.go:117] "RemoveContainer" containerID="1607cc55942ceb3893ac08637da2c4a153cccacd3b923d1455e68ad18f1ab592"
Jul 08 16:41:04.789387 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.789365    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1607cc55942ceb3893ac08637da2c4a153cccacd3b923d1455e68ad18f1ab592"} err="failed to get container status \"1607cc55942ceb3893ac08637da2c4a153cccacd3b923d1455e68ad18f1ab592\": rpc error: code = NotFound desc = could not find container \"1607cc55942ceb3893ac08637da2c4a153cccacd3b923d1455e68ad18f1ab592\": container with ID starting with 1607cc55942ceb3893ac08637da2c4a153cccacd3b923d1455e68ad18f1ab592 not found: ID does not exist"
Jul 08 16:41:04.789432 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.789389    2578 scope.go:117] "RemoveContainer" containerID="dc1c001ce3b12211c906684fb8b9745ecba4003ec2cbb7dcd84cb73d23b6bbad"
Jul 08 16:41:04.789619 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.789598    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc1c001ce3b12211c906684fb8b9745ecba4003ec2cbb7dcd84cb73d23b6bbad"} err="failed to get container status \"dc1c001ce3b12211c906684fb8b9745ecba4003ec2cbb7dcd84cb73d23b6bbad\": rpc error: code = NotFound desc = could not find container \"dc1c001ce3b12211c906684fb8b9745ecba4003ec2cbb7dcd84cb73d23b6bbad\": container with ID starting with dc1c001ce3b12211c906684fb8b9745ecba4003ec2cbb7dcd84cb73d23b6bbad not found: ID does not exist"
Jul 08 16:41:04.789736 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.789621    2578 scope.go:117] "RemoveContainer" containerID="76e9b89a240d4de422fd5cf0f8ee5fb74813264e7f1916150d0c88679fc20ba1"
Jul 08 16:41:04.789886 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.789866    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76e9b89a240d4de422fd5cf0f8ee5fb74813264e7f1916150d0c88679fc20ba1"} err="failed to get container status \"76e9b89a240d4de422fd5cf0f8ee5fb74813264e7f1916150d0c88679fc20ba1\": rpc error: code = NotFound desc = could not find container \"76e9b89a240d4de422fd5cf0f8ee5fb74813264e7f1916150d0c88679fc20ba1\": container with ID starting with 76e9b89a240d4de422fd5cf0f8ee5fb74813264e7f1916150d0c88679fc20ba1 not found: ID does not exist"
Jul 08 16:41:04.789922 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.789901    2578 scope.go:117] "RemoveContainer" containerID="842190fc6d43e8a898f199c3898234ec3030bcc746cc29768f267650d0a61a70"
Jul 08 16:41:04.790085 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.790067    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"842190fc6d43e8a898f199c3898234ec3030bcc746cc29768f267650d0a61a70"} err="failed to get container status \"842190fc6d43e8a898f199c3898234ec3030bcc746cc29768f267650d0a61a70\": rpc error: code = NotFound desc = could not find container \"842190fc6d43e8a898f199c3898234ec3030bcc746cc29768f267650d0a61a70\": container with ID starting with 842190fc6d43e8a898f199c3898234ec3030bcc746cc29768f267650d0a61a70 not found: ID does not exist"
Jul 08 16:41:04.790154 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.790087    2578 scope.go:117] "RemoveContainer" containerID="f47703b3d7319ee5ac3cbb855c6933caa47cd1d77c04034eeac1a9ba499cd5cc"
Jul 08 16:41:04.790281 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.790264    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f47703b3d7319ee5ac3cbb855c6933caa47cd1d77c04034eeac1a9ba499cd5cc"} err="failed to get container status \"f47703b3d7319ee5ac3cbb855c6933caa47cd1d77c04034eeac1a9ba499cd5cc\": rpc error: code = NotFound desc = could not find container \"f47703b3d7319ee5ac3cbb855c6933caa47cd1d77c04034eeac1a9ba499cd5cc\": container with ID starting with f47703b3d7319ee5ac3cbb855c6933caa47cd1d77c04034eeac1a9ba499cd5cc not found: ID does not exist"
Jul 08 16:41:04.790331 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.790283    2578 scope.go:117] "RemoveContainer" containerID="532f13eb8e95ab15daa585aad29fe8faefdaff7b69eb8c5b0f20eeecbced013c"
Jul 08 16:41:04.790485 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.790467    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"532f13eb8e95ab15daa585aad29fe8faefdaff7b69eb8c5b0f20eeecbced013c"} err="failed to get container status \"532f13eb8e95ab15daa585aad29fe8faefdaff7b69eb8c5b0f20eeecbced013c\": rpc error: code = NotFound desc = could not find container \"532f13eb8e95ab15daa585aad29fe8faefdaff7b69eb8c5b0f20eeecbced013c\": container with ID starting with 532f13eb8e95ab15daa585aad29fe8faefdaff7b69eb8c5b0f20eeecbced013c not found: ID does not exist"
Jul 08 16:41:04.790544 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.790486    2578 scope.go:117] "RemoveContainer" containerID="88ab3216165b920c9c4b15b4d30c8aa1a2ca71c8997664bed119fe1aa2865d36"
Jul 08 16:41:04.790589 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.790565    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:04.790713 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.790696    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88ab3216165b920c9c4b15b4d30c8aa1a2ca71c8997664bed119fe1aa2865d36"} err="failed to get container status \"88ab3216165b920c9c4b15b4d30c8aa1a2ca71c8997664bed119fe1aa2865d36\": rpc error: code = NotFound desc = could not find container \"88ab3216165b920c9c4b15b4d30c8aa1a2ca71c8997664bed119fe1aa2865d36\": container with ID starting with 88ab3216165b920c9c4b15b4d30c8aa1a2ca71c8997664bed119fe1aa2865d36 not found: ID does not exist"
Jul 08 16:41:04.790767 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.790713    2578 scope.go:117] "RemoveContainer" containerID="1607cc55942ceb3893ac08637da2c4a153cccacd3b923d1455e68ad18f1ab592"
Jul 08 16:41:04.791169 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.791150    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1607cc55942ceb3893ac08637da2c4a153cccacd3b923d1455e68ad18f1ab592"} err="failed to get container status \"1607cc55942ceb3893ac08637da2c4a153cccacd3b923d1455e68ad18f1ab592\": rpc error: code = NotFound desc = could not find container \"1607cc55942ceb3893ac08637da2c4a153cccacd3b923d1455e68ad18f1ab592\": container with ID starting with 1607cc55942ceb3893ac08637da2c4a153cccacd3b923d1455e68ad18f1ab592 not found: ID does not exist"
Jul 08 16:41:04.791169 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.791169    2578 scope.go:117] "RemoveContainer" containerID="dc1c001ce3b12211c906684fb8b9745ecba4003ec2cbb7dcd84cb73d23b6bbad"
Jul 08 16:41:04.791395 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.791376    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc1c001ce3b12211c906684fb8b9745ecba4003ec2cbb7dcd84cb73d23b6bbad"} err="failed to get container status \"dc1c001ce3b12211c906684fb8b9745ecba4003ec2cbb7dcd84cb73d23b6bbad\": rpc error: code = NotFound desc = could not find container \"dc1c001ce3b12211c906684fb8b9745ecba4003ec2cbb7dcd84cb73d23b6bbad\": container with ID starting with dc1c001ce3b12211c906684fb8b9745ecba4003ec2cbb7dcd84cb73d23b6bbad not found: ID does not exist"
Jul 08 16:41:04.791443 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.791398    2578 scope.go:117] "RemoveContainer" containerID="76e9b89a240d4de422fd5cf0f8ee5fb74813264e7f1916150d0c88679fc20ba1"
Jul 08 16:41:04.791617 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.791602    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76e9b89a240d4de422fd5cf0f8ee5fb74813264e7f1916150d0c88679fc20ba1"} err="failed to get container status \"76e9b89a240d4de422fd5cf0f8ee5fb74813264e7f1916150d0c88679fc20ba1\": rpc error: code = NotFound desc = could not find container \"76e9b89a240d4de422fd5cf0f8ee5fb74813264e7f1916150d0c88679fc20ba1\": container with ID starting with 76e9b89a240d4de422fd5cf0f8ee5fb74813264e7f1916150d0c88679fc20ba1 not found: ID does not exist"
Jul 08 16:41:04.791681 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.791617    2578 scope.go:117] "RemoveContainer" containerID="842190fc6d43e8a898f199c3898234ec3030bcc746cc29768f267650d0a61a70"
Jul 08 16:41:04.791854 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.791838    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"842190fc6d43e8a898f199c3898234ec3030bcc746cc29768f267650d0a61a70"} err="failed to get container status \"842190fc6d43e8a898f199c3898234ec3030bcc746cc29768f267650d0a61a70\": rpc error: code = NotFound desc = could not find container \"842190fc6d43e8a898f199c3898234ec3030bcc746cc29768f267650d0a61a70\": container with ID starting with 842190fc6d43e8a898f199c3898234ec3030bcc746cc29768f267650d0a61a70 not found: ID does not exist"
Jul 08 16:41:04.791898 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.791855    2578 scope.go:117] "RemoveContainer" containerID="f47703b3d7319ee5ac3cbb855c6933caa47cd1d77c04034eeac1a9ba499cd5cc"
Jul 08 16:41:04.792086 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.792070    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f47703b3d7319ee5ac3cbb855c6933caa47cd1d77c04034eeac1a9ba499cd5cc"} err="failed to get container status \"f47703b3d7319ee5ac3cbb855c6933caa47cd1d77c04034eeac1a9ba499cd5cc\": rpc error: code = NotFound desc = could not find container \"f47703b3d7319ee5ac3cbb855c6933caa47cd1d77c04034eeac1a9ba499cd5cc\": container with ID starting with f47703b3d7319ee5ac3cbb855c6933caa47cd1d77c04034eeac1a9ba499cd5cc not found: ID does not exist"
Jul 08 16:41:04.792132 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.792087    2578 scope.go:117] "RemoveContainer" containerID="532f13eb8e95ab15daa585aad29fe8faefdaff7b69eb8c5b0f20eeecbced013c"
Jul 08 16:41:04.792297 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.792275    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"532f13eb8e95ab15daa585aad29fe8faefdaff7b69eb8c5b0f20eeecbced013c"} err="failed to get container status \"532f13eb8e95ab15daa585aad29fe8faefdaff7b69eb8c5b0f20eeecbced013c\": rpc error: code = NotFound desc = could not find container \"532f13eb8e95ab15daa585aad29fe8faefdaff7b69eb8c5b0f20eeecbced013c\": container with ID starting with 532f13eb8e95ab15daa585aad29fe8faefdaff7b69eb8c5b0f20eeecbced013c not found: ID does not exist"
Jul 08 16:41:04.792372 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.792301    2578 scope.go:117] "RemoveContainer" containerID="88ab3216165b920c9c4b15b4d30c8aa1a2ca71c8997664bed119fe1aa2865d36"
Jul 08 16:41:04.792513 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.792492    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88ab3216165b920c9c4b15b4d30c8aa1a2ca71c8997664bed119fe1aa2865d36"} err="failed to get container status \"88ab3216165b920c9c4b15b4d30c8aa1a2ca71c8997664bed119fe1aa2865d36\": rpc error: code = NotFound desc = could not find container \"88ab3216165b920c9c4b15b4d30c8aa1a2ca71c8997664bed119fe1aa2865d36\": container with ID starting with 88ab3216165b920c9c4b15b4d30c8aa1a2ca71c8997664bed119fe1aa2865d36 not found: ID does not exist"
Jul 08 16:41:04.794555 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.794535    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 08 16:41:04.794777 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.794686    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 08 16:41:04.797737 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.795459    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 08 16:41:04.797737 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.795511    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 08 16:41:04.797737 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.795625    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 08 16:41:04.797737 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.795724    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 08 16:41:04.797737 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.795746    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 08 16:41:04.797737 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.795960    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 08 16:41:04.797737 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.795970    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 08 16:41:04.797737 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.795980    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 08 16:41:04.797737 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.796109    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 08 16:41:04.797737 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.796309    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-bm1io90ktn00n\""
Jul 08 16:41:04.797737 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.796573    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-ntzrr\""
Jul 08 16:41:04.798240 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.797804    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 08 16:41:04.799114 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.799092    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 08 16:41:04.802696 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.802677    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 08 16:41:04.809307 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.809277    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 08 16:41:04.887192 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.887153    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c01e663f-14c6-424b-8b7e-c1809069d1a1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c01e663f-14c6-424b-8b7e-c1809069d1a1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:04.887192 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.887191    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c01e663f-14c6-424b-8b7e-c1809069d1a1-web-config\") pod \"prometheus-k8s-0\" (UID: \"c01e663f-14c6-424b-8b7e-c1809069d1a1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:04.887409 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.887209    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c01e663f-14c6-424b-8b7e-c1809069d1a1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c01e663f-14c6-424b-8b7e-c1809069d1a1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:04.887409 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.887227    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c01e663f-14c6-424b-8b7e-c1809069d1a1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c01e663f-14c6-424b-8b7e-c1809069d1a1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:04.887409 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.887278    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c01e663f-14c6-424b-8b7e-c1809069d1a1-config-out\") pod \"prometheus-k8s-0\" (UID: \"c01e663f-14c6-424b-8b7e-c1809069d1a1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:04.887409 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.887325    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c01e663f-14c6-424b-8b7e-c1809069d1a1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c01e663f-14c6-424b-8b7e-c1809069d1a1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:04.887409 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.887364    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c01e663f-14c6-424b-8b7e-c1809069d1a1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c01e663f-14c6-424b-8b7e-c1809069d1a1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:04.887409 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.887384    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c01e663f-14c6-424b-8b7e-c1809069d1a1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c01e663f-14c6-424b-8b7e-c1809069d1a1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:04.887598 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.887421    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c01e663f-14c6-424b-8b7e-c1809069d1a1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c01e663f-14c6-424b-8b7e-c1809069d1a1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:04.887598 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.887450    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c01e663f-14c6-424b-8b7e-c1809069d1a1-config\") pod \"prometheus-k8s-0\" (UID: \"c01e663f-14c6-424b-8b7e-c1809069d1a1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:04.887598 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.887480    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c01e663f-14c6-424b-8b7e-c1809069d1a1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c01e663f-14c6-424b-8b7e-c1809069d1a1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:04.887598 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.887499    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c01e663f-14c6-424b-8b7e-c1809069d1a1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c01e663f-14c6-424b-8b7e-c1809069d1a1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:04.887598 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.887514    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c01e663f-14c6-424b-8b7e-c1809069d1a1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c01e663f-14c6-424b-8b7e-c1809069d1a1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:04.887598 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.887539    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c01e663f-14c6-424b-8b7e-c1809069d1a1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c01e663f-14c6-424b-8b7e-c1809069d1a1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:04.887598 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.887567    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c01e663f-14c6-424b-8b7e-c1809069d1a1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c01e663f-14c6-424b-8b7e-c1809069d1a1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:04.887598 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.887591    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c01e663f-14c6-424b-8b7e-c1809069d1a1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c01e663f-14c6-424b-8b7e-c1809069d1a1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:04.887867 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.887617    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c01e663f-14c6-424b-8b7e-c1809069d1a1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c01e663f-14c6-424b-8b7e-c1809069d1a1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:04.887867 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.887664    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6292\" (UniqueName: \"kubernetes.io/projected/c01e663f-14c6-424b-8b7e-c1809069d1a1-kube-api-access-s6292\") pod \"prometheus-k8s-0\" (UID: \"c01e663f-14c6-424b-8b7e-c1809069d1a1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:04.887867 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.887704    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c01e663f-14c6-424b-8b7e-c1809069d1a1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c01e663f-14c6-424b-8b7e-c1809069d1a1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:04.887867 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.887739    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c01e663f-14c6-424b-8b7e-c1809069d1a1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c01e663f-14c6-424b-8b7e-c1809069d1a1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:04.989157 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.989076    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c01e663f-14c6-424b-8b7e-c1809069d1a1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c01e663f-14c6-424b-8b7e-c1809069d1a1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:04.989157 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.989119    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c01e663f-14c6-424b-8b7e-c1809069d1a1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c01e663f-14c6-424b-8b7e-c1809069d1a1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:04.989157 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.989138    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c01e663f-14c6-424b-8b7e-c1809069d1a1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c01e663f-14c6-424b-8b7e-c1809069d1a1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:04.989308 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.989161    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6292\" (UniqueName: \"kubernetes.io/projected/c01e663f-14c6-424b-8b7e-c1809069d1a1-kube-api-access-s6292\") pod \"prometheus-k8s-0\" (UID: \"c01e663f-14c6-424b-8b7e-c1809069d1a1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:04.989387 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.989370    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c01e663f-14c6-424b-8b7e-c1809069d1a1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c01e663f-14c6-424b-8b7e-c1809069d1a1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:04.989438 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.989409    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c01e663f-14c6-424b-8b7e-c1809069d1a1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c01e663f-14c6-424b-8b7e-c1809069d1a1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:04.989484 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.989451    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c01e663f-14c6-424b-8b7e-c1809069d1a1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c01e663f-14c6-424b-8b7e-c1809069d1a1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:04.989484 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.989477    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c01e663f-14c6-424b-8b7e-c1809069d1a1-web-config\") pod \"prometheus-k8s-0\" (UID: \"c01e663f-14c6-424b-8b7e-c1809069d1a1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:04.989579 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.989497    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c01e663f-14c6-424b-8b7e-c1809069d1a1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c01e663f-14c6-424b-8b7e-c1809069d1a1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:04.989579 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.989518    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c01e663f-14c6-424b-8b7e-c1809069d1a1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c01e663f-14c6-424b-8b7e-c1809069d1a1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:04.989579 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.989545    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c01e663f-14c6-424b-8b7e-c1809069d1a1-config-out\") pod \"prometheus-k8s-0\" (UID: \"c01e663f-14c6-424b-8b7e-c1809069d1a1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:04.989579 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.989567    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c01e663f-14c6-424b-8b7e-c1809069d1a1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c01e663f-14c6-424b-8b7e-c1809069d1a1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:04.989808 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.989619    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c01e663f-14c6-424b-8b7e-c1809069d1a1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c01e663f-14c6-424b-8b7e-c1809069d1a1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:04.989808 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.989670    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c01e663f-14c6-424b-8b7e-c1809069d1a1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c01e663f-14c6-424b-8b7e-c1809069d1a1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:04.989808 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.989698    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c01e663f-14c6-424b-8b7e-c1809069d1a1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c01e663f-14c6-424b-8b7e-c1809069d1a1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:04.989808 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.989750    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c01e663f-14c6-424b-8b7e-c1809069d1a1-config\") pod \"prometheus-k8s-0\" (UID: \"c01e663f-14c6-424b-8b7e-c1809069d1a1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:04.989808 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.989778    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c01e663f-14c6-424b-8b7e-c1809069d1a1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c01e663f-14c6-424b-8b7e-c1809069d1a1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:04.990042 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.989811    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c01e663f-14c6-424b-8b7e-c1809069d1a1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c01e663f-14c6-424b-8b7e-c1809069d1a1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:04.990042 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.989839    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c01e663f-14c6-424b-8b7e-c1809069d1a1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c01e663f-14c6-424b-8b7e-c1809069d1a1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:04.990042 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.989863    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c01e663f-14c6-424b-8b7e-c1809069d1a1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c01e663f-14c6-424b-8b7e-c1809069d1a1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:04.991569 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.990301    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c01e663f-14c6-424b-8b7e-c1809069d1a1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c01e663f-14c6-424b-8b7e-c1809069d1a1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:04.991569 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.990373    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c01e663f-14c6-424b-8b7e-c1809069d1a1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c01e663f-14c6-424b-8b7e-c1809069d1a1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:04.991569 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.990777    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c01e663f-14c6-424b-8b7e-c1809069d1a1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c01e663f-14c6-424b-8b7e-c1809069d1a1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:04.991569 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.991112    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c01e663f-14c6-424b-8b7e-c1809069d1a1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c01e663f-14c6-424b-8b7e-c1809069d1a1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:04.991569 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.991248    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c01e663f-14c6-424b-8b7e-c1809069d1a1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c01e663f-14c6-424b-8b7e-c1809069d1a1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:04.991937 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.991597    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c01e663f-14c6-424b-8b7e-c1809069d1a1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c01e663f-14c6-424b-8b7e-c1809069d1a1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:04.992615 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.992581    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c01e663f-14c6-424b-8b7e-c1809069d1a1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c01e663f-14c6-424b-8b7e-c1809069d1a1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:04.992757 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.992685    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c01e663f-14c6-424b-8b7e-c1809069d1a1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c01e663f-14c6-424b-8b7e-c1809069d1a1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:04.993010 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.992984    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c01e663f-14c6-424b-8b7e-c1809069d1a1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c01e663f-14c6-424b-8b7e-c1809069d1a1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:04.993091 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.992993    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c01e663f-14c6-424b-8b7e-c1809069d1a1-web-config\") pod \"prometheus-k8s-0\" (UID: \"c01e663f-14c6-424b-8b7e-c1809069d1a1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:04.993180 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.993156    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c01e663f-14c6-424b-8b7e-c1809069d1a1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c01e663f-14c6-424b-8b7e-c1809069d1a1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:04.993530 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.993501    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c01e663f-14c6-424b-8b7e-c1809069d1a1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c01e663f-14c6-424b-8b7e-c1809069d1a1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:04.994544 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.994522    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c01e663f-14c6-424b-8b7e-c1809069d1a1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c01e663f-14c6-424b-8b7e-c1809069d1a1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:04.994728 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.994702    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c01e663f-14c6-424b-8b7e-c1809069d1a1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c01e663f-14c6-424b-8b7e-c1809069d1a1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:04.995030 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.994996    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c01e663f-14c6-424b-8b7e-c1809069d1a1-config-out\") pod \"prometheus-k8s-0\" (UID: \"c01e663f-14c6-424b-8b7e-c1809069d1a1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:04.995572 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.995550    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c01e663f-14c6-424b-8b7e-c1809069d1a1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c01e663f-14c6-424b-8b7e-c1809069d1a1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:04.995669 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.995616    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c01e663f-14c6-424b-8b7e-c1809069d1a1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c01e663f-14c6-424b-8b7e-c1809069d1a1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:04.995776 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.995758    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c01e663f-14c6-424b-8b7e-c1809069d1a1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c01e663f-14c6-424b-8b7e-c1809069d1a1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:04.995949 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.995934    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/c01e663f-14c6-424b-8b7e-c1809069d1a1-config\") pod \"prometheus-k8s-0\" (UID: \"c01e663f-14c6-424b-8b7e-c1809069d1a1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:04.997619 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:04.997602    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6292\" (UniqueName: \"kubernetes.io/projected/c01e663f-14c6-424b-8b7e-c1809069d1a1-kube-api-access-s6292\") pod \"prometheus-k8s-0\" (UID: \"c01e663f-14c6-424b-8b7e-c1809069d1a1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:05.102054 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:05.102009    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:41:05.249595 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:05.249443    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 08 16:41:05.253098 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:41:05.253073    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc01e663f_14c6_424b_8b7e_c1809069d1a1.slice/crio-0f667a5c38f211db8e55a04846031d3ea6c33ce83b918d2847a6da79d6809522 WatchSource:0}: Error finding container 0f667a5c38f211db8e55a04846031d3ea6c33ce83b918d2847a6da79d6809522: Status 404 returned error can't find the container with id 0f667a5c38f211db8e55a04846031d3ea6c33ce83b918d2847a6da79d6809522
Jul 08 16:41:05.736763 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:05.736729    2578 generic.go:356] "Generic (PLEG): container finished" podID="c01e663f-14c6-424b-8b7e-c1809069d1a1" containerID="e7ba498b3ff2fee9931548940ff808fb3b4e16af587b018f7d694fef506f1e05" exitCode=0
Jul 08 16:41:05.737185 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:05.736765    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c01e663f-14c6-424b-8b7e-c1809069d1a1","Type":"ContainerDied","Data":"e7ba498b3ff2fee9931548940ff808fb3b4e16af587b018f7d694fef506f1e05"}
Jul 08 16:41:05.737185 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:05.736846    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c01e663f-14c6-424b-8b7e-c1809069d1a1","Type":"ContainerStarted","Data":"0f667a5c38f211db8e55a04846031d3ea6c33ce83b918d2847a6da79d6809522"}
Jul 08 16:41:06.286006 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:06.285930    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ac190a4-147f-4895-90e9-9ecc1c0df60f" path="/var/lib/kubelet/pods/2ac190a4-147f-4895-90e9-9ecc1c0df60f/volumes"
Jul 08 16:41:06.744832 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:06.744795    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c01e663f-14c6-424b-8b7e-c1809069d1a1","Type":"ContainerStarted","Data":"7bcde517aee969cb6e8dd38875abe4e36946985e2614dfac8a1f32ec9e458ab2"}
Jul 08 16:41:06.745183 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:06.744886    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c01e663f-14c6-424b-8b7e-c1809069d1a1","Type":"ContainerStarted","Data":"a858d3f44e5940100ff09c1466b8d5e555aed1cebc7e240b6543aa8c18348ea9"}
Jul 08 16:41:06.745183 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:06.744930    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c01e663f-14c6-424b-8b7e-c1809069d1a1","Type":"ContainerStarted","Data":"2031679dc83cb2dc54e9b3c4a1a5da3c1eb529f06ed6bb93706006d87d8b83dd"}
Jul 08 16:41:06.745183 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:06.744969    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c01e663f-14c6-424b-8b7e-c1809069d1a1","Type":"ContainerStarted","Data":"88dd8e7095fb41229bb0db40d3a6cf532e09ae10fe33ddf92ca214c61d16ee11"}
Jul 08 16:41:06.745183 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:06.745007    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c01e663f-14c6-424b-8b7e-c1809069d1a1","Type":"ContainerStarted","Data":"dd5f524aa9cb58738adf2ba9ed1e9482c5bfaa5a857ad8f312bbddcbdc04e57c"}
Jul 08 16:41:06.745183 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:06.745064    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c01e663f-14c6-424b-8b7e-c1809069d1a1","Type":"ContainerStarted","Data":"9985238975e16cfe6942c15469540c348798f011036253edf6a40c820558f88a"}
Jul 08 16:41:06.774686 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:06.774609    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.7551662649999997 podStartE2EDuration="2.774587095s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.42083ms" isStatefulPod=false podCreationTimestamp="2026-07-08 16:41:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:41:06.772514259 +0000 UTC m=+379.043963801" watchObservedRunningTime="2026-07-08 16:41:06.774587095 +0000 UTC m=+379.046036637"
Jul 08 16:41:10.102907 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:41:10.102853    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:42:05.102496 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:42:05.102457    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:42:05.118392 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:42:05.118362    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:42:05.994703 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:42:05.994676    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:44:48.154308 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:44:48.154276    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-qn2bx_4717bdee-2017-4551-b0ec-683ab07f7660/console-operator/1.log"
Jul 08 16:44:48.155872 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:44:48.155846    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-qn2bx_4717bdee-2017-4551-b0ec-683ab07f7660/console-operator/1.log"
Jul 08 16:44:48.162103 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:44:48.162077    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2shvk_f08a1b00-78fd-43ef-b2aa-2712d09fb1b3/ovn-acl-logging/0.log"
Jul 08 16:44:48.163312 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:44:48.163294    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2shvk_f08a1b00-78fd-43ef-b2aa-2712d09fb1b3/ovn-acl-logging/0.log"
Jul 08 16:46:48.971980 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:46:48.971899    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-v2jqw"]
Jul 08 16:46:48.975007 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:46:48.974990    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-v2jqw"
Jul 08 16:46:48.977429 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:46:48.977403    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 08 16:46:48.977604 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:46:48.977404    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 08 16:46:48.978337 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:46:48.978319    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 08 16:46:48.978440 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:46:48.978322    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-znsfd\""
Jul 08 16:46:48.984892 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:46:48.984868    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-v2jqw"]
Jul 08 16:46:49.079061 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:46:49.079031    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zl5q\" (UniqueName: \"kubernetes.io/projected/2d7d32de-a61f-4512-a31c-e7aff61cf161-kube-api-access-8zl5q\") pod \"seaweedfs-64568bcd49-v2jqw\" (UID: \"2d7d32de-a61f-4512-a31c-e7aff61cf161\") " pod="kserve/seaweedfs-64568bcd49-v2jqw"
Jul 08 16:46:49.079222 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:46:49.079094    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2d7d32de-a61f-4512-a31c-e7aff61cf161-data\") pod \"seaweedfs-64568bcd49-v2jqw\" (UID: \"2d7d32de-a61f-4512-a31c-e7aff61cf161\") " pod="kserve/seaweedfs-64568bcd49-v2jqw"
Jul 08 16:46:49.179650 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:46:49.179604    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zl5q\" (UniqueName: \"kubernetes.io/projected/2d7d32de-a61f-4512-a31c-e7aff61cf161-kube-api-access-8zl5q\") pod \"seaweedfs-64568bcd49-v2jqw\" (UID: \"2d7d32de-a61f-4512-a31c-e7aff61cf161\") " pod="kserve/seaweedfs-64568bcd49-v2jqw"
Jul 08 16:46:49.179839 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:46:49.179705    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2d7d32de-a61f-4512-a31c-e7aff61cf161-data\") pod \"seaweedfs-64568bcd49-v2jqw\" (UID: \"2d7d32de-a61f-4512-a31c-e7aff61cf161\") " pod="kserve/seaweedfs-64568bcd49-v2jqw"
Jul 08 16:46:49.180020 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:46:49.180003    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2d7d32de-a61f-4512-a31c-e7aff61cf161-data\") pod \"seaweedfs-64568bcd49-v2jqw\" (UID: \"2d7d32de-a61f-4512-a31c-e7aff61cf161\") " pod="kserve/seaweedfs-64568bcd49-v2jqw"
Jul 08 16:46:49.191299 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:46:49.191271    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zl5q\" (UniqueName: \"kubernetes.io/projected/2d7d32de-a61f-4512-a31c-e7aff61cf161-kube-api-access-8zl5q\") pod \"seaweedfs-64568bcd49-v2jqw\" (UID: \"2d7d32de-a61f-4512-a31c-e7aff61cf161\") " pod="kserve/seaweedfs-64568bcd49-v2jqw"
Jul 08 16:46:49.286601 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:46:49.286524    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-v2jqw"
Jul 08 16:46:49.412232 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:46:49.412206    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-v2jqw"]
Jul 08 16:46:49.414838 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:46:49.414811    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d7d32de_a61f_4512_a31c_e7aff61cf161.slice/crio-437abab32de47bc8c1a0f2b2612f711abfff48c0f6521b7083f83e7778facc12 WatchSource:0}: Error finding container 437abab32de47bc8c1a0f2b2612f711abfff48c0f6521b7083f83e7778facc12: Status 404 returned error can't find the container with id 437abab32de47bc8c1a0f2b2612f711abfff48c0f6521b7083f83e7778facc12
Jul 08 16:46:49.416433 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:46:49.416419    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 08 16:46:50.011863 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:46:50.011832    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-v2jqw" event={"ID":"2d7d32de-a61f-4512-a31c-e7aff61cf161","Type":"ContainerStarted","Data":"437abab32de47bc8c1a0f2b2612f711abfff48c0f6521b7083f83e7778facc12"}
Jul 08 16:46:53.028741 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:46:53.028627    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-v2jqw" event={"ID":"2d7d32de-a61f-4512-a31c-e7aff61cf161","Type":"ContainerStarted","Data":"6a1364e88672fcb35ca23661a9afa1448860e7f71a5e741267f928ae2767ecc6"}
Jul 08 16:46:53.029092 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:46:53.028738    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-v2jqw"
Jul 08 16:46:53.046924 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:46:53.046877    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-v2jqw" podStartSLOduration=1.794232719 podStartE2EDuration="5.046859835s" totalImagesPullingTime="3.252627116s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:46:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:46:53.045576972 +0000 UTC m=+725.317026511" watchObservedRunningTime="2026-07-08 16:46:53.046859835 +0000 UTC m=+725.318309375"
Jul 08 16:46:59.035124 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:46:59.035096    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-v2jqw"
Jul 08 16:48:16.447332 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:48:16.447237    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-kpvjv"]
Jul 08 16:48:16.449517 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:48:16.449496    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-kpvjv"
Jul 08 16:48:16.459275 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:48:16.459244    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-kpvjv"]
Jul 08 16:48:16.603017 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:48:16.602974    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrct8\" (UniqueName: \"kubernetes.io/projected/91781367-864b-494a-b12d-1b1cd3f9158b-kube-api-access-hrct8\") pod \"s3-init-kpvjv\" (UID: \"91781367-864b-494a-b12d-1b1cd3f9158b\") " pod="kserve/s3-init-kpvjv"
Jul 08 16:48:16.603205 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:48:16.603051    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91781367-864b-494a-b12d-1b1cd3f9158b-model-cache\") pod \"s3-init-kpvjv\" (UID: \"91781367-864b-494a-b12d-1b1cd3f9158b\") " pod="kserve/s3-init-kpvjv"
Jul 08 16:48:16.703696 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:48:16.703579    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91781367-864b-494a-b12d-1b1cd3f9158b-model-cache\") pod \"s3-init-kpvjv\" (UID: \"91781367-864b-494a-b12d-1b1cd3f9158b\") " pod="kserve/s3-init-kpvjv"
Jul 08 16:48:16.703810 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:48:16.703701    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrct8\" (UniqueName: \"kubernetes.io/projected/91781367-864b-494a-b12d-1b1cd3f9158b-kube-api-access-hrct8\") pod \"s3-init-kpvjv\" (UID: \"91781367-864b-494a-b12d-1b1cd3f9158b\") " pod="kserve/s3-init-kpvjv"
Jul 08 16:48:16.704054 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:48:16.704030    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91781367-864b-494a-b12d-1b1cd3f9158b-model-cache\") pod \"s3-init-kpvjv\" (UID: \"91781367-864b-494a-b12d-1b1cd3f9158b\") " pod="kserve/s3-init-kpvjv"
Jul 08 16:48:16.715175 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:48:16.715145    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrct8\" (UniqueName: \"kubernetes.io/projected/91781367-864b-494a-b12d-1b1cd3f9158b-kube-api-access-hrct8\") pod \"s3-init-kpvjv\" (UID: \"91781367-864b-494a-b12d-1b1cd3f9158b\") " pod="kserve/s3-init-kpvjv"
Jul 08 16:48:16.770768 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:48:16.770739    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-kpvjv"
Jul 08 16:48:16.908682 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:48:16.908613    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-kpvjv"]
Jul 08 16:48:16.911057 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:48:16.911025    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91781367_864b_494a_b12d_1b1cd3f9158b.slice/crio-7ce65fc5ca48ff7f2354e4d2700d0fd8eda482caeced08c3faf3e876fa5f2fb6 WatchSource:0}: Error finding container 7ce65fc5ca48ff7f2354e4d2700d0fd8eda482caeced08c3faf3e876fa5f2fb6: Status 404 returned error can't find the container with id 7ce65fc5ca48ff7f2354e4d2700d0fd8eda482caeced08c3faf3e876fa5f2fb6
Jul 08 16:48:17.348546 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:48:17.348508    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-kpvjv" event={"ID":"91781367-864b-494a-b12d-1b1cd3f9158b","Type":"ContainerStarted","Data":"7ce65fc5ca48ff7f2354e4d2700d0fd8eda482caeced08c3faf3e876fa5f2fb6"}
Jul 08 16:48:21.366885 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:48:21.366843    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-kpvjv" event={"ID":"91781367-864b-494a-b12d-1b1cd3f9158b","Type":"ContainerStarted","Data":"897290ea0a1833fa319c4678d7c86707e9c109424164b104268bcd590ae419d2"}
Jul 08 16:48:23.376840 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:48:23.376744    2578 generic.go:356] "Generic (PLEG): container finished" podID="91781367-864b-494a-b12d-1b1cd3f9158b" containerID="897290ea0a1833fa319c4678d7c86707e9c109424164b104268bcd590ae419d2" exitCode=0
Jul 08 16:48:23.376840 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:48:23.376825    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-kpvjv" event={"ID":"91781367-864b-494a-b12d-1b1cd3f9158b","Type":"ContainerDied","Data":"897290ea0a1833fa319c4678d7c86707e9c109424164b104268bcd590ae419d2"}
Jul 08 16:48:28.401418 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:48:28.401381    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-kpvjv" event={"ID":"91781367-864b-494a-b12d-1b1cd3f9158b","Type":"ContainerStarted","Data":"d3f7a59522a9cfd208adacfd89ffc86e2cd2126d89a41ee4f7249cc151554a2e"}
Jul 08 16:48:28.420475 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:48:28.420406    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-kpvjv" podStartSLOduration=1.50818542 podStartE2EDuration="12.420384343s" totalImagesPullingTime="8.517339225s" totalInitContainerRuntime="2.394859698s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:48:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:48:28.41823245 +0000 UTC m=+820.689681992" watchObservedRunningTime="2026-07-08 16:48:28.420384343 +0000 UTC m=+820.691833885"
Jul 08 16:49:33.654273 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:49:33.654190    2578 generic.go:356] "Generic (PLEG): container finished" podID="91781367-864b-494a-b12d-1b1cd3f9158b" containerID="d3f7a59522a9cfd208adacfd89ffc86e2cd2126d89a41ee4f7249cc151554a2e" exitCode=0
Jul 08 16:49:33.654273 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:49:33.654236    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-kpvjv" event={"ID":"91781367-864b-494a-b12d-1b1cd3f9158b","Type":"ContainerDied","Data":"d3f7a59522a9cfd208adacfd89ffc86e2cd2126d89a41ee4f7249cc151554a2e"}
Jul 08 16:49:34.790951 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:49:34.790924    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-kpvjv"
Jul 08 16:49:34.877980 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:49:34.877945    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91781367-864b-494a-b12d-1b1cd3f9158b-model-cache\") pod \"91781367-864b-494a-b12d-1b1cd3f9158b\" (UID: \"91781367-864b-494a-b12d-1b1cd3f9158b\") "
Jul 08 16:49:34.878177 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:49:34.878022    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrct8\" (UniqueName: \"kubernetes.io/projected/91781367-864b-494a-b12d-1b1cd3f9158b-kube-api-access-hrct8\") pod \"91781367-864b-494a-b12d-1b1cd3f9158b\" (UID: \"91781367-864b-494a-b12d-1b1cd3f9158b\") "
Jul 08 16:49:34.880521 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:49:34.880492    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91781367-864b-494a-b12d-1b1cd3f9158b-kube-api-access-hrct8" (OuterVolumeSpecName: "kube-api-access-hrct8") pod "91781367-864b-494a-b12d-1b1cd3f9158b" (UID: "91781367-864b-494a-b12d-1b1cd3f9158b"). InnerVolumeSpecName "kube-api-access-hrct8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:49:34.896449 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:49:34.896419    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91781367-864b-494a-b12d-1b1cd3f9158b-model-cache" (OuterVolumeSpecName: "model-cache") pod "91781367-864b-494a-b12d-1b1cd3f9158b" (UID: "91781367-864b-494a-b12d-1b1cd3f9158b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:49:34.979356 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:49:34.979274    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91781367-864b-494a-b12d-1b1cd3f9158b-model-cache\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:49:34.979356 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:49:34.979301    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrct8\" (UniqueName: \"kubernetes.io/projected/91781367-864b-494a-b12d-1b1cd3f9158b-kube-api-access-hrct8\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:49:35.663520 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:49:35.663484    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-kpvjv"
Jul 08 16:49:35.663520 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:49:35.663497    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-kpvjv" event={"ID":"91781367-864b-494a-b12d-1b1cd3f9158b","Type":"ContainerDied","Data":"7ce65fc5ca48ff7f2354e4d2700d0fd8eda482caeced08c3faf3e876fa5f2fb6"}
Jul 08 16:49:35.663757 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:49:35.663532    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ce65fc5ca48ff7f2354e4d2700d0fd8eda482caeced08c3faf3e876fa5f2fb6"
Jul 08 16:49:47.666832 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:49:47.666749    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46"]
Jul 08 16:49:47.667258 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:49:47.667059    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91781367-864b-494a-b12d-1b1cd3f9158b" containerName="s3-init"
Jul 08 16:49:47.667258 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:49:47.667070    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="91781367-864b-494a-b12d-1b1cd3f9158b" containerName="s3-init"
Jul 08 16:49:47.667258 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:49:47.667087    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91781367-864b-494a-b12d-1b1cd3f9158b" containerName="download-hf-model"
Jul 08 16:49:47.667258 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:49:47.667092    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="91781367-864b-494a-b12d-1b1cd3f9158b" containerName="download-hf-model"
Jul 08 16:49:47.667258 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:49:47.667152    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="91781367-864b-494a-b12d-1b1cd3f9158b" containerName="s3-init"
Jul 08 16:49:47.670610 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:49:47.670591    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46"
Jul 08 16:49:47.673043 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:49:47.673015    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-03e1d-kube-rbac-proxy-sar-config\""
Jul 08 16:49:47.673185 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:49:47.673120    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 08 16:49:47.673185 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:49:47.673130    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-03e1d-predictor-serving-cert\""
Jul 08 16:49:47.673185 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:49:47.673125    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 08 16:49:47.673898 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:49:47.673882    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 08 16:49:47.673980 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:49:47.673954    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-vkqpp\""
Jul 08 16:49:47.674126 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:49:47.674110    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 08 16:49:47.680176 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:49:47.680151    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46"]
Jul 08 16:49:47.791956 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:49:47.791917    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d9691d9-c2fa-42de-8f3b-4851317a6bac-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46\" (UID: \"2d9691d9-c2fa-42de-8f3b-4851317a6bac\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46"
Jul 08 16:49:47.792139 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:49:47.791966    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d9691d9-c2fa-42de-8f3b-4851317a6bac-proxy-tls\") pod \"isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46\" (UID: \"2d9691d9-c2fa-42de-8f3b-4851317a6bac\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46"
Jul 08 16:49:47.792139 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:49:47.792023    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d9691d9-c2fa-42de-8f3b-4851317a6bac-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46\" (UID: \"2d9691d9-c2fa-42de-8f3b-4851317a6bac\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46"
Jul 08 16:49:47.792139 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:49:47.792062    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d9691d9-c2fa-42de-8f3b-4851317a6bac-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46\" (UID: \"2d9691d9-c2fa-42de-8f3b-4851317a6bac\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46"
Jul 08 16:49:47.792139 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:49:47.792120    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jbf7\" (UniqueName: \"kubernetes.io/projected/2d9691d9-c2fa-42de-8f3b-4851317a6bac-kube-api-access-7jbf7\") pod \"isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46\" (UID: \"2d9691d9-c2fa-42de-8f3b-4851317a6bac\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46"
Jul 08 16:49:47.893262 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:49:47.893218    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d9691d9-c2fa-42de-8f3b-4851317a6bac-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46\" (UID: \"2d9691d9-c2fa-42de-8f3b-4851317a6bac\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46"
Jul 08 16:49:47.893463 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:49:47.893268    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d9691d9-c2fa-42de-8f3b-4851317a6bac-proxy-tls\") pod \"isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46\" (UID: \"2d9691d9-c2fa-42de-8f3b-4851317a6bac\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46"
Jul 08 16:49:47.893463 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:49:47.893318    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d9691d9-c2fa-42de-8f3b-4851317a6bac-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46\" (UID: \"2d9691d9-c2fa-42de-8f3b-4851317a6bac\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46"
Jul 08 16:49:47.893463 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:49:47.893358    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d9691d9-c2fa-42de-8f3b-4851317a6bac-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46\" (UID: \"2d9691d9-c2fa-42de-8f3b-4851317a6bac\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46"
Jul 08 16:49:47.893463 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:49:47.893413    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jbf7\" (UniqueName: \"kubernetes.io/projected/2d9691d9-c2fa-42de-8f3b-4851317a6bac-kube-api-access-7jbf7\") pod \"isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46\" (UID: \"2d9691d9-c2fa-42de-8f3b-4851317a6bac\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46"
Jul 08 16:49:47.893708 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:49:47.893689    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d9691d9-c2fa-42de-8f3b-4851317a6bac-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46\" (UID: \"2d9691d9-c2fa-42de-8f3b-4851317a6bac\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46"
Jul 08 16:49:47.894176 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:49:47.894151    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d9691d9-c2fa-42de-8f3b-4851317a6bac-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46\" (UID: \"2d9691d9-c2fa-42de-8f3b-4851317a6bac\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46"
Jul 08 16:49:47.894235 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:49:47.894151    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d9691d9-c2fa-42de-8f3b-4851317a6bac-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46\" (UID: \"2d9691d9-c2fa-42de-8f3b-4851317a6bac\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46"
Jul 08 16:49:47.895913 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:49:47.895897    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d9691d9-c2fa-42de-8f3b-4851317a6bac-proxy-tls\") pod \"isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46\" (UID: \"2d9691d9-c2fa-42de-8f3b-4851317a6bac\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46"
Jul 08 16:49:47.902964 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:49:47.902936    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jbf7\" (UniqueName: \"kubernetes.io/projected/2d9691d9-c2fa-42de-8f3b-4851317a6bac-kube-api-access-7jbf7\") pod \"isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46\" (UID: \"2d9691d9-c2fa-42de-8f3b-4851317a6bac\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46"
Jul 08 16:49:47.982025 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:49:47.981932    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46"
Jul 08 16:49:48.124415 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:49:48.124390    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46"]
Jul 08 16:49:48.127399 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:49:48.127371    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d9691d9_c2fa_42de_8f3b_4851317a6bac.slice/crio-37632a289db878eefea1ca8c962c94c7c7674abdc5dfb722280e1bee1e9b474c WatchSource:0}: Error finding container 37632a289db878eefea1ca8c962c94c7c7674abdc5dfb722280e1bee1e9b474c: Status 404 returned error can't find the container with id 37632a289db878eefea1ca8c962c94c7c7674abdc5dfb722280e1bee1e9b474c
Jul 08 16:49:48.131758 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:49:48.131736    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 08 16:49:48.190903 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:49:48.190876    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-qn2bx_4717bdee-2017-4551-b0ec-683ab07f7660/console-operator/1.log"
Jul 08 16:49:48.192204 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:49:48.192181    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-qn2bx_4717bdee-2017-4551-b0ec-683ab07f7660/console-operator/1.log"
Jul 08 16:49:48.196300 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:49:48.196276    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2shvk_f08a1b00-78fd-43ef-b2aa-2712d09fb1b3/ovn-acl-logging/0.log"
Jul 08 16:49:48.196930 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:49:48.196910    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2shvk_f08a1b00-78fd-43ef-b2aa-2712d09fb1b3/ovn-acl-logging/0.log"
Jul 08 16:49:48.715226 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:49:48.715189    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46" event={"ID":"2d9691d9-c2fa-42de-8f3b-4851317a6bac","Type":"ContainerStarted","Data":"78a7f2500636d47195b5c7578802842c7339afeb56c9d09ae9c3f66c0e393636"}
Jul 08 16:49:48.715620 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:49:48.715267    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46" event={"ID":"2d9691d9-c2fa-42de-8f3b-4851317a6bac","Type":"ContainerStarted","Data":"37632a289db878eefea1ca8c962c94c7c7674abdc5dfb722280e1bee1e9b474c"}
Jul 08 16:49:52.732456 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:49:52.732424    2578 generic.go:356] "Generic (PLEG): container finished" podID="2d9691d9-c2fa-42de-8f3b-4851317a6bac" containerID="78a7f2500636d47195b5c7578802842c7339afeb56c9d09ae9c3f66c0e393636" exitCode=0
Jul 08 16:49:52.732873 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:49:52.732506    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46" event={"ID":"2d9691d9-c2fa-42de-8f3b-4851317a6bac","Type":"ContainerDied","Data":"78a7f2500636d47195b5c7578802842c7339afeb56c9d09ae9c3f66c0e393636"}
Jul 08 16:50:07.813367 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:50:07.813321    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46" event={"ID":"2d9691d9-c2fa-42de-8f3b-4851317a6bac","Type":"ContainerStarted","Data":"2a9ca09f13f46e0baa6db0de851e6569bf2473fc8dea34ede3f9dc92e6a1c9ec"}
Jul 08 16:50:10.828808 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:50:10.828773    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46" event={"ID":"2d9691d9-c2fa-42de-8f3b-4851317a6bac","Type":"ContainerStarted","Data":"36c623d02147f7bc28e08bfae7dd17f250ecca50cb9cfffca660c2ac98de730e"}
Jul 08 16:50:13.844021 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:50:13.843985    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46" event={"ID":"2d9691d9-c2fa-42de-8f3b-4851317a6bac","Type":"ContainerStarted","Data":"6dcceacab1ef12e163420f5b5e44f324ddb0d3b3b518cffe5944891ea7124f80"}
Jul 08 16:50:13.844467 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:50:13.844230    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46"
Jul 08 16:50:13.844467 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:50:13.844295    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46"
Jul 08 16:50:13.844467 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:50:13.844332    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46"
Jul 08 16:50:13.845952 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:50:13.845908    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46" podUID="2d9691d9-c2fa-42de-8f3b-4851317a6bac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 08 16:50:13.846556 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:50:13.846528    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46" podUID="2d9691d9-c2fa-42de-8f3b-4851317a6bac" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:50:13.866086 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:50:13.866038    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46" podStartSLOduration=2.205913916 podStartE2EDuration="26.866023252s" totalImagesPullingTime="20.830222276s" totalInitContainerRuntime="3.82988706s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:49:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:50:13.864575824 +0000 UTC m=+926.136025370" watchObservedRunningTime="2026-07-08 16:50:13.866023252 +0000 UTC m=+926.137472793"
Jul 08 16:50:14.848096 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:50:14.848045    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46" podUID="2d9691d9-c2fa-42de-8f3b-4851317a6bac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 08 16:50:14.848555 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:50:14.848521    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46" podUID="2d9691d9-c2fa-42de-8f3b-4851317a6bac" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:50:19.853590 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:50:19.853559    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46"
Jul 08 16:50:19.854211 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:50:19.854183    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46" podUID="2d9691d9-c2fa-42de-8f3b-4851317a6bac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 08 16:50:19.854547 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:50:19.854523    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46" podUID="2d9691d9-c2fa-42de-8f3b-4851317a6bac" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:50:29.854943 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:50:29.854896    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46" podUID="2d9691d9-c2fa-42de-8f3b-4851317a6bac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 08 16:50:29.855482 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:50:29.855393    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46" podUID="2d9691d9-c2fa-42de-8f3b-4851317a6bac" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:50:39.854526 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:50:39.854480    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46" podUID="2d9691d9-c2fa-42de-8f3b-4851317a6bac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 08 16:50:39.855063 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:50:39.855035    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46" podUID="2d9691d9-c2fa-42de-8f3b-4851317a6bac" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:50:49.854125 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:50:49.854076    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46" podUID="2d9691d9-c2fa-42de-8f3b-4851317a6bac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 08 16:50:49.854678 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:50:49.854533    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46" podUID="2d9691d9-c2fa-42de-8f3b-4851317a6bac" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:50:59.854784 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:50:59.854732    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46" podUID="2d9691d9-c2fa-42de-8f3b-4851317a6bac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 08 16:50:59.855301 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:50:59.855258    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46" podUID="2d9691d9-c2fa-42de-8f3b-4851317a6bac" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:51:09.854302 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:09.854207    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46" podUID="2d9691d9-c2fa-42de-8f3b-4851317a6bac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 08 16:51:09.854778 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:09.854752    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46" podUID="2d9691d9-c2fa-42de-8f3b-4851317a6bac" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:51:19.854593 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:19.854550    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46" podUID="2d9691d9-c2fa-42de-8f3b-4851317a6bac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 08 16:51:19.855158 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:19.855105    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46" podUID="2d9691d9-c2fa-42de-8f3b-4851317a6bac" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:51:29.855222 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:29.855189    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46"
Jul 08 16:51:29.855852 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:29.855364    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46"
Jul 08 16:51:42.711926 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:42.711886    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46"]
Jul 08 16:51:42.712405 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:42.712166    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46" podUID="2d9691d9-c2fa-42de-8f3b-4851317a6bac" containerName="kserve-container" containerID="cri-o://2a9ca09f13f46e0baa6db0de851e6569bf2473fc8dea34ede3f9dc92e6a1c9ec" gracePeriod=30
Jul 08 16:51:42.712405 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:42.712206    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46" podUID="2d9691d9-c2fa-42de-8f3b-4851317a6bac" containerName="agent" containerID="cri-o://6dcceacab1ef12e163420f5b5e44f324ddb0d3b3b518cffe5944891ea7124f80" gracePeriod=30
Jul 08 16:51:42.712405 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:42.712269    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46" podUID="2d9691d9-c2fa-42de-8f3b-4851317a6bac" containerName="kube-rbac-proxy" containerID="cri-o://36c623d02147f7bc28e08bfae7dd17f250ecca50cb9cfffca660c2ac98de730e" gracePeriod=30
Jul 08 16:51:42.885250 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:42.885204    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs"]
Jul 08 16:51:42.888035 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:42.888011    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs"
Jul 08 16:51:42.891503 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:42.891237    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-5d281-kube-rbac-proxy-sar-config\""
Jul 08 16:51:42.891503 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:42.891427    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-5d281-predictor-serving-cert\""
Jul 08 16:51:42.899965 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:42.898890    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs"]
Jul 08 16:51:42.955697 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:42.955655    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz"]
Jul 08 16:51:42.958114 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:42.958088    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz"
Jul 08 16:51:42.960594 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:42.960570    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-5d281-predictor-serving-cert\""
Jul 08 16:51:42.960756 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:42.960601    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-5d281-kube-rbac-proxy-sar-config\""
Jul 08 16:51:42.972138 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:42.972108    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz"]
Jul 08 16:51:43.001151 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:43.001105    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa6f46ad-52fc-46d4-a305-394892cb18a1-proxy-tls\") pod \"isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs\" (UID: \"aa6f46ad-52fc-46d4-a305-394892cb18a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs"
Jul 08 16:51:43.001151 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:43.001154    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/435847a0-f882-4d91-9d21-e96f272fe5fb-cabundle-cert\") pod \"isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz\" (UID: \"435847a0-f882-4d91-9d21-e96f272fe5fb\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz"
Jul 08 16:51:43.001423 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:43.001208    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtb6v\" (UniqueName: \"kubernetes.io/projected/aa6f46ad-52fc-46d4-a305-394892cb18a1-kube-api-access-xtb6v\") pod \"isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs\" (UID: \"aa6f46ad-52fc-46d4-a305-394892cb18a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs"
Jul 08 16:51:43.001423 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:43.001245    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aa6f46ad-52fc-46d4-a305-394892cb18a1-cabundle-cert\") pod \"isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs\" (UID: \"aa6f46ad-52fc-46d4-a305-394892cb18a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs"
Jul 08 16:51:43.001423 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:43.001297    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa6f46ad-52fc-46d4-a305-394892cb18a1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs\" (UID: \"aa6f46ad-52fc-46d4-a305-394892cb18a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs"
Jul 08 16:51:43.001423 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:43.001340    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa6f46ad-52fc-46d4-a305-394892cb18a1-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs\" (UID: \"aa6f46ad-52fc-46d4-a305-394892cb18a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs"
Jul 08 16:51:43.001423 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:43.001359    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx5rx\" (UniqueName: \"kubernetes.io/projected/435847a0-f882-4d91-9d21-e96f272fe5fb-kube-api-access-kx5rx\") pod \"isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz\" (UID: \"435847a0-f882-4d91-9d21-e96f272fe5fb\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz"
Jul 08 16:51:43.001423 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:43.001409    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/435847a0-f882-4d91-9d21-e96f272fe5fb-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz\" (UID: \"435847a0-f882-4d91-9d21-e96f272fe5fb\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz"
Jul 08 16:51:43.001749 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:43.001438    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/435847a0-f882-4d91-9d21-e96f272fe5fb-proxy-tls\") pod \"isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz\" (UID: \"435847a0-f882-4d91-9d21-e96f272fe5fb\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz"
Jul 08 16:51:43.001749 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:43.001464    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/435847a0-f882-4d91-9d21-e96f272fe5fb-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz\" (UID: \"435847a0-f882-4d91-9d21-e96f272fe5fb\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz"
Jul 08 16:51:43.103010 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:43.102959    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/435847a0-f882-4d91-9d21-e96f272fe5fb-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz\" (UID: \"435847a0-f882-4d91-9d21-e96f272fe5fb\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz"
Jul 08 16:51:43.103010 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:43.103014    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/435847a0-f882-4d91-9d21-e96f272fe5fb-proxy-tls\") pod \"isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz\" (UID: \"435847a0-f882-4d91-9d21-e96f272fe5fb\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz"
Jul 08 16:51:43.103267 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:51:43.103129    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-5d281-predictor-serving-cert: secret "isvc-xgboost-graph-raw-5d281-predictor-serving-cert" not found
Jul 08 16:51:43.103267 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:43.103150    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/435847a0-f882-4d91-9d21-e96f272fe5fb-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz\" (UID: \"435847a0-f882-4d91-9d21-e96f272fe5fb\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz"
Jul 08 16:51:43.103267 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:51:43.103197    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/435847a0-f882-4d91-9d21-e96f272fe5fb-proxy-tls podName:435847a0-f882-4d91-9d21-e96f272fe5fb nodeName:}" failed. No retries permitted until 2026-07-08 16:51:43.603175803 +0000 UTC m=+1015.874625325 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/435847a0-f882-4d91-9d21-e96f272fe5fb-proxy-tls") pod "isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz" (UID: "435847a0-f882-4d91-9d21-e96f272fe5fb") : secret "isvc-xgboost-graph-raw-5d281-predictor-serving-cert" not found
Jul 08 16:51:43.103427 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:43.103276    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa6f46ad-52fc-46d4-a305-394892cb18a1-proxy-tls\") pod \"isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs\" (UID: \"aa6f46ad-52fc-46d4-a305-394892cb18a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs"
Jul 08 16:51:43.103427 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:43.103309    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/435847a0-f882-4d91-9d21-e96f272fe5fb-cabundle-cert\") pod \"isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz\" (UID: \"435847a0-f882-4d91-9d21-e96f272fe5fb\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz"
Jul 08 16:51:43.103427 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:43.103340    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtb6v\" (UniqueName: \"kubernetes.io/projected/aa6f46ad-52fc-46d4-a305-394892cb18a1-kube-api-access-xtb6v\") pod \"isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs\" (UID: \"aa6f46ad-52fc-46d4-a305-394892cb18a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs"
Jul 08 16:51:43.103427 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:43.103387    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aa6f46ad-52fc-46d4-a305-394892cb18a1-cabundle-cert\") pod \"isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs\" (UID: \"aa6f46ad-52fc-46d4-a305-394892cb18a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs"
Jul 08 16:51:43.103427 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:43.103421    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa6f46ad-52fc-46d4-a305-394892cb18a1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs\" (UID: \"aa6f46ad-52fc-46d4-a305-394892cb18a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs"
Jul 08 16:51:43.103692 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:51:43.103448    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-5d281-predictor-serving-cert: secret "isvc-sklearn-graph-raw-5d281-predictor-serving-cert" not found
Jul 08 16:51:43.103692 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:43.103459    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa6f46ad-52fc-46d4-a305-394892cb18a1-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs\" (UID: \"aa6f46ad-52fc-46d4-a305-394892cb18a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs"
Jul 08 16:51:43.103692 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:43.103491    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx5rx\" (UniqueName: \"kubernetes.io/projected/435847a0-f882-4d91-9d21-e96f272fe5fb-kube-api-access-kx5rx\") pod \"isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz\" (UID: \"435847a0-f882-4d91-9d21-e96f272fe5fb\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz"
Jul 08 16:51:43.103692 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:51:43.103505    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa6f46ad-52fc-46d4-a305-394892cb18a1-proxy-tls podName:aa6f46ad-52fc-46d4-a305-394892cb18a1 nodeName:}" failed. No retries permitted until 2026-07-08 16:51:43.603489558 +0000 UTC m=+1015.874939110 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/aa6f46ad-52fc-46d4-a305-394892cb18a1-proxy-tls") pod "isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs" (UID: "aa6f46ad-52fc-46d4-a305-394892cb18a1") : secret "isvc-sklearn-graph-raw-5d281-predictor-serving-cert" not found
Jul 08 16:51:43.103692 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:43.103600    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/435847a0-f882-4d91-9d21-e96f272fe5fb-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz\" (UID: \"435847a0-f882-4d91-9d21-e96f272fe5fb\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz"
Jul 08 16:51:43.104040 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:43.104014    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/435847a0-f882-4d91-9d21-e96f272fe5fb-cabundle-cert\") pod \"isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz\" (UID: \"435847a0-f882-4d91-9d21-e96f272fe5fb\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz"
Jul 08 16:51:43.104107 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:43.104040    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa6f46ad-52fc-46d4-a305-394892cb18a1-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs\" (UID: \"aa6f46ad-52fc-46d4-a305-394892cb18a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs"
Jul 08 16:51:43.104276 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:43.104254    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa6f46ad-52fc-46d4-a305-394892cb18a1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs\" (UID: \"aa6f46ad-52fc-46d4-a305-394892cb18a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs"
Jul 08 16:51:43.104386 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:43.104370    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/435847a0-f882-4d91-9d21-e96f272fe5fb-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz\" (UID: \"435847a0-f882-4d91-9d21-e96f272fe5fb\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz"
Jul 08 16:51:43.104497 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:43.104480    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aa6f46ad-52fc-46d4-a305-394892cb18a1-cabundle-cert\") pod \"isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs\" (UID: \"aa6f46ad-52fc-46d4-a305-394892cb18a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs"
Jul 08 16:51:43.111568 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:43.111534    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtb6v\" (UniqueName: \"kubernetes.io/projected/aa6f46ad-52fc-46d4-a305-394892cb18a1-kube-api-access-xtb6v\") pod \"isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs\" (UID: \"aa6f46ad-52fc-46d4-a305-394892cb18a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs"
Jul 08 16:51:43.111936 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:43.111915    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx5rx\" (UniqueName: \"kubernetes.io/projected/435847a0-f882-4d91-9d21-e96f272fe5fb-kube-api-access-kx5rx\") pod \"isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz\" (UID: \"435847a0-f882-4d91-9d21-e96f272fe5fb\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz"
Jul 08 16:51:43.196511 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:43.196470    2578 generic.go:356] "Generic (PLEG): container finished" podID="2d9691d9-c2fa-42de-8f3b-4851317a6bac" containerID="36c623d02147f7bc28e08bfae7dd17f250ecca50cb9cfffca660c2ac98de730e" exitCode=2
Jul 08 16:51:43.196511 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:43.196512    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46" event={"ID":"2d9691d9-c2fa-42de-8f3b-4851317a6bac","Type":"ContainerDied","Data":"36c623d02147f7bc28e08bfae7dd17f250ecca50cb9cfffca660c2ac98de730e"}
Jul 08 16:51:43.608095 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:43.608054    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/435847a0-f882-4d91-9d21-e96f272fe5fb-proxy-tls\") pod \"isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz\" (UID: \"435847a0-f882-4d91-9d21-e96f272fe5fb\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz"
Jul 08 16:51:43.608273 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:43.608132    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa6f46ad-52fc-46d4-a305-394892cb18a1-proxy-tls\") pod \"isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs\" (UID: \"aa6f46ad-52fc-46d4-a305-394892cb18a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs"
Jul 08 16:51:43.610781 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:43.610744    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa6f46ad-52fc-46d4-a305-394892cb18a1-proxy-tls\") pod \"isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs\" (UID: \"aa6f46ad-52fc-46d4-a305-394892cb18a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs"
Jul 08 16:51:43.610917 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:43.610804    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/435847a0-f882-4d91-9d21-e96f272fe5fb-proxy-tls\") pod \"isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz\" (UID: \"435847a0-f882-4d91-9d21-e96f272fe5fb\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz"
Jul 08 16:51:43.804233 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:43.804193    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs"
Jul 08 16:51:43.872456 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:43.872363    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz"
Jul 08 16:51:43.953951 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:43.953919    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs"]
Jul 08 16:51:43.956446 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:51:43.956413    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa6f46ad_52fc_46d4_a305_394892cb18a1.slice/crio-06f0b0a1c8939d3a1f82e00f08fcffa374054fc54f4a75ba4c608b78e2532627 WatchSource:0}: Error finding container 06f0b0a1c8939d3a1f82e00f08fcffa374054fc54f4a75ba4c608b78e2532627: Status 404 returned error can't find the container with id 06f0b0a1c8939d3a1f82e00f08fcffa374054fc54f4a75ba4c608b78e2532627
Jul 08 16:51:44.031990 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:44.031944    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz"]
Jul 08 16:51:44.034571 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:51:44.034544    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod435847a0_f882_4d91_9d21_e96f272fe5fb.slice/crio-6ceafe65554336469af33929aca1dcbd0f106f72e9519fa5761aa727e24552b6 WatchSource:0}: Error finding container 6ceafe65554336469af33929aca1dcbd0f106f72e9519fa5761aa727e24552b6: Status 404 returned error can't find the container with id 6ceafe65554336469af33929aca1dcbd0f106f72e9519fa5761aa727e24552b6
Jul 08 16:51:44.204854 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:44.204759    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz" event={"ID":"435847a0-f882-4d91-9d21-e96f272fe5fb","Type":"ContainerStarted","Data":"2d939a10878fa117088c0f30349dd6b23e3f40841d1f230be3872533ef993ba1"}
Jul 08 16:51:44.204854 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:44.204837    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz" event={"ID":"435847a0-f882-4d91-9d21-e96f272fe5fb","Type":"ContainerStarted","Data":"6ceafe65554336469af33929aca1dcbd0f106f72e9519fa5761aa727e24552b6"}
Jul 08 16:51:44.206601 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:44.206570    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs" event={"ID":"aa6f46ad-52fc-46d4-a305-394892cb18a1","Type":"ContainerStarted","Data":"cc33fed9c8297fe392d4e5dd748fce31254c13352c39e9fa6cb7e106acc854ce"}
Jul 08 16:51:44.206758 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:44.206656    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs" event={"ID":"aa6f46ad-52fc-46d4-a305-394892cb18a1","Type":"ContainerStarted","Data":"06f0b0a1c8939d3a1f82e00f08fcffa374054fc54f4a75ba4c608b78e2532627"}
Jul 08 16:51:44.848893 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:44.848849    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46" podUID="2d9691d9-c2fa-42de-8f3b-4851317a6bac" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jul 08 16:51:48.224771 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:48.224731    2578 generic.go:356] "Generic (PLEG): container finished" podID="2d9691d9-c2fa-42de-8f3b-4851317a6bac" containerID="2a9ca09f13f46e0baa6db0de851e6569bf2473fc8dea34ede3f9dc92e6a1c9ec" exitCode=0
Jul 08 16:51:48.225195 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:48.224800    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46" event={"ID":"2d9691d9-c2fa-42de-8f3b-4851317a6bac","Type":"ContainerDied","Data":"2a9ca09f13f46e0baa6db0de851e6569bf2473fc8dea34ede3f9dc92e6a1c9ec"}
Jul 08 16:51:48.226288 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:48.226266    2578 generic.go:356] "Generic (PLEG): container finished" podID="435847a0-f882-4d91-9d21-e96f272fe5fb" containerID="2d939a10878fa117088c0f30349dd6b23e3f40841d1f230be3872533ef993ba1" exitCode=0
Jul 08 16:51:48.226408 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:48.226326    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz" event={"ID":"435847a0-f882-4d91-9d21-e96f272fe5fb","Type":"ContainerDied","Data":"2d939a10878fa117088c0f30349dd6b23e3f40841d1f230be3872533ef993ba1"}
Jul 08 16:51:48.227912 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:48.227892    2578 generic.go:356] "Generic (PLEG): container finished" podID="aa6f46ad-52fc-46d4-a305-394892cb18a1" containerID="cc33fed9c8297fe392d4e5dd748fce31254c13352c39e9fa6cb7e106acc854ce" exitCode=0
Jul 08 16:51:48.227980 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:48.227960    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs" event={"ID":"aa6f46ad-52fc-46d4-a305-394892cb18a1","Type":"ContainerDied","Data":"cc33fed9c8297fe392d4e5dd748fce31254c13352c39e9fa6cb7e106acc854ce"}
Jul 08 16:51:49.237551 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:49.237506    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs" event={"ID":"aa6f46ad-52fc-46d4-a305-394892cb18a1","Type":"ContainerStarted","Data":"a9d45f76e70be2592eb9715c7486111b0ae65c1488efa0a76c481910dd8d40f2"}
Jul 08 16:51:49.238035 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:49.237602    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs" event={"ID":"aa6f46ad-52fc-46d4-a305-394892cb18a1","Type":"ContainerStarted","Data":"e7c899cccd620c6184bdfe7e4a67cb6a663818a7ee5a3118aa5e233f89ae21fe"}
Jul 08 16:51:49.238035 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:49.237930    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs"
Jul 08 16:51:49.238162 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:49.238105    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs"
Jul 08 16:51:49.239721 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:49.239687    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs" podUID="aa6f46ad-52fc-46d4-a305-394892cb18a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 08 16:51:49.257647 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:49.257527    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs" podStartSLOduration=3.313367147 podStartE2EDuration="7.257505362s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.944138215s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:51:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:51:49.255362094 +0000 UTC m=+1021.526811636" watchObservedRunningTime="2026-07-08 16:51:49.257505362 +0000 UTC m=+1021.528954904"
Jul 08 16:51:49.848510 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:49.848464    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46" podUID="2d9691d9-c2fa-42de-8f3b-4851317a6bac" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jul 08 16:51:49.855070 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:49.854959    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46" podUID="2d9691d9-c2fa-42de-8f3b-4851317a6bac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 08 16:51:49.855338 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:49.855297    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46" podUID="2d9691d9-c2fa-42de-8f3b-4851317a6bac" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:51:50.244405 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:50.244293    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs" podUID="aa6f46ad-52fc-46d4-a305-394892cb18a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 08 16:51:54.848626 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:54.848571    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46" podUID="2d9691d9-c2fa-42de-8f3b-4851317a6bac" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jul 08 16:51:54.849120 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:54.848803    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46"
Jul 08 16:51:55.250127 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:55.250043    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs"
Jul 08 16:51:55.250879 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:55.250835    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs" podUID="aa6f46ad-52fc-46d4-a305-394892cb18a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 08 16:51:59.848477 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:59.848423    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46" podUID="2d9691d9-c2fa-42de-8f3b-4851317a6bac" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jul 08 16:51:59.855140 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:59.855078    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46" podUID="2d9691d9-c2fa-42de-8f3b-4851317a6bac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 08 16:51:59.855526 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:51:59.855491    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46" podUID="2d9691d9-c2fa-42de-8f3b-4851317a6bac" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:52:04.848404 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:52:04.848346    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46" podUID="2d9691d9-c2fa-42de-8f3b-4851317a6bac" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jul 08 16:52:05.250943 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:52:05.250833    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs" podUID="aa6f46ad-52fc-46d4-a305-394892cb18a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 08 16:52:07.325837 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:52:07.325806    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz" event={"ID":"435847a0-f882-4d91-9d21-e96f272fe5fb","Type":"ContainerStarted","Data":"6317ac1c278094a68083b352d17264a56ceadc0a26d9ef42a46cd4b61987ce81"}
Jul 08 16:52:08.332006 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:52:08.331974    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz" event={"ID":"435847a0-f882-4d91-9d21-e96f272fe5fb","Type":"ContainerStarted","Data":"cb8b42f3bc08a45b2b6ef5d2d5920d1d350af3787950ac46537a8a7d2cd7fac7"}
Jul 08 16:52:08.332423 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:52:08.332227    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz"
Jul 08 16:52:08.357299 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:52:08.357246    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz" podStartSLOduration=3.440750386 podStartE2EDuration="26.357228811s" totalImagesPullingTime="19.006998251s" totalInitContainerRuntime="3.909480174s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:51:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:52:08.355062453 +0000 UTC m=+1040.626511993" watchObservedRunningTime="2026-07-08 16:52:08.357228811 +0000 UTC m=+1040.628678350"
Jul 08 16:52:09.336680 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:52:09.336621    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz"
Jul 08 16:52:09.338068 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:52:09.338033    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz" podUID="435847a0-f882-4d91-9d21-e96f272fe5fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 08 16:52:09.849151 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:52:09.849110    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46" podUID="2d9691d9-c2fa-42de-8f3b-4851317a6bac" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jul 08 16:52:09.854581 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:52:09.854550    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46" podUID="2d9691d9-c2fa-42de-8f3b-4851317a6bac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 08 16:52:09.854755 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:52:09.854715    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46"
Jul 08 16:52:09.854903 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:52:09.854873    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46" podUID="2d9691d9-c2fa-42de-8f3b-4851317a6bac" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:52:09.855011 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:52:09.854999    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46"
Jul 08 16:52:10.340691 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:52:10.340614    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz" podUID="435847a0-f882-4d91-9d21-e96f272fe5fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 08 16:52:13.358027 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:52:13.357928    2578 generic.go:356] "Generic (PLEG): container finished" podID="2d9691d9-c2fa-42de-8f3b-4851317a6bac" containerID="6dcceacab1ef12e163420f5b5e44f324ddb0d3b3b518cffe5944891ea7124f80" exitCode=0
Jul 08 16:52:13.358027 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:52:13.357998    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46" event={"ID":"2d9691d9-c2fa-42de-8f3b-4851317a6bac","Type":"ContainerDied","Data":"6dcceacab1ef12e163420f5b5e44f324ddb0d3b3b518cffe5944891ea7124f80"}
Jul 08 16:52:13.390889 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:52:13.390854    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46"
Jul 08 16:52:13.488692 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:52:13.488619    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d9691d9-c2fa-42de-8f3b-4851317a6bac-cabundle-cert\") pod \"2d9691d9-c2fa-42de-8f3b-4851317a6bac\" (UID: \"2d9691d9-c2fa-42de-8f3b-4851317a6bac\") "
Jul 08 16:52:13.488890 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:52:13.488715    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d9691d9-c2fa-42de-8f3b-4851317a6bac-kserve-provision-location\") pod \"2d9691d9-c2fa-42de-8f3b-4851317a6bac\" (UID: \"2d9691d9-c2fa-42de-8f3b-4851317a6bac\") "
Jul 08 16:52:13.488890 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:52:13.488763    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d9691d9-c2fa-42de-8f3b-4851317a6bac-proxy-tls\") pod \"2d9691d9-c2fa-42de-8f3b-4851317a6bac\" (UID: \"2d9691d9-c2fa-42de-8f3b-4851317a6bac\") "
Jul 08 16:52:13.488890 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:52:13.488788    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7jbf7\" (UniqueName: \"kubernetes.io/projected/2d9691d9-c2fa-42de-8f3b-4851317a6bac-kube-api-access-7jbf7\") pod \"2d9691d9-c2fa-42de-8f3b-4851317a6bac\" (UID: \"2d9691d9-c2fa-42de-8f3b-4851317a6bac\") "
Jul 08 16:52:13.489059 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:52:13.488939    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d9691d9-c2fa-42de-8f3b-4851317a6bac-kube-rbac-proxy-sar-config\") pod \"2d9691d9-c2fa-42de-8f3b-4851317a6bac\" (UID: \"2d9691d9-c2fa-42de-8f3b-4851317a6bac\") "
Jul 08 16:52:13.489218 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:52:13.489182    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d9691d9-c2fa-42de-8f3b-4851317a6bac-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2d9691d9-c2fa-42de-8f3b-4851317a6bac" (UID: "2d9691d9-c2fa-42de-8f3b-4851317a6bac"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:52:13.489350 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:52:13.489195    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d9691d9-c2fa-42de-8f3b-4851317a6bac-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2d9691d9-c2fa-42de-8f3b-4851317a6bac" (UID: "2d9691d9-c2fa-42de-8f3b-4851317a6bac"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:52:13.489350 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:52:13.489308    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d9691d9-c2fa-42de-8f3b-4851317a6bac-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2d9691d9-c2fa-42de-8f3b-4851317a6bac" (UID: "2d9691d9-c2fa-42de-8f3b-4851317a6bac"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:52:13.491233 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:52:13.491206    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d9691d9-c2fa-42de-8f3b-4851317a6bac-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2d9691d9-c2fa-42de-8f3b-4851317a6bac" (UID: "2d9691d9-c2fa-42de-8f3b-4851317a6bac"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:52:13.491233 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:52:13.491217    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d9691d9-c2fa-42de-8f3b-4851317a6bac-kube-api-access-7jbf7" (OuterVolumeSpecName: "kube-api-access-7jbf7") pod "2d9691d9-c2fa-42de-8f3b-4851317a6bac" (UID: "2d9691d9-c2fa-42de-8f3b-4851317a6bac"). InnerVolumeSpecName "kube-api-access-7jbf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:52:13.590010 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:52:13.589974    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d9691d9-c2fa-42de-8f3b-4851317a6bac-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:52:13.590010 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:52:13.590006    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d9691d9-c2fa-42de-8f3b-4851317a6bac-cabundle-cert\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:52:13.590010 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:52:13.590017    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d9691d9-c2fa-42de-8f3b-4851317a6bac-kserve-provision-location\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:52:13.590248 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:52:13.590025    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d9691d9-c2fa-42de-8f3b-4851317a6bac-proxy-tls\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:52:13.590248 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:52:13.590036    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jbf7\" (UniqueName: \"kubernetes.io/projected/2d9691d9-c2fa-42de-8f3b-4851317a6bac-kube-api-access-7jbf7\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:52:14.365599 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:52:14.365562    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46"
Jul 08 16:52:14.366119 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:52:14.365559    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46" event={"ID":"2d9691d9-c2fa-42de-8f3b-4851317a6bac","Type":"ContainerDied","Data":"37632a289db878eefea1ca8c962c94c7c7674abdc5dfb722280e1bee1e9b474c"}
Jul 08 16:52:14.366119 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:52:14.365708    2578 scope.go:117] "RemoveContainer" containerID="6dcceacab1ef12e163420f5b5e44f324ddb0d3b3b518cffe5944891ea7124f80"
Jul 08 16:52:14.377002 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:52:14.376978    2578 scope.go:117] "RemoveContainer" containerID="36c623d02147f7bc28e08bfae7dd17f250ecca50cb9cfffca660c2ac98de730e"
Jul 08 16:52:14.386036 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:52:14.385999    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46"]
Jul 08 16:52:14.386892 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:52:14.386873    2578 scope.go:117] "RemoveContainer" containerID="2a9ca09f13f46e0baa6db0de851e6569bf2473fc8dea34ede3f9dc92e6a1c9ec"
Jul 08 16:52:14.393864 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:52:14.393827    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03e1d-predictor-7c6b878b5d-qss46"]
Jul 08 16:52:14.395793 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:52:14.395767    2578 scope.go:117] "RemoveContainer" containerID="78a7f2500636d47195b5c7578802842c7339afeb56c9d09ae9c3f66c0e393636"
Jul 08 16:52:15.251573 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:52:15.251528    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs" podUID="aa6f46ad-52fc-46d4-a305-394892cb18a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 08 16:52:15.344765 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:52:15.344737    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz"
Jul 08 16:52:15.345241 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:52:15.345213    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz" podUID="435847a0-f882-4d91-9d21-e96f272fe5fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 08 16:52:16.285380 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:52:16.285346    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d9691d9-c2fa-42de-8f3b-4851317a6bac" path="/var/lib/kubelet/pods/2d9691d9-c2fa-42de-8f3b-4851317a6bac/volumes"
Jul 08 16:52:25.251005 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:52:25.250962    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs" podUID="aa6f46ad-52fc-46d4-a305-394892cb18a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 08 16:52:25.345531 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:52:25.345495    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz" podUID="435847a0-f882-4d91-9d21-e96f272fe5fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 08 16:52:35.251231 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:52:35.251191    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs" podUID="aa6f46ad-52fc-46d4-a305-394892cb18a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 08 16:52:35.345291 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:52:35.345252    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz" podUID="435847a0-f882-4d91-9d21-e96f272fe5fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 08 16:52:45.251052 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:52:45.251008    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs" podUID="aa6f46ad-52fc-46d4-a305-394892cb18a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 08 16:52:45.345223 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:52:45.345186    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz" podUID="435847a0-f882-4d91-9d21-e96f272fe5fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 08 16:52:55.251102 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:52:55.251064    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs" podUID="aa6f46ad-52fc-46d4-a305-394892cb18a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 08 16:52:55.346119 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:52:55.346077    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz" podUID="435847a0-f882-4d91-9d21-e96f272fe5fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 08 16:53:05.251837 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:05.251806    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs"
Jul 08 16:53:05.346683 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:05.346660    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz"
Jul 08 16:53:33.221511 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:33.221473    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs"]
Jul 08 16:53:33.221943 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:33.221794    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs" podUID="aa6f46ad-52fc-46d4-a305-394892cb18a1" containerName="kserve-container" containerID="cri-o://e7c899cccd620c6184bdfe7e4a67cb6a663818a7ee5a3118aa5e233f89ae21fe" gracePeriod=30
Jul 08 16:53:33.221943 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:33.221803    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs" podUID="aa6f46ad-52fc-46d4-a305-394892cb18a1" containerName="kube-rbac-proxy" containerID="cri-o://a9d45f76e70be2592eb9715c7486111b0ae65c1488efa0a76c481910dd8d40f2" gracePeriod=30
Jul 08 16:53:33.322157 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:33.322125    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb"]
Jul 08 16:53:33.322452 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:33.322440    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d9691d9-c2fa-42de-8f3b-4851317a6bac" containerName="agent"
Jul 08 16:53:33.322494 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:33.322455    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d9691d9-c2fa-42de-8f3b-4851317a6bac" containerName="agent"
Jul 08 16:53:33.322494 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:33.322467    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d9691d9-c2fa-42de-8f3b-4851317a6bac" containerName="storage-initializer"
Jul 08 16:53:33.322494 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:33.322474    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d9691d9-c2fa-42de-8f3b-4851317a6bac" containerName="storage-initializer"
Jul 08 16:53:33.322494 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:33.322493    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d9691d9-c2fa-42de-8f3b-4851317a6bac" containerName="kserve-container"
Jul 08 16:53:33.322617 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:33.322500    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d9691d9-c2fa-42de-8f3b-4851317a6bac" containerName="kserve-container"
Jul 08 16:53:33.322617 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:33.322515    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d9691d9-c2fa-42de-8f3b-4851317a6bac" containerName="kube-rbac-proxy"
Jul 08 16:53:33.322617 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:33.322520    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d9691d9-c2fa-42de-8f3b-4851317a6bac" containerName="kube-rbac-proxy"
Jul 08 16:53:33.322617 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:33.322575    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d9691d9-c2fa-42de-8f3b-4851317a6bac" containerName="kserve-container"
Jul 08 16:53:33.322617 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:33.322584    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d9691d9-c2fa-42de-8f3b-4851317a6bac" containerName="kube-rbac-proxy"
Jul 08 16:53:33.322617 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:33.322590    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d9691d9-c2fa-42de-8f3b-4851317a6bac" containerName="agent"
Jul 08 16:53:33.324844 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:33.324827    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb"
Jul 08 16:53:33.328142 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:33.328112    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-3e871-predictor-serving-cert\""
Jul 08 16:53:33.328267 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:33.328114    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-3e871-kube-rbac-proxy-sar-config\""
Jul 08 16:53:33.343772 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:33.343741    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb"]
Jul 08 16:53:33.370420 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:33.370385    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb53586a-ea54-4def-87dc-9bdc6baa1761-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb\" (UID: \"eb53586a-ea54-4def-87dc-9bdc6baa1761\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb"
Jul 08 16:53:33.370607 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:33.370456    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb53586a-ea54-4def-87dc-9bdc6baa1761-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb\" (UID: \"eb53586a-ea54-4def-87dc-9bdc6baa1761\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb"
Jul 08 16:53:33.370738 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:33.370711    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb53586a-ea54-4def-87dc-9bdc6baa1761-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb\" (UID: \"eb53586a-ea54-4def-87dc-9bdc6baa1761\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb"
Jul 08 16:53:33.370830 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:33.370758    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz7g4\" (UniqueName: \"kubernetes.io/projected/eb53586a-ea54-4def-87dc-9bdc6baa1761-kube-api-access-dz7g4\") pod \"isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb\" (UID: \"eb53586a-ea54-4def-87dc-9bdc6baa1761\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb"
Jul 08 16:53:33.370830 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:33.370822    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb53586a-ea54-4def-87dc-9bdc6baa1761-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb\" (UID: \"eb53586a-ea54-4def-87dc-9bdc6baa1761\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb"
Jul 08 16:53:33.431799 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:33.431765    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd"]
Jul 08 16:53:33.437490 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:33.437474    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd"
Jul 08 16:53:33.441193 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:33.441167    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-3e871-kube-rbac-proxy-sar-config\""
Jul 08 16:53:33.441193 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:33.441183    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-3e871-predictor-serving-cert\""
Jul 08 16:53:33.452771 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:33.452750    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd"]
Jul 08 16:53:33.471869 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:33.471795    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb53586a-ea54-4def-87dc-9bdc6baa1761-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb\" (UID: \"eb53586a-ea54-4def-87dc-9bdc6baa1761\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb"
Jul 08 16:53:33.471869 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:33.471845    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb53586a-ea54-4def-87dc-9bdc6baa1761-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb\" (UID: \"eb53586a-ea54-4def-87dc-9bdc6baa1761\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb"
Jul 08 16:53:33.472060 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:33.471882    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81937630-42ac-4e5d-8427-14687fe57624-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd\" (UID: \"81937630-42ac-4e5d-8427-14687fe57624\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd"
Jul 08 16:53:33.472060 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:33.471909    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81937630-42ac-4e5d-8427-14687fe57624-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd\" (UID: \"81937630-42ac-4e5d-8427-14687fe57624\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd"
Jul 08 16:53:33.472060 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:33.471986    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb53586a-ea54-4def-87dc-9bdc6baa1761-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb\" (UID: \"eb53586a-ea54-4def-87dc-9bdc6baa1761\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb"
Jul 08 16:53:33.472060 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:33.472016    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz7g4\" (UniqueName: \"kubernetes.io/projected/eb53586a-ea54-4def-87dc-9bdc6baa1761-kube-api-access-dz7g4\") pod \"isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb\" (UID: \"eb53586a-ea54-4def-87dc-9bdc6baa1761\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb"
Jul 08 16:53:33.472060 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:33.472046    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb53586a-ea54-4def-87dc-9bdc6baa1761-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb\" (UID: \"eb53586a-ea54-4def-87dc-9bdc6baa1761\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb"
Jul 08 16:53:33.472313 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:33.472077    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnfmz\" (UniqueName: \"kubernetes.io/projected/81937630-42ac-4e5d-8427-14687fe57624-kube-api-access-vnfmz\") pod \"isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd\" (UID: \"81937630-42ac-4e5d-8427-14687fe57624\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd"
Jul 08 16:53:33.472313 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:33.472137    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81937630-42ac-4e5d-8427-14687fe57624-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd\" (UID: \"81937630-42ac-4e5d-8427-14687fe57624\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd"
Jul 08 16:53:33.472313 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:33.472162    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81937630-42ac-4e5d-8427-14687fe57624-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd\" (UID: \"81937630-42ac-4e5d-8427-14687fe57624\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd"
Jul 08 16:53:33.472698 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:33.472665    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb53586a-ea54-4def-87dc-9bdc6baa1761-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb\" (UID: \"eb53586a-ea54-4def-87dc-9bdc6baa1761\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb"
Jul 08 16:53:33.472801 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:33.472678    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb53586a-ea54-4def-87dc-9bdc6baa1761-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb\" (UID: \"eb53586a-ea54-4def-87dc-9bdc6baa1761\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb"
Jul 08 16:53:33.473413 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:33.473390    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb53586a-ea54-4def-87dc-9bdc6baa1761-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb\" (UID: \"eb53586a-ea54-4def-87dc-9bdc6baa1761\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb"
Jul 08 16:53:33.475531 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:33.475502    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb53586a-ea54-4def-87dc-9bdc6baa1761-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb\" (UID: \"eb53586a-ea54-4def-87dc-9bdc6baa1761\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb"
Jul 08 16:53:33.483931 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:33.483910    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz"]
Jul 08 16:53:33.484164 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:33.484142    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz" podUID="435847a0-f882-4d91-9d21-e96f272fe5fb" containerName="kserve-container" containerID="cri-o://6317ac1c278094a68083b352d17264a56ceadc0a26d9ef42a46cd4b61987ce81" gracePeriod=30
Jul 08 16:53:33.484261 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:33.484233    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz" podUID="435847a0-f882-4d91-9d21-e96f272fe5fb" containerName="kube-rbac-proxy" containerID="cri-o://cb8b42f3bc08a45b2b6ef5d2d5920d1d350af3787950ac46537a8a7d2cd7fac7" gracePeriod=30
Jul 08 16:53:33.503763 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:33.503740    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz7g4\" (UniqueName: \"kubernetes.io/projected/eb53586a-ea54-4def-87dc-9bdc6baa1761-kube-api-access-dz7g4\") pod \"isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb\" (UID: \"eb53586a-ea54-4def-87dc-9bdc6baa1761\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb"
Jul 08 16:53:33.573416 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:33.573382    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnfmz\" (UniqueName: \"kubernetes.io/projected/81937630-42ac-4e5d-8427-14687fe57624-kube-api-access-vnfmz\") pod \"isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd\" (UID: \"81937630-42ac-4e5d-8427-14687fe57624\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd"
Jul 08 16:53:33.573603 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:33.573447    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81937630-42ac-4e5d-8427-14687fe57624-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd\" (UID: \"81937630-42ac-4e5d-8427-14687fe57624\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd"
Jul 08 16:53:33.573603 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:33.573473    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81937630-42ac-4e5d-8427-14687fe57624-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd\" (UID: \"81937630-42ac-4e5d-8427-14687fe57624\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd"
Jul 08 16:53:33.573603 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:33.573582    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81937630-42ac-4e5d-8427-14687fe57624-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd\" (UID: \"81937630-42ac-4e5d-8427-14687fe57624\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd"
Jul 08 16:53:33.573853 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:33.573609    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81937630-42ac-4e5d-8427-14687fe57624-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd\" (UID: \"81937630-42ac-4e5d-8427-14687fe57624\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd"
Jul 08 16:53:33.573853 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:53:33.573748    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-3e871-predictor-serving-cert: secret "isvc-xgboost-graph-raw-hpa-3e871-predictor-serving-cert" not found
Jul 08 16:53:33.573853 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:53:33.573818    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81937630-42ac-4e5d-8427-14687fe57624-proxy-tls podName:81937630-42ac-4e5d-8427-14687fe57624 nodeName:}" failed. No retries permitted until 2026-07-08 16:53:34.07379382 +0000 UTC m=+1126.345243342 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/81937630-42ac-4e5d-8427-14687fe57624-proxy-tls") pod "isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd" (UID: "81937630-42ac-4e5d-8427-14687fe57624") : secret "isvc-xgboost-graph-raw-hpa-3e871-predictor-serving-cert" not found
Jul 08 16:53:33.573990 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:33.573901    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81937630-42ac-4e5d-8427-14687fe57624-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd\" (UID: \"81937630-42ac-4e5d-8427-14687fe57624\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd"
Jul 08 16:53:33.574170 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:33.574153    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81937630-42ac-4e5d-8427-14687fe57624-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd\" (UID: \"81937630-42ac-4e5d-8427-14687fe57624\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd"
Jul 08 16:53:33.574292 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:33.574275    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81937630-42ac-4e5d-8427-14687fe57624-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd\" (UID: \"81937630-42ac-4e5d-8427-14687fe57624\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd"
Jul 08 16:53:33.585489 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:33.585465    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnfmz\" (UniqueName: \"kubernetes.io/projected/81937630-42ac-4e5d-8427-14687fe57624-kube-api-access-vnfmz\") pod \"isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd\" (UID: \"81937630-42ac-4e5d-8427-14687fe57624\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd"
Jul 08 16:53:33.636526 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:33.636495    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb"
Jul 08 16:53:33.692726 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:33.692692    2578 generic.go:356] "Generic (PLEG): container finished" podID="435847a0-f882-4d91-9d21-e96f272fe5fb" containerID="cb8b42f3bc08a45b2b6ef5d2d5920d1d350af3787950ac46537a8a7d2cd7fac7" exitCode=2
Jul 08 16:53:33.692893 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:33.692794    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz" event={"ID":"435847a0-f882-4d91-9d21-e96f272fe5fb","Type":"ContainerDied","Data":"cb8b42f3bc08a45b2b6ef5d2d5920d1d350af3787950ac46537a8a7d2cd7fac7"}
Jul 08 16:53:33.696108 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:33.695975    2578 generic.go:356] "Generic (PLEG): container finished" podID="aa6f46ad-52fc-46d4-a305-394892cb18a1" containerID="a9d45f76e70be2592eb9715c7486111b0ae65c1488efa0a76c481910dd8d40f2" exitCode=2
Jul 08 16:53:33.696108 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:33.696041    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs" event={"ID":"aa6f46ad-52fc-46d4-a305-394892cb18a1","Type":"ContainerDied","Data":"a9d45f76e70be2592eb9715c7486111b0ae65c1488efa0a76c481910dd8d40f2"}
Jul 08 16:53:33.780321 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:33.780247    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb"]
Jul 08 16:53:33.782802 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:53:33.782775    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb53586a_ea54_4def_87dc_9bdc6baa1761.slice/crio-a8c312c276d0285d3c9fb5fc4f49d5be885e83823a12ae1d533e4f4d02fabe24 WatchSource:0}: Error finding container a8c312c276d0285d3c9fb5fc4f49d5be885e83823a12ae1d533e4f4d02fabe24: Status 404 returned error can't find the container with id a8c312c276d0285d3c9fb5fc4f49d5be885e83823a12ae1d533e4f4d02fabe24
Jul 08 16:53:34.078445 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:34.078415    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81937630-42ac-4e5d-8427-14687fe57624-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd\" (UID: \"81937630-42ac-4e5d-8427-14687fe57624\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd"
Jul 08 16:53:34.080847 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:34.080818    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81937630-42ac-4e5d-8427-14687fe57624-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd\" (UID: \"81937630-42ac-4e5d-8427-14687fe57624\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd"
Jul 08 16:53:34.348005 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:34.347921    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd"
Jul 08 16:53:34.483501 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:34.483478    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd"]
Jul 08 16:53:34.485209 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:53:34.485183    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81937630_42ac_4e5d_8427_14687fe57624.slice/crio-702810faa075a9f36f6741f998dedc485ecb7128e2f0105fac425251923b250a WatchSource:0}: Error finding container 702810faa075a9f36f6741f998dedc485ecb7128e2f0105fac425251923b250a: Status 404 returned error can't find the container with id 702810faa075a9f36f6741f998dedc485ecb7128e2f0105fac425251923b250a
Jul 08 16:53:34.703183 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:34.703087    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb" event={"ID":"eb53586a-ea54-4def-87dc-9bdc6baa1761","Type":"ContainerStarted","Data":"d5ed5ce94f93e5c58bd89712d9852f7c1f75de3960d8c245d97922e362ad2c5d"}
Jul 08 16:53:34.703183 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:34.703166    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb" event={"ID":"eb53586a-ea54-4def-87dc-9bdc6baa1761","Type":"ContainerStarted","Data":"a8c312c276d0285d3c9fb5fc4f49d5be885e83823a12ae1d533e4f4d02fabe24"}
Jul 08 16:53:34.704621 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:34.704594    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd" event={"ID":"81937630-42ac-4e5d-8427-14687fe57624","Type":"ContainerStarted","Data":"96f7d09ad5eddf2112a9211df325489c2114f61514cd21e6ffa9d50d4df846ac"}
Jul 08 16:53:34.704768 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:34.704665    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd" event={"ID":"81937630-42ac-4e5d-8427-14687fe57624","Type":"ContainerStarted","Data":"702810faa075a9f36f6741f998dedc485ecb7128e2f0105fac425251923b250a"}
Jul 08 16:53:35.245114 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:35.245079    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs" podUID="aa6f46ad-52fc-46d4-a305-394892cb18a1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jul 08 16:53:35.250587 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:35.250565    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs" podUID="aa6f46ad-52fc-46d4-a305-394892cb18a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 08 16:53:35.341518 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:35.341485    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz" podUID="435847a0-f882-4d91-9d21-e96f272fe5fb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jul 08 16:53:35.345833 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:35.345812    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz" podUID="435847a0-f882-4d91-9d21-e96f272fe5fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 08 16:53:37.124896 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:37.124874    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz"
Jul 08 16:53:37.207053 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:37.206970    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/435847a0-f882-4d91-9d21-e96f272fe5fb-cabundle-cert\") pod \"435847a0-f882-4d91-9d21-e96f272fe5fb\" (UID: \"435847a0-f882-4d91-9d21-e96f272fe5fb\") "
Jul 08 16:53:37.207053 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:37.207015    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kx5rx\" (UniqueName: \"kubernetes.io/projected/435847a0-f882-4d91-9d21-e96f272fe5fb-kube-api-access-kx5rx\") pod \"435847a0-f882-4d91-9d21-e96f272fe5fb\" (UID: \"435847a0-f882-4d91-9d21-e96f272fe5fb\") "
Jul 08 16:53:37.207053 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:37.207049    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/435847a0-f882-4d91-9d21-e96f272fe5fb-kube-rbac-proxy-sar-config\") pod \"435847a0-f882-4d91-9d21-e96f272fe5fb\" (UID: \"435847a0-f882-4d91-9d21-e96f272fe5fb\") "
Jul 08 16:53:37.207308 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:37.207107    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/435847a0-f882-4d91-9d21-e96f272fe5fb-proxy-tls\") pod \"435847a0-f882-4d91-9d21-e96f272fe5fb\" (UID: \"435847a0-f882-4d91-9d21-e96f272fe5fb\") "
Jul 08 16:53:37.207308 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:37.207164    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/435847a0-f882-4d91-9d21-e96f272fe5fb-kserve-provision-location\") pod \"435847a0-f882-4d91-9d21-e96f272fe5fb\" (UID: \"435847a0-f882-4d91-9d21-e96f272fe5fb\") "
Jul 08 16:53:37.207403 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:37.207310    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/435847a0-f882-4d91-9d21-e96f272fe5fb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "435847a0-f882-4d91-9d21-e96f272fe5fb" (UID: "435847a0-f882-4d91-9d21-e96f272fe5fb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:53:37.207458 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:37.207447    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/435847a0-f882-4d91-9d21-e96f272fe5fb-cabundle-cert\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:53:37.207523 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:37.207495    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/435847a0-f882-4d91-9d21-e96f272fe5fb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "435847a0-f882-4d91-9d21-e96f272fe5fb" (UID: "435847a0-f882-4d91-9d21-e96f272fe5fb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:53:37.207560 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:37.207525    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/435847a0-f882-4d91-9d21-e96f272fe5fb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "435847a0-f882-4d91-9d21-e96f272fe5fb" (UID: "435847a0-f882-4d91-9d21-e96f272fe5fb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:53:37.209197 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:37.209173    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/435847a0-f882-4d91-9d21-e96f272fe5fb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "435847a0-f882-4d91-9d21-e96f272fe5fb" (UID: "435847a0-f882-4d91-9d21-e96f272fe5fb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:53:37.209284 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:37.209266    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/435847a0-f882-4d91-9d21-e96f272fe5fb-kube-api-access-kx5rx" (OuterVolumeSpecName: "kube-api-access-kx5rx") pod "435847a0-f882-4d91-9d21-e96f272fe5fb" (UID: "435847a0-f882-4d91-9d21-e96f272fe5fb"). InnerVolumeSpecName "kube-api-access-kx5rx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:53:37.308010 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:37.307973    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/435847a0-f882-4d91-9d21-e96f272fe5fb-proxy-tls\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:53:37.308010 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:37.308000    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/435847a0-f882-4d91-9d21-e96f272fe5fb-kserve-provision-location\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:53:37.308010 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:37.308009    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kx5rx\" (UniqueName: \"kubernetes.io/projected/435847a0-f882-4d91-9d21-e96f272fe5fb-kube-api-access-kx5rx\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:53:37.308010 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:37.308019    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/435847a0-f882-4d91-9d21-e96f272fe5fb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:53:37.720083 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:37.720048    2578 generic.go:356] "Generic (PLEG): container finished" podID="eb53586a-ea54-4def-87dc-9bdc6baa1761" containerID="d5ed5ce94f93e5c58bd89712d9852f7c1f75de3960d8c245d97922e362ad2c5d" exitCode=0
Jul 08 16:53:37.720221 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:37.720117    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb" event={"ID":"eb53586a-ea54-4def-87dc-9bdc6baa1761","Type":"ContainerDied","Data":"d5ed5ce94f93e5c58bd89712d9852f7c1f75de3960d8c245d97922e362ad2c5d"}
Jul 08 16:53:37.722242 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:37.722161    2578 generic.go:356] "Generic (PLEG): container finished" podID="435847a0-f882-4d91-9d21-e96f272fe5fb" containerID="6317ac1c278094a68083b352d17264a56ceadc0a26d9ef42a46cd4b61987ce81" exitCode=0
Jul 08 16:53:37.722242 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:37.722202    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz" event={"ID":"435847a0-f882-4d91-9d21-e96f272fe5fb","Type":"ContainerDied","Data":"6317ac1c278094a68083b352d17264a56ceadc0a26d9ef42a46cd4b61987ce81"}
Jul 08 16:53:37.722328 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:37.722263    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz" event={"ID":"435847a0-f882-4d91-9d21-e96f272fe5fb","Type":"ContainerDied","Data":"6ceafe65554336469af33929aca1dcbd0f106f72e9519fa5761aa727e24552b6"}
Jul 08 16:53:37.722328 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:37.722302    2578 scope.go:117] "RemoveContainer" containerID="cb8b42f3bc08a45b2b6ef5d2d5920d1d350af3787950ac46537a8a7d2cd7fac7"
Jul 08 16:53:37.722328 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:37.722320    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz"
Jul 08 16:53:37.746685 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:37.746664    2578 scope.go:117] "RemoveContainer" containerID="6317ac1c278094a68083b352d17264a56ceadc0a26d9ef42a46cd4b61987ce81"
Jul 08 16:53:37.759689 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:37.759668    2578 scope.go:117] "RemoveContainer" containerID="2d939a10878fa117088c0f30349dd6b23e3f40841d1f230be3872533ef993ba1"
Jul 08 16:53:37.772882 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:37.772847    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz"]
Jul 08 16:53:37.777337 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:37.777314    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-5d281-predictor-5b6dbc44d5-pmrpz"]
Jul 08 16:53:37.778827 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:37.778808    2578 scope.go:117] "RemoveContainer" containerID="cb8b42f3bc08a45b2b6ef5d2d5920d1d350af3787950ac46537a8a7d2cd7fac7"
Jul 08 16:53:37.779135 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:53:37.779116    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb8b42f3bc08a45b2b6ef5d2d5920d1d350af3787950ac46537a8a7d2cd7fac7\": container with ID starting with cb8b42f3bc08a45b2b6ef5d2d5920d1d350af3787950ac46537a8a7d2cd7fac7 not found: ID does not exist" containerID="cb8b42f3bc08a45b2b6ef5d2d5920d1d350af3787950ac46537a8a7d2cd7fac7"
Jul 08 16:53:37.779190 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:37.779144    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb8b42f3bc08a45b2b6ef5d2d5920d1d350af3787950ac46537a8a7d2cd7fac7"} err="failed to get container status \"cb8b42f3bc08a45b2b6ef5d2d5920d1d350af3787950ac46537a8a7d2cd7fac7\": rpc error: code = NotFound desc = could not find container \"cb8b42f3bc08a45b2b6ef5d2d5920d1d350af3787950ac46537a8a7d2cd7fac7\": container with ID starting with cb8b42f3bc08a45b2b6ef5d2d5920d1d350af3787950ac46537a8a7d2cd7fac7 not found: ID does not exist"
Jul 08 16:53:37.779190 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:37.779164    2578 scope.go:117] "RemoveContainer" containerID="6317ac1c278094a68083b352d17264a56ceadc0a26d9ef42a46cd4b61987ce81"
Jul 08 16:53:37.779449 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:53:37.779428    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6317ac1c278094a68083b352d17264a56ceadc0a26d9ef42a46cd4b61987ce81\": container with ID starting with 6317ac1c278094a68083b352d17264a56ceadc0a26d9ef42a46cd4b61987ce81 not found: ID does not exist" containerID="6317ac1c278094a68083b352d17264a56ceadc0a26d9ef42a46cd4b61987ce81"
Jul 08 16:53:37.779537 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:37.779455    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6317ac1c278094a68083b352d17264a56ceadc0a26d9ef42a46cd4b61987ce81"} err="failed to get container status \"6317ac1c278094a68083b352d17264a56ceadc0a26d9ef42a46cd4b61987ce81\": rpc error: code = NotFound desc = could not find container \"6317ac1c278094a68083b352d17264a56ceadc0a26d9ef42a46cd4b61987ce81\": container with ID starting with 6317ac1c278094a68083b352d17264a56ceadc0a26d9ef42a46cd4b61987ce81 not found: ID does not exist"
Jul 08 16:53:37.779537 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:37.779474    2578 scope.go:117] "RemoveContainer" containerID="2d939a10878fa117088c0f30349dd6b23e3f40841d1f230be3872533ef993ba1"
Jul 08 16:53:37.779784 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:53:37.779765    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d939a10878fa117088c0f30349dd6b23e3f40841d1f230be3872533ef993ba1\": container with ID starting with 2d939a10878fa117088c0f30349dd6b23e3f40841d1f230be3872533ef993ba1 not found: ID does not exist" containerID="2d939a10878fa117088c0f30349dd6b23e3f40841d1f230be3872533ef993ba1"
Jul 08 16:53:37.779839 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:37.779791    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d939a10878fa117088c0f30349dd6b23e3f40841d1f230be3872533ef993ba1"} err="failed to get container status \"2d939a10878fa117088c0f30349dd6b23e3f40841d1f230be3872533ef993ba1\": rpc error: code = NotFound desc = could not find container \"2d939a10878fa117088c0f30349dd6b23e3f40841d1f230be3872533ef993ba1\": container with ID starting with 2d939a10878fa117088c0f30349dd6b23e3f40841d1f230be3872533ef993ba1 not found: ID does not exist"
Jul 08 16:53:37.862380 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:37.862359    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs"
Jul 08 16:53:37.912693 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:37.912664    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa6f46ad-52fc-46d4-a305-394892cb18a1-kube-rbac-proxy-sar-config\") pod \"aa6f46ad-52fc-46d4-a305-394892cb18a1\" (UID: \"aa6f46ad-52fc-46d4-a305-394892cb18a1\") "
Jul 08 16:53:37.912806 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:37.912779    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa6f46ad-52fc-46d4-a305-394892cb18a1-proxy-tls\") pod \"aa6f46ad-52fc-46d4-a305-394892cb18a1\" (UID: \"aa6f46ad-52fc-46d4-a305-394892cb18a1\") "
Jul 08 16:53:37.912865 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:37.912806    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aa6f46ad-52fc-46d4-a305-394892cb18a1-cabundle-cert\") pod \"aa6f46ad-52fc-46d4-a305-394892cb18a1\" (UID: \"aa6f46ad-52fc-46d4-a305-394892cb18a1\") "
Jul 08 16:53:37.912919 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:37.912862    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa6f46ad-52fc-46d4-a305-394892cb18a1-kserve-provision-location\") pod \"aa6f46ad-52fc-46d4-a305-394892cb18a1\" (UID: \"aa6f46ad-52fc-46d4-a305-394892cb18a1\") "
Jul 08 16:53:37.912919 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:37.912898    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xtb6v\" (UniqueName: \"kubernetes.io/projected/aa6f46ad-52fc-46d4-a305-394892cb18a1-kube-api-access-xtb6v\") pod \"aa6f46ad-52fc-46d4-a305-394892cb18a1\" (UID: \"aa6f46ad-52fc-46d4-a305-394892cb18a1\") "
Jul 08 16:53:37.913016 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:37.912908    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa6f46ad-52fc-46d4-a305-394892cb18a1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "aa6f46ad-52fc-46d4-a305-394892cb18a1" (UID: "aa6f46ad-52fc-46d4-a305-394892cb18a1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:53:37.913152 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:37.913129    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa6f46ad-52fc-46d4-a305-394892cb18a1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "aa6f46ad-52fc-46d4-a305-394892cb18a1" (UID: "aa6f46ad-52fc-46d4-a305-394892cb18a1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:53:37.913228 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:37.913133    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa6f46ad-52fc-46d4-a305-394892cb18a1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:53:37.913228 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:37.913212    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa6f46ad-52fc-46d4-a305-394892cb18a1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "aa6f46ad-52fc-46d4-a305-394892cb18a1" (UID: "aa6f46ad-52fc-46d4-a305-394892cb18a1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:53:37.914993 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:37.914972    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa6f46ad-52fc-46d4-a305-394892cb18a1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "aa6f46ad-52fc-46d4-a305-394892cb18a1" (UID: "aa6f46ad-52fc-46d4-a305-394892cb18a1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:53:37.915083 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:37.914974    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa6f46ad-52fc-46d4-a305-394892cb18a1-kube-api-access-xtb6v" (OuterVolumeSpecName: "kube-api-access-xtb6v") pod "aa6f46ad-52fc-46d4-a305-394892cb18a1" (UID: "aa6f46ad-52fc-46d4-a305-394892cb18a1"). InnerVolumeSpecName "kube-api-access-xtb6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:53:38.013740 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:38.013701    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa6f46ad-52fc-46d4-a305-394892cb18a1-proxy-tls\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:53:38.013740 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:38.013726    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aa6f46ad-52fc-46d4-a305-394892cb18a1-cabundle-cert\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:53:38.013740 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:38.013743    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa6f46ad-52fc-46d4-a305-394892cb18a1-kserve-provision-location\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:53:38.013740 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:38.013751    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtb6v\" (UniqueName: \"kubernetes.io/projected/aa6f46ad-52fc-46d4-a305-394892cb18a1-kube-api-access-xtb6v\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:53:38.285569 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:38.285538    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="435847a0-f882-4d91-9d21-e96f272fe5fb" path="/var/lib/kubelet/pods/435847a0-f882-4d91-9d21-e96f272fe5fb/volumes"
Jul 08 16:53:38.727713 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:38.727684    2578 generic.go:356] "Generic (PLEG): container finished" podID="81937630-42ac-4e5d-8427-14687fe57624" containerID="96f7d09ad5eddf2112a9211df325489c2114f61514cd21e6ffa9d50d4df846ac" exitCode=0
Jul 08 16:53:38.727881 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:38.727760    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd" event={"ID":"81937630-42ac-4e5d-8427-14687fe57624","Type":"ContainerDied","Data":"96f7d09ad5eddf2112a9211df325489c2114f61514cd21e6ffa9d50d4df846ac"}
Jul 08 16:53:38.730527 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:38.730506    2578 generic.go:356] "Generic (PLEG): container finished" podID="aa6f46ad-52fc-46d4-a305-394892cb18a1" containerID="e7c899cccd620c6184bdfe7e4a67cb6a663818a7ee5a3118aa5e233f89ae21fe" exitCode=0
Jul 08 16:53:38.730616 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:38.730553    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs"
Jul 08 16:53:38.730616 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:38.730589    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs" event={"ID":"aa6f46ad-52fc-46d4-a305-394892cb18a1","Type":"ContainerDied","Data":"e7c899cccd620c6184bdfe7e4a67cb6a663818a7ee5a3118aa5e233f89ae21fe"}
Jul 08 16:53:38.730756 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:38.730661    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs" event={"ID":"aa6f46ad-52fc-46d4-a305-394892cb18a1","Type":"ContainerDied","Data":"06f0b0a1c8939d3a1f82e00f08fcffa374054fc54f4a75ba4c608b78e2532627"}
Jul 08 16:53:38.730756 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:38.730687    2578 scope.go:117] "RemoveContainer" containerID="a9d45f76e70be2592eb9715c7486111b0ae65c1488efa0a76c481910dd8d40f2"
Jul 08 16:53:38.733254 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:38.733230    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb" event={"ID":"eb53586a-ea54-4def-87dc-9bdc6baa1761","Type":"ContainerStarted","Data":"30fcfa6e579def4071541ef2edec36cf2bc5dbb69269d12c010005598a9dcba0"}
Jul 08 16:53:38.733354 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:38.733287    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb" event={"ID":"eb53586a-ea54-4def-87dc-9bdc6baa1761","Type":"ContainerStarted","Data":"3a9051349ecd553772e04a15304f46b4005f27904266fcfda0e28e78a0e9d3ff"}
Jul 08 16:53:38.733533 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:38.733519    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb"
Jul 08 16:53:38.733617 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:38.733560    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb"
Jul 08 16:53:38.735091 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:38.735065    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb" podUID="eb53586a-ea54-4def-87dc-9bdc6baa1761" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 08 16:53:38.739490 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:38.739474    2578 scope.go:117] "RemoveContainer" containerID="e7c899cccd620c6184bdfe7e4a67cb6a663818a7ee5a3118aa5e233f89ae21fe"
Jul 08 16:53:38.753795 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:38.753775    2578 scope.go:117] "RemoveContainer" containerID="cc33fed9c8297fe392d4e5dd748fce31254c13352c39e9fa6cb7e106acc854ce"
Jul 08 16:53:38.768902 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:38.768877    2578 scope.go:117] "RemoveContainer" containerID="a9d45f76e70be2592eb9715c7486111b0ae65c1488efa0a76c481910dd8d40f2"
Jul 08 16:53:38.769168 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:53:38.769151    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9d45f76e70be2592eb9715c7486111b0ae65c1488efa0a76c481910dd8d40f2\": container with ID starting with a9d45f76e70be2592eb9715c7486111b0ae65c1488efa0a76c481910dd8d40f2 not found: ID does not exist" containerID="a9d45f76e70be2592eb9715c7486111b0ae65c1488efa0a76c481910dd8d40f2"
Jul 08 16:53:38.769222 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:38.769175    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9d45f76e70be2592eb9715c7486111b0ae65c1488efa0a76c481910dd8d40f2"} err="failed to get container status \"a9d45f76e70be2592eb9715c7486111b0ae65c1488efa0a76c481910dd8d40f2\": rpc error: code = NotFound desc = could not find container \"a9d45f76e70be2592eb9715c7486111b0ae65c1488efa0a76c481910dd8d40f2\": container with ID starting with a9d45f76e70be2592eb9715c7486111b0ae65c1488efa0a76c481910dd8d40f2 not found: ID does not exist"
Jul 08 16:53:38.769222 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:38.769194    2578 scope.go:117] "RemoveContainer" containerID="e7c899cccd620c6184bdfe7e4a67cb6a663818a7ee5a3118aa5e233f89ae21fe"
Jul 08 16:53:38.769429 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:53:38.769413    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7c899cccd620c6184bdfe7e4a67cb6a663818a7ee5a3118aa5e233f89ae21fe\": container with ID starting with e7c899cccd620c6184bdfe7e4a67cb6a663818a7ee5a3118aa5e233f89ae21fe not found: ID does not exist" containerID="e7c899cccd620c6184bdfe7e4a67cb6a663818a7ee5a3118aa5e233f89ae21fe"
Jul 08 16:53:38.769481 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:38.769439    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7c899cccd620c6184bdfe7e4a67cb6a663818a7ee5a3118aa5e233f89ae21fe"} err="failed to get container status \"e7c899cccd620c6184bdfe7e4a67cb6a663818a7ee5a3118aa5e233f89ae21fe\": rpc error: code = NotFound desc = could not find container \"e7c899cccd620c6184bdfe7e4a67cb6a663818a7ee5a3118aa5e233f89ae21fe\": container with ID starting with e7c899cccd620c6184bdfe7e4a67cb6a663818a7ee5a3118aa5e233f89ae21fe not found: ID does not exist"
Jul 08 16:53:38.769481 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:38.769451    2578 scope.go:117] "RemoveContainer" containerID="cc33fed9c8297fe392d4e5dd748fce31254c13352c39e9fa6cb7e106acc854ce"
Jul 08 16:53:38.769703 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:53:38.769687    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc33fed9c8297fe392d4e5dd748fce31254c13352c39e9fa6cb7e106acc854ce\": container with ID starting with cc33fed9c8297fe392d4e5dd748fce31254c13352c39e9fa6cb7e106acc854ce not found: ID does not exist" containerID="cc33fed9c8297fe392d4e5dd748fce31254c13352c39e9fa6cb7e106acc854ce"
Jul 08 16:53:38.769780 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:38.769706    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc33fed9c8297fe392d4e5dd748fce31254c13352c39e9fa6cb7e106acc854ce"} err="failed to get container status \"cc33fed9c8297fe392d4e5dd748fce31254c13352c39e9fa6cb7e106acc854ce\": rpc error: code = NotFound desc = could not find container \"cc33fed9c8297fe392d4e5dd748fce31254c13352c39e9fa6cb7e106acc854ce\": container with ID starting with cc33fed9c8297fe392d4e5dd748fce31254c13352c39e9fa6cb7e106acc854ce not found: ID does not exist"
Jul 08 16:53:38.775870 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:38.775850    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs"]
Jul 08 16:53:38.781364 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:38.781344    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-5d281-predictor-6d6796fdb-xxbfs"]
Jul 08 16:53:38.803621 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:38.803575    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb" podStartSLOduration=2.107572887 podStartE2EDuration="5.803561601s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.695988714s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:53:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:53:38.802881891 +0000 UTC m=+1131.074331434" watchObservedRunningTime="2026-07-08 16:53:38.803561601 +0000 UTC m=+1131.075011141"
Jul 08 16:53:39.740135 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:39.740102    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd" event={"ID":"81937630-42ac-4e5d-8427-14687fe57624","Type":"ContainerStarted","Data":"1b3dabcc911ab2bf4e856a8345289cb2dc8f91e75cbb105b5ce04e02543dbac8"}
Jul 08 16:53:39.740534 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:39.740166    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd" event={"ID":"81937630-42ac-4e5d-8427-14687fe57624","Type":"ContainerStarted","Data":"82ed0018a83d42f3773d63ece789f5b8fe244ede307385cf095b2a32adce92a6"}
Jul 08 16:53:39.740534 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:39.740320    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb" podUID="eb53586a-ea54-4def-87dc-9bdc6baa1761" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 08 16:53:39.740619 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:39.740598    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd"
Jul 08 16:53:39.740739 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:39.740722    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd"
Jul 08 16:53:39.742070 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:39.742046    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd" podUID="81937630-42ac-4e5d-8427-14687fe57624" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 08 16:53:40.285453 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:40.285421    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa6f46ad-52fc-46d4-a305-394892cb18a1" path="/var/lib/kubelet/pods/aa6f46ad-52fc-46d4-a305-394892cb18a1/volumes"
Jul 08 16:53:40.744126 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:40.744081    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd" podUID="81937630-42ac-4e5d-8427-14687fe57624" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 08 16:53:44.745313 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:44.745281    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb"
Jul 08 16:53:44.745839 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:44.745812    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb" podUID="eb53586a-ea54-4def-87dc-9bdc6baa1761" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 08 16:53:44.766432 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:44.766377    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd" podStartSLOduration=8.170112347 podStartE2EDuration="11.766361249s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.596248902s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:53:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:53:39.772907627 +0000 UTC m=+1132.044357166" watchObservedRunningTime="2026-07-08 16:53:44.766361249 +0000 UTC m=+1137.037810788"
Jul 08 16:53:45.749549 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:45.749516    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd"
Jul 08 16:53:45.750055 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:45.750025    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd" podUID="81937630-42ac-4e5d-8427-14687fe57624" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 08 16:53:54.746237 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:54.746200    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb" podUID="eb53586a-ea54-4def-87dc-9bdc6baa1761" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 08 16:53:55.750372 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:53:55.750329    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd" podUID="81937630-42ac-4e5d-8427-14687fe57624" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 08 16:54:04.746763 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:54:04.746719    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb" podUID="eb53586a-ea54-4def-87dc-9bdc6baa1761" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 08 16:54:05.750989 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:54:05.750944    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd" podUID="81937630-42ac-4e5d-8427-14687fe57624" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 08 16:54:14.746651 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:54:14.746552    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb" podUID="eb53586a-ea54-4def-87dc-9bdc6baa1761" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 08 16:54:15.750708 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:54:15.750665    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd" podUID="81937630-42ac-4e5d-8427-14687fe57624" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 08 16:54:24.746331 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:54:24.746284    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb" podUID="eb53586a-ea54-4def-87dc-9bdc6baa1761" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 08 16:54:25.750042 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:54:25.750004    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd" podUID="81937630-42ac-4e5d-8427-14687fe57624" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 08 16:54:34.746279 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:54:34.746240    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb" podUID="eb53586a-ea54-4def-87dc-9bdc6baa1761" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 08 16:54:35.750855 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:54:35.750814    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd" podUID="81937630-42ac-4e5d-8427-14687fe57624" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 08 16:54:44.746743 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:54:44.746704    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb" podUID="eb53586a-ea54-4def-87dc-9bdc6baa1761" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 08 16:54:45.750795 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:54:45.750763    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd"
Jul 08 16:54:48.219930 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:54:48.219902    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-qn2bx_4717bdee-2017-4551-b0ec-683ab07f7660/console-operator/1.log"
Jul 08 16:54:48.220370 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:54:48.220093    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-qn2bx_4717bdee-2017-4551-b0ec-683ab07f7660/console-operator/1.log"
Jul 08 16:54:48.224552 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:54:48.224529    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2shvk_f08a1b00-78fd-43ef-b2aa-2712d09fb1b3/ovn-acl-logging/0.log"
Jul 08 16:54:48.224762 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:54:48.224748    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2shvk_f08a1b00-78fd-43ef-b2aa-2712d09fb1b3/ovn-acl-logging/0.log"
Jul 08 16:54:54.746426 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:54:54.746397    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb"
Jul 08 16:55:23.512954 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:23.512921    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb"]
Jul 08 16:55:23.513461 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:23.513238    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb" podUID="eb53586a-ea54-4def-87dc-9bdc6baa1761" containerName="kube-rbac-proxy" containerID="cri-o://30fcfa6e579def4071541ef2edec36cf2bc5dbb69269d12c010005598a9dcba0" gracePeriod=30
Jul 08 16:55:23.513461 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:23.513335    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb" podUID="eb53586a-ea54-4def-87dc-9bdc6baa1761" containerName="kserve-container" containerID="cri-o://3a9051349ecd553772e04a15304f46b4005f27904266fcfda0e28e78a0e9d3ff" gracePeriod=30
Jul 08 16:55:23.566288 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:23.566256    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-9f040-predictor-5458f85b97-zlgtn"]
Jul 08 16:55:23.566560 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:23.566547    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa6f46ad-52fc-46d4-a305-394892cb18a1" containerName="kserve-container"
Jul 08 16:55:23.566602 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:23.566562    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa6f46ad-52fc-46d4-a305-394892cb18a1" containerName="kserve-container"
Jul 08 16:55:23.566602 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:23.566574    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa6f46ad-52fc-46d4-a305-394892cb18a1" containerName="kube-rbac-proxy"
Jul 08 16:55:23.566602 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:23.566581    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa6f46ad-52fc-46d4-a305-394892cb18a1" containerName="kube-rbac-proxy"
Jul 08 16:55:23.566602 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:23.566594    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="435847a0-f882-4d91-9d21-e96f272fe5fb" containerName="kserve-container"
Jul 08 16:55:23.566602 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:23.566599    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="435847a0-f882-4d91-9d21-e96f272fe5fb" containerName="kserve-container"
Jul 08 16:55:23.566777 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:23.566608    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa6f46ad-52fc-46d4-a305-394892cb18a1" containerName="storage-initializer"
Jul 08 16:55:23.566777 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:23.566613    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa6f46ad-52fc-46d4-a305-394892cb18a1" containerName="storage-initializer"
Jul 08 16:55:23.566777 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:23.566624    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="435847a0-f882-4d91-9d21-e96f272fe5fb" containerName="storage-initializer"
Jul 08 16:55:23.566777 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:23.566645    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="435847a0-f882-4d91-9d21-e96f272fe5fb" containerName="storage-initializer"
Jul 08 16:55:23.566777 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:23.566653    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="435847a0-f882-4d91-9d21-e96f272fe5fb" containerName="kube-rbac-proxy"
Jul 08 16:55:23.566777 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:23.566658    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="435847a0-f882-4d91-9d21-e96f272fe5fb" containerName="kube-rbac-proxy"
Jul 08 16:55:23.566777 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:23.566705    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa6f46ad-52fc-46d4-a305-394892cb18a1" containerName="kserve-container"
Jul 08 16:55:23.566777 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:23.566713    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa6f46ad-52fc-46d4-a305-394892cb18a1" containerName="kube-rbac-proxy"
Jul 08 16:55:23.566777 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:23.566721    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="435847a0-f882-4d91-9d21-e96f272fe5fb" containerName="kube-rbac-proxy"
Jul 08 16:55:23.566777 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:23.566727    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="435847a0-f882-4d91-9d21-e96f272fe5fb" containerName="kserve-container"
Jul 08 16:55:23.569806 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:23.569790    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-9f040-predictor-5458f85b97-zlgtn"
Jul 08 16:55:23.572183 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:23.572162    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-9f040-predictor-serving-cert\""
Jul 08 16:55:23.572281 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:23.572162    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-9f040-kube-rbac-proxy-sar-config\""
Jul 08 16:55:23.580459 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:23.580438    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-9f040-predictor-5458f85b97-zlgtn"]
Jul 08 16:55:23.614316 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:23.614287    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2cgj\" (UniqueName: \"kubernetes.io/projected/4c6d57fc-3f16-4060-8fff-5279965ae841-kube-api-access-r2cgj\") pod \"message-dumper-raw-9f040-predictor-5458f85b97-zlgtn\" (UID: \"4c6d57fc-3f16-4060-8fff-5279965ae841\") " pod="kserve-ci-e2e-test/message-dumper-raw-9f040-predictor-5458f85b97-zlgtn"
Jul 08 16:55:23.614486 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:23.614331    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c6d57fc-3f16-4060-8fff-5279965ae841-proxy-tls\") pod \"message-dumper-raw-9f040-predictor-5458f85b97-zlgtn\" (UID: \"4c6d57fc-3f16-4060-8fff-5279965ae841\") " pod="kserve-ci-e2e-test/message-dumper-raw-9f040-predictor-5458f85b97-zlgtn"
Jul 08 16:55:23.614486 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:23.614463    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4c6d57fc-3f16-4060-8fff-5279965ae841-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-9f040-predictor-5458f85b97-zlgtn\" (UID: \"4c6d57fc-3f16-4060-8fff-5279965ae841\") " pod="kserve-ci-e2e-test/message-dumper-raw-9f040-predictor-5458f85b97-zlgtn"
Jul 08 16:55:23.629034 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:23.629003    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd"]
Jul 08 16:55:23.629325 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:23.629294    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd" podUID="81937630-42ac-4e5d-8427-14687fe57624" containerName="kserve-container" containerID="cri-o://82ed0018a83d42f3773d63ece789f5b8fe244ede307385cf095b2a32adce92a6" gracePeriod=30
Jul 08 16:55:23.629458 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:23.629306    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd" podUID="81937630-42ac-4e5d-8427-14687fe57624" containerName="kube-rbac-proxy" containerID="cri-o://1b3dabcc911ab2bf4e856a8345289cb2dc8f91e75cbb105b5ce04e02543dbac8" gracePeriod=30
Jul 08 16:55:23.715300 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:23.715266    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4c6d57fc-3f16-4060-8fff-5279965ae841-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-9f040-predictor-5458f85b97-zlgtn\" (UID: \"4c6d57fc-3f16-4060-8fff-5279965ae841\") " pod="kserve-ci-e2e-test/message-dumper-raw-9f040-predictor-5458f85b97-zlgtn"
Jul 08 16:55:23.715507 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:23.715332    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2cgj\" (UniqueName: \"kubernetes.io/projected/4c6d57fc-3f16-4060-8fff-5279965ae841-kube-api-access-r2cgj\") pod \"message-dumper-raw-9f040-predictor-5458f85b97-zlgtn\" (UID: \"4c6d57fc-3f16-4060-8fff-5279965ae841\") " pod="kserve-ci-e2e-test/message-dumper-raw-9f040-predictor-5458f85b97-zlgtn"
Jul 08 16:55:23.715507 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:23.715371    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c6d57fc-3f16-4060-8fff-5279965ae841-proxy-tls\") pod \"message-dumper-raw-9f040-predictor-5458f85b97-zlgtn\" (UID: \"4c6d57fc-3f16-4060-8fff-5279965ae841\") " pod="kserve-ci-e2e-test/message-dumper-raw-9f040-predictor-5458f85b97-zlgtn"
Jul 08 16:55:23.715660 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:55:23.715508    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-raw-9f040-predictor-serving-cert: secret "message-dumper-raw-9f040-predictor-serving-cert" not found
Jul 08 16:55:23.715660 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:55:23.715581    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c6d57fc-3f16-4060-8fff-5279965ae841-proxy-tls podName:4c6d57fc-3f16-4060-8fff-5279965ae841 nodeName:}" failed. No retries permitted until 2026-07-08 16:55:24.215559485 +0000 UTC m=+1236.487009014 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4c6d57fc-3f16-4060-8fff-5279965ae841-proxy-tls") pod "message-dumper-raw-9f040-predictor-5458f85b97-zlgtn" (UID: "4c6d57fc-3f16-4060-8fff-5279965ae841") : secret "message-dumper-raw-9f040-predictor-serving-cert" not found
Jul 08 16:55:23.715960 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:23.715939    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4c6d57fc-3f16-4060-8fff-5279965ae841-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-9f040-predictor-5458f85b97-zlgtn\" (UID: \"4c6d57fc-3f16-4060-8fff-5279965ae841\") " pod="kserve-ci-e2e-test/message-dumper-raw-9f040-predictor-5458f85b97-zlgtn"
Jul 08 16:55:23.727139 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:23.727108    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2cgj\" (UniqueName: \"kubernetes.io/projected/4c6d57fc-3f16-4060-8fff-5279965ae841-kube-api-access-r2cgj\") pod \"message-dumper-raw-9f040-predictor-5458f85b97-zlgtn\" (UID: \"4c6d57fc-3f16-4060-8fff-5279965ae841\") " pod="kserve-ci-e2e-test/message-dumper-raw-9f040-predictor-5458f85b97-zlgtn"
Jul 08 16:55:24.166964 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:24.166932    2578 generic.go:356] "Generic (PLEG): container finished" podID="eb53586a-ea54-4def-87dc-9bdc6baa1761" containerID="30fcfa6e579def4071541ef2edec36cf2bc5dbb69269d12c010005598a9dcba0" exitCode=2
Jul 08 16:55:24.167160 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:24.167007    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb" event={"ID":"eb53586a-ea54-4def-87dc-9bdc6baa1761","Type":"ContainerDied","Data":"30fcfa6e579def4071541ef2edec36cf2bc5dbb69269d12c010005598a9dcba0"}
Jul 08 16:55:24.168980 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:24.168960    2578 generic.go:356] "Generic (PLEG): container finished" podID="81937630-42ac-4e5d-8427-14687fe57624" containerID="1b3dabcc911ab2bf4e856a8345289cb2dc8f91e75cbb105b5ce04e02543dbac8" exitCode=2
Jul 08 16:55:24.169085 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:24.169000    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd" event={"ID":"81937630-42ac-4e5d-8427-14687fe57624","Type":"ContainerDied","Data":"1b3dabcc911ab2bf4e856a8345289cb2dc8f91e75cbb105b5ce04e02543dbac8"}
Jul 08 16:55:24.219618 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:24.219594    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c6d57fc-3f16-4060-8fff-5279965ae841-proxy-tls\") pod \"message-dumper-raw-9f040-predictor-5458f85b97-zlgtn\" (UID: \"4c6d57fc-3f16-4060-8fff-5279965ae841\") " pod="kserve-ci-e2e-test/message-dumper-raw-9f040-predictor-5458f85b97-zlgtn"
Jul 08 16:55:24.222080 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:24.222054    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c6d57fc-3f16-4060-8fff-5279965ae841-proxy-tls\") pod \"message-dumper-raw-9f040-predictor-5458f85b97-zlgtn\" (UID: \"4c6d57fc-3f16-4060-8fff-5279965ae841\") " pod="kserve-ci-e2e-test/message-dumper-raw-9f040-predictor-5458f85b97-zlgtn"
Jul 08 16:55:24.480449 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:24.480371    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-9f040-predictor-5458f85b97-zlgtn"
Jul 08 16:55:24.614248 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:24.614221    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-9f040-predictor-5458f85b97-zlgtn"]
Jul 08 16:55:24.616823 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:55:24.616795    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c6d57fc_3f16_4060_8fff_5279965ae841.slice/crio-88fbe7131ee8df4eafcfaa41ff91ccae4646ef390adc4e980da0d3240bbd712a WatchSource:0}: Error finding container 88fbe7131ee8df4eafcfaa41ff91ccae4646ef390adc4e980da0d3240bbd712a: Status 404 returned error can't find the container with id 88fbe7131ee8df4eafcfaa41ff91ccae4646ef390adc4e980da0d3240bbd712a
Jul 08 16:55:24.618824 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:24.618808    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 08 16:55:24.741444 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:24.741336    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb" podUID="eb53586a-ea54-4def-87dc-9bdc6baa1761" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 08 16:55:24.746763 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:24.746727    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb" podUID="eb53586a-ea54-4def-87dc-9bdc6baa1761" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 08 16:55:25.174701 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:25.174662    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-9f040-predictor-5458f85b97-zlgtn" event={"ID":"4c6d57fc-3f16-4060-8fff-5279965ae841","Type":"ContainerStarted","Data":"88fbe7131ee8df4eafcfaa41ff91ccae4646ef390adc4e980da0d3240bbd712a"}
Jul 08 16:55:25.745082 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:25.745032    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd" podUID="81937630-42ac-4e5d-8427-14687fe57624" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Jul 08 16:55:25.751032 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:25.750968    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd" podUID="81937630-42ac-4e5d-8427-14687fe57624" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 08 16:55:26.180757 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:26.180716    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-9f040-predictor-5458f85b97-zlgtn" event={"ID":"4c6d57fc-3f16-4060-8fff-5279965ae841","Type":"ContainerStarted","Data":"af87b51d0602d46abcbcb581e74c2a7437943f1d613e410657b3f6878ad919b1"}
Jul 08 16:55:26.180950 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:26.180804    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-9f040-predictor-5458f85b97-zlgtn" event={"ID":"4c6d57fc-3f16-4060-8fff-5279965ae841","Type":"ContainerStarted","Data":"9b3bc331b69d35e4578326be042d9de394749da889c53ffb4854e17eb963367b"}
Jul 08 16:55:26.180950 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:26.180833    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-9f040-predictor-5458f85b97-zlgtn"
Jul 08 16:55:26.180950 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:26.180872    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-9f040-predictor-5458f85b97-zlgtn"
Jul 08 16:55:26.182539 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:26.182513    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-9f040-predictor-5458f85b97-zlgtn"
Jul 08 16:55:26.200890 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:26.200837    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-9f040-predictor-5458f85b97-zlgtn" podStartSLOduration=1.8166100699999999 podStartE2EDuration="3.200814132s" totalImagesPullingTime="1.384204062s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:55:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:55:26.198408697 +0000 UTC m=+1238.469858239" watchObservedRunningTime="2026-07-08 16:55:26.200814132 +0000 UTC m=+1238.472263674"
Jul 08 16:55:27.578065 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:27.578039    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd"
Jul 08 16:55:27.648978 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:27.648886    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vnfmz\" (UniqueName: \"kubernetes.io/projected/81937630-42ac-4e5d-8427-14687fe57624-kube-api-access-vnfmz\") pod \"81937630-42ac-4e5d-8427-14687fe57624\" (UID: \"81937630-42ac-4e5d-8427-14687fe57624\") "
Jul 08 16:55:27.648978 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:27.648971    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81937630-42ac-4e5d-8427-14687fe57624-proxy-tls\") pod \"81937630-42ac-4e5d-8427-14687fe57624\" (UID: \"81937630-42ac-4e5d-8427-14687fe57624\") "
Jul 08 16:55:27.649169 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:27.649003    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81937630-42ac-4e5d-8427-14687fe57624-kserve-provision-location\") pod \"81937630-42ac-4e5d-8427-14687fe57624\" (UID: \"81937630-42ac-4e5d-8427-14687fe57624\") "
Jul 08 16:55:27.649169 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:27.649045    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81937630-42ac-4e5d-8427-14687fe57624-kube-rbac-proxy-sar-config\") pod \"81937630-42ac-4e5d-8427-14687fe57624\" (UID: \"81937630-42ac-4e5d-8427-14687fe57624\") "
Jul 08 16:55:27.649169 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:27.649091    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81937630-42ac-4e5d-8427-14687fe57624-cabundle-cert\") pod \"81937630-42ac-4e5d-8427-14687fe57624\" (UID: \"81937630-42ac-4e5d-8427-14687fe57624\") "
Jul 08 16:55:27.649391 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:27.649356    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81937630-42ac-4e5d-8427-14687fe57624-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "81937630-42ac-4e5d-8427-14687fe57624" (UID: "81937630-42ac-4e5d-8427-14687fe57624"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:55:27.649508 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:27.649479    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81937630-42ac-4e5d-8427-14687fe57624-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "81937630-42ac-4e5d-8427-14687fe57624" (UID: "81937630-42ac-4e5d-8427-14687fe57624"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:55:27.649619 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:27.649589    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81937630-42ac-4e5d-8427-14687fe57624-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "81937630-42ac-4e5d-8427-14687fe57624" (UID: "81937630-42ac-4e5d-8427-14687fe57624"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:55:27.651221 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:27.651193    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81937630-42ac-4e5d-8427-14687fe57624-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "81937630-42ac-4e5d-8427-14687fe57624" (UID: "81937630-42ac-4e5d-8427-14687fe57624"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:55:27.651307 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:27.651271    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81937630-42ac-4e5d-8427-14687fe57624-kube-api-access-vnfmz" (OuterVolumeSpecName: "kube-api-access-vnfmz") pod "81937630-42ac-4e5d-8427-14687fe57624" (UID: "81937630-42ac-4e5d-8427-14687fe57624"). InnerVolumeSpecName "kube-api-access-vnfmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:55:27.749895 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:27.749855    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81937630-42ac-4e5d-8427-14687fe57624-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:55:27.749895 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:27.749889    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81937630-42ac-4e5d-8427-14687fe57624-cabundle-cert\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:55:27.749895 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:27.749899    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnfmz\" (UniqueName: \"kubernetes.io/projected/81937630-42ac-4e5d-8427-14687fe57624-kube-api-access-vnfmz\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:55:27.750126 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:27.749908    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81937630-42ac-4e5d-8427-14687fe57624-proxy-tls\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:55:27.750126 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:27.749919    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81937630-42ac-4e5d-8427-14687fe57624-kserve-provision-location\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:55:28.191109 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:28.191065    2578 generic.go:356] "Generic (PLEG): container finished" podID="81937630-42ac-4e5d-8427-14687fe57624" containerID="82ed0018a83d42f3773d63ece789f5b8fe244ede307385cf095b2a32adce92a6" exitCode=0
Jul 08 16:55:28.191294 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:28.191125    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd"
Jul 08 16:55:28.191294 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:28.191156    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd" event={"ID":"81937630-42ac-4e5d-8427-14687fe57624","Type":"ContainerDied","Data":"82ed0018a83d42f3773d63ece789f5b8fe244ede307385cf095b2a32adce92a6"}
Jul 08 16:55:28.191294 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:28.191211    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd" event={"ID":"81937630-42ac-4e5d-8427-14687fe57624","Type":"ContainerDied","Data":"702810faa075a9f36f6741f998dedc485ecb7128e2f0105fac425251923b250a"}
Jul 08 16:55:28.191294 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:28.191237    2578 scope.go:117] "RemoveContainer" containerID="1b3dabcc911ab2bf4e856a8345289cb2dc8f91e75cbb105b5ce04e02543dbac8"
Jul 08 16:55:28.200589 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:28.200567    2578 scope.go:117] "RemoveContainer" containerID="82ed0018a83d42f3773d63ece789f5b8fe244ede307385cf095b2a32adce92a6"
Jul 08 16:55:28.208776 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:28.208756    2578 scope.go:117] "RemoveContainer" containerID="96f7d09ad5eddf2112a9211df325489c2114f61514cd21e6ffa9d50d4df846ac"
Jul 08 16:55:28.214270 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:28.214245    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd"]
Jul 08 16:55:28.217742 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:28.217705    2578 scope.go:117] "RemoveContainer" containerID="1b3dabcc911ab2bf4e856a8345289cb2dc8f91e75cbb105b5ce04e02543dbac8"
Jul 08 16:55:28.218095 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:55:28.218074    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b3dabcc911ab2bf4e856a8345289cb2dc8f91e75cbb105b5ce04e02543dbac8\": container with ID starting with 1b3dabcc911ab2bf4e856a8345289cb2dc8f91e75cbb105b5ce04e02543dbac8 not found: ID does not exist" containerID="1b3dabcc911ab2bf4e856a8345289cb2dc8f91e75cbb105b5ce04e02543dbac8"
Jul 08 16:55:28.218166 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:28.218104    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b3dabcc911ab2bf4e856a8345289cb2dc8f91e75cbb105b5ce04e02543dbac8"} err="failed to get container status \"1b3dabcc911ab2bf4e856a8345289cb2dc8f91e75cbb105b5ce04e02543dbac8\": rpc error: code = NotFound desc = could not find container \"1b3dabcc911ab2bf4e856a8345289cb2dc8f91e75cbb105b5ce04e02543dbac8\": container with ID starting with 1b3dabcc911ab2bf4e856a8345289cb2dc8f91e75cbb105b5ce04e02543dbac8 not found: ID does not exist"
Jul 08 16:55:28.218166 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:28.218126    2578 scope.go:117] "RemoveContainer" containerID="82ed0018a83d42f3773d63ece789f5b8fe244ede307385cf095b2a32adce92a6"
Jul 08 16:55:28.218351 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:28.218328    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-3e871-predictor-f9756c6d9-h6bkd"]
Jul 08 16:55:28.218402 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:55:28.218386    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82ed0018a83d42f3773d63ece789f5b8fe244ede307385cf095b2a32adce92a6\": container with ID starting with 82ed0018a83d42f3773d63ece789f5b8fe244ede307385cf095b2a32adce92a6 not found: ID does not exist" containerID="82ed0018a83d42f3773d63ece789f5b8fe244ede307385cf095b2a32adce92a6"
Jul 08 16:55:28.218439 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:28.218410    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82ed0018a83d42f3773d63ece789f5b8fe244ede307385cf095b2a32adce92a6"} err="failed to get container status \"82ed0018a83d42f3773d63ece789f5b8fe244ede307385cf095b2a32adce92a6\": rpc error: code = NotFound desc = could not find container \"82ed0018a83d42f3773d63ece789f5b8fe244ede307385cf095b2a32adce92a6\": container with ID starting with 82ed0018a83d42f3773d63ece789f5b8fe244ede307385cf095b2a32adce92a6 not found: ID does not exist"
Jul 08 16:55:28.218439 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:28.218423    2578 scope.go:117] "RemoveContainer" containerID="96f7d09ad5eddf2112a9211df325489c2114f61514cd21e6ffa9d50d4df846ac"
Jul 08 16:55:28.218675 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:55:28.218659    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96f7d09ad5eddf2112a9211df325489c2114f61514cd21e6ffa9d50d4df846ac\": container with ID starting with 96f7d09ad5eddf2112a9211df325489c2114f61514cd21e6ffa9d50d4df846ac not found: ID does not exist" containerID="96f7d09ad5eddf2112a9211df325489c2114f61514cd21e6ffa9d50d4df846ac"
Jul 08 16:55:28.218729 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:28.218678    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96f7d09ad5eddf2112a9211df325489c2114f61514cd21e6ffa9d50d4df846ac"} err="failed to get container status \"96f7d09ad5eddf2112a9211df325489c2114f61514cd21e6ffa9d50d4df846ac\": rpc error: code = NotFound desc = could not find container \"96f7d09ad5eddf2112a9211df325489c2114f61514cd21e6ffa9d50d4df846ac\": container with ID starting with 96f7d09ad5eddf2112a9211df325489c2114f61514cd21e6ffa9d50d4df846ac not found: ID does not exist"
Jul 08 16:55:28.285095 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:28.285066    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81937630-42ac-4e5d-8427-14687fe57624" path="/var/lib/kubelet/pods/81937630-42ac-4e5d-8427-14687fe57624/volumes"
Jul 08 16:55:28.658791 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:28.658767    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb"
Jul 08 16:55:28.757258 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:28.757153    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb53586a-ea54-4def-87dc-9bdc6baa1761-kube-rbac-proxy-sar-config\") pod \"eb53586a-ea54-4def-87dc-9bdc6baa1761\" (UID: \"eb53586a-ea54-4def-87dc-9bdc6baa1761\") "
Jul 08 16:55:28.757258 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:28.757209    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb53586a-ea54-4def-87dc-9bdc6baa1761-proxy-tls\") pod \"eb53586a-ea54-4def-87dc-9bdc6baa1761\" (UID: \"eb53586a-ea54-4def-87dc-9bdc6baa1761\") "
Jul 08 16:55:28.757258 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:28.757242    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dz7g4\" (UniqueName: \"kubernetes.io/projected/eb53586a-ea54-4def-87dc-9bdc6baa1761-kube-api-access-dz7g4\") pod \"eb53586a-ea54-4def-87dc-9bdc6baa1761\" (UID: \"eb53586a-ea54-4def-87dc-9bdc6baa1761\") "
Jul 08 16:55:28.757554 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:28.757311    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb53586a-ea54-4def-87dc-9bdc6baa1761-cabundle-cert\") pod \"eb53586a-ea54-4def-87dc-9bdc6baa1761\" (UID: \"eb53586a-ea54-4def-87dc-9bdc6baa1761\") "
Jul 08 16:55:28.757554 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:28.757361    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb53586a-ea54-4def-87dc-9bdc6baa1761-kserve-provision-location\") pod \"eb53586a-ea54-4def-87dc-9bdc6baa1761\" (UID: \"eb53586a-ea54-4def-87dc-9bdc6baa1761\") "
Jul 08 16:55:28.757776 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:28.757597    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb53586a-ea54-4def-87dc-9bdc6baa1761-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "eb53586a-ea54-4def-87dc-9bdc6baa1761" (UID: "eb53586a-ea54-4def-87dc-9bdc6baa1761"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:55:28.757776 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:28.757709    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb53586a-ea54-4def-87dc-9bdc6baa1761-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "eb53586a-ea54-4def-87dc-9bdc6baa1761" (UID: "eb53586a-ea54-4def-87dc-9bdc6baa1761"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:55:28.757863 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:28.757786    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb53586a-ea54-4def-87dc-9bdc6baa1761-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "eb53586a-ea54-4def-87dc-9bdc6baa1761" (UID: "eb53586a-ea54-4def-87dc-9bdc6baa1761"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:55:28.759762 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:28.759724    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb53586a-ea54-4def-87dc-9bdc6baa1761-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "eb53586a-ea54-4def-87dc-9bdc6baa1761" (UID: "eb53586a-ea54-4def-87dc-9bdc6baa1761"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:55:28.759884 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:28.759776    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb53586a-ea54-4def-87dc-9bdc6baa1761-kube-api-access-dz7g4" (OuterVolumeSpecName: "kube-api-access-dz7g4") pod "eb53586a-ea54-4def-87dc-9bdc6baa1761" (UID: "eb53586a-ea54-4def-87dc-9bdc6baa1761"). InnerVolumeSpecName "kube-api-access-dz7g4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:55:28.858683 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:28.858616    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb53586a-ea54-4def-87dc-9bdc6baa1761-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:55:28.858683 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:28.858677    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb53586a-ea54-4def-87dc-9bdc6baa1761-proxy-tls\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:55:28.858683 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:28.858688    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dz7g4\" (UniqueName: \"kubernetes.io/projected/eb53586a-ea54-4def-87dc-9bdc6baa1761-kube-api-access-dz7g4\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:55:28.858928 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:28.858697    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb53586a-ea54-4def-87dc-9bdc6baa1761-cabundle-cert\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:55:28.858928 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:28.858706    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb53586a-ea54-4def-87dc-9bdc6baa1761-kserve-provision-location\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:55:29.198403 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:29.198367    2578 generic.go:356] "Generic (PLEG): container finished" podID="eb53586a-ea54-4def-87dc-9bdc6baa1761" containerID="3a9051349ecd553772e04a15304f46b4005f27904266fcfda0e28e78a0e9d3ff" exitCode=0
Jul 08 16:55:29.198612 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:29.198414    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb" event={"ID":"eb53586a-ea54-4def-87dc-9bdc6baa1761","Type":"ContainerDied","Data":"3a9051349ecd553772e04a15304f46b4005f27904266fcfda0e28e78a0e9d3ff"}
Jul 08 16:55:29.198612 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:29.198433    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb"
Jul 08 16:55:29.198612 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:29.198461    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb" event={"ID":"eb53586a-ea54-4def-87dc-9bdc6baa1761","Type":"ContainerDied","Data":"a8c312c276d0285d3c9fb5fc4f49d5be885e83823a12ae1d533e4f4d02fabe24"}
Jul 08 16:55:29.198612 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:29.198490    2578 scope.go:117] "RemoveContainer" containerID="30fcfa6e579def4071541ef2edec36cf2bc5dbb69269d12c010005598a9dcba0"
Jul 08 16:55:29.206870 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:29.206851    2578 scope.go:117] "RemoveContainer" containerID="3a9051349ecd553772e04a15304f46b4005f27904266fcfda0e28e78a0e9d3ff"
Jul 08 16:55:29.214772 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:29.214753    2578 scope.go:117] "RemoveContainer" containerID="d5ed5ce94f93e5c58bd89712d9852f7c1f75de3960d8c245d97922e362ad2c5d"
Jul 08 16:55:29.222555 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:29.222527    2578 scope.go:117] "RemoveContainer" containerID="30fcfa6e579def4071541ef2edec36cf2bc5dbb69269d12c010005598a9dcba0"
Jul 08 16:55:29.222689 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:29.222587    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb"]
Jul 08 16:55:29.222933 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:55:29.222912    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30fcfa6e579def4071541ef2edec36cf2bc5dbb69269d12c010005598a9dcba0\": container with ID starting with 30fcfa6e579def4071541ef2edec36cf2bc5dbb69269d12c010005598a9dcba0 not found: ID does not exist" containerID="30fcfa6e579def4071541ef2edec36cf2bc5dbb69269d12c010005598a9dcba0"
Jul 08 16:55:29.223011 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:29.222945    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30fcfa6e579def4071541ef2edec36cf2bc5dbb69269d12c010005598a9dcba0"} err="failed to get container status \"30fcfa6e579def4071541ef2edec36cf2bc5dbb69269d12c010005598a9dcba0\": rpc error: code = NotFound desc = could not find container \"30fcfa6e579def4071541ef2edec36cf2bc5dbb69269d12c010005598a9dcba0\": container with ID starting with 30fcfa6e579def4071541ef2edec36cf2bc5dbb69269d12c010005598a9dcba0 not found: ID does not exist"
Jul 08 16:55:29.223011 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:29.222973    2578 scope.go:117] "RemoveContainer" containerID="3a9051349ecd553772e04a15304f46b4005f27904266fcfda0e28e78a0e9d3ff"
Jul 08 16:55:29.223307 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:55:29.223290    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a9051349ecd553772e04a15304f46b4005f27904266fcfda0e28e78a0e9d3ff\": container with ID starting with 3a9051349ecd553772e04a15304f46b4005f27904266fcfda0e28e78a0e9d3ff not found: ID does not exist" containerID="3a9051349ecd553772e04a15304f46b4005f27904266fcfda0e28e78a0e9d3ff"
Jul 08 16:55:29.223352 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:29.223314    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a9051349ecd553772e04a15304f46b4005f27904266fcfda0e28e78a0e9d3ff"} err="failed to get container status \"3a9051349ecd553772e04a15304f46b4005f27904266fcfda0e28e78a0e9d3ff\": rpc error: code = NotFound desc = could not find container \"3a9051349ecd553772e04a15304f46b4005f27904266fcfda0e28e78a0e9d3ff\": container with ID starting with 3a9051349ecd553772e04a15304f46b4005f27904266fcfda0e28e78a0e9d3ff not found: ID does not exist"
Jul 08 16:55:29.223352 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:29.223334    2578 scope.go:117] "RemoveContainer" containerID="d5ed5ce94f93e5c58bd89712d9852f7c1f75de3960d8c245d97922e362ad2c5d"
Jul 08 16:55:29.223625 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:55:29.223606    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5ed5ce94f93e5c58bd89712d9852f7c1f75de3960d8c245d97922e362ad2c5d\": container with ID starting with d5ed5ce94f93e5c58bd89712d9852f7c1f75de3960d8c245d97922e362ad2c5d not found: ID does not exist" containerID="d5ed5ce94f93e5c58bd89712d9852f7c1f75de3960d8c245d97922e362ad2c5d"
Jul 08 16:55:29.223688 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:29.223657    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5ed5ce94f93e5c58bd89712d9852f7c1f75de3960d8c245d97922e362ad2c5d"} err="failed to get container status \"d5ed5ce94f93e5c58bd89712d9852f7c1f75de3960d8c245d97922e362ad2c5d\": rpc error: code = NotFound desc = could not find container \"d5ed5ce94f93e5c58bd89712d9852f7c1f75de3960d8c245d97922e362ad2c5d\": container with ID starting with d5ed5ce94f93e5c58bd89712d9852f7c1f75de3960d8c245d97922e362ad2c5d not found: ID does not exist"
Jul 08 16:55:29.229672 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:29.229622    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3e871-predictor-78f774c878-5v2kb"]
Jul 08 16:55:30.285803 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:30.285763    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb53586a-ea54-4def-87dc-9bdc6baa1761" path="/var/lib/kubelet/pods/eb53586a-ea54-4def-87dc-9bdc6baa1761/volumes"
Jul 08 16:55:33.197173 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:33.197142    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-9f040-predictor-5458f85b97-zlgtn"
Jul 08 16:55:33.603078 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:33.603039    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c"]
Jul 08 16:55:33.603352 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:33.603339    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb53586a-ea54-4def-87dc-9bdc6baa1761" containerName="kserve-container"
Jul 08 16:55:33.603410 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:33.603355    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb53586a-ea54-4def-87dc-9bdc6baa1761" containerName="kserve-container"
Jul 08 16:55:33.603410 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:33.603364    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb53586a-ea54-4def-87dc-9bdc6baa1761" containerName="kube-rbac-proxy"
Jul 08 16:55:33.603410 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:33.603370    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb53586a-ea54-4def-87dc-9bdc6baa1761" containerName="kube-rbac-proxy"
Jul 08 16:55:33.603410 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:33.603382    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81937630-42ac-4e5d-8427-14687fe57624" containerName="kserve-container"
Jul 08 16:55:33.603410 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:33.603387    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="81937630-42ac-4e5d-8427-14687fe57624" containerName="kserve-container"
Jul 08 16:55:33.603410 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:33.603399    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81937630-42ac-4e5d-8427-14687fe57624" containerName="storage-initializer"
Jul 08 16:55:33.603410 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:33.603404    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="81937630-42ac-4e5d-8427-14687fe57624" containerName="storage-initializer"
Jul 08 16:55:33.603410 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:33.603413    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb53586a-ea54-4def-87dc-9bdc6baa1761" containerName="storage-initializer"
Jul 08 16:55:33.603711 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:33.603418    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb53586a-ea54-4def-87dc-9bdc6baa1761" containerName="storage-initializer"
Jul 08 16:55:33.603711 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:33.603429    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81937630-42ac-4e5d-8427-14687fe57624" containerName="kube-rbac-proxy"
Jul 08 16:55:33.603711 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:33.603434    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="81937630-42ac-4e5d-8427-14687fe57624" containerName="kube-rbac-proxy"
Jul 08 16:55:33.603711 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:33.603479    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb53586a-ea54-4def-87dc-9bdc6baa1761" containerName="kube-rbac-proxy"
Jul 08 16:55:33.603711 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:33.603488    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="81937630-42ac-4e5d-8427-14687fe57624" containerName="kserve-container"
Jul 08 16:55:33.603711 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:33.603496    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="81937630-42ac-4e5d-8427-14687fe57624" containerName="kube-rbac-proxy"
Jul 08 16:55:33.603711 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:33.603503    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb53586a-ea54-4def-87dc-9bdc6baa1761" containerName="kserve-container"
Jul 08 16:55:33.608194 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:33.608170    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c"
Jul 08 16:55:33.610660 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:33.610608    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-9f040-predictor-serving-cert\""
Jul 08 16:55:33.610811 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:33.610659    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-9f040-kube-rbac-proxy-sar-config\""
Jul 08 16:55:33.610811 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:33.610676    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 08 16:55:33.610811 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:33.610664    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 08 16:55:33.617249 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:33.617213    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c"]
Jul 08 16:55:33.698496 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:33.698443    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kthqd\" (UniqueName: \"kubernetes.io/projected/150db917-4ee1-488e-8070-c11d7057d3e1-kube-api-access-kthqd\") pod \"isvc-logger-raw-9f040-predictor-758d49c58-7gb6c\" (UID: \"150db917-4ee1-488e-8070-c11d7057d3e1\") " pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c"
Jul 08 16:55:33.698768 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:33.698506    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/150db917-4ee1-488e-8070-c11d7057d3e1-cabundle-cert\") pod \"isvc-logger-raw-9f040-predictor-758d49c58-7gb6c\" (UID: \"150db917-4ee1-488e-8070-c11d7057d3e1\") " pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c"
Jul 08 16:55:33.698768 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:33.698583    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/150db917-4ee1-488e-8070-c11d7057d3e1-kserve-provision-location\") pod \"isvc-logger-raw-9f040-predictor-758d49c58-7gb6c\" (UID: \"150db917-4ee1-488e-8070-c11d7057d3e1\") " pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c"
Jul 08 16:55:33.698768 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:33.698679    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/150db917-4ee1-488e-8070-c11d7057d3e1-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-9f040-predictor-758d49c58-7gb6c\" (UID: \"150db917-4ee1-488e-8070-c11d7057d3e1\") " pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c"
Jul 08 16:55:33.698768 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:33.698718    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/150db917-4ee1-488e-8070-c11d7057d3e1-proxy-tls\") pod \"isvc-logger-raw-9f040-predictor-758d49c58-7gb6c\" (UID: \"150db917-4ee1-488e-8070-c11d7057d3e1\") " pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c"
Jul 08 16:55:33.799816 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:33.799764    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kthqd\" (UniqueName: \"kubernetes.io/projected/150db917-4ee1-488e-8070-c11d7057d3e1-kube-api-access-kthqd\") pod \"isvc-logger-raw-9f040-predictor-758d49c58-7gb6c\" (UID: \"150db917-4ee1-488e-8070-c11d7057d3e1\") " pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c"
Jul 08 16:55:33.800032 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:33.799830    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/150db917-4ee1-488e-8070-c11d7057d3e1-cabundle-cert\") pod \"isvc-logger-raw-9f040-predictor-758d49c58-7gb6c\" (UID: \"150db917-4ee1-488e-8070-c11d7057d3e1\") " pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c"
Jul 08 16:55:33.800032 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:33.799865    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/150db917-4ee1-488e-8070-c11d7057d3e1-kserve-provision-location\") pod \"isvc-logger-raw-9f040-predictor-758d49c58-7gb6c\" (UID: \"150db917-4ee1-488e-8070-c11d7057d3e1\") " pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c"
Jul 08 16:55:33.800032 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:33.799898    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/150db917-4ee1-488e-8070-c11d7057d3e1-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-9f040-predictor-758d49c58-7gb6c\" (UID: \"150db917-4ee1-488e-8070-c11d7057d3e1\") " pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c"
Jul 08 16:55:33.800032 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:33.799928    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/150db917-4ee1-488e-8070-c11d7057d3e1-proxy-tls\") pod \"isvc-logger-raw-9f040-predictor-758d49c58-7gb6c\" (UID: \"150db917-4ee1-488e-8070-c11d7057d3e1\") " pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c"
Jul 08 16:55:33.800256 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:55:33.800094    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-serving-cert: secret "isvc-logger-raw-9f040-predictor-serving-cert" not found
Jul 08 16:55:33.800256 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:55:33.800169    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/150db917-4ee1-488e-8070-c11d7057d3e1-proxy-tls podName:150db917-4ee1-488e-8070-c11d7057d3e1 nodeName:}" failed. No retries permitted until 2026-07-08 16:55:34.300148239 +0000 UTC m=+1246.571597773 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/150db917-4ee1-488e-8070-c11d7057d3e1-proxy-tls") pod "isvc-logger-raw-9f040-predictor-758d49c58-7gb6c" (UID: "150db917-4ee1-488e-8070-c11d7057d3e1") : secret "isvc-logger-raw-9f040-predictor-serving-cert" not found
Jul 08 16:55:33.800365 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:33.800350    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/150db917-4ee1-488e-8070-c11d7057d3e1-kserve-provision-location\") pod \"isvc-logger-raw-9f040-predictor-758d49c58-7gb6c\" (UID: \"150db917-4ee1-488e-8070-c11d7057d3e1\") " pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c"
Jul 08 16:55:33.800556 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:33.800537    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/150db917-4ee1-488e-8070-c11d7057d3e1-cabundle-cert\") pod \"isvc-logger-raw-9f040-predictor-758d49c58-7gb6c\" (UID: \"150db917-4ee1-488e-8070-c11d7057d3e1\") " pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c"
Jul 08 16:55:33.800599 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:33.800589    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/150db917-4ee1-488e-8070-c11d7057d3e1-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-9f040-predictor-758d49c58-7gb6c\" (UID: \"150db917-4ee1-488e-8070-c11d7057d3e1\") " pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c"
Jul 08 16:55:33.808880 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:33.808849    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kthqd\" (UniqueName: \"kubernetes.io/projected/150db917-4ee1-488e-8070-c11d7057d3e1-kube-api-access-kthqd\") pod \"isvc-logger-raw-9f040-predictor-758d49c58-7gb6c\" (UID: \"150db917-4ee1-488e-8070-c11d7057d3e1\") " pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c"
Jul 08 16:55:34.304406 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:34.304362    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/150db917-4ee1-488e-8070-c11d7057d3e1-proxy-tls\") pod \"isvc-logger-raw-9f040-predictor-758d49c58-7gb6c\" (UID: \"150db917-4ee1-488e-8070-c11d7057d3e1\") " pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c"
Jul 08 16:55:34.307006 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:34.306967    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/150db917-4ee1-488e-8070-c11d7057d3e1-proxy-tls\") pod \"isvc-logger-raw-9f040-predictor-758d49c58-7gb6c\" (UID: \"150db917-4ee1-488e-8070-c11d7057d3e1\") " pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c"
Jul 08 16:55:34.518810 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:34.518766    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c"
Jul 08 16:55:34.664830 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:34.664782    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c"]
Jul 08 16:55:34.667285 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:55:34.667255    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod150db917_4ee1_488e_8070_c11d7057d3e1.slice/crio-01feaf62d406abfd024f72502ef159c6e9662a40caf0c0983932b428e5e89265 WatchSource:0}: Error finding container 01feaf62d406abfd024f72502ef159c6e9662a40caf0c0983932b428e5e89265: Status 404 returned error can't find the container with id 01feaf62d406abfd024f72502ef159c6e9662a40caf0c0983932b428e5e89265
Jul 08 16:55:35.228124 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:35.228083    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c" event={"ID":"150db917-4ee1-488e-8070-c11d7057d3e1","Type":"ContainerStarted","Data":"fe04b1c71c5c2cf7fc8b9ccd106c08184f78d7a36b414387d9700b18b4c9787d"}
Jul 08 16:55:35.228311 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:35.228161    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c" event={"ID":"150db917-4ee1-488e-8070-c11d7057d3e1","Type":"ContainerStarted","Data":"01feaf62d406abfd024f72502ef159c6e9662a40caf0c0983932b428e5e89265"}
Jul 08 16:55:39.246343 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:39.246305    2578 generic.go:356] "Generic (PLEG): container finished" podID="150db917-4ee1-488e-8070-c11d7057d3e1" containerID="fe04b1c71c5c2cf7fc8b9ccd106c08184f78d7a36b414387d9700b18b4c9787d" exitCode=0
Jul 08 16:55:39.246863 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:39.246369    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c" event={"ID":"150db917-4ee1-488e-8070-c11d7057d3e1","Type":"ContainerDied","Data":"fe04b1c71c5c2cf7fc8b9ccd106c08184f78d7a36b414387d9700b18b4c9787d"}
Jul 08 16:55:40.253227 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:40.253191    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c" event={"ID":"150db917-4ee1-488e-8070-c11d7057d3e1","Type":"ContainerStarted","Data":"7ff041221171e314457b4ef728aeaf6eca8f0cb21fa72d43b83276f793f7f12c"}
Jul 08 16:55:40.253674 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:40.253259    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c" event={"ID":"150db917-4ee1-488e-8070-c11d7057d3e1","Type":"ContainerStarted","Data":"a0e0fe74b051ecb0fb0507e0b52610f0d00325ae9d7b3f0e8a1e6cbfe64aef7f"}
Jul 08 16:55:40.253674 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:40.253283    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c" event={"ID":"150db917-4ee1-488e-8070-c11d7057d3e1","Type":"ContainerStarted","Data":"7dfebaa3db6433550031e135c90b2daaf0c18f92f2445f114855697f5756db6f"}
Jul 08 16:55:40.253794 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:40.253726    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c"
Jul 08 16:55:40.253848 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:40.253795    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c"
Jul 08 16:55:40.253848 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:40.253839    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c"
Jul 08 16:55:40.255331 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:40.255298    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c" podUID="150db917-4ee1-488e-8070-c11d7057d3e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 08 16:55:40.256107 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:40.256083    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c" podUID="150db917-4ee1-488e-8070-c11d7057d3e1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:55:40.281077 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:40.281026    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c" podStartSLOduration=3.508782284 podStartE2EDuration="7.281010854s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.77222857s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:55:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:55:40.279536911 +0000 UTC m=+1252.550986451" watchObservedRunningTime="2026-07-08 16:55:40.281010854 +0000 UTC m=+1252.552460394"
Jul 08 16:55:41.257820 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:41.257767    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c" podUID="150db917-4ee1-488e-8070-c11d7057d3e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 08 16:55:41.258340 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:41.258312    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c" podUID="150db917-4ee1-488e-8070-c11d7057d3e1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:55:46.262531 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:46.262496    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c"
Jul 08 16:55:46.263148 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:46.263098    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c" podUID="150db917-4ee1-488e-8070-c11d7057d3e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 08 16:55:46.263428 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:46.263403    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c" podUID="150db917-4ee1-488e-8070-c11d7057d3e1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:55:56.263322 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:56.263272    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c" podUID="150db917-4ee1-488e-8070-c11d7057d3e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 08 16:55:56.263853 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:55:56.263769    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c" podUID="150db917-4ee1-488e-8070-c11d7057d3e1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:56:06.263931 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:56:06.263879    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c" podUID="150db917-4ee1-488e-8070-c11d7057d3e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 08 16:56:06.264419 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:56:06.264357    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c" podUID="150db917-4ee1-488e-8070-c11d7057d3e1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:56:16.263175 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:56:16.263129    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c" podUID="150db917-4ee1-488e-8070-c11d7057d3e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 08 16:56:16.263700 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:56:16.263678    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c" podUID="150db917-4ee1-488e-8070-c11d7057d3e1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:56:26.263335 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:56:26.263290    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c" podUID="150db917-4ee1-488e-8070-c11d7057d3e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 08 16:56:26.263785 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:56:26.263763    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c" podUID="150db917-4ee1-488e-8070-c11d7057d3e1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:56:36.263090 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:56:36.263042    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c" podUID="150db917-4ee1-488e-8070-c11d7057d3e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 08 16:56:36.263615 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:56:36.263508    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c" podUID="150db917-4ee1-488e-8070-c11d7057d3e1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:56:46.263341 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:56:46.263292    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c" podUID="150db917-4ee1-488e-8070-c11d7057d3e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 08 16:56:46.263871 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:56:46.263847    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c" podUID="150db917-4ee1-488e-8070-c11d7057d3e1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:56:56.264458 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:56:56.264422    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c"
Jul 08 16:56:56.264979 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:56:56.264675    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c"
Jul 08 16:57:08.638570 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:08.638481    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-9f040-predictor-5458f85b97-zlgtn_4c6d57fc-3f16-4060-8fff-5279965ae841/kserve-container/0.log"
Jul 08 16:57:08.871132 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:08.871097    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c"]
Jul 08 16:57:08.871534 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:08.871474    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c" podUID="150db917-4ee1-488e-8070-c11d7057d3e1" containerName="agent" containerID="cri-o://7ff041221171e314457b4ef728aeaf6eca8f0cb21fa72d43b83276f793f7f12c" gracePeriod=30
Jul 08 16:57:08.871690 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:08.871521    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c" podUID="150db917-4ee1-488e-8070-c11d7057d3e1" containerName="kube-rbac-proxy" containerID="cri-o://a0e0fe74b051ecb0fb0507e0b52610f0d00325ae9d7b3f0e8a1e6cbfe64aef7f" gracePeriod=30
Jul 08 16:57:08.871690 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:08.871487    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c" podUID="150db917-4ee1-488e-8070-c11d7057d3e1" containerName="kserve-container" containerID="cri-o://7dfebaa3db6433550031e135c90b2daaf0c18f92f2445f114855697f5756db6f" gracePeriod=30
Jul 08 16:57:08.976113 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:08.976069    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr"]
Jul 08 16:57:08.982170 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:08.982139    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr"
Jul 08 16:57:08.984664 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:08.984621    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-c7447-predictor-serving-cert\""
Jul 08 16:57:08.984812 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:08.984622    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-c7447-kube-rbac-proxy-sar-config\""
Jul 08 16:57:08.990464 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:08.990430    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr"]
Jul 08 16:57:09.002735 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:09.002698    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3b76551-8e16-4370-a7f3-a5a91577cd3b-cabundle-cert\") pod \"isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr\" (UID: \"b3b76551-8e16-4370-a7f3-a5a91577cd3b\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr"
Jul 08 16:57:09.002942 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:09.002765    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3b76551-8e16-4370-a7f3-a5a91577cd3b-proxy-tls\") pod \"isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr\" (UID: \"b3b76551-8e16-4370-a7f3-a5a91577cd3b\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr"
Jul 08 16:57:09.002942 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:09.002864    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2cjn\" (UniqueName: \"kubernetes.io/projected/b3b76551-8e16-4370-a7f3-a5a91577cd3b-kube-api-access-z2cjn\") pod \"isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr\" (UID: \"b3b76551-8e16-4370-a7f3-a5a91577cd3b\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr"
Jul 08 16:57:09.002942 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:09.002908    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3b76551-8e16-4370-a7f3-a5a91577cd3b-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr\" (UID: \"b3b76551-8e16-4370-a7f3-a5a91577cd3b\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr"
Jul 08 16:57:09.003114 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:09.002968    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3b76551-8e16-4370-a7f3-a5a91577cd3b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr\" (UID: \"b3b76551-8e16-4370-a7f3-a5a91577cd3b\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr"
Jul 08 16:57:09.011203 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:09.011165    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-9f040-predictor-5458f85b97-zlgtn"]
Jul 08 16:57:09.011496 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:09.011449    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-9f040-predictor-5458f85b97-zlgtn" podUID="4c6d57fc-3f16-4060-8fff-5279965ae841" containerName="kserve-container" containerID="cri-o://9b3bc331b69d35e4578326be042d9de394749da889c53ffb4854e17eb963367b" gracePeriod=30
Jul 08 16:57:09.011659 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:09.011468    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-9f040-predictor-5458f85b97-zlgtn" podUID="4c6d57fc-3f16-4060-8fff-5279965ae841" containerName="kube-rbac-proxy" containerID="cri-o://af87b51d0602d46abcbcb581e74c2a7437943f1d613e410657b3f6878ad919b1" gracePeriod=30
Jul 08 16:57:09.103815 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:09.103770    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3b76551-8e16-4370-a7f3-a5a91577cd3b-cabundle-cert\") pod \"isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr\" (UID: \"b3b76551-8e16-4370-a7f3-a5a91577cd3b\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr"
Jul 08 16:57:09.104023 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:09.103832    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3b76551-8e16-4370-a7f3-a5a91577cd3b-proxy-tls\") pod \"isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr\" (UID: \"b3b76551-8e16-4370-a7f3-a5a91577cd3b\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr"
Jul 08 16:57:09.104023 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:09.103887    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2cjn\" (UniqueName: \"kubernetes.io/projected/b3b76551-8e16-4370-a7f3-a5a91577cd3b-kube-api-access-z2cjn\") pod \"isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr\" (UID: \"b3b76551-8e16-4370-a7f3-a5a91577cd3b\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr"
Jul 08 16:57:09.104023 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:09.103914    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3b76551-8e16-4370-a7f3-a5a91577cd3b-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr\" (UID: \"b3b76551-8e16-4370-a7f3-a5a91577cd3b\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr"
Jul 08 16:57:09.104023 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:09.103959    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3b76551-8e16-4370-a7f3-a5a91577cd3b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr\" (UID: \"b3b76551-8e16-4370-a7f3-a5a91577cd3b\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr"
Jul 08 16:57:09.104282 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:57:09.104021    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-scale-raw-c7447-predictor-serving-cert: secret "isvc-sklearn-scale-raw-c7447-predictor-serving-cert" not found
Jul 08 16:57:09.104282 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:57:09.104102    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3b76551-8e16-4370-a7f3-a5a91577cd3b-proxy-tls podName:b3b76551-8e16-4370-a7f3-a5a91577cd3b nodeName:}" failed. No retries permitted until 2026-07-08 16:57:09.604080745 +0000 UTC m=+1341.875530265 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b3b76551-8e16-4370-a7f3-a5a91577cd3b-proxy-tls") pod "isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr" (UID: "b3b76551-8e16-4370-a7f3-a5a91577cd3b") : secret "isvc-sklearn-scale-raw-c7447-predictor-serving-cert" not found
Jul 08 16:57:09.104435 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:09.104360    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3b76551-8e16-4370-a7f3-a5a91577cd3b-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr\" (UID: \"b3b76551-8e16-4370-a7f3-a5a91577cd3b\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr"
Jul 08 16:57:09.104575 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:09.104550    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3b76551-8e16-4370-a7f3-a5a91577cd3b-cabundle-cert\") pod \"isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr\" (UID: \"b3b76551-8e16-4370-a7f3-a5a91577cd3b\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr"
Jul 08 16:57:09.104677 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:09.104654    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3b76551-8e16-4370-a7f3-a5a91577cd3b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr\" (UID: \"b3b76551-8e16-4370-a7f3-a5a91577cd3b\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr"
Jul 08 16:57:09.113191 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:09.113156    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2cjn\" (UniqueName: \"kubernetes.io/projected/b3b76551-8e16-4370-a7f3-a5a91577cd3b-kube-api-access-z2cjn\") pod \"isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr\" (UID: \"b3b76551-8e16-4370-a7f3-a5a91577cd3b\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr"
Jul 08 16:57:09.260264 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:09.260230    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-9f040-predictor-5458f85b97-zlgtn"
Jul 08 16:57:09.306273 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:09.306234    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4c6d57fc-3f16-4060-8fff-5279965ae841-kube-rbac-proxy-sar-config\") pod \"4c6d57fc-3f16-4060-8fff-5279965ae841\" (UID: \"4c6d57fc-3f16-4060-8fff-5279965ae841\") "
Jul 08 16:57:09.306450 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:09.306285    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c6d57fc-3f16-4060-8fff-5279965ae841-proxy-tls\") pod \"4c6d57fc-3f16-4060-8fff-5279965ae841\" (UID: \"4c6d57fc-3f16-4060-8fff-5279965ae841\") "
Jul 08 16:57:09.306450 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:09.306316    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r2cgj\" (UniqueName: \"kubernetes.io/projected/4c6d57fc-3f16-4060-8fff-5279965ae841-kube-api-access-r2cgj\") pod \"4c6d57fc-3f16-4060-8fff-5279965ae841\" (UID: \"4c6d57fc-3f16-4060-8fff-5279965ae841\") "
Jul 08 16:57:09.306758 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:09.306727    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c6d57fc-3f16-4060-8fff-5279965ae841-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4c6d57fc-3f16-4060-8fff-5279965ae841" (UID: "4c6d57fc-3f16-4060-8fff-5279965ae841"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:57:09.308700 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:09.308668    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c6d57fc-3f16-4060-8fff-5279965ae841-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4c6d57fc-3f16-4060-8fff-5279965ae841" (UID: "4c6d57fc-3f16-4060-8fff-5279965ae841"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:57:09.308832 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:09.308714    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c6d57fc-3f16-4060-8fff-5279965ae841-kube-api-access-r2cgj" (OuterVolumeSpecName: "kube-api-access-r2cgj") pod "4c6d57fc-3f16-4060-8fff-5279965ae841" (UID: "4c6d57fc-3f16-4060-8fff-5279965ae841"). InnerVolumeSpecName "kube-api-access-r2cgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:57:09.407330 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:09.407285    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4c6d57fc-3f16-4060-8fff-5279965ae841-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:57:09.407330 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:09.407320    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c6d57fc-3f16-4060-8fff-5279965ae841-proxy-tls\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:57:09.407330 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:09.407332    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2cgj\" (UniqueName: \"kubernetes.io/projected/4c6d57fc-3f16-4060-8fff-5279965ae841-kube-api-access-r2cgj\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:57:09.609611 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:09.609563    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3b76551-8e16-4370-a7f3-a5a91577cd3b-proxy-tls\") pod \"isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr\" (UID: \"b3b76551-8e16-4370-a7f3-a5a91577cd3b\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr"
Jul 08 16:57:09.612208 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:09.612177    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3b76551-8e16-4370-a7f3-a5a91577cd3b-proxy-tls\") pod \"isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr\" (UID: \"b3b76551-8e16-4370-a7f3-a5a91577cd3b\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr"
Jul 08 16:57:09.621298 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:09.621265    2578 generic.go:356] "Generic (PLEG): container finished" podID="4c6d57fc-3f16-4060-8fff-5279965ae841" containerID="af87b51d0602d46abcbcb581e74c2a7437943f1d613e410657b3f6878ad919b1" exitCode=2
Jul 08 16:57:09.621298 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:09.621295    2578 generic.go:356] "Generic (PLEG): container finished" podID="4c6d57fc-3f16-4060-8fff-5279965ae841" containerID="9b3bc331b69d35e4578326be042d9de394749da889c53ffb4854e17eb963367b" exitCode=2
Jul 08 16:57:09.621526 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:09.621313    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-9f040-predictor-5458f85b97-zlgtn"
Jul 08 16:57:09.621526 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:09.621339    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-9f040-predictor-5458f85b97-zlgtn" event={"ID":"4c6d57fc-3f16-4060-8fff-5279965ae841","Type":"ContainerDied","Data":"af87b51d0602d46abcbcb581e74c2a7437943f1d613e410657b3f6878ad919b1"}
Jul 08 16:57:09.621526 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:09.621392    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-9f040-predictor-5458f85b97-zlgtn" event={"ID":"4c6d57fc-3f16-4060-8fff-5279965ae841","Type":"ContainerDied","Data":"9b3bc331b69d35e4578326be042d9de394749da889c53ffb4854e17eb963367b"}
Jul 08 16:57:09.621526 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:09.621412    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-9f040-predictor-5458f85b97-zlgtn" event={"ID":"4c6d57fc-3f16-4060-8fff-5279965ae841","Type":"ContainerDied","Data":"88fbe7131ee8df4eafcfaa41ff91ccae4646ef390adc4e980da0d3240bbd712a"}
Jul 08 16:57:09.621526 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:09.621437    2578 scope.go:117] "RemoveContainer" containerID="af87b51d0602d46abcbcb581e74c2a7437943f1d613e410657b3f6878ad919b1"
Jul 08 16:57:09.624197 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:09.624172    2578 generic.go:356] "Generic (PLEG): container finished" podID="150db917-4ee1-488e-8070-c11d7057d3e1" containerID="a0e0fe74b051ecb0fb0507e0b52610f0d00325ae9d7b3f0e8a1e6cbfe64aef7f" exitCode=2
Jul 08 16:57:09.624320 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:09.624248    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c" event={"ID":"150db917-4ee1-488e-8070-c11d7057d3e1","Type":"ContainerDied","Data":"a0e0fe74b051ecb0fb0507e0b52610f0d00325ae9d7b3f0e8a1e6cbfe64aef7f"}
Jul 08 16:57:09.630733 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:09.630710    2578 scope.go:117] "RemoveContainer" containerID="9b3bc331b69d35e4578326be042d9de394749da889c53ffb4854e17eb963367b"
Jul 08 16:57:09.638983 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:09.638958    2578 scope.go:117] "RemoveContainer" containerID="af87b51d0602d46abcbcb581e74c2a7437943f1d613e410657b3f6878ad919b1"
Jul 08 16:57:09.639319 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:57:09.639295    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af87b51d0602d46abcbcb581e74c2a7437943f1d613e410657b3f6878ad919b1\": container with ID starting with af87b51d0602d46abcbcb581e74c2a7437943f1d613e410657b3f6878ad919b1 not found: ID does not exist" containerID="af87b51d0602d46abcbcb581e74c2a7437943f1d613e410657b3f6878ad919b1"
Jul 08 16:57:09.639363 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:09.639331    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af87b51d0602d46abcbcb581e74c2a7437943f1d613e410657b3f6878ad919b1"} err="failed to get container status \"af87b51d0602d46abcbcb581e74c2a7437943f1d613e410657b3f6878ad919b1\": rpc error: code = NotFound desc = could not find container \"af87b51d0602d46abcbcb581e74c2a7437943f1d613e410657b3f6878ad919b1\": container with ID starting with af87b51d0602d46abcbcb581e74c2a7437943f1d613e410657b3f6878ad919b1 not found: ID does not exist"
Jul 08 16:57:09.639363 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:09.639352    2578 scope.go:117] "RemoveContainer" containerID="9b3bc331b69d35e4578326be042d9de394749da889c53ffb4854e17eb963367b"
Jul 08 16:57:09.639660 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:57:09.639618    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b3bc331b69d35e4578326be042d9de394749da889c53ffb4854e17eb963367b\": container with ID starting with 9b3bc331b69d35e4578326be042d9de394749da889c53ffb4854e17eb963367b not found: ID does not exist" containerID="9b3bc331b69d35e4578326be042d9de394749da889c53ffb4854e17eb963367b"
Jul 08 16:57:09.639713 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:09.639667    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b3bc331b69d35e4578326be042d9de394749da889c53ffb4854e17eb963367b"} err="failed to get container status \"9b3bc331b69d35e4578326be042d9de394749da889c53ffb4854e17eb963367b\": rpc error: code = NotFound desc = could not find container \"9b3bc331b69d35e4578326be042d9de394749da889c53ffb4854e17eb963367b\": container with ID starting with 9b3bc331b69d35e4578326be042d9de394749da889c53ffb4854e17eb963367b not found: ID does not exist"
Jul 08 16:57:09.639713 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:09.639684    2578 scope.go:117] "RemoveContainer" containerID="af87b51d0602d46abcbcb581e74c2a7437943f1d613e410657b3f6878ad919b1"
Jul 08 16:57:09.639916 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:09.639900    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af87b51d0602d46abcbcb581e74c2a7437943f1d613e410657b3f6878ad919b1"} err="failed to get container status \"af87b51d0602d46abcbcb581e74c2a7437943f1d613e410657b3f6878ad919b1\": rpc error: code = NotFound desc = could not find container \"af87b51d0602d46abcbcb581e74c2a7437943f1d613e410657b3f6878ad919b1\": container with ID starting with af87b51d0602d46abcbcb581e74c2a7437943f1d613e410657b3f6878ad919b1 not found: ID does not exist"
Jul 08 16:57:09.639961 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:09.639916    2578 scope.go:117] "RemoveContainer" containerID="9b3bc331b69d35e4578326be042d9de394749da889c53ffb4854e17eb963367b"
Jul 08 16:57:09.640168 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:09.640150    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b3bc331b69d35e4578326be042d9de394749da889c53ffb4854e17eb963367b"} err="failed to get container status \"9b3bc331b69d35e4578326be042d9de394749da889c53ffb4854e17eb963367b\": rpc error: code = NotFound desc = could not find container \"9b3bc331b69d35e4578326be042d9de394749da889c53ffb4854e17eb963367b\": container with ID starting with 9b3bc331b69d35e4578326be042d9de394749da889c53ffb4854e17eb963367b not found: ID does not exist"
Jul 08 16:57:09.643381 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:09.643360    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-9f040-predictor-5458f85b97-zlgtn"]
Jul 08 16:57:09.649014 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:09.648991    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-9f040-predictor-5458f85b97-zlgtn"]
Jul 08 16:57:09.900249 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:09.900207    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr"
Jul 08 16:57:10.051800 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:10.051764    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr"]
Jul 08 16:57:10.055178 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:57:10.055146    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3b76551_8e16_4370_a7f3_a5a91577cd3b.slice/crio-e6e793f8cfed447865c3022511c553b7a00cfa720944ca56289f96547358e4f8 WatchSource:0}: Error finding container e6e793f8cfed447865c3022511c553b7a00cfa720944ca56289f96547358e4f8: Status 404 returned error can't find the container with id e6e793f8cfed447865c3022511c553b7a00cfa720944ca56289f96547358e4f8
Jul 08 16:57:10.285801 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:10.285708    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c6d57fc-3f16-4060-8fff-5279965ae841" path="/var/lib/kubelet/pods/4c6d57fc-3f16-4060-8fff-5279965ae841/volumes"
Jul 08 16:57:10.629751 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:10.629699    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr" event={"ID":"b3b76551-8e16-4370-a7f3-a5a91577cd3b","Type":"ContainerStarted","Data":"f6f05c7de4a8fc1d51f1217cc62e3636557f06312484db063fe9276becb2e336"}
Jul 08 16:57:10.629961 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:10.629778    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr" event={"ID":"b3b76551-8e16-4370-a7f3-a5a91577cd3b","Type":"ContainerStarted","Data":"e6e793f8cfed447865c3022511c553b7a00cfa720944ca56289f96547358e4f8"}
Jul 08 16:57:11.258964 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:11.258916    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c" podUID="150db917-4ee1-488e-8070-c11d7057d3e1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 08 16:57:14.649766 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:14.649732    2578 generic.go:356] "Generic (PLEG): container finished" podID="b3b76551-8e16-4370-a7f3-a5a91577cd3b" containerID="f6f05c7de4a8fc1d51f1217cc62e3636557f06312484db063fe9276becb2e336" exitCode=0
Jul 08 16:57:14.650478 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:14.649809    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr" event={"ID":"b3b76551-8e16-4370-a7f3-a5a91577cd3b","Type":"ContainerDied","Data":"f6f05c7de4a8fc1d51f1217cc62e3636557f06312484db063fe9276becb2e336"}
Jul 08 16:57:14.652711 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:14.652567    2578 generic.go:356] "Generic (PLEG): container finished" podID="150db917-4ee1-488e-8070-c11d7057d3e1" containerID="7dfebaa3db6433550031e135c90b2daaf0c18f92f2445f114855697f5756db6f" exitCode=0
Jul 08 16:57:14.652711 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:14.652623    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c" event={"ID":"150db917-4ee1-488e-8070-c11d7057d3e1","Type":"ContainerDied","Data":"7dfebaa3db6433550031e135c90b2daaf0c18f92f2445f114855697f5756db6f"}
Jul 08 16:57:15.659869 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:15.659832    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr" event={"ID":"b3b76551-8e16-4370-a7f3-a5a91577cd3b","Type":"ContainerStarted","Data":"2ec0ead5dc021ee363373c507bec1d1c1fa59cf860ad0214cdd03fb917f35c99"}
Jul 08 16:57:15.660259 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:15.659904    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr" event={"ID":"b3b76551-8e16-4370-a7f3-a5a91577cd3b","Type":"ContainerStarted","Data":"2d952194c2f93c29c6dfdff925911e2c48676ab86da96d92b43d8942e2533813"}
Jul 08 16:57:15.660259 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:15.660101    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr"
Jul 08 16:57:15.680713 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:15.680627    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr" podStartSLOduration=4.02913888 podStartE2EDuration="7.68060662s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.65146774s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:57:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:57:15.678691205 +0000 UTC m=+1347.950140747" watchObservedRunningTime="2026-07-08 16:57:15.68060662 +0000 UTC m=+1347.952056160"
Jul 08 16:57:16.258164 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:16.258125    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c" podUID="150db917-4ee1-488e-8070-c11d7057d3e1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 08 16:57:16.263569 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:16.263544    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c" podUID="150db917-4ee1-488e-8070-c11d7057d3e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 08 16:57:16.263975 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:16.263952    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c" podUID="150db917-4ee1-488e-8070-c11d7057d3e1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:57:16.664668 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:16.664624    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr"
Jul 08 16:57:16.665867 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:16.665838    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr" podUID="b3b76551-8e16-4370-a7f3-a5a91577cd3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 08 16:57:17.668564 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:17.668524    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr" podUID="b3b76551-8e16-4370-a7f3-a5a91577cd3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 08 16:57:21.258596 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:21.258552    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c" podUID="150db917-4ee1-488e-8070-c11d7057d3e1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 08 16:57:21.259084 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:21.258712    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c"
Jul 08 16:57:22.672898 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:22.672869    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr"
Jul 08 16:57:22.673553 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:22.673524    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr" podUID="b3b76551-8e16-4370-a7f3-a5a91577cd3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 08 16:57:26.258888 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:26.258838    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c" podUID="150db917-4ee1-488e-8070-c11d7057d3e1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 08 16:57:26.263267 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:26.263231    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c" podUID="150db917-4ee1-488e-8070-c11d7057d3e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 08 16:57:26.263587 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:26.263564    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c" podUID="150db917-4ee1-488e-8070-c11d7057d3e1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:57:31.258694 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:31.258621    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c" podUID="150db917-4ee1-488e-8070-c11d7057d3e1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 08 16:57:32.673865 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:32.673816    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr" podUID="b3b76551-8e16-4370-a7f3-a5a91577cd3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 08 16:57:36.258343 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:36.258293    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c" podUID="150db917-4ee1-488e-8070-c11d7057d3e1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 08 16:57:36.263827 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:36.263786    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c" podUID="150db917-4ee1-488e-8070-c11d7057d3e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 08 16:57:36.263988 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:36.263921    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c"
Jul 08 16:57:36.264216 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:36.264191    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c" podUID="150db917-4ee1-488e-8070-c11d7057d3e1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:57:36.264324 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:36.264310    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c"
Jul 08 16:57:39.040263 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:39.040236    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c"
Jul 08 16:57:39.170237 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:39.170144    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/150db917-4ee1-488e-8070-c11d7057d3e1-kube-rbac-proxy-sar-config\") pod \"150db917-4ee1-488e-8070-c11d7057d3e1\" (UID: \"150db917-4ee1-488e-8070-c11d7057d3e1\") "
Jul 08 16:57:39.170237 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:39.170201    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/150db917-4ee1-488e-8070-c11d7057d3e1-kserve-provision-location\") pod \"150db917-4ee1-488e-8070-c11d7057d3e1\" (UID: \"150db917-4ee1-488e-8070-c11d7057d3e1\") "
Jul 08 16:57:39.170480 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:39.170242    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/150db917-4ee1-488e-8070-c11d7057d3e1-proxy-tls\") pod \"150db917-4ee1-488e-8070-c11d7057d3e1\" (UID: \"150db917-4ee1-488e-8070-c11d7057d3e1\") "
Jul 08 16:57:39.170480 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:39.170265    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kthqd\" (UniqueName: \"kubernetes.io/projected/150db917-4ee1-488e-8070-c11d7057d3e1-kube-api-access-kthqd\") pod \"150db917-4ee1-488e-8070-c11d7057d3e1\" (UID: \"150db917-4ee1-488e-8070-c11d7057d3e1\") "
Jul 08 16:57:39.170480 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:39.170309    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/150db917-4ee1-488e-8070-c11d7057d3e1-cabundle-cert\") pod \"150db917-4ee1-488e-8070-c11d7057d3e1\" (UID: \"150db917-4ee1-488e-8070-c11d7057d3e1\") "
Jul 08 16:57:39.170681 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:39.170581    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/150db917-4ee1-488e-8070-c11d7057d3e1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "150db917-4ee1-488e-8070-c11d7057d3e1" (UID: "150db917-4ee1-488e-8070-c11d7057d3e1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:57:39.170681 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:39.170619    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/150db917-4ee1-488e-8070-c11d7057d3e1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "150db917-4ee1-488e-8070-c11d7057d3e1" (UID: "150db917-4ee1-488e-8070-c11d7057d3e1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:57:39.170681 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:39.170627    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/150db917-4ee1-488e-8070-c11d7057d3e1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "150db917-4ee1-488e-8070-c11d7057d3e1" (UID: "150db917-4ee1-488e-8070-c11d7057d3e1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:57:39.172618 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:39.172595    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/150db917-4ee1-488e-8070-c11d7057d3e1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "150db917-4ee1-488e-8070-c11d7057d3e1" (UID: "150db917-4ee1-488e-8070-c11d7057d3e1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:57:39.172618 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:39.172601    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/150db917-4ee1-488e-8070-c11d7057d3e1-kube-api-access-kthqd" (OuterVolumeSpecName: "kube-api-access-kthqd") pod "150db917-4ee1-488e-8070-c11d7057d3e1" (UID: "150db917-4ee1-488e-8070-c11d7057d3e1"). InnerVolumeSpecName "kube-api-access-kthqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:57:39.271309 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:39.271258    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/150db917-4ee1-488e-8070-c11d7057d3e1-cabundle-cert\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:57:39.271309 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:39.271306    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/150db917-4ee1-488e-8070-c11d7057d3e1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:57:39.271309 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:39.271316    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/150db917-4ee1-488e-8070-c11d7057d3e1-kserve-provision-location\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:57:39.271309 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:39.271326    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/150db917-4ee1-488e-8070-c11d7057d3e1-proxy-tls\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:57:39.271596 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:39.271360    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kthqd\" (UniqueName: \"kubernetes.io/projected/150db917-4ee1-488e-8070-c11d7057d3e1-kube-api-access-kthqd\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:57:39.760498 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:39.760461    2578 generic.go:356] "Generic (PLEG): container finished" podID="150db917-4ee1-488e-8070-c11d7057d3e1" containerID="7ff041221171e314457b4ef728aeaf6eca8f0cb21fa72d43b83276f793f7f12c" exitCode=0
Jul 08 16:57:39.760835 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:39.760536    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c"
Jul 08 16:57:39.760835 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:39.760547    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c" event={"ID":"150db917-4ee1-488e-8070-c11d7057d3e1","Type":"ContainerDied","Data":"7ff041221171e314457b4ef728aeaf6eca8f0cb21fa72d43b83276f793f7f12c"}
Jul 08 16:57:39.760835 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:39.760618    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c" event={"ID":"150db917-4ee1-488e-8070-c11d7057d3e1","Type":"ContainerDied","Data":"01feaf62d406abfd024f72502ef159c6e9662a40caf0c0983932b428e5e89265"}
Jul 08 16:57:39.760835 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:39.760657    2578 scope.go:117] "RemoveContainer" containerID="7ff041221171e314457b4ef728aeaf6eca8f0cb21fa72d43b83276f793f7f12c"
Jul 08 16:57:39.769948 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:39.769927    2578 scope.go:117] "RemoveContainer" containerID="a0e0fe74b051ecb0fb0507e0b52610f0d00325ae9d7b3f0e8a1e6cbfe64aef7f"
Jul 08 16:57:39.777734 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:39.777714    2578 scope.go:117] "RemoveContainer" containerID="7dfebaa3db6433550031e135c90b2daaf0c18f92f2445f114855697f5756db6f"
Jul 08 16:57:39.782580 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:39.782555    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c"]
Jul 08 16:57:39.785934 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:39.785908    2578 scope.go:117] "RemoveContainer" containerID="fe04b1c71c5c2cf7fc8b9ccd106c08184f78d7a36b414387d9700b18b4c9787d"
Jul 08 16:57:39.787308 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:39.787282    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-9f040-predictor-758d49c58-7gb6c"]
Jul 08 16:57:39.794132 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:39.794112    2578 scope.go:117] "RemoveContainer" containerID="7ff041221171e314457b4ef728aeaf6eca8f0cb21fa72d43b83276f793f7f12c"
Jul 08 16:57:39.794466 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:57:39.794445    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ff041221171e314457b4ef728aeaf6eca8f0cb21fa72d43b83276f793f7f12c\": container with ID starting with 7ff041221171e314457b4ef728aeaf6eca8f0cb21fa72d43b83276f793f7f12c not found: ID does not exist" containerID="7ff041221171e314457b4ef728aeaf6eca8f0cb21fa72d43b83276f793f7f12c"
Jul 08 16:57:39.794535 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:39.794476    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ff041221171e314457b4ef728aeaf6eca8f0cb21fa72d43b83276f793f7f12c"} err="failed to get container status \"7ff041221171e314457b4ef728aeaf6eca8f0cb21fa72d43b83276f793f7f12c\": rpc error: code = NotFound desc = could not find container \"7ff041221171e314457b4ef728aeaf6eca8f0cb21fa72d43b83276f793f7f12c\": container with ID starting with 7ff041221171e314457b4ef728aeaf6eca8f0cb21fa72d43b83276f793f7f12c not found: ID does not exist"
Jul 08 16:57:39.794535 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:39.794495    2578 scope.go:117] "RemoveContainer" containerID="a0e0fe74b051ecb0fb0507e0b52610f0d00325ae9d7b3f0e8a1e6cbfe64aef7f"
Jul 08 16:57:39.794820 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:57:39.794794    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0e0fe74b051ecb0fb0507e0b52610f0d00325ae9d7b3f0e8a1e6cbfe64aef7f\": container with ID starting with a0e0fe74b051ecb0fb0507e0b52610f0d00325ae9d7b3f0e8a1e6cbfe64aef7f not found: ID does not exist" containerID="a0e0fe74b051ecb0fb0507e0b52610f0d00325ae9d7b3f0e8a1e6cbfe64aef7f"
Jul 08 16:57:39.794862 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:39.794831    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0e0fe74b051ecb0fb0507e0b52610f0d00325ae9d7b3f0e8a1e6cbfe64aef7f"} err="failed to get container status \"a0e0fe74b051ecb0fb0507e0b52610f0d00325ae9d7b3f0e8a1e6cbfe64aef7f\": rpc error: code = NotFound desc = could not find container \"a0e0fe74b051ecb0fb0507e0b52610f0d00325ae9d7b3f0e8a1e6cbfe64aef7f\": container with ID starting with a0e0fe74b051ecb0fb0507e0b52610f0d00325ae9d7b3f0e8a1e6cbfe64aef7f not found: ID does not exist"
Jul 08 16:57:39.794862 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:39.794854    2578 scope.go:117] "RemoveContainer" containerID="7dfebaa3db6433550031e135c90b2daaf0c18f92f2445f114855697f5756db6f"
Jul 08 16:57:39.795159 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:57:39.795141    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dfebaa3db6433550031e135c90b2daaf0c18f92f2445f114855697f5756db6f\": container with ID starting with 7dfebaa3db6433550031e135c90b2daaf0c18f92f2445f114855697f5756db6f not found: ID does not exist" containerID="7dfebaa3db6433550031e135c90b2daaf0c18f92f2445f114855697f5756db6f"
Jul 08 16:57:39.795226 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:39.795164    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dfebaa3db6433550031e135c90b2daaf0c18f92f2445f114855697f5756db6f"} err="failed to get container status \"7dfebaa3db6433550031e135c90b2daaf0c18f92f2445f114855697f5756db6f\": rpc error: code = NotFound desc = could not find container \"7dfebaa3db6433550031e135c90b2daaf0c18f92f2445f114855697f5756db6f\": container with ID starting with 7dfebaa3db6433550031e135c90b2daaf0c18f92f2445f114855697f5756db6f not found: ID does not exist"
Jul 08 16:57:39.795226 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:39.795179    2578 scope.go:117] "RemoveContainer" containerID="fe04b1c71c5c2cf7fc8b9ccd106c08184f78d7a36b414387d9700b18b4c9787d"
Jul 08 16:57:39.795428 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:57:39.795413    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe04b1c71c5c2cf7fc8b9ccd106c08184f78d7a36b414387d9700b18b4c9787d\": container with ID starting with fe04b1c71c5c2cf7fc8b9ccd106c08184f78d7a36b414387d9700b18b4c9787d not found: ID does not exist" containerID="fe04b1c71c5c2cf7fc8b9ccd106c08184f78d7a36b414387d9700b18b4c9787d"
Jul 08 16:57:39.795471 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:39.795431    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe04b1c71c5c2cf7fc8b9ccd106c08184f78d7a36b414387d9700b18b4c9787d"} err="failed to get container status \"fe04b1c71c5c2cf7fc8b9ccd106c08184f78d7a36b414387d9700b18b4c9787d\": rpc error: code = NotFound desc = could not find container \"fe04b1c71c5c2cf7fc8b9ccd106c08184f78d7a36b414387d9700b18b4c9787d\": container with ID starting with fe04b1c71c5c2cf7fc8b9ccd106c08184f78d7a36b414387d9700b18b4c9787d not found: ID does not exist"
Jul 08 16:57:40.284946 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:40.284909    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="150db917-4ee1-488e-8070-c11d7057d3e1" path="/var/lib/kubelet/pods/150db917-4ee1-488e-8070-c11d7057d3e1/volumes"
Jul 08 16:57:42.673900 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:42.673862    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr" podUID="b3b76551-8e16-4370-a7f3-a5a91577cd3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 08 16:57:52.674177 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:57:52.674129    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr" podUID="b3b76551-8e16-4370-a7f3-a5a91577cd3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 08 16:58:02.673677 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:58:02.673616    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr" podUID="b3b76551-8e16-4370-a7f3-a5a91577cd3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 08 16:58:12.673687 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:58:12.673623    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr" podUID="b3b76551-8e16-4370-a7f3-a5a91577cd3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 08 16:58:22.673402 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:58:22.673355    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr" podUID="b3b76551-8e16-4370-a7f3-a5a91577cd3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 08 16:58:32.673440 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:58:32.673403    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr" podUID="b3b76551-8e16-4370-a7f3-a5a91577cd3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 08 16:58:42.673425 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:58:42.673340    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr" podUID="b3b76551-8e16-4370-a7f3-a5a91577cd3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 08 16:58:48.282733 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:58:48.282685    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr" podUID="b3b76551-8e16-4370-a7f3-a5a91577cd3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 08 16:58:58.282747 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:58:58.282705    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr" podUID="b3b76551-8e16-4370-a7f3-a5a91577cd3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 08 16:59:08.282676 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:08.282623    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr" podUID="b3b76551-8e16-4370-a7f3-a5a91577cd3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 08 16:59:18.282652 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:18.282600    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr" podUID="b3b76551-8e16-4370-a7f3-a5a91577cd3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 08 16:59:28.282586 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:28.282544    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr" podUID="b3b76551-8e16-4370-a7f3-a5a91577cd3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 08 16:59:38.282765 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:38.282721    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr" podUID="b3b76551-8e16-4370-a7f3-a5a91577cd3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 08 16:59:48.245121 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:48.245085    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-qn2bx_4717bdee-2017-4551-b0ec-683ab07f7660/console-operator/1.log"
Jul 08 16:59:48.246032 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:48.246008    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-qn2bx_4717bdee-2017-4551-b0ec-683ab07f7660/console-operator/1.log"
Jul 08 16:59:48.249591 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:48.249569    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2shvk_f08a1b00-78fd-43ef-b2aa-2712d09fb1b3/ovn-acl-logging/0.log"
Jul 08 16:59:48.250671 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:48.250654    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2shvk_f08a1b00-78fd-43ef-b2aa-2712d09fb1b3/ovn-acl-logging/0.log"
Jul 08 16:59:48.285247 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:48.285216    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr"
Jul 08 16:59:49.135505 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:49.135459    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr"]
Jul 08 16:59:49.266234 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:49.266186    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr" podUID="b3b76551-8e16-4370-a7f3-a5a91577cd3b" containerName="kserve-container" containerID="cri-o://2d952194c2f93c29c6dfdff925911e2c48676ab86da96d92b43d8942e2533813" gracePeriod=30
Jul 08 16:59:49.266792 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:49.266189    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr" podUID="b3b76551-8e16-4370-a7f3-a5a91577cd3b" containerName="kube-rbac-proxy" containerID="cri-o://2ec0ead5dc021ee363373c507bec1d1c1fa59cf860ad0214cdd03fb917f35c99" gracePeriod=30
Jul 08 16:59:50.273743 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:50.273704    2578 generic.go:356] "Generic (PLEG): container finished" podID="b3b76551-8e16-4370-a7f3-a5a91577cd3b" containerID="2ec0ead5dc021ee363373c507bec1d1c1fa59cf860ad0214cdd03fb917f35c99" exitCode=2
Jul 08 16:59:50.274119 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:50.273773    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr" event={"ID":"b3b76551-8e16-4370-a7f3-a5a91577cd3b","Type":"ContainerDied","Data":"2ec0ead5dc021ee363373c507bec1d1c1fa59cf860ad0214cdd03fb917f35c99"}
Jul 08 16:59:52.669332 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:52.669283    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr" podUID="b3b76551-8e16-4370-a7f3-a5a91577cd3b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Jul 08 16:59:54.301134 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:54.301099    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj"]
Jul 08 16:59:54.301509 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:54.301376    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="150db917-4ee1-488e-8070-c11d7057d3e1" containerName="agent"
Jul 08 16:59:54.301509 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:54.301386    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="150db917-4ee1-488e-8070-c11d7057d3e1" containerName="agent"
Jul 08 16:59:54.301509 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:54.301398    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c6d57fc-3f16-4060-8fff-5279965ae841" containerName="kserve-container"
Jul 08 16:59:54.301509 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:54.301403    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c6d57fc-3f16-4060-8fff-5279965ae841" containerName="kserve-container"
Jul 08 16:59:54.301509 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:54.301411    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c6d57fc-3f16-4060-8fff-5279965ae841" containerName="kube-rbac-proxy"
Jul 08 16:59:54.301509 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:54.301417    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c6d57fc-3f16-4060-8fff-5279965ae841" containerName="kube-rbac-proxy"
Jul 08 16:59:54.301509 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:54.301428    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="150db917-4ee1-488e-8070-c11d7057d3e1" containerName="kserve-container"
Jul 08 16:59:54.301509 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:54.301433    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="150db917-4ee1-488e-8070-c11d7057d3e1" containerName="kserve-container"
Jul 08 16:59:54.301509 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:54.301438    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="150db917-4ee1-488e-8070-c11d7057d3e1" containerName="storage-initializer"
Jul 08 16:59:54.301509 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:54.301443    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="150db917-4ee1-488e-8070-c11d7057d3e1" containerName="storage-initializer"
Jul 08 16:59:54.301509 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:54.301447    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="150db917-4ee1-488e-8070-c11d7057d3e1" containerName="kube-rbac-proxy"
Jul 08 16:59:54.301509 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:54.301453    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="150db917-4ee1-488e-8070-c11d7057d3e1" containerName="kube-rbac-proxy"
Jul 08 16:59:54.301509 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:54.301494    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="150db917-4ee1-488e-8070-c11d7057d3e1" containerName="agent"
Jul 08 16:59:54.301509 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:54.301501    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="150db917-4ee1-488e-8070-c11d7057d3e1" containerName="kube-rbac-proxy"
Jul 08 16:59:54.301509 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:54.301507    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c6d57fc-3f16-4060-8fff-5279965ae841" containerName="kserve-container"
Jul 08 16:59:54.301509 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:54.301514    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c6d57fc-3f16-4060-8fff-5279965ae841" containerName="kube-rbac-proxy"
Jul 08 16:59:54.302012 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:54.301521    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="150db917-4ee1-488e-8070-c11d7057d3e1" containerName="kserve-container"
Jul 08 16:59:54.304433 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:54.304417    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj"
Jul 08 16:59:54.307059 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:54.307033    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-ff3865-kube-rbac-proxy-sar-config\""
Jul 08 16:59:54.307173 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:54.307062    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-ff3865-predictor-serving-cert\""
Jul 08 16:59:54.313420 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:54.313394    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj"]
Jul 08 16:59:54.398385 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:54.398341    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/612009f3-1621-48f9-a0aa-838e07da4734-kube-rbac-proxy-sar-config\") pod \"isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj\" (UID: \"612009f3-1621-48f9-a0aa-838e07da4734\") " pod="kserve-ci-e2e-test/isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj"
Jul 08 16:59:54.398580 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:54.398401    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/612009f3-1621-48f9-a0aa-838e07da4734-proxy-tls\") pod \"isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj\" (UID: \"612009f3-1621-48f9-a0aa-838e07da4734\") " pod="kserve-ci-e2e-test/isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj"
Jul 08 16:59:54.398580 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:54.398441    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/612009f3-1621-48f9-a0aa-838e07da4734-kserve-provision-location\") pod \"isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj\" (UID: \"612009f3-1621-48f9-a0aa-838e07da4734\") " pod="kserve-ci-e2e-test/isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj"
Jul 08 16:59:54.398580 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:54.398461    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/612009f3-1621-48f9-a0aa-838e07da4734-cabundle-cert\") pod \"isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj\" (UID: \"612009f3-1621-48f9-a0aa-838e07da4734\") " pod="kserve-ci-e2e-test/isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj"
Jul 08 16:59:54.398580 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:54.398486    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj64c\" (UniqueName: \"kubernetes.io/projected/612009f3-1621-48f9-a0aa-838e07da4734-kube-api-access-bj64c\") pod \"isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj\" (UID: \"612009f3-1621-48f9-a0aa-838e07da4734\") " pod="kserve-ci-e2e-test/isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj"
Jul 08 16:59:54.499920 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:54.499886    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/612009f3-1621-48f9-a0aa-838e07da4734-kserve-provision-location\") pod \"isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj\" (UID: \"612009f3-1621-48f9-a0aa-838e07da4734\") " pod="kserve-ci-e2e-test/isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj"
Jul 08 16:59:54.500112 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:54.499927    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/612009f3-1621-48f9-a0aa-838e07da4734-cabundle-cert\") pod \"isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj\" (UID: \"612009f3-1621-48f9-a0aa-838e07da4734\") " pod="kserve-ci-e2e-test/isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj"
Jul 08 16:59:54.500112 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:54.499962    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj64c\" (UniqueName: \"kubernetes.io/projected/612009f3-1621-48f9-a0aa-838e07da4734-kube-api-access-bj64c\") pod \"isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj\" (UID: \"612009f3-1621-48f9-a0aa-838e07da4734\") " pod="kserve-ci-e2e-test/isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj"
Jul 08 16:59:54.500112 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:54.500020    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/612009f3-1621-48f9-a0aa-838e07da4734-kube-rbac-proxy-sar-config\") pod \"isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj\" (UID: \"612009f3-1621-48f9-a0aa-838e07da4734\") " pod="kserve-ci-e2e-test/isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj"
Jul 08 16:59:54.500112 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:54.500083    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/612009f3-1621-48f9-a0aa-838e07da4734-proxy-tls\") pod \"isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj\" (UID: \"612009f3-1621-48f9-a0aa-838e07da4734\") " pod="kserve-ci-e2e-test/isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj"
Jul 08 16:59:54.500334 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:59:54.500194    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-primary-ff3865-predictor-serving-cert: secret "isvc-primary-ff3865-predictor-serving-cert" not found
Jul 08 16:59:54.500334 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:59:54.500276    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/612009f3-1621-48f9-a0aa-838e07da4734-proxy-tls podName:612009f3-1621-48f9-a0aa-838e07da4734 nodeName:}" failed. No retries permitted until 2026-07-08 16:59:55.000254825 +0000 UTC m=+1507.271704347 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/612009f3-1621-48f9-a0aa-838e07da4734-proxy-tls") pod "isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj" (UID: "612009f3-1621-48f9-a0aa-838e07da4734") : secret "isvc-primary-ff3865-predictor-serving-cert" not found
Jul 08 16:59:54.500450 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:54.500343    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/612009f3-1621-48f9-a0aa-838e07da4734-kserve-provision-location\") pod \"isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj\" (UID: \"612009f3-1621-48f9-a0aa-838e07da4734\") " pod="kserve-ci-e2e-test/isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj"
Jul 08 16:59:54.500667 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:54.500649    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/612009f3-1621-48f9-a0aa-838e07da4734-kube-rbac-proxy-sar-config\") pod \"isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj\" (UID: \"612009f3-1621-48f9-a0aa-838e07da4734\") " pod="kserve-ci-e2e-test/isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj"
Jul 08 16:59:54.500770 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:54.500751    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/612009f3-1621-48f9-a0aa-838e07da4734-cabundle-cert\") pod \"isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj\" (UID: \"612009f3-1621-48f9-a0aa-838e07da4734\") " pod="kserve-ci-e2e-test/isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj"
Jul 08 16:59:54.508742 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:54.508716    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj64c\" (UniqueName: \"kubernetes.io/projected/612009f3-1621-48f9-a0aa-838e07da4734-kube-api-access-bj64c\") pod \"isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj\" (UID: \"612009f3-1621-48f9-a0aa-838e07da4734\") " pod="kserve-ci-e2e-test/isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj"
Jul 08 16:59:55.004900 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:55.004860    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/612009f3-1621-48f9-a0aa-838e07da4734-proxy-tls\") pod \"isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj\" (UID: \"612009f3-1621-48f9-a0aa-838e07da4734\") " pod="kserve-ci-e2e-test/isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj"
Jul 08 16:59:55.006776 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:59:55.004993    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-primary-ff3865-predictor-serving-cert: secret "isvc-primary-ff3865-predictor-serving-cert" not found
Jul 08 16:59:55.006776 ip-10-0-142-94 kubenswrapper[2578]: E0708 16:59:55.005060    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/612009f3-1621-48f9-a0aa-838e07da4734-proxy-tls podName:612009f3-1621-48f9-a0aa-838e07da4734 nodeName:}" failed. No retries permitted until 2026-07-08 16:59:56.005045619 +0000 UTC m=+1508.276495138 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/612009f3-1621-48f9-a0aa-838e07da4734-proxy-tls") pod "isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj" (UID: "612009f3-1621-48f9-a0aa-838e07da4734") : secret "isvc-primary-ff3865-predictor-serving-cert" not found
Jul 08 16:59:56.013470 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:56.013431    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/612009f3-1621-48f9-a0aa-838e07da4734-proxy-tls\") pod \"isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj\" (UID: \"612009f3-1621-48f9-a0aa-838e07da4734\") " pod="kserve-ci-e2e-test/isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj"
Jul 08 16:59:56.016025 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:56.015998    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/612009f3-1621-48f9-a0aa-838e07da4734-proxy-tls\") pod \"isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj\" (UID: \"612009f3-1621-48f9-a0aa-838e07da4734\") " pod="kserve-ci-e2e-test/isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj"
Jul 08 16:59:56.115188 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:56.115149    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj"
Jul 08 16:59:56.252888 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:56.252862    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj"]
Jul 08 16:59:56.255655 ip-10-0-142-94 kubenswrapper[2578]: W0708 16:59:56.255602    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod612009f3_1621_48f9_a0aa_838e07da4734.slice/crio-e085f0e2795dae629a8ea04ea80c935d6116b9731686e2d0643c8d9b126dcbf8 WatchSource:0}: Error finding container e085f0e2795dae629a8ea04ea80c935d6116b9731686e2d0643c8d9b126dcbf8: Status 404 returned error can't find the container with id e085f0e2795dae629a8ea04ea80c935d6116b9731686e2d0643c8d9b126dcbf8
Jul 08 16:59:56.299961 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:56.299924    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj" event={"ID":"612009f3-1621-48f9-a0aa-838e07da4734","Type":"ContainerStarted","Data":"e085f0e2795dae629a8ea04ea80c935d6116b9731686e2d0643c8d9b126dcbf8"}
Jul 08 16:59:57.305375 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:57.305340    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj" event={"ID":"612009f3-1621-48f9-a0aa-838e07da4734","Type":"ContainerStarted","Data":"a30df5b08411c01869d9ec5c5968c1e1eceea31dc397f72d462d063e97ba6b1d"}
Jul 08 16:59:57.669780 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:57.669736    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr" podUID="b3b76551-8e16-4370-a7f3-a5a91577cd3b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Jul 08 16:59:58.283114 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:58.283068    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr" podUID="b3b76551-8e16-4370-a7f3-a5a91577cd3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 08 16:59:59.519014 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:59.518987    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr"
Jul 08 16:59:59.542954 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:59.542487    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3b76551-8e16-4370-a7f3-a5a91577cd3b-cabundle-cert\") pod \"b3b76551-8e16-4370-a7f3-a5a91577cd3b\" (UID: \"b3b76551-8e16-4370-a7f3-a5a91577cd3b\") "
Jul 08 16:59:59.542954 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:59.542545    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z2cjn\" (UniqueName: \"kubernetes.io/projected/b3b76551-8e16-4370-a7f3-a5a91577cd3b-kube-api-access-z2cjn\") pod \"b3b76551-8e16-4370-a7f3-a5a91577cd3b\" (UID: \"b3b76551-8e16-4370-a7f3-a5a91577cd3b\") "
Jul 08 16:59:59.542954 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:59.542618    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3b76551-8e16-4370-a7f3-a5a91577cd3b-kserve-provision-location\") pod \"b3b76551-8e16-4370-a7f3-a5a91577cd3b\" (UID: \"b3b76551-8e16-4370-a7f3-a5a91577cd3b\") "
Jul 08 16:59:59.542954 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:59.542683    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3b76551-8e16-4370-a7f3-a5a91577cd3b-proxy-tls\") pod \"b3b76551-8e16-4370-a7f3-a5a91577cd3b\" (UID: \"b3b76551-8e16-4370-a7f3-a5a91577cd3b\") "
Jul 08 16:59:59.542954 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:59.542744    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3b76551-8e16-4370-a7f3-a5a91577cd3b-kube-rbac-proxy-sar-config\") pod \"b3b76551-8e16-4370-a7f3-a5a91577cd3b\" (UID: \"b3b76551-8e16-4370-a7f3-a5a91577cd3b\") "
Jul 08 16:59:59.543383 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:59.543368    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3b76551-8e16-4370-a7f3-a5a91577cd3b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b3b76551-8e16-4370-a7f3-a5a91577cd3b" (UID: "b3b76551-8e16-4370-a7f3-a5a91577cd3b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:59:59.543383 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:59.543365    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3b76551-8e16-4370-a7f3-a5a91577cd3b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b3b76551-8e16-4370-a7f3-a5a91577cd3b" (UID: "b3b76551-8e16-4370-a7f3-a5a91577cd3b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:59:59.543501 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:59.543369    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3b76551-8e16-4370-a7f3-a5a91577cd3b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b3b76551-8e16-4370-a7f3-a5a91577cd3b" (UID: "b3b76551-8e16-4370-a7f3-a5a91577cd3b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:59:59.545481 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:59.545395    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3b76551-8e16-4370-a7f3-a5a91577cd3b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b3b76551-8e16-4370-a7f3-a5a91577cd3b" (UID: "b3b76551-8e16-4370-a7f3-a5a91577cd3b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:59:59.545613 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:59.545593    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3b76551-8e16-4370-a7f3-a5a91577cd3b-kube-api-access-z2cjn" (OuterVolumeSpecName: "kube-api-access-z2cjn") pod "b3b76551-8e16-4370-a7f3-a5a91577cd3b" (UID: "b3b76551-8e16-4370-a7f3-a5a91577cd3b"). InnerVolumeSpecName "kube-api-access-z2cjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:59:59.644285 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:59.644244    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3b76551-8e16-4370-a7f3-a5a91577cd3b-proxy-tls\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:59:59.644285 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:59.644279    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3b76551-8e16-4370-a7f3-a5a91577cd3b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:59:59.644285 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:59.644290    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3b76551-8e16-4370-a7f3-a5a91577cd3b-cabundle-cert\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:59:59.644526 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:59.644300    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2cjn\" (UniqueName: \"kubernetes.io/projected/b3b76551-8e16-4370-a7f3-a5a91577cd3b-kube-api-access-z2cjn\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 16:59:59.644526 ip-10-0-142-94 kubenswrapper[2578]: I0708 16:59:59.644309    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3b76551-8e16-4370-a7f3-a5a91577cd3b-kserve-provision-location\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 17:00:00.319322 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:00:00.319288    2578 generic.go:356] "Generic (PLEG): container finished" podID="b3b76551-8e16-4370-a7f3-a5a91577cd3b" containerID="2d952194c2f93c29c6dfdff925911e2c48676ab86da96d92b43d8942e2533813" exitCode=0
Jul 08 17:00:00.319511 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:00:00.319347    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr"
Jul 08 17:00:00.319511 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:00:00.319377    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr" event={"ID":"b3b76551-8e16-4370-a7f3-a5a91577cd3b","Type":"ContainerDied","Data":"2d952194c2f93c29c6dfdff925911e2c48676ab86da96d92b43d8942e2533813"}
Jul 08 17:00:00.319511 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:00:00.319447    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr" event={"ID":"b3b76551-8e16-4370-a7f3-a5a91577cd3b","Type":"ContainerDied","Data":"e6e793f8cfed447865c3022511c553b7a00cfa720944ca56289f96547358e4f8"}
Jul 08 17:00:00.319511 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:00:00.319479    2578 scope.go:117] "RemoveContainer" containerID="2ec0ead5dc021ee363373c507bec1d1c1fa59cf860ad0214cdd03fb917f35c99"
Jul 08 17:00:00.328178 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:00:00.328155    2578 scope.go:117] "RemoveContainer" containerID="2d952194c2f93c29c6dfdff925911e2c48676ab86da96d92b43d8942e2533813"
Jul 08 17:00:00.337273 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:00:00.337248    2578 scope.go:117] "RemoveContainer" containerID="f6f05c7de4a8fc1d51f1217cc62e3636557f06312484db063fe9276becb2e336"
Jul 08 17:00:00.339196 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:00:00.339170    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr"]
Jul 08 17:00:00.343350 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:00:00.343320    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-c7447-predictor-86797847c6-zdmwr"]
Jul 08 17:00:00.347138 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:00:00.347108    2578 scope.go:117] "RemoveContainer" containerID="2ec0ead5dc021ee363373c507bec1d1c1fa59cf860ad0214cdd03fb917f35c99"
Jul 08 17:00:00.347471 ip-10-0-142-94 kubenswrapper[2578]: E0708 17:00:00.347443    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ec0ead5dc021ee363373c507bec1d1c1fa59cf860ad0214cdd03fb917f35c99\": container with ID starting with 2ec0ead5dc021ee363373c507bec1d1c1fa59cf860ad0214cdd03fb917f35c99 not found: ID does not exist" containerID="2ec0ead5dc021ee363373c507bec1d1c1fa59cf860ad0214cdd03fb917f35c99"
Jul 08 17:00:00.347544 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:00:00.347477    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ec0ead5dc021ee363373c507bec1d1c1fa59cf860ad0214cdd03fb917f35c99"} err="failed to get container status \"2ec0ead5dc021ee363373c507bec1d1c1fa59cf860ad0214cdd03fb917f35c99\": rpc error: code = NotFound desc = could not find container \"2ec0ead5dc021ee363373c507bec1d1c1fa59cf860ad0214cdd03fb917f35c99\": container with ID starting with 2ec0ead5dc021ee363373c507bec1d1c1fa59cf860ad0214cdd03fb917f35c99 not found: ID does not exist"
Jul 08 17:00:00.347544 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:00:00.347513    2578 scope.go:117] "RemoveContainer" containerID="2d952194c2f93c29c6dfdff925911e2c48676ab86da96d92b43d8942e2533813"
Jul 08 17:00:00.347879 ip-10-0-142-94 kubenswrapper[2578]: E0708 17:00:00.347859    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d952194c2f93c29c6dfdff925911e2c48676ab86da96d92b43d8942e2533813\": container with ID starting with 2d952194c2f93c29c6dfdff925911e2c48676ab86da96d92b43d8942e2533813 not found: ID does not exist" containerID="2d952194c2f93c29c6dfdff925911e2c48676ab86da96d92b43d8942e2533813"
Jul 08 17:00:00.347956 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:00:00.347888    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d952194c2f93c29c6dfdff925911e2c48676ab86da96d92b43d8942e2533813"} err="failed to get container status \"2d952194c2f93c29c6dfdff925911e2c48676ab86da96d92b43d8942e2533813\": rpc error: code = NotFound desc = could not find container \"2d952194c2f93c29c6dfdff925911e2c48676ab86da96d92b43d8942e2533813\": container with ID starting with 2d952194c2f93c29c6dfdff925911e2c48676ab86da96d92b43d8942e2533813 not found: ID does not exist"
Jul 08 17:00:00.347956 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:00:00.347914    2578 scope.go:117] "RemoveContainer" containerID="f6f05c7de4a8fc1d51f1217cc62e3636557f06312484db063fe9276becb2e336"
Jul 08 17:00:00.348257 ip-10-0-142-94 kubenswrapper[2578]: E0708 17:00:00.348239    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6f05c7de4a8fc1d51f1217cc62e3636557f06312484db063fe9276becb2e336\": container with ID starting with f6f05c7de4a8fc1d51f1217cc62e3636557f06312484db063fe9276becb2e336 not found: ID does not exist" containerID="f6f05c7de4a8fc1d51f1217cc62e3636557f06312484db063fe9276becb2e336"
Jul 08 17:00:00.348319 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:00:00.348265    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6f05c7de4a8fc1d51f1217cc62e3636557f06312484db063fe9276becb2e336"} err="failed to get container status \"f6f05c7de4a8fc1d51f1217cc62e3636557f06312484db063fe9276becb2e336\": rpc error: code = NotFound desc = could not find container \"f6f05c7de4a8fc1d51f1217cc62e3636557f06312484db063fe9276becb2e336\": container with ID starting with f6f05c7de4a8fc1d51f1217cc62e3636557f06312484db063fe9276becb2e336 not found: ID does not exist"
Jul 08 17:00:01.325707 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:00:01.325660    2578 generic.go:356] "Generic (PLEG): container finished" podID="612009f3-1621-48f9-a0aa-838e07da4734" containerID="a30df5b08411c01869d9ec5c5968c1e1eceea31dc397f72d462d063e97ba6b1d" exitCode=0
Jul 08 17:00:01.326119 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:00:01.325726    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj" event={"ID":"612009f3-1621-48f9-a0aa-838e07da4734","Type":"ContainerDied","Data":"a30df5b08411c01869d9ec5c5968c1e1eceea31dc397f72d462d063e97ba6b1d"}
Jul 08 17:00:02.285092 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:00:02.285050    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3b76551-8e16-4370-a7f3-a5a91577cd3b" path="/var/lib/kubelet/pods/b3b76551-8e16-4370-a7f3-a5a91577cd3b/volumes"
Jul 08 17:00:02.331409 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:00:02.331378    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj" event={"ID":"612009f3-1621-48f9-a0aa-838e07da4734","Type":"ContainerStarted","Data":"6e9609c2ceb6a476d48c049f3ad4ecd080bfd122cb7bd6eaff6aab93fe3ceb60"}
Jul 08 17:00:02.331823 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:00:02.331439    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj" event={"ID":"612009f3-1621-48f9-a0aa-838e07da4734","Type":"ContainerStarted","Data":"41e5a21e2834b11a3eda16e69f45b50cc061f72e78dbb8b9d7e9cb0d2747ddb1"}
Jul 08 17:00:02.331823 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:00:02.331623    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj"
Jul 08 17:00:02.355128 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:00:02.355059    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj" podStartSLOduration=4.296484472 podStartE2EDuration="8.355038745s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.058554273s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:59:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:00:02.353858943 +0000 UTC m=+1514.625308484" watchObservedRunningTime="2026-07-08 17:00:02.355038745 +0000 UTC m=+1514.626488286"
Jul 08 17:00:03.335567 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:00:03.335528    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj"
Jul 08 17:00:03.336605 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:00:03.336579    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj" podUID="612009f3-1621-48f9-a0aa-838e07da4734" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 08 17:00:04.339124 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:00:04.339087    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj" podUID="612009f3-1621-48f9-a0aa-838e07da4734" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 08 17:00:09.344089 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:00:09.344003    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj"
Jul 08 17:00:09.344693 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:00:09.344665    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj" podUID="612009f3-1621-48f9-a0aa-838e07da4734" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 08 17:00:19.344728 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:00:19.344689    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj" podUID="612009f3-1621-48f9-a0aa-838e07da4734" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 08 17:00:29.345551 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:00:29.345508    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj" podUID="612009f3-1621-48f9-a0aa-838e07da4734" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 08 17:00:39.344577 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:00:39.344537    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj" podUID="612009f3-1621-48f9-a0aa-838e07da4734" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 08 17:00:49.344583 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:00:49.344544    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj" podUID="612009f3-1621-48f9-a0aa-838e07da4734" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 08 17:00:59.345513 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:00:59.345474    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj" podUID="612009f3-1621-48f9-a0aa-838e07da4734" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 08 17:01:09.344999 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:09.344951    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj" podUID="612009f3-1621-48f9-a0aa-838e07da4734" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 08 17:01:19.345312 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:19.345280    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj"
Jul 08 17:01:24.533463 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:24.533426    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-ff3865-predictor-6b98f7f6fb-krwgl"]
Jul 08 17:01:24.533883 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:24.533753    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3b76551-8e16-4370-a7f3-a5a91577cd3b" containerName="kserve-container"
Jul 08 17:01:24.533883 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:24.533767    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3b76551-8e16-4370-a7f3-a5a91577cd3b" containerName="kserve-container"
Jul 08 17:01:24.533883 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:24.533778    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3b76551-8e16-4370-a7f3-a5a91577cd3b" containerName="storage-initializer"
Jul 08 17:01:24.533883 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:24.533784    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3b76551-8e16-4370-a7f3-a5a91577cd3b" containerName="storage-initializer"
Jul 08 17:01:24.533883 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:24.533790    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3b76551-8e16-4370-a7f3-a5a91577cd3b" containerName="kube-rbac-proxy"
Jul 08 17:01:24.533883 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:24.533796    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3b76551-8e16-4370-a7f3-a5a91577cd3b" containerName="kube-rbac-proxy"
Jul 08 17:01:24.533883 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:24.533854    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3b76551-8e16-4370-a7f3-a5a91577cd3b" containerName="kube-rbac-proxy"
Jul 08 17:01:24.533883 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:24.533863    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3b76551-8e16-4370-a7f3-a5a91577cd3b" containerName="kserve-container"
Jul 08 17:01:24.536882 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:24.536866    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-ff3865-predictor-6b98f7f6fb-krwgl"
Jul 08 17:01:24.539258 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:24.539222    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-ff3865-predictor-serving-cert\""
Jul 08 17:01:24.539431 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:24.539411    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-ff3865-kube-rbac-proxy-sar-config\""
Jul 08 17:01:24.539513 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:24.539433    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-ff3865-dockercfg-gwwm2\""
Jul 08 17:01:24.539513 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:24.539487    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-ff3865\""
Jul 08 17:01:24.547682 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:24.547661    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-ff3865-predictor-6b98f7f6fb-krwgl"]
Jul 08 17:01:24.674228 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:24.674190    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db0b0dfb-c87f-4735-b90f-558a56c7df01-proxy-tls\") pod \"isvc-secondary-ff3865-predictor-6b98f7f6fb-krwgl\" (UID: \"db0b0dfb-c87f-4735-b90f-558a56c7df01\") " pod="kserve-ci-e2e-test/isvc-secondary-ff3865-predictor-6b98f7f6fb-krwgl"
Jul 08 17:01:24.674430 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:24.674242    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db0b0dfb-c87f-4735-b90f-558a56c7df01-kserve-provision-location\") pod \"isvc-secondary-ff3865-predictor-6b98f7f6fb-krwgl\" (UID: \"db0b0dfb-c87f-4735-b90f-558a56c7df01\") " pod="kserve-ci-e2e-test/isvc-secondary-ff3865-predictor-6b98f7f6fb-krwgl"
Jul 08 17:01:24.674430 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:24.674354    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db0b0dfb-c87f-4735-b90f-558a56c7df01-cabundle-cert\") pod \"isvc-secondary-ff3865-predictor-6b98f7f6fb-krwgl\" (UID: \"db0b0dfb-c87f-4735-b90f-558a56c7df01\") " pod="kserve-ci-e2e-test/isvc-secondary-ff3865-predictor-6b98f7f6fb-krwgl"
Jul 08 17:01:24.674430 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:24.674412    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db0b0dfb-c87f-4735-b90f-558a56c7df01-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-ff3865-predictor-6b98f7f6fb-krwgl\" (UID: \"db0b0dfb-c87f-4735-b90f-558a56c7df01\") " pod="kserve-ci-e2e-test/isvc-secondary-ff3865-predictor-6b98f7f6fb-krwgl"
Jul 08 17:01:24.674583 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:24.674450    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spc54\" (UniqueName: \"kubernetes.io/projected/db0b0dfb-c87f-4735-b90f-558a56c7df01-kube-api-access-spc54\") pod \"isvc-secondary-ff3865-predictor-6b98f7f6fb-krwgl\" (UID: \"db0b0dfb-c87f-4735-b90f-558a56c7df01\") " pod="kserve-ci-e2e-test/isvc-secondary-ff3865-predictor-6b98f7f6fb-krwgl"
Jul 08 17:01:24.775503 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:24.775463    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db0b0dfb-c87f-4735-b90f-558a56c7df01-proxy-tls\") pod \"isvc-secondary-ff3865-predictor-6b98f7f6fb-krwgl\" (UID: \"db0b0dfb-c87f-4735-b90f-558a56c7df01\") " pod="kserve-ci-e2e-test/isvc-secondary-ff3865-predictor-6b98f7f6fb-krwgl"
Jul 08 17:01:24.775696 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:24.775509    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db0b0dfb-c87f-4735-b90f-558a56c7df01-kserve-provision-location\") pod \"isvc-secondary-ff3865-predictor-6b98f7f6fb-krwgl\" (UID: \"db0b0dfb-c87f-4735-b90f-558a56c7df01\") " pod="kserve-ci-e2e-test/isvc-secondary-ff3865-predictor-6b98f7f6fb-krwgl"
Jul 08 17:01:24.775696 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:24.775615    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db0b0dfb-c87f-4735-b90f-558a56c7df01-cabundle-cert\") pod \"isvc-secondary-ff3865-predictor-6b98f7f6fb-krwgl\" (UID: \"db0b0dfb-c87f-4735-b90f-558a56c7df01\") " pod="kserve-ci-e2e-test/isvc-secondary-ff3865-predictor-6b98f7f6fb-krwgl"
Jul 08 17:01:24.775696 ip-10-0-142-94 kubenswrapper[2578]: E0708 17:01:24.775678    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-secondary-ff3865-predictor-serving-cert: secret "isvc-secondary-ff3865-predictor-serving-cert" not found
Jul 08 17:01:24.775696 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:24.775692    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db0b0dfb-c87f-4735-b90f-558a56c7df01-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-ff3865-predictor-6b98f7f6fb-krwgl\" (UID: \"db0b0dfb-c87f-4735-b90f-558a56c7df01\") " pod="kserve-ci-e2e-test/isvc-secondary-ff3865-predictor-6b98f7f6fb-krwgl"
Jul 08 17:01:24.775933 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:24.775732    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spc54\" (UniqueName: \"kubernetes.io/projected/db0b0dfb-c87f-4735-b90f-558a56c7df01-kube-api-access-spc54\") pod \"isvc-secondary-ff3865-predictor-6b98f7f6fb-krwgl\" (UID: \"db0b0dfb-c87f-4735-b90f-558a56c7df01\") " pod="kserve-ci-e2e-test/isvc-secondary-ff3865-predictor-6b98f7f6fb-krwgl"
Jul 08 17:01:24.775933 ip-10-0-142-94 kubenswrapper[2578]: E0708 17:01:24.775780    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db0b0dfb-c87f-4735-b90f-558a56c7df01-proxy-tls podName:db0b0dfb-c87f-4735-b90f-558a56c7df01 nodeName:}" failed. No retries permitted until 2026-07-08 17:01:25.275757352 +0000 UTC m=+1597.547206884 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/db0b0dfb-c87f-4735-b90f-558a56c7df01-proxy-tls") pod "isvc-secondary-ff3865-predictor-6b98f7f6fb-krwgl" (UID: "db0b0dfb-c87f-4735-b90f-558a56c7df01") : secret "isvc-secondary-ff3865-predictor-serving-cert" not found
Jul 08 17:01:24.775933 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:24.775884    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db0b0dfb-c87f-4735-b90f-558a56c7df01-kserve-provision-location\") pod \"isvc-secondary-ff3865-predictor-6b98f7f6fb-krwgl\" (UID: \"db0b0dfb-c87f-4735-b90f-558a56c7df01\") " pod="kserve-ci-e2e-test/isvc-secondary-ff3865-predictor-6b98f7f6fb-krwgl"
Jul 08 17:01:24.776369 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:24.776353    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db0b0dfb-c87f-4735-b90f-558a56c7df01-cabundle-cert\") pod \"isvc-secondary-ff3865-predictor-6b98f7f6fb-krwgl\" (UID: \"db0b0dfb-c87f-4735-b90f-558a56c7df01\") " pod="kserve-ci-e2e-test/isvc-secondary-ff3865-predictor-6b98f7f6fb-krwgl"
Jul 08 17:01:24.776421 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:24.776400    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db0b0dfb-c87f-4735-b90f-558a56c7df01-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-ff3865-predictor-6b98f7f6fb-krwgl\" (UID: \"db0b0dfb-c87f-4735-b90f-558a56c7df01\") " pod="kserve-ci-e2e-test/isvc-secondary-ff3865-predictor-6b98f7f6fb-krwgl"
Jul 08 17:01:24.785580 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:24.785514    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spc54\" (UniqueName: \"kubernetes.io/projected/db0b0dfb-c87f-4735-b90f-558a56c7df01-kube-api-access-spc54\") pod \"isvc-secondary-ff3865-predictor-6b98f7f6fb-krwgl\" (UID: \"db0b0dfb-c87f-4735-b90f-558a56c7df01\") " pod="kserve-ci-e2e-test/isvc-secondary-ff3865-predictor-6b98f7f6fb-krwgl"
Jul 08 17:01:25.281112 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:25.281068    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db0b0dfb-c87f-4735-b90f-558a56c7df01-proxy-tls\") pod \"isvc-secondary-ff3865-predictor-6b98f7f6fb-krwgl\" (UID: \"db0b0dfb-c87f-4735-b90f-558a56c7df01\") " pod="kserve-ci-e2e-test/isvc-secondary-ff3865-predictor-6b98f7f6fb-krwgl"
Jul 08 17:01:25.283584 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:25.283558    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db0b0dfb-c87f-4735-b90f-558a56c7df01-proxy-tls\") pod \"isvc-secondary-ff3865-predictor-6b98f7f6fb-krwgl\" (UID: \"db0b0dfb-c87f-4735-b90f-558a56c7df01\") " pod="kserve-ci-e2e-test/isvc-secondary-ff3865-predictor-6b98f7f6fb-krwgl"
Jul 08 17:01:25.447754 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:25.447712    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-ff3865-predictor-6b98f7f6fb-krwgl"
Jul 08 17:01:25.578033 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:25.577940    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-ff3865-predictor-6b98f7f6fb-krwgl"]
Jul 08 17:01:25.580726 ip-10-0-142-94 kubenswrapper[2578]: W0708 17:01:25.580700    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb0b0dfb_c87f_4735_b90f_558a56c7df01.slice/crio-9fe626aa36d7a94159ed5a9c11a836a558b9eccfc6740e9424aaacf7b7a186f8 WatchSource:0}: Error finding container 9fe626aa36d7a94159ed5a9c11a836a558b9eccfc6740e9424aaacf7b7a186f8: Status 404 returned error can't find the container with id 9fe626aa36d7a94159ed5a9c11a836a558b9eccfc6740e9424aaacf7b7a186f8
Jul 08 17:01:25.659835 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:25.659800    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-ff3865-predictor-6b98f7f6fb-krwgl" event={"ID":"db0b0dfb-c87f-4735-b90f-558a56c7df01","Type":"ContainerStarted","Data":"ef2a9aab8afdfe18d2fc762be74357bc4e32b594652cbbfa590566d17a9ec925"}
Jul 08 17:01:25.659948 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:25.659865    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-ff3865-predictor-6b98f7f6fb-krwgl" event={"ID":"db0b0dfb-c87f-4735-b90f-558a56c7df01","Type":"ContainerStarted","Data":"9fe626aa36d7a94159ed5a9c11a836a558b9eccfc6740e9424aaacf7b7a186f8"}
Jul 08 17:01:28.672722 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:28.672625    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-ff3865-predictor-6b98f7f6fb-krwgl_db0b0dfb-c87f-4735-b90f-558a56c7df01/storage-initializer/0.log"
Jul 08 17:01:28.672722 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:28.672680    2578 generic.go:356] "Generic (PLEG): container finished" podID="db0b0dfb-c87f-4735-b90f-558a56c7df01" containerID="ef2a9aab8afdfe18d2fc762be74357bc4e32b594652cbbfa590566d17a9ec925" exitCode=1
Jul 08 17:01:28.673198 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:28.672760    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-ff3865-predictor-6b98f7f6fb-krwgl" event={"ID":"db0b0dfb-c87f-4735-b90f-558a56c7df01","Type":"ContainerDied","Data":"ef2a9aab8afdfe18d2fc762be74357bc4e32b594652cbbfa590566d17a9ec925"}
Jul 08 17:01:29.679206 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:29.679180    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-ff3865-predictor-6b98f7f6fb-krwgl_db0b0dfb-c87f-4735-b90f-558a56c7df01/storage-initializer/0.log"
Jul 08 17:01:29.679579 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:29.679265    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-ff3865-predictor-6b98f7f6fb-krwgl" event={"ID":"db0b0dfb-c87f-4735-b90f-558a56c7df01","Type":"ContainerStarted","Data":"bf38db215e90ac3ef71c600ff0d3617e9579cbb175f3062dc39e2c3cfc32cd9f"}
Jul 08 17:01:34.700655 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:34.700609    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-ff3865-predictor-6b98f7f6fb-krwgl_db0b0dfb-c87f-4735-b90f-558a56c7df01/storage-initializer/1.log"
Jul 08 17:01:34.701071 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:34.700989    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-ff3865-predictor-6b98f7f6fb-krwgl_db0b0dfb-c87f-4735-b90f-558a56c7df01/storage-initializer/0.log"
Jul 08 17:01:34.701071 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:34.701020    2578 generic.go:356] "Generic (PLEG): container finished" podID="db0b0dfb-c87f-4735-b90f-558a56c7df01" containerID="bf38db215e90ac3ef71c600ff0d3617e9579cbb175f3062dc39e2c3cfc32cd9f" exitCode=1
Jul 08 17:01:34.701157 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:34.701082    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-ff3865-predictor-6b98f7f6fb-krwgl" event={"ID":"db0b0dfb-c87f-4735-b90f-558a56c7df01","Type":"ContainerDied","Data":"bf38db215e90ac3ef71c600ff0d3617e9579cbb175f3062dc39e2c3cfc32cd9f"}
Jul 08 17:01:34.701157 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:34.701136    2578 scope.go:117] "RemoveContainer" containerID="ef2a9aab8afdfe18d2fc762be74357bc4e32b594652cbbfa590566d17a9ec925"
Jul 08 17:01:34.701546 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:34.701524    2578 scope.go:117] "RemoveContainer" containerID="ef2a9aab8afdfe18d2fc762be74357bc4e32b594652cbbfa590566d17a9ec925"
Jul 08 17:01:34.712310 ip-10-0-142-94 kubenswrapper[2578]: E0708 17:01:34.712289    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-ff3865-predictor-6b98f7f6fb-krwgl_kserve-ci-e2e-test(db0b0dfb-c87f-4735-b90f-558a56c7df01)\"" pod="kserve-ci-e2e-test/isvc-secondary-ff3865-predictor-6b98f7f6fb-krwgl" podUID="db0b0dfb-c87f-4735-b90f-558a56c7df01"
Jul 08 17:01:34.712432 ip-10-0-142-94 kubenswrapper[2578]: E0708 17:01:34.712403    2578 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-ff3865-predictor-6b98f7f6fb-krwgl_kserve-ci-e2e-test_db0b0dfb-c87f-4735-b90f-558a56c7df01_0 in pod sandbox 9fe626aa36d7a94159ed5a9c11a836a558b9eccfc6740e9424aaacf7b7a186f8 from index: no such id: 'ef2a9aab8afdfe18d2fc762be74357bc4e32b594652cbbfa590566d17a9ec925'" containerID="ef2a9aab8afdfe18d2fc762be74357bc4e32b594652cbbfa590566d17a9ec925"
Jul 08 17:01:34.712478 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:34.712442    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef2a9aab8afdfe18d2fc762be74357bc4e32b594652cbbfa590566d17a9ec925"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-ff3865-predictor-6b98f7f6fb-krwgl_kserve-ci-e2e-test_db0b0dfb-c87f-4735-b90f-558a56c7df01_0 in pod sandbox 9fe626aa36d7a94159ed5a9c11a836a558b9eccfc6740e9424aaacf7b7a186f8 from index: no such id: 'ef2a9aab8afdfe18d2fc762be74357bc4e32b594652cbbfa590566d17a9ec925'"
Jul 08 17:01:35.706329 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:35.706298    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-ff3865-predictor-6b98f7f6fb-krwgl_db0b0dfb-c87f-4735-b90f-558a56c7df01/storage-initializer/1.log"
Jul 08 17:01:40.545856 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:40.545774    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj"]
Jul 08 17:01:40.546241 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:40.546078    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj" podUID="612009f3-1621-48f9-a0aa-838e07da4734" containerName="kserve-container" containerID="cri-o://41e5a21e2834b11a3eda16e69f45b50cc061f72e78dbb8b9d7e9cb0d2747ddb1" gracePeriod=30
Jul 08 17:01:40.546241 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:40.546112    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj" podUID="612009f3-1621-48f9-a0aa-838e07da4734" containerName="kube-rbac-proxy" containerID="cri-o://6e9609c2ceb6a476d48c049f3ad4ecd080bfd122cb7bd6eaff6aab93fe3ceb60" gracePeriod=30
Jul 08 17:01:40.616390 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:40.616355    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-ff3865-predictor-6b98f7f6fb-krwgl"]
Jul 08 17:01:40.728088 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:40.728057    2578 generic.go:356] "Generic (PLEG): container finished" podID="612009f3-1621-48f9-a0aa-838e07da4734" containerID="6e9609c2ceb6a476d48c049f3ad4ecd080bfd122cb7bd6eaff6aab93fe3ceb60" exitCode=2
Jul 08 17:01:40.728281 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:40.728117    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj" event={"ID":"612009f3-1621-48f9-a0aa-838e07da4734","Type":"ContainerDied","Data":"6e9609c2ceb6a476d48c049f3ad4ecd080bfd122cb7bd6eaff6aab93fe3ceb60"}
Jul 08 17:01:40.758322 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:40.758299    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-ff3865-predictor-6b98f7f6fb-krwgl_db0b0dfb-c87f-4735-b90f-558a56c7df01/storage-initializer/1.log"
Jul 08 17:01:40.760651 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:40.760600    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-ff3865-predictor-6b98f7f6fb-krwgl"
Jul 08 17:01:40.811336 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:40.811307    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db0b0dfb-c87f-4735-b90f-558a56c7df01-kube-rbac-proxy-sar-config\") pod \"db0b0dfb-c87f-4735-b90f-558a56c7df01\" (UID: \"db0b0dfb-c87f-4735-b90f-558a56c7df01\") "
Jul 08 17:01:40.811511 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:40.811351    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db0b0dfb-c87f-4735-b90f-558a56c7df01-kserve-provision-location\") pod \"db0b0dfb-c87f-4735-b90f-558a56c7df01\" (UID: \"db0b0dfb-c87f-4735-b90f-558a56c7df01\") "
Jul 08 17:01:40.811511 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:40.811389    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db0b0dfb-c87f-4735-b90f-558a56c7df01-cabundle-cert\") pod \"db0b0dfb-c87f-4735-b90f-558a56c7df01\" (UID: \"db0b0dfb-c87f-4735-b90f-558a56c7df01\") "
Jul 08 17:01:40.811511 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:40.811441    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db0b0dfb-c87f-4735-b90f-558a56c7df01-proxy-tls\") pod \"db0b0dfb-c87f-4735-b90f-558a56c7df01\" (UID: \"db0b0dfb-c87f-4735-b90f-558a56c7df01\") "
Jul 08 17:01:40.811511 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:40.811456    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-spc54\" (UniqueName: \"kubernetes.io/projected/db0b0dfb-c87f-4735-b90f-558a56c7df01-kube-api-access-spc54\") pod \"db0b0dfb-c87f-4735-b90f-558a56c7df01\" (UID: \"db0b0dfb-c87f-4735-b90f-558a56c7df01\") "
Jul 08 17:01:40.811772 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:40.811714    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db0b0dfb-c87f-4735-b90f-558a56c7df01-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "db0b0dfb-c87f-4735-b90f-558a56c7df01" (UID: "db0b0dfb-c87f-4735-b90f-558a56c7df01"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:01:40.811772 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:40.811744    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db0b0dfb-c87f-4735-b90f-558a56c7df01-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "db0b0dfb-c87f-4735-b90f-558a56c7df01" (UID: "db0b0dfb-c87f-4735-b90f-558a56c7df01"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 17:01:40.811887 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:40.811859    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db0b0dfb-c87f-4735-b90f-558a56c7df01-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "db0b0dfb-c87f-4735-b90f-558a56c7df01" (UID: "db0b0dfb-c87f-4735-b90f-558a56c7df01"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 17:01:40.813726 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:40.813696    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db0b0dfb-c87f-4735-b90f-558a56c7df01-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "db0b0dfb-c87f-4735-b90f-558a56c7df01" (UID: "db0b0dfb-c87f-4735-b90f-558a56c7df01"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:01:40.813824 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:40.813760    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db0b0dfb-c87f-4735-b90f-558a56c7df01-kube-api-access-spc54" (OuterVolumeSpecName: "kube-api-access-spc54") pod "db0b0dfb-c87f-4735-b90f-558a56c7df01" (UID: "db0b0dfb-c87f-4735-b90f-558a56c7df01"). InnerVolumeSpecName "kube-api-access-spc54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:01:40.912001 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:40.911963    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db0b0dfb-c87f-4735-b90f-558a56c7df01-proxy-tls\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 17:01:40.912001 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:40.911995    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spc54\" (UniqueName: \"kubernetes.io/projected/db0b0dfb-c87f-4735-b90f-558a56c7df01-kube-api-access-spc54\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 17:01:40.912001 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:40.912005    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db0b0dfb-c87f-4735-b90f-558a56c7df01-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 17:01:40.912001 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:40.912013    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db0b0dfb-c87f-4735-b90f-558a56c7df01-kserve-provision-location\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 17:01:40.912269 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:40.912022    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db0b0dfb-c87f-4735-b90f-558a56c7df01-cabundle-cert\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 17:01:41.732926 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:41.732899    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-ff3865-predictor-6b98f7f6fb-krwgl_db0b0dfb-c87f-4735-b90f-558a56c7df01/storage-initializer/1.log"
Jul 08 17:01:41.733321 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:41.732969    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-ff3865-predictor-6b98f7f6fb-krwgl" event={"ID":"db0b0dfb-c87f-4735-b90f-558a56c7df01","Type":"ContainerDied","Data":"9fe626aa36d7a94159ed5a9c11a836a558b9eccfc6740e9424aaacf7b7a186f8"}
Jul 08 17:01:41.733321 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:41.733000    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-ff3865-predictor-6b98f7f6fb-krwgl"
Jul 08 17:01:41.733321 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:41.733037    2578 scope.go:117] "RemoveContainer" containerID="bf38db215e90ac3ef71c600ff0d3617e9579cbb175f3062dc39e2c3cfc32cd9f"
Jul 08 17:01:41.770987 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:41.770957    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-ff3865-predictor-6b98f7f6fb-krwgl"]
Jul 08 17:01:41.774541 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:41.774516    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-ff3865-predictor-6b98f7f6fb-krwgl"]
Jul 08 17:01:42.285670 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:42.285616    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db0b0dfb-c87f-4735-b90f-558a56c7df01" path="/var/lib/kubelet/pods/db0b0dfb-c87f-4735-b90f-558a56c7df01/volumes"
Jul 08 17:01:44.339849 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:44.339802    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj" podUID="612009f3-1621-48f9-a0aa-838e07da4734" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 08 17:01:45.087327 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.087304    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj"
Jul 08 17:01:45.144877 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.144811    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/612009f3-1621-48f9-a0aa-838e07da4734-kube-rbac-proxy-sar-config\") pod \"612009f3-1621-48f9-a0aa-838e07da4734\" (UID: \"612009f3-1621-48f9-a0aa-838e07da4734\") "
Jul 08 17:01:45.144877 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.144864    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/612009f3-1621-48f9-a0aa-838e07da4734-cabundle-cert\") pod \"612009f3-1621-48f9-a0aa-838e07da4734\" (UID: \"612009f3-1621-48f9-a0aa-838e07da4734\") "
Jul 08 17:01:45.145040 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.144897    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/612009f3-1621-48f9-a0aa-838e07da4734-proxy-tls\") pod \"612009f3-1621-48f9-a0aa-838e07da4734\" (UID: \"612009f3-1621-48f9-a0aa-838e07da4734\") "
Jul 08 17:01:45.145040 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.144923    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/612009f3-1621-48f9-a0aa-838e07da4734-kserve-provision-location\") pod \"612009f3-1621-48f9-a0aa-838e07da4734\" (UID: \"612009f3-1621-48f9-a0aa-838e07da4734\") "
Jul 08 17:01:45.145040 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.144940    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bj64c\" (UniqueName: \"kubernetes.io/projected/612009f3-1621-48f9-a0aa-838e07da4734-kube-api-access-bj64c\") pod \"612009f3-1621-48f9-a0aa-838e07da4734\" (UID: \"612009f3-1621-48f9-a0aa-838e07da4734\") "
Jul 08 17:01:45.145252 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.145226    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/612009f3-1621-48f9-a0aa-838e07da4734-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "612009f3-1621-48f9-a0aa-838e07da4734" (UID: "612009f3-1621-48f9-a0aa-838e07da4734"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 17:01:45.145306 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.145282    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/612009f3-1621-48f9-a0aa-838e07da4734-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "612009f3-1621-48f9-a0aa-838e07da4734" (UID: "612009f3-1621-48f9-a0aa-838e07da4734"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:01:45.145350 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.145311    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/612009f3-1621-48f9-a0aa-838e07da4734-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "612009f3-1621-48f9-a0aa-838e07da4734" (UID: "612009f3-1621-48f9-a0aa-838e07da4734"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 17:01:45.147039 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.147010    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/612009f3-1621-48f9-a0aa-838e07da4734-kube-api-access-bj64c" (OuterVolumeSpecName: "kube-api-access-bj64c") pod "612009f3-1621-48f9-a0aa-838e07da4734" (UID: "612009f3-1621-48f9-a0aa-838e07da4734"). InnerVolumeSpecName "kube-api-access-bj64c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:01:45.147230 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.147210    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/612009f3-1621-48f9-a0aa-838e07da4734-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "612009f3-1621-48f9-a0aa-838e07da4734" (UID: "612009f3-1621-48f9-a0aa-838e07da4734"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:01:45.246259 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.246231    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/612009f3-1621-48f9-a0aa-838e07da4734-cabundle-cert\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 17:01:45.246259 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.246253    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/612009f3-1621-48f9-a0aa-838e07da4734-proxy-tls\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 17:01:45.246259 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.246262    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/612009f3-1621-48f9-a0aa-838e07da4734-kserve-provision-location\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 17:01:45.246460 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.246270    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bj64c\" (UniqueName: \"kubernetes.io/projected/612009f3-1621-48f9-a0aa-838e07da4734-kube-api-access-bj64c\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 17:01:45.246460 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.246280    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/612009f3-1621-48f9-a0aa-838e07da4734-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 17:01:45.752204 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.752169    2578 generic.go:356] "Generic (PLEG): container finished" podID="612009f3-1621-48f9-a0aa-838e07da4734" containerID="41e5a21e2834b11a3eda16e69f45b50cc061f72e78dbb8b9d7e9cb0d2747ddb1" exitCode=0
Jul 08 17:01:45.752588 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.752234    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj"
Jul 08 17:01:45.752588 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.752248    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj" event={"ID":"612009f3-1621-48f9-a0aa-838e07da4734","Type":"ContainerDied","Data":"41e5a21e2834b11a3eda16e69f45b50cc061f72e78dbb8b9d7e9cb0d2747ddb1"}
Jul 08 17:01:45.752588 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.752314    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj" event={"ID":"612009f3-1621-48f9-a0aa-838e07da4734","Type":"ContainerDied","Data":"e085f0e2795dae629a8ea04ea80c935d6116b9731686e2d0643c8d9b126dcbf8"}
Jul 08 17:01:45.752588 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.752344    2578 scope.go:117] "RemoveContainer" containerID="6e9609c2ceb6a476d48c049f3ad4ecd080bfd122cb7bd6eaff6aab93fe3ceb60"
Jul 08 17:01:45.768093 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.768065    2578 scope.go:117] "RemoveContainer" containerID="41e5a21e2834b11a3eda16e69f45b50cc061f72e78dbb8b9d7e9cb0d2747ddb1"
Jul 08 17:01:45.775346 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.775329    2578 scope.go:117] "RemoveContainer" containerID="a30df5b08411c01869d9ec5c5968c1e1eceea31dc397f72d462d063e97ba6b1d"
Jul 08 17:01:45.777613 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.777592    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj"]
Jul 08 17:01:45.782104 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.782083    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-ff3865-predictor-6c5bfbff9f-jspsj"]
Jul 08 17:01:45.783744 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.783729    2578 scope.go:117] "RemoveContainer" containerID="6e9609c2ceb6a476d48c049f3ad4ecd080bfd122cb7bd6eaff6aab93fe3ceb60"
Jul 08 17:01:45.784003 ip-10-0-142-94 kubenswrapper[2578]: E0708 17:01:45.783983    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e9609c2ceb6a476d48c049f3ad4ecd080bfd122cb7bd6eaff6aab93fe3ceb60\": container with ID starting with 6e9609c2ceb6a476d48c049f3ad4ecd080bfd122cb7bd6eaff6aab93fe3ceb60 not found: ID does not exist" containerID="6e9609c2ceb6a476d48c049f3ad4ecd080bfd122cb7bd6eaff6aab93fe3ceb60"
Jul 08 17:01:45.784045 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.784013    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e9609c2ceb6a476d48c049f3ad4ecd080bfd122cb7bd6eaff6aab93fe3ceb60"} err="failed to get container status \"6e9609c2ceb6a476d48c049f3ad4ecd080bfd122cb7bd6eaff6aab93fe3ceb60\": rpc error: code = NotFound desc = could not find container \"6e9609c2ceb6a476d48c049f3ad4ecd080bfd122cb7bd6eaff6aab93fe3ceb60\": container with ID starting with 6e9609c2ceb6a476d48c049f3ad4ecd080bfd122cb7bd6eaff6aab93fe3ceb60 not found: ID does not exist"
Jul 08 17:01:45.784045 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.784030    2578 scope.go:117] "RemoveContainer" containerID="41e5a21e2834b11a3eda16e69f45b50cc061f72e78dbb8b9d7e9cb0d2747ddb1"
Jul 08 17:01:45.784295 ip-10-0-142-94 kubenswrapper[2578]: E0708 17:01:45.784277    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41e5a21e2834b11a3eda16e69f45b50cc061f72e78dbb8b9d7e9cb0d2747ddb1\": container with ID starting with 41e5a21e2834b11a3eda16e69f45b50cc061f72e78dbb8b9d7e9cb0d2747ddb1 not found: ID does not exist" containerID="41e5a21e2834b11a3eda16e69f45b50cc061f72e78dbb8b9d7e9cb0d2747ddb1"
Jul 08 17:01:45.784348 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.784299    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41e5a21e2834b11a3eda16e69f45b50cc061f72e78dbb8b9d7e9cb0d2747ddb1"} err="failed to get container status \"41e5a21e2834b11a3eda16e69f45b50cc061f72e78dbb8b9d7e9cb0d2747ddb1\": rpc error: code = NotFound desc = could not find container \"41e5a21e2834b11a3eda16e69f45b50cc061f72e78dbb8b9d7e9cb0d2747ddb1\": container with ID starting with 41e5a21e2834b11a3eda16e69f45b50cc061f72e78dbb8b9d7e9cb0d2747ddb1 not found: ID does not exist"
Jul 08 17:01:45.784348 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.784313    2578 scope.go:117] "RemoveContainer" containerID="a30df5b08411c01869d9ec5c5968c1e1eceea31dc397f72d462d063e97ba6b1d"
Jul 08 17:01:45.784572 ip-10-0-142-94 kubenswrapper[2578]: E0708 17:01:45.784549    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a30df5b08411c01869d9ec5c5968c1e1eceea31dc397f72d462d063e97ba6b1d\": container with ID starting with a30df5b08411c01869d9ec5c5968c1e1eceea31dc397f72d462d063e97ba6b1d not found: ID does not exist" containerID="a30df5b08411c01869d9ec5c5968c1e1eceea31dc397f72d462d063e97ba6b1d"
Jul 08 17:01:45.784620 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.784581    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a30df5b08411c01869d9ec5c5968c1e1eceea31dc397f72d462d063e97ba6b1d"} err="failed to get container status \"a30df5b08411c01869d9ec5c5968c1e1eceea31dc397f72d462d063e97ba6b1d\": rpc error: code = NotFound desc = could not find container \"a30df5b08411c01869d9ec5c5968c1e1eceea31dc397f72d462d063e97ba6b1d\": container with ID starting with a30df5b08411c01869d9ec5c5968c1e1eceea31dc397f72d462d063e97ba6b1d not found: ID does not exist"
Jul 08 17:01:45.795336 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.795315    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-491244-predictor-7c78c9fc8-wgp5f"]
Jul 08 17:01:45.795598 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.795587    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="612009f3-1621-48f9-a0aa-838e07da4734" containerName="kube-rbac-proxy"
Jul 08 17:01:45.795676 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.795600    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="612009f3-1621-48f9-a0aa-838e07da4734" containerName="kube-rbac-proxy"
Jul 08 17:01:45.795676 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.795612    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="612009f3-1621-48f9-a0aa-838e07da4734" containerName="storage-initializer"
Jul 08 17:01:45.795676 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.795617    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="612009f3-1621-48f9-a0aa-838e07da4734" containerName="storage-initializer"
Jul 08 17:01:45.795676 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.795647    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="612009f3-1621-48f9-a0aa-838e07da4734" containerName="kserve-container"
Jul 08 17:01:45.795676 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.795656    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="612009f3-1621-48f9-a0aa-838e07da4734" containerName="kserve-container"
Jul 08 17:01:45.795824 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.795682    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db0b0dfb-c87f-4735-b90f-558a56c7df01" containerName="storage-initializer"
Jul 08 17:01:45.795824 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.795692    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="db0b0dfb-c87f-4735-b90f-558a56c7df01" containerName="storage-initializer"
Jul 08 17:01:45.795824 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.795700    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db0b0dfb-c87f-4735-b90f-558a56c7df01" containerName="storage-initializer"
Jul 08 17:01:45.795824 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.795705    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="db0b0dfb-c87f-4735-b90f-558a56c7df01" containerName="storage-initializer"
Jul 08 17:01:45.795824 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.795751    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="db0b0dfb-c87f-4735-b90f-558a56c7df01" containerName="storage-initializer"
Jul 08 17:01:45.795824 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.795758    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="612009f3-1621-48f9-a0aa-838e07da4734" containerName="kserve-container"
Jul 08 17:01:45.795824 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.795766    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="db0b0dfb-c87f-4735-b90f-558a56c7df01" containerName="storage-initializer"
Jul 08 17:01:45.795824 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.795771    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="612009f3-1621-48f9-a0aa-838e07da4734" containerName="kube-rbac-proxy"
Jul 08 17:01:45.799907 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.799893    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-491244-predictor-7c78c9fc8-wgp5f"
Jul 08 17:01:45.802258 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.802240    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-491244-predictor-serving-cert\""
Jul 08 17:01:45.802356 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.802246    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-491244-kube-rbac-proxy-sar-config\""
Jul 08 17:01:45.802706 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.802684    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 08 17:01:45.802706 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.802697    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 08 17:01:45.802860 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.802827    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-491244-dockercfg-fsfff\""
Jul 08 17:01:45.802922 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.802889    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 08 17:01:45.802995 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.802933    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-491244\""
Jul 08 17:01:45.810020 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.809998    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-491244-predictor-7c78c9fc8-wgp5f"]
Jul 08 17:01:45.851709 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.851626    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e3a8f33-eba7-449f-969d-a2bd1b73f5a7-kserve-provision-location\") pod \"isvc-init-fail-491244-predictor-7c78c9fc8-wgp5f\" (UID: \"9e3a8f33-eba7-449f-969d-a2bd1b73f5a7\") " pod="kserve-ci-e2e-test/isvc-init-fail-491244-predictor-7c78c9fc8-wgp5f"
Jul 08 17:01:45.851853 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.851740    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghdtv\" (UniqueName: \"kubernetes.io/projected/9e3a8f33-eba7-449f-969d-a2bd1b73f5a7-kube-api-access-ghdtv\") pod \"isvc-init-fail-491244-predictor-7c78c9fc8-wgp5f\" (UID: \"9e3a8f33-eba7-449f-969d-a2bd1b73f5a7\") " pod="kserve-ci-e2e-test/isvc-init-fail-491244-predictor-7c78c9fc8-wgp5f"
Jul 08 17:01:45.851853 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.851783    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e3a8f33-eba7-449f-969d-a2bd1b73f5a7-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-491244-predictor-7c78c9fc8-wgp5f\" (UID: \"9e3a8f33-eba7-449f-969d-a2bd1b73f5a7\") " pod="kserve-ci-e2e-test/isvc-init-fail-491244-predictor-7c78c9fc8-wgp5f"
Jul 08 17:01:45.851853 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.851810    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9e3a8f33-eba7-449f-969d-a2bd1b73f5a7-cabundle-cert\") pod \"isvc-init-fail-491244-predictor-7c78c9fc8-wgp5f\" (UID: \"9e3a8f33-eba7-449f-969d-a2bd1b73f5a7\") " pod="kserve-ci-e2e-test/isvc-init-fail-491244-predictor-7c78c9fc8-wgp5f"
Jul 08 17:01:45.851978 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.851861    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e3a8f33-eba7-449f-969d-a2bd1b73f5a7-proxy-tls\") pod \"isvc-init-fail-491244-predictor-7c78c9fc8-wgp5f\" (UID: \"9e3a8f33-eba7-449f-969d-a2bd1b73f5a7\") " pod="kserve-ci-e2e-test/isvc-init-fail-491244-predictor-7c78c9fc8-wgp5f"
Jul 08 17:01:45.952793 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.952757    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e3a8f33-eba7-449f-969d-a2bd1b73f5a7-proxy-tls\") pod \"isvc-init-fail-491244-predictor-7c78c9fc8-wgp5f\" (UID: \"9e3a8f33-eba7-449f-969d-a2bd1b73f5a7\") " pod="kserve-ci-e2e-test/isvc-init-fail-491244-predictor-7c78c9fc8-wgp5f"
Jul 08 17:01:45.953027 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.952804    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e3a8f33-eba7-449f-969d-a2bd1b73f5a7-kserve-provision-location\") pod \"isvc-init-fail-491244-predictor-7c78c9fc8-wgp5f\" (UID: \"9e3a8f33-eba7-449f-969d-a2bd1b73f5a7\") " pod="kserve-ci-e2e-test/isvc-init-fail-491244-predictor-7c78c9fc8-wgp5f"
Jul 08 17:01:45.953027 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.952847    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghdtv\" (UniqueName: \"kubernetes.io/projected/9e3a8f33-eba7-449f-969d-a2bd1b73f5a7-kube-api-access-ghdtv\") pod \"isvc-init-fail-491244-predictor-7c78c9fc8-wgp5f\" (UID: \"9e3a8f33-eba7-449f-969d-a2bd1b73f5a7\") " pod="kserve-ci-e2e-test/isvc-init-fail-491244-predictor-7c78c9fc8-wgp5f"
Jul 08 17:01:45.953027 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.952879    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e3a8f33-eba7-449f-969d-a2bd1b73f5a7-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-491244-predictor-7c78c9fc8-wgp5f\" (UID: \"9e3a8f33-eba7-449f-969d-a2bd1b73f5a7\") " pod="kserve-ci-e2e-test/isvc-init-fail-491244-predictor-7c78c9fc8-wgp5f"
Jul 08 17:01:45.953027 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.952907    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9e3a8f33-eba7-449f-969d-a2bd1b73f5a7-cabundle-cert\") pod \"isvc-init-fail-491244-predictor-7c78c9fc8-wgp5f\" (UID: \"9e3a8f33-eba7-449f-969d-a2bd1b73f5a7\") " pod="kserve-ci-e2e-test/isvc-init-fail-491244-predictor-7c78c9fc8-wgp5f"
Jul 08 17:01:45.953340 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.953319    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e3a8f33-eba7-449f-969d-a2bd1b73f5a7-kserve-provision-location\") pod \"isvc-init-fail-491244-predictor-7c78c9fc8-wgp5f\" (UID: \"9e3a8f33-eba7-449f-969d-a2bd1b73f5a7\") " pod="kserve-ci-e2e-test/isvc-init-fail-491244-predictor-7c78c9fc8-wgp5f"
Jul 08 17:01:45.953600 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.953581    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e3a8f33-eba7-449f-969d-a2bd1b73f5a7-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-491244-predictor-7c78c9fc8-wgp5f\" (UID: \"9e3a8f33-eba7-449f-969d-a2bd1b73f5a7\") " pod="kserve-ci-e2e-test/isvc-init-fail-491244-predictor-7c78c9fc8-wgp5f"
Jul 08 17:01:45.953654 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.953608    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9e3a8f33-eba7-449f-969d-a2bd1b73f5a7-cabundle-cert\") pod \"isvc-init-fail-491244-predictor-7c78c9fc8-wgp5f\" (UID: \"9e3a8f33-eba7-449f-969d-a2bd1b73f5a7\") " pod="kserve-ci-e2e-test/isvc-init-fail-491244-predictor-7c78c9fc8-wgp5f"
Jul 08 17:01:45.955386 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.955363    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e3a8f33-eba7-449f-969d-a2bd1b73f5a7-proxy-tls\") pod \"isvc-init-fail-491244-predictor-7c78c9fc8-wgp5f\" (UID: \"9e3a8f33-eba7-449f-969d-a2bd1b73f5a7\") " pod="kserve-ci-e2e-test/isvc-init-fail-491244-predictor-7c78c9fc8-wgp5f"
Jul 08 17:01:45.961082 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:45.961051    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghdtv\" (UniqueName: \"kubernetes.io/projected/9e3a8f33-eba7-449f-969d-a2bd1b73f5a7-kube-api-access-ghdtv\") pod \"isvc-init-fail-491244-predictor-7c78c9fc8-wgp5f\" (UID: \"9e3a8f33-eba7-449f-969d-a2bd1b73f5a7\") " pod="kserve-ci-e2e-test/isvc-init-fail-491244-predictor-7c78c9fc8-wgp5f"
Jul 08 17:01:46.110518 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:46.110472    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-491244-predictor-7c78c9fc8-wgp5f"
Jul 08 17:01:46.246163 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:46.246080    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-491244-predictor-7c78c9fc8-wgp5f"]
Jul 08 17:01:46.248954 ip-10-0-142-94 kubenswrapper[2578]: W0708 17:01:46.248926    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e3a8f33_eba7_449f_969d_a2bd1b73f5a7.slice/crio-f51385db1df0baad917011063c600faf83d472bbd1d5d50d7368bee0efa2cc73 WatchSource:0}: Error finding container f51385db1df0baad917011063c600faf83d472bbd1d5d50d7368bee0efa2cc73: Status 404 returned error can't find the container with id f51385db1df0baad917011063c600faf83d472bbd1d5d50d7368bee0efa2cc73
Jul 08 17:01:46.284851 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:46.284818    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="612009f3-1621-48f9-a0aa-838e07da4734" path="/var/lib/kubelet/pods/612009f3-1621-48f9-a0aa-838e07da4734/volumes"
Jul 08 17:01:46.758482 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:46.758437    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-491244-predictor-7c78c9fc8-wgp5f" event={"ID":"9e3a8f33-eba7-449f-969d-a2bd1b73f5a7","Type":"ContainerStarted","Data":"a3eb1cb021ca41a1d6bca9bc653887a010416a66a8c115c84a5b7cf953f97040"}
Jul 08 17:01:46.758979 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:46.758510    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-491244-predictor-7c78c9fc8-wgp5f" event={"ID":"9e3a8f33-eba7-449f-969d-a2bd1b73f5a7","Type":"ContainerStarted","Data":"f51385db1df0baad917011063c600faf83d472bbd1d5d50d7368bee0efa2cc73"}
Jul 08 17:01:51.779721 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:51.779694    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-491244-predictor-7c78c9fc8-wgp5f_9e3a8f33-eba7-449f-969d-a2bd1b73f5a7/storage-initializer/0.log"
Jul 08 17:01:51.780137 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:51.779736    2578 generic.go:356] "Generic (PLEG): container finished" podID="9e3a8f33-eba7-449f-969d-a2bd1b73f5a7" containerID="a3eb1cb021ca41a1d6bca9bc653887a010416a66a8c115c84a5b7cf953f97040" exitCode=1
Jul 08 17:01:51.780137 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:51.779813    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-491244-predictor-7c78c9fc8-wgp5f" event={"ID":"9e3a8f33-eba7-449f-969d-a2bd1b73f5a7","Type":"ContainerDied","Data":"a3eb1cb021ca41a1d6bca9bc653887a010416a66a8c115c84a5b7cf953f97040"}
Jul 08 17:01:52.785591 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:52.785563    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-491244-predictor-7c78c9fc8-wgp5f_9e3a8f33-eba7-449f-969d-a2bd1b73f5a7/storage-initializer/0.log"
Jul 08 17:01:52.786027 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:52.785691    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-491244-predictor-7c78c9fc8-wgp5f" event={"ID":"9e3a8f33-eba7-449f-969d-a2bd1b73f5a7","Type":"ContainerStarted","Data":"0d4206ac66c016488434e3d09035d0c180a14eee75615dbb544b548ad126078a"}
Jul 08 17:01:55.792658 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:55.792605    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-491244-predictor-7c78c9fc8-wgp5f"]
Jul 08 17:01:55.793074 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:55.792849    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-491244-predictor-7c78c9fc8-wgp5f" podUID="9e3a8f33-eba7-449f-969d-a2bd1b73f5a7" containerName="storage-initializer" containerID="cri-o://0d4206ac66c016488434e3d09035d0c180a14eee75615dbb544b548ad126078a" gracePeriod=30
Jul 08 17:01:56.531338 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:56.531314    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-491244-predictor-7c78c9fc8-wgp5f_9e3a8f33-eba7-449f-969d-a2bd1b73f5a7/storage-initializer/1.log"
Jul 08 17:01:56.531732 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:56.531712    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-491244-predictor-7c78c9fc8-wgp5f_9e3a8f33-eba7-449f-969d-a2bd1b73f5a7/storage-initializer/0.log"
Jul 08 17:01:56.534057 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:56.534039    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-491244-predictor-7c78c9fc8-wgp5f"
Jul 08 17:01:56.644923 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:56.644821    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ghdtv\" (UniqueName: \"kubernetes.io/projected/9e3a8f33-eba7-449f-969d-a2bd1b73f5a7-kube-api-access-ghdtv\") pod \"9e3a8f33-eba7-449f-969d-a2bd1b73f5a7\" (UID: \"9e3a8f33-eba7-449f-969d-a2bd1b73f5a7\") "
Jul 08 17:01:56.645116 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:56.644931    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e3a8f33-eba7-449f-969d-a2bd1b73f5a7-proxy-tls\") pod \"9e3a8f33-eba7-449f-969d-a2bd1b73f5a7\" (UID: \"9e3a8f33-eba7-449f-969d-a2bd1b73f5a7\") "
Jul 08 17:01:56.645116 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:56.644961    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e3a8f33-eba7-449f-969d-a2bd1b73f5a7-kserve-provision-location\") pod \"9e3a8f33-eba7-449f-969d-a2bd1b73f5a7\" (UID: \"9e3a8f33-eba7-449f-969d-a2bd1b73f5a7\") "
Jul 08 17:01:56.645116 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:56.644992    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9e3a8f33-eba7-449f-969d-a2bd1b73f5a7-cabundle-cert\") pod \"9e3a8f33-eba7-449f-969d-a2bd1b73f5a7\" (UID: \"9e3a8f33-eba7-449f-969d-a2bd1b73f5a7\") "
Jul 08 17:01:56.645116 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:56.645032    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e3a8f33-eba7-449f-969d-a2bd1b73f5a7-kube-rbac-proxy-sar-config\") pod \"9e3a8f33-eba7-449f-969d-a2bd1b73f5a7\" (UID: \"9e3a8f33-eba7-449f-969d-a2bd1b73f5a7\") "
Jul 08 17:01:56.645328 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:56.645261    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e3a8f33-eba7-449f-969d-a2bd1b73f5a7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9e3a8f33-eba7-449f-969d-a2bd1b73f5a7" (UID: "9e3a8f33-eba7-449f-969d-a2bd1b73f5a7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:01:56.645463 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:56.645436    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e3a8f33-eba7-449f-969d-a2bd1b73f5a7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9e3a8f33-eba7-449f-969d-a2bd1b73f5a7" (UID: "9e3a8f33-eba7-449f-969d-a2bd1b73f5a7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 17:01:56.645588 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:56.645440    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e3a8f33-eba7-449f-969d-a2bd1b73f5a7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9e3a8f33-eba7-449f-969d-a2bd1b73f5a7" (UID: "9e3a8f33-eba7-449f-969d-a2bd1b73f5a7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 17:01:56.647258 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:56.647217    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e3a8f33-eba7-449f-969d-a2bd1b73f5a7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9e3a8f33-eba7-449f-969d-a2bd1b73f5a7" (UID: "9e3a8f33-eba7-449f-969d-a2bd1b73f5a7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:01:56.647377 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:56.647261    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e3a8f33-eba7-449f-969d-a2bd1b73f5a7-kube-api-access-ghdtv" (OuterVolumeSpecName: "kube-api-access-ghdtv") pod "9e3a8f33-eba7-449f-969d-a2bd1b73f5a7" (UID: "9e3a8f33-eba7-449f-969d-a2bd1b73f5a7"). InnerVolumeSpecName "kube-api-access-ghdtv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:01:56.745601 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:56.745560    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghdtv\" (UniqueName: \"kubernetes.io/projected/9e3a8f33-eba7-449f-969d-a2bd1b73f5a7-kube-api-access-ghdtv\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 17:01:56.745601 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:56.745597    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e3a8f33-eba7-449f-969d-a2bd1b73f5a7-proxy-tls\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 17:01:56.745601 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:56.745608    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e3a8f33-eba7-449f-969d-a2bd1b73f5a7-kserve-provision-location\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 17:01:56.745887 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:56.745617    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9e3a8f33-eba7-449f-969d-a2bd1b73f5a7-cabundle-cert\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 17:01:56.745887 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:56.745625    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e3a8f33-eba7-449f-969d-a2bd1b73f5a7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 17:01:56.801594 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:56.801563    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-491244-predictor-7c78c9fc8-wgp5f_9e3a8f33-eba7-449f-969d-a2bd1b73f5a7/storage-initializer/1.log"
Jul 08 17:01:56.802089 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:56.802000    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-491244-predictor-7c78c9fc8-wgp5f_9e3a8f33-eba7-449f-969d-a2bd1b73f5a7/storage-initializer/0.log"
Jul 08 17:01:56.802089 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:56.802041    2578 generic.go:356] "Generic (PLEG): container finished" podID="9e3a8f33-eba7-449f-969d-a2bd1b73f5a7" containerID="0d4206ac66c016488434e3d09035d0c180a14eee75615dbb544b548ad126078a" exitCode=1
Jul 08 17:01:56.802089 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:56.802075    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-491244-predictor-7c78c9fc8-wgp5f" event={"ID":"9e3a8f33-eba7-449f-969d-a2bd1b73f5a7","Type":"ContainerDied","Data":"0d4206ac66c016488434e3d09035d0c180a14eee75615dbb544b548ad126078a"}
Jul 08 17:01:56.802254 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:56.802096    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-491244-predictor-7c78c9fc8-wgp5f"
Jul 08 17:01:56.802254 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:56.802183    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-491244-predictor-7c78c9fc8-wgp5f" event={"ID":"9e3a8f33-eba7-449f-969d-a2bd1b73f5a7","Type":"ContainerDied","Data":"f51385db1df0baad917011063c600faf83d472bbd1d5d50d7368bee0efa2cc73"}
Jul 08 17:01:56.802254 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:56.802228    2578 scope.go:117] "RemoveContainer" containerID="0d4206ac66c016488434e3d09035d0c180a14eee75615dbb544b548ad126078a"
Jul 08 17:01:56.811937 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:56.811918    2578 scope.go:117] "RemoveContainer" containerID="a3eb1cb021ca41a1d6bca9bc653887a010416a66a8c115c84a5b7cf953f97040"
Jul 08 17:01:56.819912 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:56.819888    2578 scope.go:117] "RemoveContainer" containerID="0d4206ac66c016488434e3d09035d0c180a14eee75615dbb544b548ad126078a"
Jul 08 17:01:56.820209 ip-10-0-142-94 kubenswrapper[2578]: E0708 17:01:56.820189    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d4206ac66c016488434e3d09035d0c180a14eee75615dbb544b548ad126078a\": container with ID starting with 0d4206ac66c016488434e3d09035d0c180a14eee75615dbb544b548ad126078a not found: ID does not exist" containerID="0d4206ac66c016488434e3d09035d0c180a14eee75615dbb544b548ad126078a"
Jul 08 17:01:56.820266 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:56.820219    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d4206ac66c016488434e3d09035d0c180a14eee75615dbb544b548ad126078a"} err="failed to get container status \"0d4206ac66c016488434e3d09035d0c180a14eee75615dbb544b548ad126078a\": rpc error: code = NotFound desc = could not find container \"0d4206ac66c016488434e3d09035d0c180a14eee75615dbb544b548ad126078a\": container with ID starting with 0d4206ac66c016488434e3d09035d0c180a14eee75615dbb544b548ad126078a not found: ID does not exist"
Jul 08 17:01:56.820266 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:56.820238    2578 scope.go:117] "RemoveContainer" containerID="a3eb1cb021ca41a1d6bca9bc653887a010416a66a8c115c84a5b7cf953f97040"
Jul 08 17:01:56.820493 ip-10-0-142-94 kubenswrapper[2578]: E0708 17:01:56.820474    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3eb1cb021ca41a1d6bca9bc653887a010416a66a8c115c84a5b7cf953f97040\": container with ID starting with a3eb1cb021ca41a1d6bca9bc653887a010416a66a8c115c84a5b7cf953f97040 not found: ID does not exist" containerID="a3eb1cb021ca41a1d6bca9bc653887a010416a66a8c115c84a5b7cf953f97040"
Jul 08 17:01:56.820551 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:56.820503    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3eb1cb021ca41a1d6bca9bc653887a010416a66a8c115c84a5b7cf953f97040"} err="failed to get container status \"a3eb1cb021ca41a1d6bca9bc653887a010416a66a8c115c84a5b7cf953f97040\": rpc error: code = NotFound desc = could not find container \"a3eb1cb021ca41a1d6bca9bc653887a010416a66a8c115c84a5b7cf953f97040\": container with ID starting with a3eb1cb021ca41a1d6bca9bc653887a010416a66a8c115c84a5b7cf953f97040 not found: ID does not exist"
Jul 08 17:01:56.836315 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:56.836275    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-491244-predictor-7c78c9fc8-wgp5f"]
Jul 08 17:01:56.842105 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:56.842072    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-491244-predictor-7c78c9fc8-wgp5f"]
Jul 08 17:01:58.285395 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:01:58.285348    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e3a8f33-eba7-449f-969d-a2bd1b73f5a7" path="/var/lib/kubelet/pods/9e3a8f33-eba7-449f-969d-a2bd1b73f5a7/volumes"
Jul 08 17:02:01.045049 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:02:01.045006    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-f05f8-predictor-54888985c8-gwssx"]
Jul 08 17:02:01.045582 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:02:01.045420    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e3a8f33-eba7-449f-969d-a2bd1b73f5a7" containerName="storage-initializer"
Jul 08 17:02:01.045582 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:02:01.045434    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e3a8f33-eba7-449f-969d-a2bd1b73f5a7" containerName="storage-initializer"
Jul 08 17:02:01.045582 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:02:01.045513    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e3a8f33-eba7-449f-969d-a2bd1b73f5a7" containerName="storage-initializer"
Jul 08 17:02:01.045582 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:02:01.045525    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e3a8f33-eba7-449f-969d-a2bd1b73f5a7" containerName="storage-initializer"
Jul 08 17:02:01.045792 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:02:01.045611    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e3a8f33-eba7-449f-969d-a2bd1b73f5a7" containerName="storage-initializer"
Jul 08 17:02:01.045792 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:02:01.045621    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e3a8f33-eba7-449f-969d-a2bd1b73f5a7" containerName="storage-initializer"
Jul 08 17:02:01.050286 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:02:01.050263    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-f05f8-predictor-54888985c8-gwssx"
Jul 08 17:02:01.053009 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:02:01.052867    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-vkqpp\""
Jul 08 17:02:01.053009 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:02:01.052880    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 08 17:02:01.053009 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:02:01.052886    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-f05f8-predictor-serving-cert\""
Jul 08 17:02:01.053009 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:02:01.052896    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 08 17:02:01.053009 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:02:01.052965    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 08 17:02:01.053368 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:02:01.053052    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-f05f8-kube-rbac-proxy-sar-config\""
Jul 08 17:02:01.053423 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:02:01.053398    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 08 17:02:01.060776 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:02:01.060750    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-f05f8-predictor-54888985c8-gwssx"]
Jul 08 17:02:01.185323 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:02:01.185282    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkgcm\" (UniqueName: \"kubernetes.io/projected/e5afccf1-df32-44fb-9ce5-e0487e67eaef-kube-api-access-qkgcm\") pod \"raw-sklearn-f05f8-predictor-54888985c8-gwssx\" (UID: \"e5afccf1-df32-44fb-9ce5-e0487e67eaef\") " pod="kserve-ci-e2e-test/raw-sklearn-f05f8-predictor-54888985c8-gwssx"
Jul 08 17:02:01.185514 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:02:01.185333    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5afccf1-df32-44fb-9ce5-e0487e67eaef-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-f05f8-predictor-54888985c8-gwssx\" (UID: \"e5afccf1-df32-44fb-9ce5-e0487e67eaef\") " pod="kserve-ci-e2e-test/raw-sklearn-f05f8-predictor-54888985c8-gwssx"
Jul 08 17:02:01.185514 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:02:01.185359    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e5afccf1-df32-44fb-9ce5-e0487e67eaef-cabundle-cert\") pod \"raw-sklearn-f05f8-predictor-54888985c8-gwssx\" (UID: \"e5afccf1-df32-44fb-9ce5-e0487e67eaef\") " pod="kserve-ci-e2e-test/raw-sklearn-f05f8-predictor-54888985c8-gwssx"
Jul 08 17:02:01.185514 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:02:01.185431    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5afccf1-df32-44fb-9ce5-e0487e67eaef-proxy-tls\") pod \"raw-sklearn-f05f8-predictor-54888985c8-gwssx\" (UID: \"e5afccf1-df32-44fb-9ce5-e0487e67eaef\") " pod="kserve-ci-e2e-test/raw-sklearn-f05f8-predictor-54888985c8-gwssx"
Jul 08 17:02:01.185514 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:02:01.185461    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5afccf1-df32-44fb-9ce5-e0487e67eaef-kserve-provision-location\") pod \"raw-sklearn-f05f8-predictor-54888985c8-gwssx\" (UID: \"e5afccf1-df32-44fb-9ce5-e0487e67eaef\") " pod="kserve-ci-e2e-test/raw-sklearn-f05f8-predictor-54888985c8-gwssx"
Jul 08 17:02:01.286814 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:02:01.286782    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5afccf1-df32-44fb-9ce5-e0487e67eaef-proxy-tls\") pod \"raw-sklearn-f05f8-predictor-54888985c8-gwssx\" (UID: \"e5afccf1-df32-44fb-9ce5-e0487e67eaef\") " pod="kserve-ci-e2e-test/raw-sklearn-f05f8-predictor-54888985c8-gwssx"
Jul 08 17:02:01.286814 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:02:01.286819    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5afccf1-df32-44fb-9ce5-e0487e67eaef-kserve-provision-location\") pod \"raw-sklearn-f05f8-predictor-54888985c8-gwssx\" (UID: \"e5afccf1-df32-44fb-9ce5-e0487e67eaef\") " pod="kserve-ci-e2e-test/raw-sklearn-f05f8-predictor-54888985c8-gwssx"
Jul 08 17:02:01.287072 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:02:01.286893    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkgcm\" (UniqueName: \"kubernetes.io/projected/e5afccf1-df32-44fb-9ce5-e0487e67eaef-kube-api-access-qkgcm\") pod \"raw-sklearn-f05f8-predictor-54888985c8-gwssx\" (UID: \"e5afccf1-df32-44fb-9ce5-e0487e67eaef\") " pod="kserve-ci-e2e-test/raw-sklearn-f05f8-predictor-54888985c8-gwssx"
Jul 08 17:02:01.287072 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:02:01.286927    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5afccf1-df32-44fb-9ce5-e0487e67eaef-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-f05f8-predictor-54888985c8-gwssx\" (UID: \"e5afccf1-df32-44fb-9ce5-e0487e67eaef\") " pod="kserve-ci-e2e-test/raw-sklearn-f05f8-predictor-54888985c8-gwssx"
Jul 08 17:02:01.287072 ip-10-0-142-94 kubenswrapper[2578]: E0708 17:02:01.286931    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/raw-sklearn-f05f8-predictor-serving-cert: secret "raw-sklearn-f05f8-predictor-serving-cert" not found
Jul 08 17:02:01.287072 ip-10-0-142-94 kubenswrapper[2578]: E0708 17:02:01.286986    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5afccf1-df32-44fb-9ce5-e0487e67eaef-proxy-tls podName:e5afccf1-df32-44fb-9ce5-e0487e67eaef nodeName:}" failed. No retries permitted until 2026-07-08 17:02:01.786971744 +0000 UTC m=+1634.058421263 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e5afccf1-df32-44fb-9ce5-e0487e67eaef-proxy-tls") pod "raw-sklearn-f05f8-predictor-54888985c8-gwssx" (UID: "e5afccf1-df32-44fb-9ce5-e0487e67eaef") : secret "raw-sklearn-f05f8-predictor-serving-cert" not found
Jul 08 17:02:01.287072 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:02:01.287045    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e5afccf1-df32-44fb-9ce5-e0487e67eaef-cabundle-cert\") pod \"raw-sklearn-f05f8-predictor-54888985c8-gwssx\" (UID: \"e5afccf1-df32-44fb-9ce5-e0487e67eaef\") " pod="kserve-ci-e2e-test/raw-sklearn-f05f8-predictor-54888985c8-gwssx"
Jul 08 17:02:01.287410 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:02:01.287392    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5afccf1-df32-44fb-9ce5-e0487e67eaef-kserve-provision-location\") pod \"raw-sklearn-f05f8-predictor-54888985c8-gwssx\" (UID: \"e5afccf1-df32-44fb-9ce5-e0487e67eaef\") " pod="kserve-ci-e2e-test/raw-sklearn-f05f8-predictor-54888985c8-gwssx"
Jul 08 17:02:01.287691 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:02:01.287667    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e5afccf1-df32-44fb-9ce5-e0487e67eaef-cabundle-cert\") pod \"raw-sklearn-f05f8-predictor-54888985c8-gwssx\" (UID: \"e5afccf1-df32-44fb-9ce5-e0487e67eaef\") " pod="kserve-ci-e2e-test/raw-sklearn-f05f8-predictor-54888985c8-gwssx"
Jul 08 17:02:01.287786 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:02:01.287750    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5afccf1-df32-44fb-9ce5-e0487e67eaef-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-f05f8-predictor-54888985c8-gwssx\" (UID: \"e5afccf1-df32-44fb-9ce5-e0487e67eaef\") " pod="kserve-ci-e2e-test/raw-sklearn-f05f8-predictor-54888985c8-gwssx"
Jul 08 17:02:01.296242 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:02:01.296188    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkgcm\" (UniqueName: \"kubernetes.io/projected/e5afccf1-df32-44fb-9ce5-e0487e67eaef-kube-api-access-qkgcm\") pod \"raw-sklearn-f05f8-predictor-54888985c8-gwssx\" (UID: \"e5afccf1-df32-44fb-9ce5-e0487e67eaef\") " pod="kserve-ci-e2e-test/raw-sklearn-f05f8-predictor-54888985c8-gwssx"
Jul 08 17:02:01.791682 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:02:01.791625    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5afccf1-df32-44fb-9ce5-e0487e67eaef-proxy-tls\") pod \"raw-sklearn-f05f8-predictor-54888985c8-gwssx\" (UID: \"e5afccf1-df32-44fb-9ce5-e0487e67eaef\") " pod="kserve-ci-e2e-test/raw-sklearn-f05f8-predictor-54888985c8-gwssx"
Jul 08 17:02:01.794349 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:02:01.794314    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5afccf1-df32-44fb-9ce5-e0487e67eaef-proxy-tls\") pod \"raw-sklearn-f05f8-predictor-54888985c8-gwssx\" (UID: \"e5afccf1-df32-44fb-9ce5-e0487e67eaef\") " pod="kserve-ci-e2e-test/raw-sklearn-f05f8-predictor-54888985c8-gwssx"
Jul 08 17:02:01.962005 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:02:01.961969    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-f05f8-predictor-54888985c8-gwssx"
Jul 08 17:02:02.100180 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:02:02.100147    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-f05f8-predictor-54888985c8-gwssx"]
Jul 08 17:02:02.103356 ip-10-0-142-94 kubenswrapper[2578]: W0708 17:02:02.103327    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5afccf1_df32_44fb_9ce5_e0487e67eaef.slice/crio-f66e7381779db067276db7ff6e45bb4be39574ae7c8b78b4da605d3446f2c899 WatchSource:0}: Error finding container f66e7381779db067276db7ff6e45bb4be39574ae7c8b78b4da605d3446f2c899: Status 404 returned error can't find the container with id f66e7381779db067276db7ff6e45bb4be39574ae7c8b78b4da605d3446f2c899
Jul 08 17:02:02.827858 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:02:02.827822    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-f05f8-predictor-54888985c8-gwssx" event={"ID":"e5afccf1-df32-44fb-9ce5-e0487e67eaef","Type":"ContainerStarted","Data":"4b11de1cda1edd93d827e1bd35cd7499901f9e181c12b9bbed5d8a3e7ef835b0"}
Jul 08 17:02:02.828046 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:02:02.827886    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-f05f8-predictor-54888985c8-gwssx" event={"ID":"e5afccf1-df32-44fb-9ce5-e0487e67eaef","Type":"ContainerStarted","Data":"f66e7381779db067276db7ff6e45bb4be39574ae7c8b78b4da605d3446f2c899"}
Jul 08 17:02:06.844770 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:02:06.844730    2578 generic.go:356] "Generic (PLEG): container finished" podID="e5afccf1-df32-44fb-9ce5-e0487e67eaef" containerID="4b11de1cda1edd93d827e1bd35cd7499901f9e181c12b9bbed5d8a3e7ef835b0" exitCode=0
Jul 08 17:02:06.844770 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:02:06.844771    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-f05f8-predictor-54888985c8-gwssx" event={"ID":"e5afccf1-df32-44fb-9ce5-e0487e67eaef","Type":"ContainerDied","Data":"4b11de1cda1edd93d827e1bd35cd7499901f9e181c12b9bbed5d8a3e7ef835b0"}
Jul 08 17:02:07.850862 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:02:07.850829    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-f05f8-predictor-54888985c8-gwssx" event={"ID":"e5afccf1-df32-44fb-9ce5-e0487e67eaef","Type":"ContainerStarted","Data":"b3b936057c5a52e7a20320dc8322459907f81e9877f6691690eb0e0ee170328f"}
Jul 08 17:02:07.851296 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:02:07.850889    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-f05f8-predictor-54888985c8-gwssx" event={"ID":"e5afccf1-df32-44fb-9ce5-e0487e67eaef","Type":"ContainerStarted","Data":"c6153e9542792ed0f7c16bcbb7c9aac058237c1a70b3ed0ee9f95d74e5180900"}
Jul 08 17:02:07.851296 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:02:07.851153    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-f05f8-predictor-54888985c8-gwssx"
Jul 08 17:02:07.851296 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:02:07.851198    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-f05f8-predictor-54888985c8-gwssx"
Jul 08 17:02:07.852413 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:02:07.852384    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-f05f8-predictor-54888985c8-gwssx" podUID="e5afccf1-df32-44fb-9ce5-e0487e67eaef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 08 17:02:07.871696 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:02:07.871650    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-f05f8-predictor-54888985c8-gwssx" podStartSLOduration=3.071691355 podStartE2EDuration="6.871615708s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.799924353s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:02:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:02:07.869942986 +0000 UTC m=+1640.141392528" watchObservedRunningTime="2026-07-08 17:02:07.871615708 +0000 UTC m=+1640.143065249"
Jul 08 17:02:08.856032 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:02:08.855986    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-f05f8-predictor-54888985c8-gwssx" podUID="e5afccf1-df32-44fb-9ce5-e0487e67eaef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 08 17:02:13.860996 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:02:13.860962    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-f05f8-predictor-54888985c8-gwssx"
Jul 08 17:02:13.861564 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:02:13.861532    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-f05f8-predictor-54888985c8-gwssx" podUID="e5afccf1-df32-44fb-9ce5-e0487e67eaef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 08 17:02:23.861902 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:02:23.861860    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-f05f8-predictor-54888985c8-gwssx" podUID="e5afccf1-df32-44fb-9ce5-e0487e67eaef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 08 17:02:33.862475 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:02:33.862427    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-f05f8-predictor-54888985c8-gwssx" podUID="e5afccf1-df32-44fb-9ce5-e0487e67eaef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 08 17:02:43.862137 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:02:43.862091    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-f05f8-predictor-54888985c8-gwssx" podUID="e5afccf1-df32-44fb-9ce5-e0487e67eaef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 08 17:02:53.861648 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:02:53.861592    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-f05f8-predictor-54888985c8-gwssx" podUID="e5afccf1-df32-44fb-9ce5-e0487e67eaef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 08 17:03:03.861533 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:03.861483    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-f05f8-predictor-54888985c8-gwssx" podUID="e5afccf1-df32-44fb-9ce5-e0487e67eaef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 08 17:03:13.861531 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:13.861448    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-f05f8-predictor-54888985c8-gwssx" podUID="e5afccf1-df32-44fb-9ce5-e0487e67eaef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 08 17:03:23.862380 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:23.862348    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-f05f8-predictor-54888985c8-gwssx"
Jul 08 17:03:31.170922 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:31.170890    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-f05f8-predictor-54888985c8-gwssx"]
Jul 08 17:03:31.171488 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:31.171168    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-f05f8-predictor-54888985c8-gwssx" podUID="e5afccf1-df32-44fb-9ce5-e0487e67eaef" containerName="kserve-container" containerID="cri-o://c6153e9542792ed0f7c16bcbb7c9aac058237c1a70b3ed0ee9f95d74e5180900" gracePeriod=30
Jul 08 17:03:31.171488 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:31.171235    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-f05f8-predictor-54888985c8-gwssx" podUID="e5afccf1-df32-44fb-9ce5-e0487e67eaef" containerName="kube-rbac-proxy" containerID="cri-o://b3b936057c5a52e7a20320dc8322459907f81e9877f6691690eb0e0ee170328f" gracePeriod=30
Jul 08 17:03:32.184585 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:32.184551    2578 generic.go:356] "Generic (PLEG): container finished" podID="e5afccf1-df32-44fb-9ce5-e0487e67eaef" containerID="b3b936057c5a52e7a20320dc8322459907f81e9877f6691690eb0e0ee170328f" exitCode=2
Jul 08 17:03:32.184978 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:32.184622    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-f05f8-predictor-54888985c8-gwssx" event={"ID":"e5afccf1-df32-44fb-9ce5-e0487e67eaef","Type":"ContainerDied","Data":"b3b936057c5a52e7a20320dc8322459907f81e9877f6691690eb0e0ee170328f"}
Jul 08 17:03:33.856296 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:33.856255    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-f05f8-predictor-54888985c8-gwssx" podUID="e5afccf1-df32-44fb-9ce5-e0487e67eaef" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Jul 08 17:03:33.861845 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:33.861821    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-f05f8-predictor-54888985c8-gwssx" podUID="e5afccf1-df32-44fb-9ce5-e0487e67eaef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 08 17:03:35.819175 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:35.819148    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-f05f8-predictor-54888985c8-gwssx"
Jul 08 17:03:35.890157 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:35.890063    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5afccf1-df32-44fb-9ce5-e0487e67eaef-kube-rbac-proxy-sar-config\") pod \"e5afccf1-df32-44fb-9ce5-e0487e67eaef\" (UID: \"e5afccf1-df32-44fb-9ce5-e0487e67eaef\") "
Jul 08 17:03:35.890157 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:35.890118    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e5afccf1-df32-44fb-9ce5-e0487e67eaef-cabundle-cert\") pod \"e5afccf1-df32-44fb-9ce5-e0487e67eaef\" (UID: \"e5afccf1-df32-44fb-9ce5-e0487e67eaef\") "
Jul 08 17:03:35.890343 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:35.890189    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qkgcm\" (UniqueName: \"kubernetes.io/projected/e5afccf1-df32-44fb-9ce5-e0487e67eaef-kube-api-access-qkgcm\") pod \"e5afccf1-df32-44fb-9ce5-e0487e67eaef\" (UID: \"e5afccf1-df32-44fb-9ce5-e0487e67eaef\") "
Jul 08 17:03:35.890343 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:35.890215    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5afccf1-df32-44fb-9ce5-e0487e67eaef-proxy-tls\") pod \"e5afccf1-df32-44fb-9ce5-e0487e67eaef\" (UID: \"e5afccf1-df32-44fb-9ce5-e0487e67eaef\") "
Jul 08 17:03:35.890343 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:35.890240    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5afccf1-df32-44fb-9ce5-e0487e67eaef-kserve-provision-location\") pod \"e5afccf1-df32-44fb-9ce5-e0487e67eaef\" (UID: \"e5afccf1-df32-44fb-9ce5-e0487e67eaef\") "
Jul 08 17:03:35.890513 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:35.890492    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5afccf1-df32-44fb-9ce5-e0487e67eaef-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e5afccf1-df32-44fb-9ce5-e0487e67eaef" (UID: "e5afccf1-df32-44fb-9ce5-e0487e67eaef"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 17:03:35.890674 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:35.890520    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5afccf1-df32-44fb-9ce5-e0487e67eaef-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e5afccf1-df32-44fb-9ce5-e0487e67eaef" (UID: "e5afccf1-df32-44fb-9ce5-e0487e67eaef"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 17:03:35.890798 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:35.890687    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5afccf1-df32-44fb-9ce5-e0487e67eaef-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e5afccf1-df32-44fb-9ce5-e0487e67eaef" (UID: "e5afccf1-df32-44fb-9ce5-e0487e67eaef"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:03:35.892482 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:35.892460    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5afccf1-df32-44fb-9ce5-e0487e67eaef-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e5afccf1-df32-44fb-9ce5-e0487e67eaef" (UID: "e5afccf1-df32-44fb-9ce5-e0487e67eaef"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:03:35.892544 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:35.892477    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5afccf1-df32-44fb-9ce5-e0487e67eaef-kube-api-access-qkgcm" (OuterVolumeSpecName: "kube-api-access-qkgcm") pod "e5afccf1-df32-44fb-9ce5-e0487e67eaef" (UID: "e5afccf1-df32-44fb-9ce5-e0487e67eaef"). InnerVolumeSpecName "kube-api-access-qkgcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:03:35.991592 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:35.991549    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5afccf1-df32-44fb-9ce5-e0487e67eaef-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 17:03:35.991592 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:35.991577    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e5afccf1-df32-44fb-9ce5-e0487e67eaef-cabundle-cert\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 17:03:35.991592 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:35.991588    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkgcm\" (UniqueName: \"kubernetes.io/projected/e5afccf1-df32-44fb-9ce5-e0487e67eaef-kube-api-access-qkgcm\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 17:03:35.991592 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:35.991597    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5afccf1-df32-44fb-9ce5-e0487e67eaef-proxy-tls\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 17:03:35.991592 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:35.991607    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5afccf1-df32-44fb-9ce5-e0487e67eaef-kserve-provision-location\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 17:03:36.203207 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:36.203118    2578 generic.go:356] "Generic (PLEG): container finished" podID="e5afccf1-df32-44fb-9ce5-e0487e67eaef" containerID="c6153e9542792ed0f7c16bcbb7c9aac058237c1a70b3ed0ee9f95d74e5180900" exitCode=0
Jul 08 17:03:36.203207 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:36.203176    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-f05f8-predictor-54888985c8-gwssx"
Jul 08 17:03:36.203207 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:36.203196    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-f05f8-predictor-54888985c8-gwssx" event={"ID":"e5afccf1-df32-44fb-9ce5-e0487e67eaef","Type":"ContainerDied","Data":"c6153e9542792ed0f7c16bcbb7c9aac058237c1a70b3ed0ee9f95d74e5180900"}
Jul 08 17:03:36.203452 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:36.203257    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-f05f8-predictor-54888985c8-gwssx" event={"ID":"e5afccf1-df32-44fb-9ce5-e0487e67eaef","Type":"ContainerDied","Data":"f66e7381779db067276db7ff6e45bb4be39574ae7c8b78b4da605d3446f2c899"}
Jul 08 17:03:36.203452 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:36.203285    2578 scope.go:117] "RemoveContainer" containerID="b3b936057c5a52e7a20320dc8322459907f81e9877f6691690eb0e0ee170328f"
Jul 08 17:03:36.211970 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:36.211951    2578 scope.go:117] "RemoveContainer" containerID="c6153e9542792ed0f7c16bcbb7c9aac058237c1a70b3ed0ee9f95d74e5180900"
Jul 08 17:03:36.220069 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:36.220050    2578 scope.go:117] "RemoveContainer" containerID="4b11de1cda1edd93d827e1bd35cd7499901f9e181c12b9bbed5d8a3e7ef835b0"
Jul 08 17:03:36.223330 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:36.223303    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-f05f8-predictor-54888985c8-gwssx"]
Jul 08 17:03:36.227927 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:36.227907    2578 scope.go:117] "RemoveContainer" containerID="b3b936057c5a52e7a20320dc8322459907f81e9877f6691690eb0e0ee170328f"
Jul 08 17:03:36.228284 ip-10-0-142-94 kubenswrapper[2578]: E0708 17:03:36.228246    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3b936057c5a52e7a20320dc8322459907f81e9877f6691690eb0e0ee170328f\": container with ID starting with b3b936057c5a52e7a20320dc8322459907f81e9877f6691690eb0e0ee170328f not found: ID does not exist" containerID="b3b936057c5a52e7a20320dc8322459907f81e9877f6691690eb0e0ee170328f"
Jul 08 17:03:36.228373 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:36.228288    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3b936057c5a52e7a20320dc8322459907f81e9877f6691690eb0e0ee170328f"} err="failed to get container status \"b3b936057c5a52e7a20320dc8322459907f81e9877f6691690eb0e0ee170328f\": rpc error: code = NotFound desc = could not find container \"b3b936057c5a52e7a20320dc8322459907f81e9877f6691690eb0e0ee170328f\": container with ID starting with b3b936057c5a52e7a20320dc8322459907f81e9877f6691690eb0e0ee170328f not found: ID does not exist"
Jul 08 17:03:36.228373 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:36.228314    2578 scope.go:117] "RemoveContainer" containerID="c6153e9542792ed0f7c16bcbb7c9aac058237c1a70b3ed0ee9f95d74e5180900"
Jul 08 17:03:36.228784 ip-10-0-142-94 kubenswrapper[2578]: E0708 17:03:36.228607    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6153e9542792ed0f7c16bcbb7c9aac058237c1a70b3ed0ee9f95d74e5180900\": container with ID starting with c6153e9542792ed0f7c16bcbb7c9aac058237c1a70b3ed0ee9f95d74e5180900 not found: ID does not exist" containerID="c6153e9542792ed0f7c16bcbb7c9aac058237c1a70b3ed0ee9f95d74e5180900"
Jul 08 17:03:36.228784 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:36.228674    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6153e9542792ed0f7c16bcbb7c9aac058237c1a70b3ed0ee9f95d74e5180900"} err="failed to get container status \"c6153e9542792ed0f7c16bcbb7c9aac058237c1a70b3ed0ee9f95d74e5180900\": rpc error: code = NotFound desc = could not find container \"c6153e9542792ed0f7c16bcbb7c9aac058237c1a70b3ed0ee9f95d74e5180900\": container with ID starting with c6153e9542792ed0f7c16bcbb7c9aac058237c1a70b3ed0ee9f95d74e5180900 not found: ID does not exist"
Jul 08 17:03:36.228784 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:36.228699    2578 scope.go:117] "RemoveContainer" containerID="4b11de1cda1edd93d827e1bd35cd7499901f9e181c12b9bbed5d8a3e7ef835b0"
Jul 08 17:03:36.228996 ip-10-0-142-94 kubenswrapper[2578]: E0708 17:03:36.228973    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b11de1cda1edd93d827e1bd35cd7499901f9e181c12b9bbed5d8a3e7ef835b0\": container with ID starting with 4b11de1cda1edd93d827e1bd35cd7499901f9e181c12b9bbed5d8a3e7ef835b0 not found: ID does not exist" containerID="4b11de1cda1edd93d827e1bd35cd7499901f9e181c12b9bbed5d8a3e7ef835b0"
Jul 08 17:03:36.229046 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:36.229004    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b11de1cda1edd93d827e1bd35cd7499901f9e181c12b9bbed5d8a3e7ef835b0"} err="failed to get container status \"4b11de1cda1edd93d827e1bd35cd7499901f9e181c12b9bbed5d8a3e7ef835b0\": rpc error: code = NotFound desc = could not find container \"4b11de1cda1edd93d827e1bd35cd7499901f9e181c12b9bbed5d8a3e7ef835b0\": container with ID starting with 4b11de1cda1edd93d827e1bd35cd7499901f9e181c12b9bbed5d8a3e7ef835b0 not found: ID does not exist"
Jul 08 17:03:36.229509 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:36.229492    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-f05f8-predictor-54888985c8-gwssx"]
Jul 08 17:03:36.286818 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:36.286790    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5afccf1-df32-44fb-9ce5-e0487e67eaef" path="/var/lib/kubelet/pods/e5afccf1-df32-44fb-9ce5-e0487e67eaef/volumes"
Jul 08 17:03:36.372921 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:36.372880    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9"]
Jul 08 17:03:36.373248 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:36.373229    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5afccf1-df32-44fb-9ce5-e0487e67eaef" containerName="kube-rbac-proxy"
Jul 08 17:03:36.373334 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:36.373251    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5afccf1-df32-44fb-9ce5-e0487e67eaef" containerName="kube-rbac-proxy"
Jul 08 17:03:36.373334 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:36.373278    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5afccf1-df32-44fb-9ce5-e0487e67eaef" containerName="storage-initializer"
Jul 08 17:03:36.373334 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:36.373287    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5afccf1-df32-44fb-9ce5-e0487e67eaef" containerName="storage-initializer"
Jul 08 17:03:36.373334 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:36.373301    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5afccf1-df32-44fb-9ce5-e0487e67eaef" containerName="kserve-container"
Jul 08 17:03:36.373334 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:36.373312    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5afccf1-df32-44fb-9ce5-e0487e67eaef" containerName="kserve-container"
Jul 08 17:03:36.373600 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:36.373413    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5afccf1-df32-44fb-9ce5-e0487e67eaef" containerName="kube-rbac-proxy"
Jul 08 17:03:36.373600 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:36.373426    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5afccf1-df32-44fb-9ce5-e0487e67eaef" containerName="kserve-container"
Jul 08 17:03:36.378040 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:36.378019    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9"
Jul 08 17:03:36.380596 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:36.380485    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 08 17:03:36.380596 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:36.380524    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 08 17:03:36.380596 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:36.380552    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-vkqpp\""
Jul 08 17:03:36.380596 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:36.380524    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 08 17:03:36.380878 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:36.380780    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-cfbd9-kube-rbac-proxy-sar-config\""
Jul 08 17:03:36.380878 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:36.380786    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-cfbd9-predictor-serving-cert\""
Jul 08 17:03:36.380878 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:36.380854    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 08 17:03:36.387624 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:36.387603    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9"]
Jul 08 17:03:36.394570 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:36.394546    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b906914b-10ad-4131-bd22-ef6bf3e03d68-cabundle-cert\") pod \"raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9\" (UID: \"b906914b-10ad-4131-bd22-ef6bf3e03d68\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9"
Jul 08 17:03:36.394711 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:36.394589    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b906914b-10ad-4131-bd22-ef6bf3e03d68-proxy-tls\") pod \"raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9\" (UID: \"b906914b-10ad-4131-bd22-ef6bf3e03d68\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9"
Jul 08 17:03:36.394711 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:36.394694    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b906914b-10ad-4131-bd22-ef6bf3e03d68-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9\" (UID: \"b906914b-10ad-4131-bd22-ef6bf3e03d68\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9"
Jul 08 17:03:36.394850 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:36.394757    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw47v\" (UniqueName: \"kubernetes.io/projected/b906914b-10ad-4131-bd22-ef6bf3e03d68-kube-api-access-tw47v\") pod \"raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9\" (UID: \"b906914b-10ad-4131-bd22-ef6bf3e03d68\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9"
Jul 08 17:03:36.394850 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:36.394784    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b906914b-10ad-4131-bd22-ef6bf3e03d68-kserve-provision-location\") pod \"raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9\" (UID: \"b906914b-10ad-4131-bd22-ef6bf3e03d68\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9"
Jul 08 17:03:36.495786 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:36.495695    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b906914b-10ad-4131-bd22-ef6bf3e03d68-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9\" (UID: \"b906914b-10ad-4131-bd22-ef6bf3e03d68\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9"
Jul 08 17:03:36.495786 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:36.495749    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw47v\" (UniqueName: \"kubernetes.io/projected/b906914b-10ad-4131-bd22-ef6bf3e03d68-kube-api-access-tw47v\") pod \"raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9\" (UID: \"b906914b-10ad-4131-bd22-ef6bf3e03d68\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9"
Jul 08 17:03:36.495786 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:36.495783    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b906914b-10ad-4131-bd22-ef6bf3e03d68-kserve-provision-location\") pod \"raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9\" (UID: \"b906914b-10ad-4131-bd22-ef6bf3e03d68\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9"
Jul 08 17:03:36.496064 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:36.495817    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b906914b-10ad-4131-bd22-ef6bf3e03d68-cabundle-cert\") pod \"raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9\" (UID: \"b906914b-10ad-4131-bd22-ef6bf3e03d68\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9"
Jul 08 17:03:36.496064 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:36.495868    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b906914b-10ad-4131-bd22-ef6bf3e03d68-proxy-tls\") pod \"raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9\" (UID: \"b906914b-10ad-4131-bd22-ef6bf3e03d68\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9"
Jul 08 17:03:36.496259 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:36.496234    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b906914b-10ad-4131-bd22-ef6bf3e03d68-kserve-provision-location\") pod \"raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9\" (UID: \"b906914b-10ad-4131-bd22-ef6bf3e03d68\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9"
Jul 08 17:03:36.496405 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:36.496387    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b906914b-10ad-4131-bd22-ef6bf3e03d68-cabundle-cert\") pod \"raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9\" (UID: \"b906914b-10ad-4131-bd22-ef6bf3e03d68\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9"
Jul 08 17:03:36.496500 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:36.496480    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b906914b-10ad-4131-bd22-ef6bf3e03d68-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9\" (UID: \"b906914b-10ad-4131-bd22-ef6bf3e03d68\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9"
Jul 08 17:03:36.498542 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:36.498516    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b906914b-10ad-4131-bd22-ef6bf3e03d68-proxy-tls\") pod \"raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9\" (UID: \"b906914b-10ad-4131-bd22-ef6bf3e03d68\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9"
Jul 08 17:03:36.503564 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:36.503541    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw47v\" (UniqueName: \"kubernetes.io/projected/b906914b-10ad-4131-bd22-ef6bf3e03d68-kube-api-access-tw47v\") pod \"raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9\" (UID: \"b906914b-10ad-4131-bd22-ef6bf3e03d68\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9"
Jul 08 17:03:36.688753 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:36.688705    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9"
Jul 08 17:03:36.823694 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:36.823669    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9"]
Jul 08 17:03:36.826560 ip-10-0-142-94 kubenswrapper[2578]: W0708 17:03:36.826533    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb906914b_10ad_4131_bd22_ef6bf3e03d68.slice/crio-e78ddd51cc25e7d692a9d06f11496ec51cceb63f07ad087507acf2856c4585bb WatchSource:0}: Error finding container e78ddd51cc25e7d692a9d06f11496ec51cceb63f07ad087507acf2856c4585bb: Status 404 returned error can't find the container with id e78ddd51cc25e7d692a9d06f11496ec51cceb63f07ad087507acf2856c4585bb
Jul 08 17:03:37.210000 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:37.209964    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9" event={"ID":"b906914b-10ad-4131-bd22-ef6bf3e03d68","Type":"ContainerStarted","Data":"09bb0b035c477ce4162f361e27c10cd51f7277b0b9c338daf54405d4e960f613"}
Jul 08 17:03:37.210332 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:37.210030    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9" event={"ID":"b906914b-10ad-4131-bd22-ef6bf3e03d68","Type":"ContainerStarted","Data":"e78ddd51cc25e7d692a9d06f11496ec51cceb63f07ad087507acf2856c4585bb"}
Jul 08 17:03:41.226055 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:41.226023    2578 generic.go:356] "Generic (PLEG): container finished" podID="b906914b-10ad-4131-bd22-ef6bf3e03d68" containerID="09bb0b035c477ce4162f361e27c10cd51f7277b0b9c338daf54405d4e960f613" exitCode=0
Jul 08 17:03:41.226441 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:41.226064    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9" event={"ID":"b906914b-10ad-4131-bd22-ef6bf3e03d68","Type":"ContainerDied","Data":"09bb0b035c477ce4162f361e27c10cd51f7277b0b9c338daf54405d4e960f613"}
Jul 08 17:03:42.232244 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:42.232211    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9" event={"ID":"b906914b-10ad-4131-bd22-ef6bf3e03d68","Type":"ContainerStarted","Data":"9ae76c886d7ae8c53f555fff1f7da3bc0ce94cb1dfb112076fab4d8715899bb0"}
Jul 08 17:03:42.232657 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:42.232271    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9" event={"ID":"b906914b-10ad-4131-bd22-ef6bf3e03d68","Type":"ContainerStarted","Data":"719ad30012032e41027d75f09f50955d06ff993ca4c457b484187faba9c4a02c"}
Jul 08 17:03:42.232657 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:42.232535    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9"
Jul 08 17:03:42.232785 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:42.232691    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9"
Jul 08 17:03:42.234079 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:42.234051    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9" podUID="b906914b-10ad-4131-bd22-ef6bf3e03d68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 08 17:03:42.252620 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:42.252572    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9" podStartSLOduration=2.630076394 podStartE2EDuration="6.252557344s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.62248095s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:03:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:03:42.251280831 +0000 UTC m=+1734.522730374" watchObservedRunningTime="2026-07-08 17:03:42.252557344 +0000 UTC m=+1734.524006885"
Jul 08 17:03:43.236318 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:43.236276    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9" podUID="b906914b-10ad-4131-bd22-ef6bf3e03d68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 08 17:03:48.242308 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:48.242277    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9"
Jul 08 17:03:48.242882 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:48.242854    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9" podUID="b906914b-10ad-4131-bd22-ef6bf3e03d68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 08 17:03:58.243787 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:03:58.243746    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9" podUID="b906914b-10ad-4131-bd22-ef6bf3e03d68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 08 17:04:08.243777 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:04:08.243731    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9" podUID="b906914b-10ad-4131-bd22-ef6bf3e03d68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 08 17:04:18.243489 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:04:18.243442    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9" podUID="b906914b-10ad-4131-bd22-ef6bf3e03d68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 08 17:04:28.243688 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:04:28.243622    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9" podUID="b906914b-10ad-4131-bd22-ef6bf3e03d68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 08 17:04:38.243649 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:04:38.243599    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9" podUID="b906914b-10ad-4131-bd22-ef6bf3e03d68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 08 17:04:48.242901 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:04:48.242858    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9" podUID="b906914b-10ad-4131-bd22-ef6bf3e03d68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 08 17:04:48.269770 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:04:48.269732    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-qn2bx_4717bdee-2017-4551-b0ec-683ab07f7660/console-operator/1.log"
Jul 08 17:04:48.272124 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:04:48.272096    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-qn2bx_4717bdee-2017-4551-b0ec-683ab07f7660/console-operator/1.log"
Jul 08 17:04:48.274706 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:04:48.274687    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2shvk_f08a1b00-78fd-43ef-b2aa-2712d09fb1b3/ovn-acl-logging/0.log"
Jul 08 17:04:48.276854 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:04:48.276835    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2shvk_f08a1b00-78fd-43ef-b2aa-2712d09fb1b3/ovn-acl-logging/0.log"
Jul 08 17:04:58.243452 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:04:58.243418    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9"
Jul 08 17:05:06.478180 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:06.478127    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9"]
Jul 08 17:05:06.478604 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:06.478484    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9" podUID="b906914b-10ad-4131-bd22-ef6bf3e03d68" containerName="kserve-container" containerID="cri-o://719ad30012032e41027d75f09f50955d06ff993ca4c457b484187faba9c4a02c" gracePeriod=30
Jul 08 17:05:06.478604 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:06.478576    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9" podUID="b906914b-10ad-4131-bd22-ef6bf3e03d68" containerName="kube-rbac-proxy" containerID="cri-o://9ae76c886d7ae8c53f555fff1f7da3bc0ce94cb1dfb112076fab4d8715899bb0" gracePeriod=30
Jul 08 17:05:07.567754 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:07.567719    2578 generic.go:356] "Generic (PLEG): container finished" podID="b906914b-10ad-4131-bd22-ef6bf3e03d68" containerID="9ae76c886d7ae8c53f555fff1f7da3bc0ce94cb1dfb112076fab4d8715899bb0" exitCode=2
Jul 08 17:05:07.568168 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:07.567793    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9" event={"ID":"b906914b-10ad-4131-bd22-ef6bf3e03d68","Type":"ContainerDied","Data":"9ae76c886d7ae8c53f555fff1f7da3bc0ce94cb1dfb112076fab4d8715899bb0"}
Jul 08 17:05:08.237080 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:08.237032    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9" podUID="b906914b-10ad-4131-bd22-ef6bf3e03d68" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Jul 08 17:05:08.243788 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:08.243751    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9" podUID="b906914b-10ad-4131-bd22-ef6bf3e03d68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 08 17:05:11.587120 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:11.587088    2578 generic.go:356] "Generic (PLEG): container finished" podID="b906914b-10ad-4131-bd22-ef6bf3e03d68" containerID="719ad30012032e41027d75f09f50955d06ff993ca4c457b484187faba9c4a02c" exitCode=0
Jul 08 17:05:11.587608 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:11.587157    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9" event={"ID":"b906914b-10ad-4131-bd22-ef6bf3e03d68","Type":"ContainerDied","Data":"719ad30012032e41027d75f09f50955d06ff993ca4c457b484187faba9c4a02c"}
Jul 08 17:05:11.640370 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:11.640344    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9"
Jul 08 17:05:11.788953 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:11.788844    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tw47v\" (UniqueName: \"kubernetes.io/projected/b906914b-10ad-4131-bd22-ef6bf3e03d68-kube-api-access-tw47v\") pod \"b906914b-10ad-4131-bd22-ef6bf3e03d68\" (UID: \"b906914b-10ad-4131-bd22-ef6bf3e03d68\") "
Jul 08 17:05:11.788953 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:11.788914    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b906914b-10ad-4131-bd22-ef6bf3e03d68-kserve-provision-location\") pod \"b906914b-10ad-4131-bd22-ef6bf3e03d68\" (UID: \"b906914b-10ad-4131-bd22-ef6bf3e03d68\") "
Jul 08 17:05:11.789197 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:11.788969    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b906914b-10ad-4131-bd22-ef6bf3e03d68-kube-rbac-proxy-sar-config\") pod \"b906914b-10ad-4131-bd22-ef6bf3e03d68\" (UID: \"b906914b-10ad-4131-bd22-ef6bf3e03d68\") "
Jul 08 17:05:11.789197 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:11.789002    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b906914b-10ad-4131-bd22-ef6bf3e03d68-proxy-tls\") pod \"b906914b-10ad-4131-bd22-ef6bf3e03d68\" (UID: \"b906914b-10ad-4131-bd22-ef6bf3e03d68\") "
Jul 08 17:05:11.789197 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:11.789062    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b906914b-10ad-4131-bd22-ef6bf3e03d68-cabundle-cert\") pod \"b906914b-10ad-4131-bd22-ef6bf3e03d68\" (UID: \"b906914b-10ad-4131-bd22-ef6bf3e03d68\") "
Jul 08 17:05:11.789362 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:11.789318    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b906914b-10ad-4131-bd22-ef6bf3e03d68-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b906914b-10ad-4131-bd22-ef6bf3e03d68" (UID: "b906914b-10ad-4131-bd22-ef6bf3e03d68"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:05:11.789362 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:11.789322    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b906914b-10ad-4131-bd22-ef6bf3e03d68-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b906914b-10ad-4131-bd22-ef6bf3e03d68" (UID: "b906914b-10ad-4131-bd22-ef6bf3e03d68"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 17:05:11.789500 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:11.789478    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b906914b-10ad-4131-bd22-ef6bf3e03d68-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b906914b-10ad-4131-bd22-ef6bf3e03d68" (UID: "b906914b-10ad-4131-bd22-ef6bf3e03d68"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 17:05:11.791287 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:11.791265    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b906914b-10ad-4131-bd22-ef6bf3e03d68-kube-api-access-tw47v" (OuterVolumeSpecName: "kube-api-access-tw47v") pod "b906914b-10ad-4131-bd22-ef6bf3e03d68" (UID: "b906914b-10ad-4131-bd22-ef6bf3e03d68"). InnerVolumeSpecName "kube-api-access-tw47v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:05:11.791287 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:11.791269    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b906914b-10ad-4131-bd22-ef6bf3e03d68-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b906914b-10ad-4131-bd22-ef6bf3e03d68" (UID: "b906914b-10ad-4131-bd22-ef6bf3e03d68"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:05:11.890018 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:11.889964    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b906914b-10ad-4131-bd22-ef6bf3e03d68-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 17:05:11.890018 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:11.890012    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b906914b-10ad-4131-bd22-ef6bf3e03d68-proxy-tls\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 17:05:11.890018 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:11.890025    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b906914b-10ad-4131-bd22-ef6bf3e03d68-cabundle-cert\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 17:05:11.890018 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:11.890035    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw47v\" (UniqueName: \"kubernetes.io/projected/b906914b-10ad-4131-bd22-ef6bf3e03d68-kube-api-access-tw47v\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 17:05:11.890313 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:11.890043    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b906914b-10ad-4131-bd22-ef6bf3e03d68-kserve-provision-location\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 17:05:12.279804 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:12.278886    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xfjmg/must-gather-wvr2p"]
Jul 08 17:05:12.279804 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:12.279549    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b906914b-10ad-4131-bd22-ef6bf3e03d68" containerName="kserve-container"
Jul 08 17:05:12.279804 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:12.279574    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b906914b-10ad-4131-bd22-ef6bf3e03d68" containerName="kserve-container"
Jul 08 17:05:12.279804 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:12.279617    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b906914b-10ad-4131-bd22-ef6bf3e03d68" containerName="kube-rbac-proxy"
Jul 08 17:05:12.279804 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:12.279657    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b906914b-10ad-4131-bd22-ef6bf3e03d68" containerName="kube-rbac-proxy"
Jul 08 17:05:12.279804 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:12.279682    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b906914b-10ad-4131-bd22-ef6bf3e03d68" containerName="storage-initializer"
Jul 08 17:05:12.279804 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:12.279693    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b906914b-10ad-4131-bd22-ef6bf3e03d68" containerName="storage-initializer"
Jul 08 17:05:12.280320 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:12.279854    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b906914b-10ad-4131-bd22-ef6bf3e03d68" containerName="kserve-container"
Jul 08 17:05:12.280320 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:12.279871    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b906914b-10ad-4131-bd22-ef6bf3e03d68" containerName="kube-rbac-proxy"
Jul 08 17:05:12.284016 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:12.283980    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xfjmg/must-gather-wvr2p"
Jul 08 17:05:12.286976 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:12.286897    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xfjmg\"/\"openshift-service-ca.crt\""
Jul 08 17:05:12.287401 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:12.287382    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xfjmg\"/\"kube-root-ca.crt\""
Jul 08 17:05:12.287722 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:12.287703    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-xfjmg\"/\"default-dockercfg-mpz2q\""
Jul 08 17:05:12.290329 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:12.290305    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xfjmg/must-gather-wvr2p"]
Jul 08 17:05:12.395445 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:12.395407    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/03b0e2eb-d41f-4f98-81bb-f0e2435b70a6-must-gather-output\") pod \"must-gather-wvr2p\" (UID: \"03b0e2eb-d41f-4f98-81bb-f0e2435b70a6\") " pod="openshift-must-gather-xfjmg/must-gather-wvr2p"
Jul 08 17:05:12.395659 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:12.395477    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjkt7\" (UniqueName: \"kubernetes.io/projected/03b0e2eb-d41f-4f98-81bb-f0e2435b70a6-kube-api-access-qjkt7\") pod \"must-gather-wvr2p\" (UID: \"03b0e2eb-d41f-4f98-81bb-f0e2435b70a6\") " pod="openshift-must-gather-xfjmg/must-gather-wvr2p"
Jul 08 17:05:12.496088 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:12.496047    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/03b0e2eb-d41f-4f98-81bb-f0e2435b70a6-must-gather-output\") pod \"must-gather-wvr2p\" (UID: \"03b0e2eb-d41f-4f98-81bb-f0e2435b70a6\") " pod="openshift-must-gather-xfjmg/must-gather-wvr2p"
Jul 08 17:05:12.496299 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:12.496157    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjkt7\" (UniqueName: \"kubernetes.io/projected/03b0e2eb-d41f-4f98-81bb-f0e2435b70a6-kube-api-access-qjkt7\") pod \"must-gather-wvr2p\" (UID: \"03b0e2eb-d41f-4f98-81bb-f0e2435b70a6\") " pod="openshift-must-gather-xfjmg/must-gather-wvr2p"
Jul 08 17:05:12.496422 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:12.496396    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/03b0e2eb-d41f-4f98-81bb-f0e2435b70a6-must-gather-output\") pod \"must-gather-wvr2p\" (UID: \"03b0e2eb-d41f-4f98-81bb-f0e2435b70a6\") " pod="openshift-must-gather-xfjmg/must-gather-wvr2p"
Jul 08 17:05:12.505254 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:12.505218    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjkt7\" (UniqueName: \"kubernetes.io/projected/03b0e2eb-d41f-4f98-81bb-f0e2435b70a6-kube-api-access-qjkt7\") pod \"must-gather-wvr2p\" (UID: \"03b0e2eb-d41f-4f98-81bb-f0e2435b70a6\") " pod="openshift-must-gather-xfjmg/must-gather-wvr2p"
Jul 08 17:05:12.593353 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:12.593321    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9" event={"ID":"b906914b-10ad-4131-bd22-ef6bf3e03d68","Type":"ContainerDied","Data":"e78ddd51cc25e7d692a9d06f11496ec51cceb63f07ad087507acf2856c4585bb"}
Jul 08 17:05:12.593353 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:12.593351    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9"
Jul 08 17:05:12.593959 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:12.593390    2578 scope.go:117] "RemoveContainer" containerID="9ae76c886d7ae8c53f555fff1f7da3bc0ce94cb1dfb112076fab4d8715899bb0"
Jul 08 17:05:12.602227 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:12.602203    2578 scope.go:117] "RemoveContainer" containerID="719ad30012032e41027d75f09f50955d06ff993ca4c457b484187faba9c4a02c"
Jul 08 17:05:12.609605 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:12.609574    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9"]
Jul 08 17:05:12.609968 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:12.609953    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xfjmg/must-gather-wvr2p"
Jul 08 17:05:12.610842 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:12.610819    2578 scope.go:117] "RemoveContainer" containerID="09bb0b035c477ce4162f361e27c10cd51f7277b0b9c338daf54405d4e960f613"
Jul 08 17:05:12.615244 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:12.615218    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-cfbd9-predictor-7ccbb7c7f8-kjtk9"]
Jul 08 17:05:12.752034 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:12.751998    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xfjmg/must-gather-wvr2p"]
Jul 08 17:05:12.756076 ip-10-0-142-94 kubenswrapper[2578]: W0708 17:05:12.756036    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod03b0e2eb_d41f_4f98_81bb_f0e2435b70a6.slice/crio-e81cb9ae5ce2ece5f12d0f2ea8f677e8609598644a7cb2e23def12847ff15528 WatchSource:0}: Error finding container e81cb9ae5ce2ece5f12d0f2ea8f677e8609598644a7cb2e23def12847ff15528: Status 404 returned error can't find the container with id e81cb9ae5ce2ece5f12d0f2ea8f677e8609598644a7cb2e23def12847ff15528
Jul 08 17:05:12.758195 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:12.758170    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 08 17:05:13.601004 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:13.600959    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xfjmg/must-gather-wvr2p" event={"ID":"03b0e2eb-d41f-4f98-81bb-f0e2435b70a6","Type":"ContainerStarted","Data":"e81cb9ae5ce2ece5f12d0f2ea8f677e8609598644a7cb2e23def12847ff15528"}
Jul 08 17:05:14.288526 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:14.288492    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b906914b-10ad-4131-bd22-ef6bf3e03d68" path="/var/lib/kubelet/pods/b906914b-10ad-4131-bd22-ef6bf3e03d68/volumes"
Jul 08 17:05:19.635526 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:19.635487    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xfjmg/must-gather-wvr2p" event={"ID":"03b0e2eb-d41f-4f98-81bb-f0e2435b70a6","Type":"ContainerStarted","Data":"2f5bdd176927b04f078580614edef999ccf5701aee5d4f7c552b9e6676fcb67f"}
Jul 08 17:05:19.636011 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:19.635536    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xfjmg/must-gather-wvr2p" event={"ID":"03b0e2eb-d41f-4f98-81bb-f0e2435b70a6","Type":"ContainerStarted","Data":"ce645a3b239c174ea555826a2ac59bc471a1e41a7b5b2c08231c03b16fb58d92"}
Jul 08 17:05:19.651910 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:19.651853    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-xfjmg/must-gather-wvr2p" podStartSLOduration=1.238888417 podStartE2EDuration="7.651838165s" totalImagesPullingTime="6.412949748s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:05:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:05:19.650154011 +0000 UTC m=+1831.921603553" watchObservedRunningTime="2026-07-08 17:05:19.651838165 +0000 UTC m=+1831.923287705"
Jul 08 17:05:41.739081 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:41.738988    2578 generic.go:356] "Generic (PLEG): container finished" podID="03b0e2eb-d41f-4f98-81bb-f0e2435b70a6" containerID="ce645a3b239c174ea555826a2ac59bc471a1e41a7b5b2c08231c03b16fb58d92" exitCode=0
Jul 08 17:05:41.739081 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:41.739066    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xfjmg/must-gather-wvr2p" event={"ID":"03b0e2eb-d41f-4f98-81bb-f0e2435b70a6","Type":"ContainerDied","Data":"ce645a3b239c174ea555826a2ac59bc471a1e41a7b5b2c08231c03b16fb58d92"}
Jul 08 17:05:41.739509 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:41.739404    2578 scope.go:117] "RemoveContainer" containerID="ce645a3b239c174ea555826a2ac59bc471a1e41a7b5b2c08231c03b16fb58d92"
Jul 08 17:05:42.359747 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:42.359713    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-xfjmg_must-gather-wvr2p_03b0e2eb-d41f-4f98-81bb-f0e2435b70a6/gather/0.log"
Jul 08 17:05:45.673121 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:45.673091    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-d9p5w_e6f14c39-461f-44d2-811a-87cfb899d92c/global-pull-secret-syncer/0.log"
Jul 08 17:05:45.864990 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:45.864960    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-ht2nh_3b861d99-7199-4174-a922-d0aae9cd010a/konnectivity-agent/0.log"
Jul 08 17:05:45.979901 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:45.979793    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-94.ec2.internal_70024d4eaabe656e42acf5ac910ecc55/haproxy/0.log"
Jul 08 17:05:47.777378 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:47.777338    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-xfjmg/must-gather-wvr2p"]
Jul 08 17:05:47.777855 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:47.777563    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-must-gather-xfjmg/must-gather-wvr2p" podUID="03b0e2eb-d41f-4f98-81bb-f0e2435b70a6" containerName="copy" containerID="cri-o://2f5bdd176927b04f078580614edef999ccf5701aee5d4f7c552b9e6676fcb67f" gracePeriod=2
Jul 08 17:05:47.780142 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:47.780118    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-xfjmg/must-gather-wvr2p"]
Jul 08 17:05:47.780576 ip-10-0-142-94 kubenswrapper[2578]: E0708 17:05:47.780551    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-wvr2p\" is forbidden: User \"system:node:ip-10-0-142-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-xfjmg\": no relationship found between node 'ip-10-0-142-94.ec2.internal' and this object" podUID="03b0e2eb-d41f-4f98-81bb-f0e2435b70a6" pod="openshift-must-gather-xfjmg/must-gather-wvr2p"
Jul 08 17:05:48.008166 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:48.008143    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-xfjmg_must-gather-wvr2p_03b0e2eb-d41f-4f98-81bb-f0e2435b70a6/copy/0.log"
Jul 08 17:05:48.009930 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:48.009908    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xfjmg/must-gather-wvr2p"
Jul 08 17:05:48.011869 ip-10-0-142-94 kubenswrapper[2578]: E0708 17:05:48.011846    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-wvr2p\" is forbidden: User \"system:node:ip-10-0-142-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-xfjmg\": no relationship found between node 'ip-10-0-142-94.ec2.internal' and this object" podUID="03b0e2eb-d41f-4f98-81bb-f0e2435b70a6" pod="openshift-must-gather-xfjmg/must-gather-wvr2p"
Jul 08 17:05:48.116556 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:48.116526    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/03b0e2eb-d41f-4f98-81bb-f0e2435b70a6-must-gather-output\") pod \"03b0e2eb-d41f-4f98-81bb-f0e2435b70a6\" (UID: \"03b0e2eb-d41f-4f98-81bb-f0e2435b70a6\") "
Jul 08 17:05:48.116936 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:48.116583    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qjkt7\" (UniqueName: \"kubernetes.io/projected/03b0e2eb-d41f-4f98-81bb-f0e2435b70a6-kube-api-access-qjkt7\") pod \"03b0e2eb-d41f-4f98-81bb-f0e2435b70a6\" (UID: \"03b0e2eb-d41f-4f98-81bb-f0e2435b70a6\") "
Jul 08 17:05:48.118487 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:48.118460    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03b0e2eb-d41f-4f98-81bb-f0e2435b70a6-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "03b0e2eb-d41f-4f98-81bb-f0e2435b70a6" (UID: "03b0e2eb-d41f-4f98-81bb-f0e2435b70a6"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:05:48.118941 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:48.118922    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03b0e2eb-d41f-4f98-81bb-f0e2435b70a6-kube-api-access-qjkt7" (OuterVolumeSpecName: "kube-api-access-qjkt7") pod "03b0e2eb-d41f-4f98-81bb-f0e2435b70a6" (UID: "03b0e2eb-d41f-4f98-81bb-f0e2435b70a6"). InnerVolumeSpecName "kube-api-access-qjkt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:05:48.217789 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:48.217755    2578 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/03b0e2eb-d41f-4f98-81bb-f0e2435b70a6-must-gather-output\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 17:05:48.217789 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:48.217783    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjkt7\" (UniqueName: \"kubernetes.io/projected/03b0e2eb-d41f-4f98-81bb-f0e2435b70a6-kube-api-access-qjkt7\") on node \"ip-10-0-142-94.ec2.internal\" DevicePath \"\""
Jul 08 17:05:48.284018 ip-10-0-142-94 kubenswrapper[2578]: E0708 17:05:48.283984    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-wvr2p\" is forbidden: User \"system:node:ip-10-0-142-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-xfjmg\": no relationship found between node 'ip-10-0-142-94.ec2.internal' and this object" podUID="03b0e2eb-d41f-4f98-81bb-f0e2435b70a6" pod="openshift-must-gather-xfjmg/must-gather-wvr2p"
Jul 08 17:05:48.290674 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:48.290617    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03b0e2eb-d41f-4f98-81bb-f0e2435b70a6" path="/var/lib/kubelet/pods/03b0e2eb-d41f-4f98-81bb-f0e2435b70a6/volumes"
Jul 08 17:05:48.768928 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:48.768893    2578 generic.go:356] "Generic (PLEG): container finished" podID="03b0e2eb-d41f-4f98-81bb-f0e2435b70a6" containerID="2f5bdd176927b04f078580614edef999ccf5701aee5d4f7c552b9e6676fcb67f" exitCode=143
Jul 08 17:05:48.769133 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:48.768960    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xfjmg/must-gather-wvr2p"
Jul 08 17:05:48.769133 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:48.768993    2578 scope.go:117] "RemoveContainer" containerID="2f5bdd176927b04f078580614edef999ccf5701aee5d4f7c552b9e6676fcb67f"
Jul 08 17:05:48.776585 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:48.776564    2578 scope.go:117] "RemoveContainer" containerID="ce645a3b239c174ea555826a2ac59bc471a1e41a7b5b2c08231c03b16fb58d92"
Jul 08 17:05:48.789099 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:48.788940    2578 scope.go:117] "RemoveContainer" containerID="2f5bdd176927b04f078580614edef999ccf5701aee5d4f7c552b9e6676fcb67f"
Jul 08 17:05:48.789325 ip-10-0-142-94 kubenswrapper[2578]: E0708 17:05:48.789185    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f5bdd176927b04f078580614edef999ccf5701aee5d4f7c552b9e6676fcb67f\": container with ID starting with 2f5bdd176927b04f078580614edef999ccf5701aee5d4f7c552b9e6676fcb67f not found: ID does not exist" containerID="2f5bdd176927b04f078580614edef999ccf5701aee5d4f7c552b9e6676fcb67f"
Jul 08 17:05:48.789325 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:48.789212    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f5bdd176927b04f078580614edef999ccf5701aee5d4f7c552b9e6676fcb67f"} err="failed to get container status \"2f5bdd176927b04f078580614edef999ccf5701aee5d4f7c552b9e6676fcb67f\": rpc error: code = NotFound desc = could not find container \"2f5bdd176927b04f078580614edef999ccf5701aee5d4f7c552b9e6676fcb67f\": container with ID starting with 2f5bdd176927b04f078580614edef999ccf5701aee5d4f7c552b9e6676fcb67f not found: ID does not exist"
Jul 08 17:05:48.789325 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:48.789230    2578 scope.go:117] "RemoveContainer" containerID="ce645a3b239c174ea555826a2ac59bc471a1e41a7b5b2c08231c03b16fb58d92"
Jul 08 17:05:48.789487 ip-10-0-142-94 kubenswrapper[2578]: E0708 17:05:48.789468    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce645a3b239c174ea555826a2ac59bc471a1e41a7b5b2c08231c03b16fb58d92\": container with ID starting with ce645a3b239c174ea555826a2ac59bc471a1e41a7b5b2c08231c03b16fb58d92 not found: ID does not exist" containerID="ce645a3b239c174ea555826a2ac59bc471a1e41a7b5b2c08231c03b16fb58d92"
Jul 08 17:05:48.789523 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:48.789495    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce645a3b239c174ea555826a2ac59bc471a1e41a7b5b2c08231c03b16fb58d92"} err="failed to get container status \"ce645a3b239c174ea555826a2ac59bc471a1e41a7b5b2c08231c03b16fb58d92\": rpc error: code = NotFound desc = could not find container \"ce645a3b239c174ea555826a2ac59bc471a1e41a7b5b2c08231c03b16fb58d92\": container with ID starting with ce645a3b239c174ea555826a2ac59bc471a1e41a7b5b2c08231c03b16fb58d92 not found: ID does not exist"
Jul 08 17:05:50.030257 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:50.030229    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5f5d97fccf-hvpl2_393b64f8-62fc-489e-9717-f63b23485906/metrics-server/0.log"
Jul 08 17:05:50.063651 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:50.063600    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-k98gm_df1c923d-81c9-4e33-939b-050ede7d6675/monitoring-plugin/0.log"
Jul 08 17:05:50.191412 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:50.191328    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cl4ch_1988f4a9-d199-4f16-9a8f-b0eaab1b21f7/node-exporter/0.log"
Jul 08 17:05:50.212019 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:50.211985    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cl4ch_1988f4a9-d199-4f16-9a8f-b0eaab1b21f7/kube-rbac-proxy/0.log"
Jul 08 17:05:50.239817 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:50.239786    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cl4ch_1988f4a9-d199-4f16-9a8f-b0eaab1b21f7/init-textfile/0.log"
Jul 08 17:05:50.430008 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:50.429977    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c01e663f-14c6-424b-8b7e-c1809069d1a1/prometheus/0.log"
Jul 08 17:05:50.452144 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:50.452058    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c01e663f-14c6-424b-8b7e-c1809069d1a1/config-reloader/0.log"
Jul 08 17:05:50.473303 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:50.473266    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c01e663f-14c6-424b-8b7e-c1809069d1a1/thanos-sidecar/0.log"
Jul 08 17:05:50.492341 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:50.492314    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c01e663f-14c6-424b-8b7e-c1809069d1a1/kube-rbac-proxy-web/0.log"
Jul 08 17:05:50.512257 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:50.512229    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c01e663f-14c6-424b-8b7e-c1809069d1a1/kube-rbac-proxy/0.log"
Jul 08 17:05:50.531287 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:50.531262    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c01e663f-14c6-424b-8b7e-c1809069d1a1/kube-rbac-proxy-thanos/0.log"
Jul 08 17:05:50.550666 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:50.550619    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c01e663f-14c6-424b-8b7e-c1809069d1a1/init-config-reloader/0.log"
Jul 08 17:05:50.580828 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:50.580798    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-j6s2h_dc428bbb-0f04-4342-b29e-9f4dfe36fdd1/prometheus-operator/0.log"
Jul 08 17:05:50.603412 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:50.603383    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-j6s2h_dc428bbb-0f04-4342-b29e-9f4dfe36fdd1/kube-rbac-proxy/0.log"
Jul 08 17:05:52.817905 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:52.817872    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-qn2bx_4717bdee-2017-4551-b0ec-683ab07f7660/console-operator/1.log"
Jul 08 17:05:52.828495 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:52.828326    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-qn2bx_4717bdee-2017-4551-b0ec-683ab07f7660/console-operator/2.log"
Jul 08 17:05:53.823389 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:53.823356    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5d456fdbcd-mn8c2_c94e9045-53b8-455e-8c97-8a19845535a0/volume-data-source-validator/0.log"
Jul 08 17:05:54.078914 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:54.078835    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mcg99/perf-node-gather-daemonset-mkf7s"]
Jul 08 17:05:54.079188 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:54.079175    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03b0e2eb-d41f-4f98-81bb-f0e2435b70a6" containerName="gather"
Jul 08 17:05:54.079250 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:54.079190    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="03b0e2eb-d41f-4f98-81bb-f0e2435b70a6" containerName="gather"
Jul 08 17:05:54.079250 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:54.079201    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03b0e2eb-d41f-4f98-81bb-f0e2435b70a6" containerName="copy"
Jul 08 17:05:54.079250 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:54.079206    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="03b0e2eb-d41f-4f98-81bb-f0e2435b70a6" containerName="copy"
Jul 08 17:05:54.079250 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:54.079249    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="03b0e2eb-d41f-4f98-81bb-f0e2435b70a6" containerName="copy"
Jul 08 17:05:54.079397 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:54.079260    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="03b0e2eb-d41f-4f98-81bb-f0e2435b70a6" containerName="gather"
Jul 08 17:05:54.081283 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:54.081261    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mcg99/perf-node-gather-daemonset-mkf7s"
Jul 08 17:05:54.083562 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:54.083539    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mcg99\"/\"openshift-service-ca.crt\""
Jul 08 17:05:54.083696 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:54.083586    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mcg99\"/\"kube-root-ca.crt\""
Jul 08 17:05:54.083696 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:54.083663    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mcg99\"/\"default-dockercfg-z6gtz\""
Jul 08 17:05:54.090750 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:54.090728    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mcg99/perf-node-gather-daemonset-mkf7s"]
Jul 08 17:05:54.169314 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:54.169272    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9a53e192-cc4c-42fa-be92-cd26dd1e8ecb-lib-modules\") pod \"perf-node-gather-daemonset-mkf7s\" (UID: \"9a53e192-cc4c-42fa-be92-cd26dd1e8ecb\") " pod="openshift-must-gather-mcg99/perf-node-gather-daemonset-mkf7s"
Jul 08 17:05:54.169501 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:54.169339    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l29dq\" (UniqueName: \"kubernetes.io/projected/9a53e192-cc4c-42fa-be92-cd26dd1e8ecb-kube-api-access-l29dq\") pod \"perf-node-gather-daemonset-mkf7s\" (UID: \"9a53e192-cc4c-42fa-be92-cd26dd1e8ecb\") " pod="openshift-must-gather-mcg99/perf-node-gather-daemonset-mkf7s"
Jul 08 17:05:54.169501 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:54.169381    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a53e192-cc4c-42fa-be92-cd26dd1e8ecb-sys\") pod \"perf-node-gather-daemonset-mkf7s\" (UID: \"9a53e192-cc4c-42fa-be92-cd26dd1e8ecb\") " pod="openshift-must-gather-mcg99/perf-node-gather-daemonset-mkf7s"
Jul 08 17:05:54.169501 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:54.169407    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9a53e192-cc4c-42fa-be92-cd26dd1e8ecb-podres\") pod \"perf-node-gather-daemonset-mkf7s\" (UID: \"9a53e192-cc4c-42fa-be92-cd26dd1e8ecb\") " pod="openshift-must-gather-mcg99/perf-node-gather-daemonset-mkf7s"
Jul 08 17:05:54.169501 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:54.169466    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9a53e192-cc4c-42fa-be92-cd26dd1e8ecb-proc\") pod \"perf-node-gather-daemonset-mkf7s\" (UID: \"9a53e192-cc4c-42fa-be92-cd26dd1e8ecb\") " pod="openshift-must-gather-mcg99/perf-node-gather-daemonset-mkf7s"
Jul 08 17:05:54.270940 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:54.270892    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9a53e192-cc4c-42fa-be92-cd26dd1e8ecb-lib-modules\") pod \"perf-node-gather-daemonset-mkf7s\" (UID: \"9a53e192-cc4c-42fa-be92-cd26dd1e8ecb\") " pod="openshift-must-gather-mcg99/perf-node-gather-daemonset-mkf7s"
Jul 08 17:05:54.271139 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:54.270958    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l29dq\" (UniqueName: \"kubernetes.io/projected/9a53e192-cc4c-42fa-be92-cd26dd1e8ecb-kube-api-access-l29dq\") pod \"perf-node-gather-daemonset-mkf7s\" (UID: \"9a53e192-cc4c-42fa-be92-cd26dd1e8ecb\") " pod="openshift-must-gather-mcg99/perf-node-gather-daemonset-mkf7s"
Jul 08 17:05:54.271139 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:54.270999    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a53e192-cc4c-42fa-be92-cd26dd1e8ecb-sys\") pod \"perf-node-gather-daemonset-mkf7s\" (UID: \"9a53e192-cc4c-42fa-be92-cd26dd1e8ecb\") " pod="openshift-must-gather-mcg99/perf-node-gather-daemonset-mkf7s"
Jul 08 17:05:54.271139 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:54.271020    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9a53e192-cc4c-42fa-be92-cd26dd1e8ecb-podres\") pod \"perf-node-gather-daemonset-mkf7s\" (UID: \"9a53e192-cc4c-42fa-be92-cd26dd1e8ecb\") " pod="openshift-must-gather-mcg99/perf-node-gather-daemonset-mkf7s"
Jul 08 17:05:54.271139 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:54.271047    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9a53e192-cc4c-42fa-be92-cd26dd1e8ecb-proc\") pod \"perf-node-gather-daemonset-mkf7s\" (UID: \"9a53e192-cc4c-42fa-be92-cd26dd1e8ecb\") " pod="openshift-must-gather-mcg99/perf-node-gather-daemonset-mkf7s"
Jul 08 17:05:54.271139 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:54.271113    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9a53e192-cc4c-42fa-be92-cd26dd1e8ecb-lib-modules\") pod \"perf-node-gather-daemonset-mkf7s\" (UID: \"9a53e192-cc4c-42fa-be92-cd26dd1e8ecb\") " pod="openshift-must-gather-mcg99/perf-node-gather-daemonset-mkf7s"
Jul 08 17:05:54.271139 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:54.271129    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9a53e192-cc4c-42fa-be92-cd26dd1e8ecb-proc\") pod \"perf-node-gather-daemonset-mkf7s\" (UID: \"9a53e192-cc4c-42fa-be92-cd26dd1e8ecb\") " pod="openshift-must-gather-mcg99/perf-node-gather-daemonset-mkf7s"
Jul 08 17:05:54.271372 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:54.271132    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a53e192-cc4c-42fa-be92-cd26dd1e8ecb-sys\") pod \"perf-node-gather-daemonset-mkf7s\" (UID: \"9a53e192-cc4c-42fa-be92-cd26dd1e8ecb\") " pod="openshift-must-gather-mcg99/perf-node-gather-daemonset-mkf7s"
Jul 08 17:05:54.271372 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:54.271158    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9a53e192-cc4c-42fa-be92-cd26dd1e8ecb-podres\") pod \"perf-node-gather-daemonset-mkf7s\" (UID: \"9a53e192-cc4c-42fa-be92-cd26dd1e8ecb\") " pod="openshift-must-gather-mcg99/perf-node-gather-daemonset-mkf7s"
Jul 08 17:05:54.279360 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:54.279332    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l29dq\" (UniqueName: \"kubernetes.io/projected/9a53e192-cc4c-42fa-be92-cd26dd1e8ecb-kube-api-access-l29dq\") pod \"perf-node-gather-daemonset-mkf7s\" (UID: \"9a53e192-cc4c-42fa-be92-cd26dd1e8ecb\") " pod="openshift-must-gather-mcg99/perf-node-gather-daemonset-mkf7s"
Jul 08 17:05:54.391407 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:54.391306    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mcg99/perf-node-gather-daemonset-mkf7s"
Jul 08 17:05:54.526551 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:54.526520    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mcg99/perf-node-gather-daemonset-mkf7s"]
Jul 08 17:05:54.529351 ip-10-0-142-94 kubenswrapper[2578]: W0708 17:05:54.529316    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9a53e192_cc4c_42fa_be92_cd26dd1e8ecb.slice/crio-321ee043fd70882854a1c5523aa8441e25edb8b6a9e22776a8491919f5562920 WatchSource:0}: Error finding container 321ee043fd70882854a1c5523aa8441e25edb8b6a9e22776a8491919f5562920: Status 404 returned error can't find the container with id 321ee043fd70882854a1c5523aa8441e25edb8b6a9e22776a8491919f5562920
Jul 08 17:05:54.792942 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:54.792851    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mcg99/perf-node-gather-daemonset-mkf7s" event={"ID":"9a53e192-cc4c-42fa-be92-cd26dd1e8ecb","Type":"ContainerStarted","Data":"45a96e15c037a918bf4937294bdc18deb01f884e960304782b4c426811ff3c73"}
Jul 08 17:05:54.792942 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:54.792901    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mcg99/perf-node-gather-daemonset-mkf7s" event={"ID":"9a53e192-cc4c-42fa-be92-cd26dd1e8ecb","Type":"ContainerStarted","Data":"321ee043fd70882854a1c5523aa8441e25edb8b6a9e22776a8491919f5562920"}
Jul 08 17:05:54.793151 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:54.792987    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mcg99/perf-node-gather-daemonset-mkf7s"
Jul 08 17:05:54.809235 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:54.809181    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mcg99/perf-node-gather-daemonset-mkf7s" podStartSLOduration=0.809165928 podStartE2EDuration="809.165928ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 17:05:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:05:54.80807963 +0000 UTC m=+1867.079529171" watchObservedRunningTime="2026-07-08 17:05:54.809165928 +0000 UTC m=+1867.080615468"
Jul 08 17:05:54.889604 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:54.889570    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kbqcx_0ee105a4-0d7b-4a50-a212-110cac5e12fb/dns/0.log"
Jul 08 17:05:54.910604 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:54.910571    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kbqcx_0ee105a4-0d7b-4a50-a212-110cac5e12fb/kube-rbac-proxy/0.log"
Jul 08 17:05:54.981700 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:54.981662    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wflvt_84d1dbea-4585-46d9-b520-1329dba8df82/dns-node-resolver/0.log"
Jul 08 17:05:55.478175 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:55.478141    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-685fc4f98d-99nqw_654454ed-ca06-46b1-a030-b33bb974dd6d/registry/0.log"
Jul 08 17:05:55.546975 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:55.546941    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vnjdb_416e11c0-3ba2-451d-a901-9681ec0d4e3e/node-ca/0.log"
Jul 08 17:05:56.903592 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:56.903557    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-clfj6_8fad727b-fceb-4ac4-b275-bd741597566e/serve-healthcheck-canary/0.log"
Jul 08 17:05:57.273247 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:57.273150    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-ctm5m_ec3a3d1a-9e90-40a7-9695-7a1c81f3c41c/insights-operator/1.log"
Jul 08 17:05:57.273419 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:57.273368    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-ctm5m_ec3a3d1a-9e90-40a7-9695-7a1c81f3c41c/insights-operator/0.log"
Jul 08 17:05:57.429805 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:57.429780    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-r6rhh_0580073c-aa77-4d66-ab87-156ce96691ba/kube-rbac-proxy/0.log"
Jul 08 17:05:57.452724 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:57.452694    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-r6rhh_0580073c-aa77-4d66-ab87-156ce96691ba/exporter/0.log"
Jul 08 17:05:57.479069 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:57.479038    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-r6rhh_0580073c-aa77-4d66-ab87-156ce96691ba/extractor/0.log"
Jul 08 17:05:59.830752 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:59.830709    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-kpvjv_91781367-864b-494a-b12d-1b1cd3f9158b/s3-init/0.log"
Jul 08 17:05:59.847611 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:59.847563    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-kpvjv_91781367-864b-494a-b12d-1b1cd3f9158b/download-hf-model/0.log"
Jul 08 17:05:59.879515 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:05:59.879482    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-v2jqw_2d7d32de-a61f-4512-a31c-e7aff61cf161/seaweedfs/0.log"
Jul 08 17:06:00.806538 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:06:00.806500    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mcg99/perf-node-gather-daemonset-mkf7s"
Jul 08 17:06:05.955491 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:06:05.955462    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g4stg_28a74820-f159-4264-af3e-c983d2c68a51/kube-multus-additional-cni-plugins/0.log"
Jul 08 17:06:05.974792 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:06:05.974765    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g4stg_28a74820-f159-4264-af3e-c983d2c68a51/egress-router-binary-copy/0.log"
Jul 08 17:06:05.993788 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:06:05.993708    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g4stg_28a74820-f159-4264-af3e-c983d2c68a51/cni-plugins/0.log"
Jul 08 17:06:06.013725 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:06:06.013702    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g4stg_28a74820-f159-4264-af3e-c983d2c68a51/bond-cni-plugin/0.log"
Jul 08 17:06:06.033670 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:06:06.033617    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g4stg_28a74820-f159-4264-af3e-c983d2c68a51/routeoverride-cni/0.log"
Jul 08 17:06:06.052747 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:06:06.052722    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g4stg_28a74820-f159-4264-af3e-c983d2c68a51/whereabouts-cni-bincopy/0.log"
Jul 08 17:06:06.072805 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:06:06.072784    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g4stg_28a74820-f159-4264-af3e-c983d2c68a51/whereabouts-cni/0.log"
Jul 08 17:06:06.109234 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:06:06.109209    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-bg589_193fd6c8-8f4f-4f11-9e06-fd7b8ee997c0/kube-multus/0.log"
Jul 08 17:06:06.163342 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:06:06.163320    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-f5jcr_0f2a26b1-8d19-4874-a633-1cd61e29091a/network-metrics-daemon/0.log"
Jul 08 17:06:06.181042 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:06:06.181016    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-f5jcr_0f2a26b1-8d19-4874-a633-1cd61e29091a/kube-rbac-proxy/0.log"
Jul 08 17:06:07.051563 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:06:07.051534    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2shvk_f08a1b00-78fd-43ef-b2aa-2712d09fb1b3/ovn-controller/0.log"
Jul 08 17:06:07.069615 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:06:07.069591    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2shvk_f08a1b00-78fd-43ef-b2aa-2712d09fb1b3/ovn-acl-logging/1.log"
Jul 08 17:06:07.069815 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:06:07.069796    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2shvk_f08a1b00-78fd-43ef-b2aa-2712d09fb1b3/ovn-acl-logging/0.log"
Jul 08 17:06:07.096977 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:06:07.096939    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2shvk_f08a1b00-78fd-43ef-b2aa-2712d09fb1b3/kube-rbac-proxy-node/0.log"
Jul 08 17:06:07.124180 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:06:07.124159    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2shvk_f08a1b00-78fd-43ef-b2aa-2712d09fb1b3/kube-rbac-proxy-ovn-metrics/0.log"
Jul 08 17:06:07.149522 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:06:07.149501    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2shvk_f08a1b00-78fd-43ef-b2aa-2712d09fb1b3/northd/0.log"
Jul 08 17:06:07.175095 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:06:07.175077    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2shvk_f08a1b00-78fd-43ef-b2aa-2712d09fb1b3/nbdb/0.log"
Jul 08 17:06:07.198614 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:06:07.198591    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2shvk_f08a1b00-78fd-43ef-b2aa-2712d09fb1b3/sbdb/0.log"
Jul 08 17:06:07.280308 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:06:07.280284    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2shvk_f08a1b00-78fd-43ef-b2aa-2712d09fb1b3/ovnkube-controller/0.log"
Jul 08 17:06:08.722222 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:06:08.722191    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-989f8b8dc-kcxv2_a09322e9-4819-42eb-b98a-cf8ffe9ad70b/check-endpoints/0.log"
Jul 08 17:06:08.744190 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:06:08.744165    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-hzjsx_9bbb98bc-99e1-4401-96ab-ad9aae630c48/network-check-target-container/0.log"
Jul 08 17:06:09.722045 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:06:09.721953    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-2bv8p_7ce643e8-a633-411e-9d39-6f349e7ed426/iptables-alerter/0.log"
Jul 08 17:06:10.431445 ip-10-0-142-94 kubenswrapper[2578]: I0708 17:06:10.431415    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-wnsct_b33e50d5-554b-4d53-aafe-3dc13ae3fa6c/tuned/0.log"